You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "eldenmoon (via GitHub)" <gi...@apache.org> on 2023/02/01 07:02:31 UTC
[GitHub] [doris] eldenmoon opened a new pull request, #16319: [DRAFT] Dynamic table
eldenmoon opened a new pull request, #16319:
URL: https://github.com/apache/doris/pull/16319
# Proposed changes
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16319: [DRAFT] Dynamic table
Posted by github-actions.
github-actions[bot] commented on code in PR #16319:
URL: https://github.com/apache/doris/pull/16319#discussion_r1092988066
##########
be/src/vec/columns/subcolumn_tree.h:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/DataTypes/Serializations/SubcolumnsTree.h
+// and modified by Doris
+
+#pragma once
+#include <vec/columns/column.h>
+#include <vec/common/hash_table/hash_map.h>
+#include <vec/data_types/data_type.h>
+#include <vec/json/path_in_data.h>
+namespace doris::vectorized {
+/// Tree that represents paths in document
+/// with additional data in nodes.
+
+template <typename NodeData>
+class SubcolumnsTree {
+public:
+ struct Node {
+ enum Kind {
+ TUPLE,
+ NESTED,
+ SCALAR,
+ };
+
+ explicit Node(Kind kind_) : kind(kind_) {}
+ Node(Kind kind_, const NodeData& data_) : kind(kind_), data(data_) {}
+ Node(Kind kind_, const NodeData& data_, const PathInData& path_)
+ : kind(kind_), data(data_), path(path_) {}
+
+ Kind kind = TUPLE;
+ const Node* parent = nullptr;
+
+ Arena strings_pool;
+ HashMapWithStackMemory<StringRef, std::shared_ptr<Node>, StringRefHash, 4> children;
+
+ NodeData data;
+ PathInData path;
+
+ bool is_nested() const { return kind == NESTED; }
+ bool is_scalar() const { return kind == SCALAR; }
+
+ void add_child(std::string_view key, std::shared_ptr<Node> next_node) {
+ next_node->parent = this;
+ StringRef key_ref {strings_pool.insert(key.data(), key.length()), key.length()};
+ children[key_ref] = std::move(next_node);
+ }
+ };
+
+ using NodeKind = typename Node::Kind;
+ using NodePtr = std::shared_ptr<Node>;
+
+ /// Add a leaf without any data in other nodes.
+ bool add(const PathInData& path, const NodeData& leaf_data) {
+ return add(path, [&](NodeKind kind, bool exists) -> NodePtr {
+ if (exists) return nullptr;
+
+ if (kind == Node::SCALAR) return std::make_shared<Node>(kind, leaf_data, path);
+
+ return std::make_shared<Node>(kind);
+ });
+ }
+
+ /// Callback for creation of node. Receives kind of node and
+ /// flag, which is true if node already exists.
+ using NodeCreator = std::function<NodePtr(NodeKind, bool)>;
+
+ bool add(const PathInData& path, const NodeCreator& node_creator) {
+ const auto& parts = path.get_parts();
+
+ if (parts.empty()) return false;
+
+ if (!root) root = std::make_shared<Node>(Node::TUPLE);
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!root) { root = std::make_shared<Node>(Node::TUPLE);
}
```
##########
be/src/vec/columns/subcolumn_tree.h:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/DataTypes/Serializations/SubcolumnsTree.h
+// and modified by Doris
+
+#pragma once
+#include <vec/columns/column.h>
+#include <vec/common/hash_table/hash_map.h>
+#include <vec/data_types/data_type.h>
+#include <vec/json/path_in_data.h>
+namespace doris::vectorized {
+/// Tree that represents paths in document
+/// with additional data in nodes.
+
+template <typename NodeData>
+class SubcolumnsTree {
+public:
+ struct Node {
+ enum Kind {
+ TUPLE,
+ NESTED,
+ SCALAR,
+ };
+
+ explicit Node(Kind kind_) : kind(kind_) {}
+ Node(Kind kind_, const NodeData& data_) : kind(kind_), data(data_) {}
+ Node(Kind kind_, const NodeData& data_, const PathInData& path_)
+ : kind(kind_), data(data_), path(path_) {}
+
+ Kind kind = TUPLE;
+ const Node* parent = nullptr;
+
+ Arena strings_pool;
+ HashMapWithStackMemory<StringRef, std::shared_ptr<Node>, StringRefHash, 4> children;
+
+ NodeData data;
+ PathInData path;
+
+ bool is_nested() const { return kind == NESTED; }
+ bool is_scalar() const { return kind == SCALAR; }
+
+ void add_child(std::string_view key, std::shared_ptr<Node> next_node) {
+ next_node->parent = this;
+ StringRef key_ref {strings_pool.insert(key.data(), key.length()), key.length()};
+ children[key_ref] = std::move(next_node);
+ }
+ };
+
+ using NodeKind = typename Node::Kind;
+ using NodePtr = std::shared_ptr<Node>;
+
+ /// Add a leaf without any data in other nodes.
+ bool add(const PathInData& path, const NodeData& leaf_data) {
+ return add(path, [&](NodeKind kind, bool exists) -> NodePtr {
+ if (exists) return nullptr;
+
+ if (kind == Node::SCALAR) return std::make_shared<Node>(kind, leaf_data, path);
+
+ return std::make_shared<Node>(kind);
+ });
+ }
+
+ /// Callback for creation of node. Receives kind of node and
+ /// flag, which is true if node already exists.
+ using NodeCreator = std::function<NodePtr(NodeKind, bool)>;
+
+ bool add(const PathInData& path, const NodeCreator& node_creator) {
+ const auto& parts = path.get_parts();
+
+ if (parts.empty()) return false;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (parts.empty()) { return false;
}
```
##########
be/src/vec/columns/subcolumn_tree.h:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/DataTypes/Serializations/SubcolumnsTree.h
+// and modified by Doris
+
+#pragma once
+#include <vec/columns/column.h>
+#include <vec/common/hash_table/hash_map.h>
+#include <vec/data_types/data_type.h>
+#include <vec/json/path_in_data.h>
+namespace doris::vectorized {
+/// Tree that represents paths in document
+/// with additional data in nodes.
+
+template <typename NodeData>
+class SubcolumnsTree {
+public:
+ struct Node {
+ enum Kind {
+ TUPLE,
+ NESTED,
+ SCALAR,
+ };
+
+ explicit Node(Kind kind_) : kind(kind_) {}
+ Node(Kind kind_, const NodeData& data_) : kind(kind_), data(data_) {}
+ Node(Kind kind_, const NodeData& data_, const PathInData& path_)
+ : kind(kind_), data(data_), path(path_) {}
+
+ Kind kind = TUPLE;
+ const Node* parent = nullptr;
+
+ Arena strings_pool;
+ HashMapWithStackMemory<StringRef, std::shared_ptr<Node>, StringRefHash, 4> children;
+
+ NodeData data;
+ PathInData path;
+
+ bool is_nested() const { return kind == NESTED; }
+ bool is_scalar() const { return kind == SCALAR; }
+
+ void add_child(std::string_view key, std::shared_ptr<Node> next_node) {
+ next_node->parent = this;
+ StringRef key_ref {strings_pool.insert(key.data(), key.length()), key.length()};
+ children[key_ref] = std::move(next_node);
+ }
+ };
+
+ using NodeKind = typename Node::Kind;
+ using NodePtr = std::shared_ptr<Node>;
+
+ /// Add a leaf without any data in other nodes.
+ bool add(const PathInData& path, const NodeData& leaf_data) {
+ return add(path, [&](NodeKind kind, bool exists) -> NodePtr {
+ if (exists) return nullptr;
+
+ if (kind == Node::SCALAR) return std::make_shared<Node>(kind, leaf_data, path);
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (kind == Node::SCALAR) { return std::make_shared<Node>(kind, leaf_data, path);
}
```
##########
be/src/vec/columns/subcolumn_tree.h:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/DataTypes/Serializations/SubcolumnsTree.h
+// and modified by Doris
+
+#pragma once
+#include <vec/columns/column.h>
+#include <vec/common/hash_table/hash_map.h>
+#include <vec/data_types/data_type.h>
+#include <vec/json/path_in_data.h>
+namespace doris::vectorized {
+/// Tree that represents paths in document
+/// with additional data in nodes.
+
+template <typename NodeData>
+class SubcolumnsTree {
+public:
+ struct Node {
+ enum Kind {
+ TUPLE,
+ NESTED,
+ SCALAR,
+ };
+
+ explicit Node(Kind kind_) : kind(kind_) {}
+ Node(Kind kind_, const NodeData& data_) : kind(kind_), data(data_) {}
+ Node(Kind kind_, const NodeData& data_, const PathInData& path_)
+ : kind(kind_), data(data_), path(path_) {}
+
+ Kind kind = TUPLE;
+ const Node* parent = nullptr;
+
+ Arena strings_pool;
+ HashMapWithStackMemory<StringRef, std::shared_ptr<Node>, StringRefHash, 4> children;
+
+ NodeData data;
+ PathInData path;
+
+ bool is_nested() const { return kind == NESTED; }
+ bool is_scalar() const { return kind == SCALAR; }
+
+ void add_child(std::string_view key, std::shared_ptr<Node> next_node) {
+ next_node->parent = this;
+ StringRef key_ref {strings_pool.insert(key.data(), key.length()), key.length()};
+ children[key_ref] = std::move(next_node);
+ }
+ };
+
+ using NodeKind = typename Node::Kind;
+ using NodePtr = std::shared_ptr<Node>;
+
+ /// Add a leaf without any data in other nodes.
+ bool add(const PathInData& path, const NodeData& leaf_data) {
+ return add(path, [&](NodeKind kind, bool exists) -> NodePtr {
+ if (exists) return nullptr;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (exists) { return nullptr;
}
```
##########
be/src/vec/columns/subcolumn_tree.h:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/DataTypes/Serializations/SubcolumnsTree.h
+// and modified by Doris
+
+#pragma once
+#include <vec/columns/column.h>
+#include <vec/common/hash_table/hash_map.h>
+#include <vec/data_types/data_type.h>
+#include <vec/json/path_in_data.h>
+namespace doris::vectorized {
+/// Tree that represents paths in document
+/// with additional data in nodes.
+
+template <typename NodeData>
+class SubcolumnsTree {
+public:
+ struct Node {
+ enum Kind {
+ TUPLE,
+ NESTED,
+ SCALAR,
+ };
+
+ explicit Node(Kind kind_) : kind(kind_) {}
+ Node(Kind kind_, const NodeData& data_) : kind(kind_), data(data_) {}
+ Node(Kind kind_, const NodeData& data_, const PathInData& path_)
+ : kind(kind_), data(data_), path(path_) {}
+
+ Kind kind = TUPLE;
+ const Node* parent = nullptr;
+
+ Arena strings_pool;
+ HashMapWithStackMemory<StringRef, std::shared_ptr<Node>, StringRefHash, 4> children;
+
+ NodeData data;
+ PathInData path;
+
+ bool is_nested() const { return kind == NESTED; }
+ bool is_scalar() const { return kind == SCALAR; }
+
+ void add_child(std::string_view key, std::shared_ptr<Node> next_node) {
+ next_node->parent = this;
+ StringRef key_ref {strings_pool.insert(key.data(), key.length()), key.length()};
+ children[key_ref] = std::move(next_node);
+ }
+ };
+
+ using NodeKind = typename Node::Kind;
+ using NodePtr = std::shared_ptr<Node>;
+
+ /// Add a leaf without any data in other nodes.
+ bool add(const PathInData& path, const NodeData& leaf_data) {
+ return add(path, [&](NodeKind kind, bool exists) -> NodePtr {
+ if (exists) return nullptr;
+
+ if (kind == Node::SCALAR) return std::make_shared<Node>(kind, leaf_data, path);
+
+ return std::make_shared<Node>(kind);
+ });
+ }
+
+ /// Callback for creation of node. Receives kind of node and
+ /// flag, which is true if node already exists.
+ using NodeCreator = std::function<NodePtr(NodeKind, bool)>;
+
+ bool add(const PathInData& path, const NodeCreator& node_creator) {
+ const auto& parts = path.get_parts();
+
+ if (parts.empty()) return false;
+
+ if (!root) root = std::make_shared<Node>(Node::TUPLE);
+
+ Node* current_node = root.get();
+ for (size_t i = 0; i < parts.size() - 1; ++i) {
+ assert(current_node->kind != Node::SCALAR);
+
+ auto it = current_node->children.find(
+ StringRef {parts[i].key.data(), parts[i].key.size()});
+ if (it != current_node->children.end()) {
+ current_node = it->get_second().get();
+ node_creator(current_node->kind, true);
+
+ if (current_node->is_nested() != parts[i].is_nested) return false;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (current_node->is_nested() != parts[i].is_nested) { return false;
}
```
##########
be/src/vec/columns/subcolumn_tree.h:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/DataTypes/Serializations/SubcolumnsTree.h
+// and modified by Doris
+
+#pragma once
+#include <vec/columns/column.h>
+#include <vec/common/hash_table/hash_map.h>
+#include <vec/data_types/data_type.h>
+#include <vec/json/path_in_data.h>
+namespace doris::vectorized {
+/// Tree that represents paths in document
+/// with additional data in nodes.
+
+template <typename NodeData>
+class SubcolumnsTree {
+public:
+ struct Node {
+ enum Kind {
+ TUPLE,
+ NESTED,
+ SCALAR,
+ };
+
+ explicit Node(Kind kind_) : kind(kind_) {}
+ Node(Kind kind_, const NodeData& data_) : kind(kind_), data(data_) {}
+ Node(Kind kind_, const NodeData& data_, const PathInData& path_)
+ : kind(kind_), data(data_), path(path_) {}
+
+ Kind kind = TUPLE;
+ const Node* parent = nullptr;
+
+ Arena strings_pool;
+ HashMapWithStackMemory<StringRef, std::shared_ptr<Node>, StringRefHash, 4> children;
+
+ NodeData data;
+ PathInData path;
+
+ bool is_nested() const { return kind == NESTED; }
+ bool is_scalar() const { return kind == SCALAR; }
+
+ void add_child(std::string_view key, std::shared_ptr<Node> next_node) {
+ next_node->parent = this;
+ StringRef key_ref {strings_pool.insert(key.data(), key.length()), key.length()};
+ children[key_ref] = std::move(next_node);
+ }
+ };
+
+ using NodeKind = typename Node::Kind;
+ using NodePtr = std::shared_ptr<Node>;
+
+ /// Add a leaf without any data in other nodes.
+ bool add(const PathInData& path, const NodeData& leaf_data) {
+ return add(path, [&](NodeKind kind, bool exists) -> NodePtr {
+ if (exists) return nullptr;
+
+ if (kind == Node::SCALAR) return std::make_shared<Node>(kind, leaf_data, path);
+
+ return std::make_shared<Node>(kind);
+ });
+ }
+
+ /// Callback for creation of node. Receives kind of node and
+ /// flag, which is true if node already exists.
+ using NodeCreator = std::function<NodePtr(NodeKind, bool)>;
+
+ bool add(const PathInData& path, const NodeCreator& node_creator) {
+ const auto& parts = path.get_parts();
+
+ if (parts.empty()) return false;
+
+ if (!root) root = std::make_shared<Node>(Node::TUPLE);
+
+ Node* current_node = root.get();
+ for (size_t i = 0; i < parts.size() - 1; ++i) {
+ assert(current_node->kind != Node::SCALAR);
+
+ auto it = current_node->children.find(
+ StringRef {parts[i].key.data(), parts[i].key.size()});
+ if (it != current_node->children.end()) {
+ current_node = it->get_second().get();
+ node_creator(current_node->kind, true);
+
+ if (current_node->is_nested() != parts[i].is_nested) return false;
+ } else {
+ auto next_kind = parts[i].is_nested ? Node::NESTED : Node::TUPLE;
+ auto next_node = node_creator(next_kind, false);
+ current_node->add_child(String(parts[i].key), next_node);
+ current_node = next_node.get();
+ }
+ }
+
+ auto it = current_node->children.find(
+ StringRef {parts.back().key.data(), parts.back().key.size()});
+ if (it != current_node->children.end()) return false;
+
+ auto next_node = node_creator(Node::SCALAR, false);
+ current_node->add_child(String(parts.back().key), next_node);
+ leaves.push_back(std::move(next_node));
+
+ return true;
+ }
+
+ /// Find node that matches the path the best.
+ const Node* find_best_match(const PathInData& path) const { return find_impl(path, false); }
+
+ /// Find node that matches the path exactly.
+ const Node* find_exact(const PathInData& path) const { return find_impl(path, true); }
+
+ /// Find leaf by path.
+ const Node* find_leaf(const PathInData& path) const {
+ const auto* candidate = find_exact(path);
+ if (!candidate || !candidate->is_scalar()) return nullptr;
+ return candidate;
+ }
+
+ using NodePredicate = std::function<bool(const Node&)>;
+
+ /// Finds leaf that satisfies the predicate.
+ const Node* find_leaf(const NodePredicate& predicate) {
+ return find_leaf(root.get(), predicate);
+ }
+
+ static const Node* find_leaf(const Node* node, const NodePredicate& predicate) {
+ if (!node) return nullptr;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!node) { return nullptr;
}
```
##########
be/src/vec/columns/subcolumn_tree.h:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/DataTypes/Serializations/SubcolumnsTree.h
+// and modified by Doris
+
+#pragma once
+#include <vec/columns/column.h>
+#include <vec/common/hash_table/hash_map.h>
+#include <vec/data_types/data_type.h>
+#include <vec/json/path_in_data.h>
+namespace doris::vectorized {
+/// Tree that represents paths in document
+/// with additional data in nodes.
+
+template <typename NodeData>
+class SubcolumnsTree {
+public:
+ struct Node {
+ enum Kind {
+ TUPLE,
+ NESTED,
+ SCALAR,
+ };
+
+ explicit Node(Kind kind_) : kind(kind_) {}
+ Node(Kind kind_, const NodeData& data_) : kind(kind_), data(data_) {}
+ Node(Kind kind_, const NodeData& data_, const PathInData& path_)
+ : kind(kind_), data(data_), path(path_) {}
+
+ Kind kind = TUPLE;
+ const Node* parent = nullptr;
+
+ Arena strings_pool;
+ HashMapWithStackMemory<StringRef, std::shared_ptr<Node>, StringRefHash, 4> children;
+
+ NodeData data;
+ PathInData path;
+
+ bool is_nested() const { return kind == NESTED; }
+ bool is_scalar() const { return kind == SCALAR; }
+
+ void add_child(std::string_view key, std::shared_ptr<Node> next_node) {
+ next_node->parent = this;
+ StringRef key_ref {strings_pool.insert(key.data(), key.length()), key.length()};
+ children[key_ref] = std::move(next_node);
+ }
+ };
+
+ using NodeKind = typename Node::Kind;
+ using NodePtr = std::shared_ptr<Node>;
+
+ /// Add a leaf without any data in other nodes.
+ bool add(const PathInData& path, const NodeData& leaf_data) {
+ return add(path, [&](NodeKind kind, bool exists) -> NodePtr {
+ if (exists) return nullptr;
+
+ if (kind == Node::SCALAR) return std::make_shared<Node>(kind, leaf_data, path);
+
+ return std::make_shared<Node>(kind);
+ });
+ }
+
+ /// Callback for creation of node. Receives kind of node and
+ /// flag, which is true if node already exists.
+ using NodeCreator = std::function<NodePtr(NodeKind, bool)>;
+
+ bool add(const PathInData& path, const NodeCreator& node_creator) {
+ const auto& parts = path.get_parts();
+
+ if (parts.empty()) return false;
+
+ if (!root) root = std::make_shared<Node>(Node::TUPLE);
+
+ Node* current_node = root.get();
+ for (size_t i = 0; i < parts.size() - 1; ++i) {
+ assert(current_node->kind != Node::SCALAR);
+
+ auto it = current_node->children.find(
+ StringRef {parts[i].key.data(), parts[i].key.size()});
+ if (it != current_node->children.end()) {
+ current_node = it->get_second().get();
+ node_creator(current_node->kind, true);
+
+ if (current_node->is_nested() != parts[i].is_nested) return false;
+ } else {
+ auto next_kind = parts[i].is_nested ? Node::NESTED : Node::TUPLE;
+ auto next_node = node_creator(next_kind, false);
+ current_node->add_child(String(parts[i].key), next_node);
+ current_node = next_node.get();
+ }
+ }
+
+ auto it = current_node->children.find(
+ StringRef {parts.back().key.data(), parts.back().key.size()});
+ if (it != current_node->children.end()) return false;
+
+ auto next_node = node_creator(Node::SCALAR, false);
+ current_node->add_child(String(parts.back().key), next_node);
+ leaves.push_back(std::move(next_node));
+
+ return true;
+ }
+
+ /// Find node that matches the path the best.
+ const Node* find_best_match(const PathInData& path) const { return find_impl(path, false); }
+
+ /// Find node that matches the path exactly.
+ const Node* find_exact(const PathInData& path) const { return find_impl(path, true); }
+
+ /// Find leaf by path.
+ const Node* find_leaf(const PathInData& path) const {
+ const auto* candidate = find_exact(path);
+ if (!candidate || !candidate->is_scalar()) return nullptr;
+ return candidate;
+ }
+
+ using NodePredicate = std::function<bool(const Node&)>;
+
+ /// Finds leaf that satisfies the predicate.
+ const Node* find_leaf(const NodePredicate& predicate) {
+ return find_leaf(root.get(), predicate);
+ }
+
+ static const Node* find_leaf(const Node* node, const NodePredicate& predicate) {
+ if (!node) return nullptr;
+
+ if (node->is_scalar()) return predicate(*node) ? node : nullptr;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (node->is_scalar()) { return predicate(*node) ? node : nullptr;
}
```
##########
be/src/vec/columns/subcolumn_tree.h:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/DataTypes/Serializations/SubcolumnsTree.h
+// and modified by Doris
+
+#pragma once
+#include <vec/columns/column.h>
+#include <vec/common/hash_table/hash_map.h>
+#include <vec/data_types/data_type.h>
+#include <vec/json/path_in_data.h>
+namespace doris::vectorized {
+/// Tree that represents paths in document
+/// with additional data in nodes.
+
+template <typename NodeData>
+class SubcolumnsTree {
+public:
+ struct Node {
+ enum Kind {
+ TUPLE,
+ NESTED,
+ SCALAR,
+ };
+
+ explicit Node(Kind kind_) : kind(kind_) {}
+ Node(Kind kind_, const NodeData& data_) : kind(kind_), data(data_) {}
+ Node(Kind kind_, const NodeData& data_, const PathInData& path_)
+ : kind(kind_), data(data_), path(path_) {}
+
+ Kind kind = TUPLE;
+ const Node* parent = nullptr;
+
+ Arena strings_pool;
+ HashMapWithStackMemory<StringRef, std::shared_ptr<Node>, StringRefHash, 4> children;
+
+ NodeData data;
+ PathInData path;
+
+ bool is_nested() const { return kind == NESTED; }
+ bool is_scalar() const { return kind == SCALAR; }
+
+ void add_child(std::string_view key, std::shared_ptr<Node> next_node) {
+ next_node->parent = this;
+ StringRef key_ref {strings_pool.insert(key.data(), key.length()), key.length()};
+ children[key_ref] = std::move(next_node);
+ }
+ };
+
+ using NodeKind = typename Node::Kind;
+ using NodePtr = std::shared_ptr<Node>;
+
+ /// Add a leaf without any data in other nodes.
+ bool add(const PathInData& path, const NodeData& leaf_data) {
+ return add(path, [&](NodeKind kind, bool exists) -> NodePtr {
+ if (exists) return nullptr;
+
+ if (kind == Node::SCALAR) return std::make_shared<Node>(kind, leaf_data, path);
+
+ return std::make_shared<Node>(kind);
+ });
+ }
+
+ /// Callback for creation of node. Receives kind of node and
+ /// flag, which is true if node already exists.
+ using NodeCreator = std::function<NodePtr(NodeKind, bool)>;
+
+ bool add(const PathInData& path, const NodeCreator& node_creator) {
+ const auto& parts = path.get_parts();
+
+ if (parts.empty()) return false;
+
+ if (!root) root = std::make_shared<Node>(Node::TUPLE);
+
+ Node* current_node = root.get();
+ for (size_t i = 0; i < parts.size() - 1; ++i) {
+ assert(current_node->kind != Node::SCALAR);
+
+ auto it = current_node->children.find(
+ StringRef {parts[i].key.data(), parts[i].key.size()});
+ if (it != current_node->children.end()) {
+ current_node = it->get_second().get();
+ node_creator(current_node->kind, true);
+
+ if (current_node->is_nested() != parts[i].is_nested) return false;
+ } else {
+ auto next_kind = parts[i].is_nested ? Node::NESTED : Node::TUPLE;
+ auto next_node = node_creator(next_kind, false);
+ current_node->add_child(String(parts[i].key), next_node);
+ current_node = next_node.get();
+ }
+ }
+
+ auto it = current_node->children.find(
+ StringRef {parts.back().key.data(), parts.back().key.size()});
+ if (it != current_node->children.end()) return false;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (it != current_node->children.end()) { return false;
}
```
##########
be/src/vec/columns/subcolumn_tree.h:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/DataTypes/Serializations/SubcolumnsTree.h
+// and modified by Doris
+
+#pragma once
+#include <vec/columns/column.h>
+#include <vec/common/hash_table/hash_map.h>
+#include <vec/data_types/data_type.h>
+#include <vec/json/path_in_data.h>
+namespace doris::vectorized {
+/// Tree that represents paths in document
+/// with additional data in nodes.
+
+template <typename NodeData>
+class SubcolumnsTree {
+public:
+ struct Node {
+ enum Kind {
+ TUPLE,
+ NESTED,
+ SCALAR,
+ };
+
+ explicit Node(Kind kind_) : kind(kind_) {}
+ Node(Kind kind_, const NodeData& data_) : kind(kind_), data(data_) {}
+ Node(Kind kind_, const NodeData& data_, const PathInData& path_)
+ : kind(kind_), data(data_), path(path_) {}
+
+ Kind kind = TUPLE;
+ const Node* parent = nullptr;
+
+ Arena strings_pool;
+ HashMapWithStackMemory<StringRef, std::shared_ptr<Node>, StringRefHash, 4> children;
+
+ NodeData data;
+ PathInData path;
+
+ bool is_nested() const { return kind == NESTED; }
+ bool is_scalar() const { return kind == SCALAR; }
+
+ void add_child(std::string_view key, std::shared_ptr<Node> next_node) {
+ next_node->parent = this;
+ StringRef key_ref {strings_pool.insert(key.data(), key.length()), key.length()};
+ children[key_ref] = std::move(next_node);
+ }
+ };
+
+ using NodeKind = typename Node::Kind;
+ using NodePtr = std::shared_ptr<Node>;
+
+ /// Add a leaf without any data in other nodes.
+ bool add(const PathInData& path, const NodeData& leaf_data) {
+ return add(path, [&](NodeKind kind, bool exists) -> NodePtr {
+ if (exists) return nullptr;
+
+ if (kind == Node::SCALAR) return std::make_shared<Node>(kind, leaf_data, path);
+
+ return std::make_shared<Node>(kind);
+ });
+ }
+
+ /// Callback for creation of node. Receives kind of node and
+ /// flag, which is true if node already exists.
+ using NodeCreator = std::function<NodePtr(NodeKind, bool)>;
+
+ bool add(const PathInData& path, const NodeCreator& node_creator) {
+ const auto& parts = path.get_parts();
+
+ if (parts.empty()) return false;
+
+ if (!root) root = std::make_shared<Node>(Node::TUPLE);
+
+ Node* current_node = root.get();
+ for (size_t i = 0; i < parts.size() - 1; ++i) {
+ assert(current_node->kind != Node::SCALAR);
+
+ auto it = current_node->children.find(
+ StringRef {parts[i].key.data(), parts[i].key.size()});
+ if (it != current_node->children.end()) {
+ current_node = it->get_second().get();
+ node_creator(current_node->kind, true);
+
+ if (current_node->is_nested() != parts[i].is_nested) return false;
+ } else {
+ auto next_kind = parts[i].is_nested ? Node::NESTED : Node::TUPLE;
+ auto next_node = node_creator(next_kind, false);
+ current_node->add_child(String(parts[i].key), next_node);
+ current_node = next_node.get();
+ }
+ }
+
+ auto it = current_node->children.find(
+ StringRef {parts.back().key.data(), parts.back().key.size()});
+ if (it != current_node->children.end()) return false;
+
+ auto next_node = node_creator(Node::SCALAR, false);
+ current_node->add_child(String(parts.back().key), next_node);
+ leaves.push_back(std::move(next_node));
+
+ return true;
+ }
+
+ /// Find node that matches the path the best.
+ const Node* find_best_match(const PathInData& path) const { return find_impl(path, false); }
+
+ /// Find node that matches the path exactly.
+ const Node* find_exact(const PathInData& path) const { return find_impl(path, true); }
+
+ /// Find leaf by path.
+ const Node* find_leaf(const PathInData& path) const {
+ const auto* candidate = find_exact(path);
+ if (!candidate || !candidate->is_scalar()) return nullptr;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!candidate || !candidate->is_scalar()) { return nullptr;
}
```
##########
be/src/vec/columns/subcolumn_tree.h:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/DataTypes/Serializations/SubcolumnsTree.h
+// and modified by Doris
+
+#pragma once
+#include <vec/columns/column.h>
+#include <vec/common/hash_table/hash_map.h>
+#include <vec/data_types/data_type.h>
+#include <vec/json/path_in_data.h>
+namespace doris::vectorized {
+/// Tree that represents paths in document
+/// with additional data in nodes.
+
+template <typename NodeData>
+class SubcolumnsTree {
+public:
+ struct Node {
+ enum Kind {
+ TUPLE,
+ NESTED,
+ SCALAR,
+ };
+
+ explicit Node(Kind kind_) : kind(kind_) {}
+ Node(Kind kind_, const NodeData& data_) : kind(kind_), data(data_) {}
+ Node(Kind kind_, const NodeData& data_, const PathInData& path_)
+ : kind(kind_), data(data_), path(path_) {}
+
+ Kind kind = TUPLE;
+ const Node* parent = nullptr;
+
+ Arena strings_pool;
+ HashMapWithStackMemory<StringRef, std::shared_ptr<Node>, StringRefHash, 4> children;
+
+ NodeData data;
+ PathInData path;
+
+ bool is_nested() const { return kind == NESTED; }
+ bool is_scalar() const { return kind == SCALAR; }
+
+ void add_child(std::string_view key, std::shared_ptr<Node> next_node) {
+ next_node->parent = this;
+ StringRef key_ref {strings_pool.insert(key.data(), key.length()), key.length()};
+ children[key_ref] = std::move(next_node);
+ }
+ };
+
+ using NodeKind = typename Node::Kind;
+ using NodePtr = std::shared_ptr<Node>;
+
+ /// Add a leaf without any data in other nodes.
+ bool add(const PathInData& path, const NodeData& leaf_data) {
+ return add(path, [&](NodeKind kind, bool exists) -> NodePtr {
+ if (exists) return nullptr;
+
+ if (kind == Node::SCALAR) return std::make_shared<Node>(kind, leaf_data, path);
+
+ return std::make_shared<Node>(kind);
+ });
+ }
+
+ /// Callback for creation of node. Receives kind of node and
+ /// flag, which is true if node already exists.
+ using NodeCreator = std::function<NodePtr(NodeKind, bool)>;
+
+ bool add(const PathInData& path, const NodeCreator& node_creator) {
+ const auto& parts = path.get_parts();
+
+ if (parts.empty()) return false;
+
+ if (!root) root = std::make_shared<Node>(Node::TUPLE);
+
+ Node* current_node = root.get();
+ for (size_t i = 0; i < parts.size() - 1; ++i) {
+ assert(current_node->kind != Node::SCALAR);
+
+ auto it = current_node->children.find(
+ StringRef {parts[i].key.data(), parts[i].key.size()});
+ if (it != current_node->children.end()) {
+ current_node = it->get_second().get();
+ node_creator(current_node->kind, true);
+
+ if (current_node->is_nested() != parts[i].is_nested) return false;
+ } else {
+ auto next_kind = parts[i].is_nested ? Node::NESTED : Node::TUPLE;
+ auto next_node = node_creator(next_kind, false);
+ current_node->add_child(String(parts[i].key), next_node);
+ current_node = next_node.get();
+ }
+ }
+
+ auto it = current_node->children.find(
+ StringRef {parts.back().key.data(), parts.back().key.size()});
+ if (it != current_node->children.end()) return false;
+
+ auto next_node = node_creator(Node::SCALAR, false);
+ current_node->add_child(String(parts.back().key), next_node);
+ leaves.push_back(std::move(next_node));
+
+ return true;
+ }
+
+ /// Find node that matches the path the best.
+ const Node* find_best_match(const PathInData& path) const { return find_impl(path, false); }
+
+ /// Find node that matches the path exactly.
+ const Node* find_exact(const PathInData& path) const { return find_impl(path, true); }
+
+ /// Find leaf by path.
+ const Node* find_leaf(const PathInData& path) const {
+ const auto* candidate = find_exact(path);
+ if (!candidate || !candidate->is_scalar()) return nullptr;
+ return candidate;
+ }
+
+ using NodePredicate = std::function<bool(const Node&)>;
+
+ /// Finds leaf that satisfies the predicate.
+ const Node* find_leaf(const NodePredicate& predicate) {
+ return find_leaf(root.get(), predicate);
+ }
+
+ static const Node* find_leaf(const Node* node, const NodePredicate& predicate) {
+ if (!node) return nullptr;
+
+ if (node->is_scalar()) return predicate(*node) ? node : nullptr;
+
+ for (auto it = node->children.begin(); it != node->children.end(); ++it) {
+ auto child = it->get_second();
+ if (const auto* leaf = find_leaf(child.get(), predicate)) return leaf;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (const auto* leaf = find_leaf(child.get(), predicate)) { return leaf;
}
```
##########
be/src/vec/common/schema_util.cpp:
##########
@@ -0,0 +1,481 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+
+#include <vec/columns/column_array.h>
+#include <vec/columns/column_object.h>
+#include <vec/common/schema_util.h>
+#include <vec/core/field.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_object.h>
+#include <vec/functions/simple_function_factory.h>
+#include <vec/json/parse2column.h>
+
+#include <vec/data_types/data_type_factory.hpp>
+
+#include "gen_cpp/FrontendService.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "olap/rowset/rowset_writer_context.h"
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "util/thrift_rpc_helper.h"
+
+namespace doris::vectorized::schema_util {
+
+size_t get_number_of_dimensions(const IDataType& type) {
+ if (const auto* type_array = typeid_cast<const DataTypeArray*>(&type))
+ return type_array->get_number_of_dimensions();
+ return 0;
+}
+size_t get_number_of_dimensions(const IColumn& column) {
+ if (const auto* column_array = check_and_get_column<ColumnArray>(column))
+ return column_array->get_number_of_dimensions();
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (const auto* column_array = check_and_get_column<ColumnArray>(column)) {
return column_array->get_number_of_dimensions();
}
```
##########
be/src/vec/columns/subcolumn_tree.h:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/DataTypes/Serializations/SubcolumnsTree.h
+// and modified by Doris
+
+#pragma once
+#include <vec/columns/column.h>
+#include <vec/common/hash_table/hash_map.h>
+#include <vec/data_types/data_type.h>
+#include <vec/json/path_in_data.h>
+namespace doris::vectorized {
+/// Tree that represents paths in document
+/// with additional data in nodes.
+
+template <typename NodeData>
+class SubcolumnsTree {
+public:
+ struct Node {
+ enum Kind {
+ TUPLE,
+ NESTED,
+ SCALAR,
+ };
+
+ explicit Node(Kind kind_) : kind(kind_) {}
+ Node(Kind kind_, const NodeData& data_) : kind(kind_), data(data_) {}
+ Node(Kind kind_, const NodeData& data_, const PathInData& path_)
+ : kind(kind_), data(data_), path(path_) {}
+
+ Kind kind = TUPLE;
+ const Node* parent = nullptr;
+
+ Arena strings_pool;
+ HashMapWithStackMemory<StringRef, std::shared_ptr<Node>, StringRefHash, 4> children;
+
+ NodeData data;
+ PathInData path;
+
+ bool is_nested() const { return kind == NESTED; }
+ bool is_scalar() const { return kind == SCALAR; }
+
+ void add_child(std::string_view key, std::shared_ptr<Node> next_node) {
+ next_node->parent = this;
+ StringRef key_ref {strings_pool.insert(key.data(), key.length()), key.length()};
+ children[key_ref] = std::move(next_node);
+ }
+ };
+
+ using NodeKind = typename Node::Kind;
+ using NodePtr = std::shared_ptr<Node>;
+
+ /// Add a leaf without any data in other nodes.
+ bool add(const PathInData& path, const NodeData& leaf_data) {
+ return add(path, [&](NodeKind kind, bool exists) -> NodePtr {
+ if (exists) return nullptr;
+
+ if (kind == Node::SCALAR) return std::make_shared<Node>(kind, leaf_data, path);
+
+ return std::make_shared<Node>(kind);
+ });
+ }
+
+ /// Callback for creation of node. Receives kind of node and
+ /// flag, which is true if node already exists.
+ using NodeCreator = std::function<NodePtr(NodeKind, bool)>;
+
+ bool add(const PathInData& path, const NodeCreator& node_creator) {
+ const auto& parts = path.get_parts();
+
+ if (parts.empty()) return false;
+
+ if (!root) root = std::make_shared<Node>(Node::TUPLE);
+
+ Node* current_node = root.get();
+ for (size_t i = 0; i < parts.size() - 1; ++i) {
+ assert(current_node->kind != Node::SCALAR);
+
+ auto it = current_node->children.find(
+ StringRef {parts[i].key.data(), parts[i].key.size()});
+ if (it != current_node->children.end()) {
+ current_node = it->get_second().get();
+ node_creator(current_node->kind, true);
+
+ if (current_node->is_nested() != parts[i].is_nested) return false;
+ } else {
+ auto next_kind = parts[i].is_nested ? Node::NESTED : Node::TUPLE;
+ auto next_node = node_creator(next_kind, false);
+ current_node->add_child(String(parts[i].key), next_node);
+ current_node = next_node.get();
+ }
+ }
+
+ auto it = current_node->children.find(
+ StringRef {parts.back().key.data(), parts.back().key.size()});
+ if (it != current_node->children.end()) return false;
+
+ auto next_node = node_creator(Node::SCALAR, false);
+ current_node->add_child(String(parts.back().key), next_node);
+ leaves.push_back(std::move(next_node));
+
+ return true;
+ }
+
+ /// Find node that matches the path the best.
+ const Node* find_best_match(const PathInData& path) const { return find_impl(path, false); }
+
+ /// Find node that matches the path exactly.
+ const Node* find_exact(const PathInData& path) const { return find_impl(path, true); }
+
+ /// Find leaf by path.
+ const Node* find_leaf(const PathInData& path) const {
+ const auto* candidate = find_exact(path);
+ if (!candidate || !candidate->is_scalar()) return nullptr;
+ return candidate;
+ }
+
+ using NodePredicate = std::function<bool(const Node&)>;
+
+ /// Finds leaf that satisfies the predicate.
+ const Node* find_leaf(const NodePredicate& predicate) {
+ return find_leaf(root.get(), predicate);
+ }
+
+ static const Node* find_leaf(const Node* node, const NodePredicate& predicate) {
+ if (!node) return nullptr;
+
+ if (node->is_scalar()) return predicate(*node) ? node : nullptr;
+
+ for (auto it = node->children.begin(); it != node->children.end(); ++it) {
+ auto child = it->get_second();
+ if (const auto* leaf = find_leaf(child.get(), predicate)) return leaf;
+ }
+ return nullptr;
+ }
+
+ /// Find first parent node that satisfies the predicate.
+ static const Node* find_parent(const Node* node, const NodePredicate& predicate) {
+ while (node && !predicate(*node)) node = node->parent;
+ return node;
+ }
+
+ bool empty() const { return root == nullptr; }
+ size_t size() const { return leaves.size(); }
+
+ using Nodes = std::vector<NodePtr>;
+
+ const Nodes& get_leaves() const { return leaves; }
+ const Node* get_root() const { return root.get(); }
+
+ using iterator = typename Nodes::iterator;
+ using const_iterator = typename Nodes::const_iterator;
+
+ iterator begin() { return leaves.begin(); }
+ iterator end() { return leaves.end(); }
+
+ const_iterator begin() const { return leaves.begin(); }
+ const_iterator end() const { return leaves.end(); }
+
+private:
+ const Node* find_impl(const PathInData& path, bool find_exact) const {
+ if (!root) return nullptr;
+
+ const auto& parts = path.get_parts();
+ const Node* current_node = root.get();
+
+ for (const auto& part : parts) {
+ auto it = current_node->children.find(StringRef {part.key.data(), part.key.size()});
+ if (it == current_node->children.end()) return find_exact ? nullptr : current_node;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (it == current_node->children.end()) { return find_exact ? nullptr : current_node;
}
```
##########
be/src/vec/columns/subcolumn_tree.h:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/DataTypes/Serializations/SubcolumnsTree.h
+// and modified by Doris
+
+#pragma once
+#include <vec/columns/column.h>
+#include <vec/common/hash_table/hash_map.h>
+#include <vec/data_types/data_type.h>
+#include <vec/json/path_in_data.h>
+namespace doris::vectorized {
+/// Tree that represents paths in document
+/// with additional data in nodes.
+
+template <typename NodeData>
+class SubcolumnsTree {
+public:
+ struct Node {
+ enum Kind {
+ TUPLE,
+ NESTED,
+ SCALAR,
+ };
+
+ explicit Node(Kind kind_) : kind(kind_) {}
+ Node(Kind kind_, const NodeData& data_) : kind(kind_), data(data_) {}
+ Node(Kind kind_, const NodeData& data_, const PathInData& path_)
+ : kind(kind_), data(data_), path(path_) {}
+
+ Kind kind = TUPLE;
+ const Node* parent = nullptr;
+
+ Arena strings_pool;
+ HashMapWithStackMemory<StringRef, std::shared_ptr<Node>, StringRefHash, 4> children;
+
+ NodeData data;
+ PathInData path;
+
+ bool is_nested() const { return kind == NESTED; }
+ bool is_scalar() const { return kind == SCALAR; }
+
+ void add_child(std::string_view key, std::shared_ptr<Node> next_node) {
+ next_node->parent = this;
+ StringRef key_ref {strings_pool.insert(key.data(), key.length()), key.length()};
+ children[key_ref] = std::move(next_node);
+ }
+ };
+
+ using NodeKind = typename Node::Kind;
+ using NodePtr = std::shared_ptr<Node>;
+
+ /// Add a leaf without any data in other nodes.
+ bool add(const PathInData& path, const NodeData& leaf_data) {
+ return add(path, [&](NodeKind kind, bool exists) -> NodePtr {
+ if (exists) return nullptr;
+
+ if (kind == Node::SCALAR) return std::make_shared<Node>(kind, leaf_data, path);
+
+ return std::make_shared<Node>(kind);
+ });
+ }
+
+ /// Callback for creation of node. Receives kind of node and
+ /// flag, which is true if node already exists.
+ using NodeCreator = std::function<NodePtr(NodeKind, bool)>;
+
+ bool add(const PathInData& path, const NodeCreator& node_creator) {
+ const auto& parts = path.get_parts();
+
+ if (parts.empty()) return false;
+
+ if (!root) root = std::make_shared<Node>(Node::TUPLE);
+
+ Node* current_node = root.get();
+ for (size_t i = 0; i < parts.size() - 1; ++i) {
+ assert(current_node->kind != Node::SCALAR);
+
+ auto it = current_node->children.find(
+ StringRef {parts[i].key.data(), parts[i].key.size()});
+ if (it != current_node->children.end()) {
+ current_node = it->get_second().get();
+ node_creator(current_node->kind, true);
+
+ if (current_node->is_nested() != parts[i].is_nested) return false;
+ } else {
+ auto next_kind = parts[i].is_nested ? Node::NESTED : Node::TUPLE;
+ auto next_node = node_creator(next_kind, false);
+ current_node->add_child(String(parts[i].key), next_node);
+ current_node = next_node.get();
+ }
+ }
+
+ auto it = current_node->children.find(
+ StringRef {parts.back().key.data(), parts.back().key.size()});
+ if (it != current_node->children.end()) return false;
+
+ auto next_node = node_creator(Node::SCALAR, false);
+ current_node->add_child(String(parts.back().key), next_node);
+ leaves.push_back(std::move(next_node));
+
+ return true;
+ }
+
+ /// Find node that matches the path the best.
+ const Node* find_best_match(const PathInData& path) const { return find_impl(path, false); }
+
+ /// Find node that matches the path exactly.
+ const Node* find_exact(const PathInData& path) const { return find_impl(path, true); }
+
+ /// Find leaf by path.
+ const Node* find_leaf(const PathInData& path) const {
+ const auto* candidate = find_exact(path);
+ if (!candidate || !candidate->is_scalar()) return nullptr;
+ return candidate;
+ }
+
+ using NodePredicate = std::function<bool(const Node&)>;
+
+ /// Finds leaf that satisfies the predicate.
+ const Node* find_leaf(const NodePredicate& predicate) {
+ return find_leaf(root.get(), predicate);
+ }
+
+ static const Node* find_leaf(const Node* node, const NodePredicate& predicate) {
+ if (!node) return nullptr;
+
+ if (node->is_scalar()) return predicate(*node) ? node : nullptr;
+
+ for (auto it = node->children.begin(); it != node->children.end(); ++it) {
+ auto child = it->get_second();
+ if (const auto* leaf = find_leaf(child.get(), predicate)) return leaf;
+ }
+ return nullptr;
+ }
+
+ /// Find first parent node that satisfies the predicate.
+ static const Node* find_parent(const Node* node, const NodePredicate& predicate) {
+ while (node && !predicate(*node)) node = node->parent;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
while (node && !predicate(*node)) { node = node->parent;
}
```
##########
be/src/vec/common/schema_util.cpp:
##########
@@ -0,0 +1,481 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+
+#include <vec/columns/column_array.h>
+#include <vec/columns/column_object.h>
+#include <vec/common/schema_util.h>
+#include <vec/core/field.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_object.h>
+#include <vec/functions/simple_function_factory.h>
+#include <vec/json/parse2column.h>
+
+#include <vec/data_types/data_type_factory.hpp>
+
+#include "gen_cpp/FrontendService.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "olap/rowset/rowset_writer_context.h"
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "util/thrift_rpc_helper.h"
+
+namespace doris::vectorized::schema_util {
+
+size_t get_number_of_dimensions(const IDataType& type) {
+ if (const auto* type_array = typeid_cast<const DataTypeArray*>(&type))
+ return type_array->get_number_of_dimensions();
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (const auto* type_array = typeid_cast<const DataTypeArray*>(&type)) {
return type_array->get_number_of_dimensions();
}
```
##########
be/src/vec/core/accurate_comparison.h:
##########
@@ -456,22 +456,36 @@ inline bool_if_safe_conversion<A, B> greaterOrEqualsOp(A a, B b) {
}
/// Converts numeric to an equal numeric of other type.
-template <typename From, typename To>
+/// When `strict` is `true` check that result exactly same as input, otherwise just check overflow
+template <typename From, typename To, bool strict = true>
inline bool convertNumeric(From value, To& result) {
/// If the type is actually the same it's not necessary to do any checks.
if constexpr (std::is_same_v<From, To>) {
result = value;
return true;
}
-
- /// Note that NaNs doesn't compare equal to anything, but they are still in range of any Float type.
- if (is_nan(value) && std::is_floating_point_v<To>) {
- result = value;
- return true;
+ if constexpr (std::is_floating_point_v<From> && std::is_floating_point_v<To>) {
+ /// Note that NaNs doesn't compare equal to anything, but they are still in range of any Float type.
+ if (is_nan(value)) {
+ result = value;
+ return true;
+ }
+ if (value == std::numeric_limits<From>::infinity()) {
+ result = std::numeric_limits<To>::infinity();
+ return true;
+ }
+ if (value == -std::numeric_limits<From>::infinity()) {
+ result = -std::numeric_limits<To>::infinity();
+ return true;
+ }
+ }
+ if (greaterOp(value, std::numeric_limits<To>::max()) ||
+ lessOp(value, std::numeric_limits<To>::lowest())) {
+ return false;
}
-
result = static_cast<To>(value);
- return equalsOp(value, result);
+ if constexpr (strict) return equalsOp(value, result);
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if constexpr (strict) { return equalsOp(value, result);
}
```
##########
be/src/vec/columns/subcolumn_tree.h:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/DataTypes/Serializations/SubcolumnsTree.h
+// and modified by Doris
+
+#pragma once
+#include <vec/columns/column.h>
+#include <vec/common/hash_table/hash_map.h>
+#include <vec/data_types/data_type.h>
+#include <vec/json/path_in_data.h>
+namespace doris::vectorized {
+/// Tree that represents paths in document
+/// with additional data in nodes.
+
+template <typename NodeData>
+class SubcolumnsTree {
+public:
+ struct Node {
+ enum Kind {
+ TUPLE,
+ NESTED,
+ SCALAR,
+ };
+
+ explicit Node(Kind kind_) : kind(kind_) {}
+ Node(Kind kind_, const NodeData& data_) : kind(kind_), data(data_) {}
+ Node(Kind kind_, const NodeData& data_, const PathInData& path_)
+ : kind(kind_), data(data_), path(path_) {}
+
+ Kind kind = TUPLE;
+ const Node* parent = nullptr;
+
+ Arena strings_pool;
+ HashMapWithStackMemory<StringRef, std::shared_ptr<Node>, StringRefHash, 4> children;
+
+ NodeData data;
+ PathInData path;
+
+ bool is_nested() const { return kind == NESTED; }
+ bool is_scalar() const { return kind == SCALAR; }
+
+ void add_child(std::string_view key, std::shared_ptr<Node> next_node) {
+ next_node->parent = this;
+ StringRef key_ref {strings_pool.insert(key.data(), key.length()), key.length()};
+ children[key_ref] = std::move(next_node);
+ }
+ };
+
+ using NodeKind = typename Node::Kind;
+ using NodePtr = std::shared_ptr<Node>;
+
+ /// Add a leaf without any data in other nodes.
+ bool add(const PathInData& path, const NodeData& leaf_data) {
+ return add(path, [&](NodeKind kind, bool exists) -> NodePtr {
+ if (exists) return nullptr;
+
+ if (kind == Node::SCALAR) return std::make_shared<Node>(kind, leaf_data, path);
+
+ return std::make_shared<Node>(kind);
+ });
+ }
+
+ /// Callback for creation of node. Receives kind of node and
+ /// flag, which is true if node already exists.
+ using NodeCreator = std::function<NodePtr(NodeKind, bool)>;
+
+ bool add(const PathInData& path, const NodeCreator& node_creator) {
+ const auto& parts = path.get_parts();
+
+ if (parts.empty()) return false;
+
+ if (!root) root = std::make_shared<Node>(Node::TUPLE);
+
+ Node* current_node = root.get();
+ for (size_t i = 0; i < parts.size() - 1; ++i) {
+ assert(current_node->kind != Node::SCALAR);
+
+ auto it = current_node->children.find(
+ StringRef {parts[i].key.data(), parts[i].key.size()});
+ if (it != current_node->children.end()) {
+ current_node = it->get_second().get();
+ node_creator(current_node->kind, true);
+
+ if (current_node->is_nested() != parts[i].is_nested) return false;
+ } else {
+ auto next_kind = parts[i].is_nested ? Node::NESTED : Node::TUPLE;
+ auto next_node = node_creator(next_kind, false);
+ current_node->add_child(String(parts[i].key), next_node);
+ current_node = next_node.get();
+ }
+ }
+
+ auto it = current_node->children.find(
+ StringRef {parts.back().key.data(), parts.back().key.size()});
+ if (it != current_node->children.end()) return false;
+
+ auto next_node = node_creator(Node::SCALAR, false);
+ current_node->add_child(String(parts.back().key), next_node);
+ leaves.push_back(std::move(next_node));
+
+ return true;
+ }
+
+ /// Find node that matches the path the best.
+ const Node* find_best_match(const PathInData& path) const { return find_impl(path, false); }
+
+ /// Find node that matches the path exactly.
+ const Node* find_exact(const PathInData& path) const { return find_impl(path, true); }
+
+ /// Find leaf by path.
+ const Node* find_leaf(const PathInData& path) const {
+ const auto* candidate = find_exact(path);
+ if (!candidate || !candidate->is_scalar()) return nullptr;
+ return candidate;
+ }
+
+ using NodePredicate = std::function<bool(const Node&)>;
+
+ /// Finds leaf that satisfies the predicate.
+ const Node* find_leaf(const NodePredicate& predicate) {
+ return find_leaf(root.get(), predicate);
+ }
+
+ static const Node* find_leaf(const Node* node, const NodePredicate& predicate) {
+ if (!node) return nullptr;
+
+ if (node->is_scalar()) return predicate(*node) ? node : nullptr;
+
+ for (auto it = node->children.begin(); it != node->children.end(); ++it) {
+ auto child = it->get_second();
+ if (const auto* leaf = find_leaf(child.get(), predicate)) return leaf;
+ }
+ return nullptr;
+ }
+
+ /// Find first parent node that satisfies the predicate.
+ static const Node* find_parent(const Node* node, const NodePredicate& predicate) {
+ while (node && !predicate(*node)) node = node->parent;
+ return node;
+ }
+
+ bool empty() const { return root == nullptr; }
+ size_t size() const { return leaves.size(); }
+
+ using Nodes = std::vector<NodePtr>;
+
+ const Nodes& get_leaves() const { return leaves; }
+ const Node* get_root() const { return root.get(); }
+
+ using iterator = typename Nodes::iterator;
+ using const_iterator = typename Nodes::const_iterator;
+
+ iterator begin() { return leaves.begin(); }
+ iterator end() { return leaves.end(); }
+
+ const_iterator begin() const { return leaves.begin(); }
+ const_iterator end() const { return leaves.end(); }
+
+private:
+ const Node* find_impl(const PathInData& path, bool find_exact) const {
+ if (!root) return nullptr;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!root) { return nullptr;
}
```
##########
be/src/vec/data_types/convert_field_to_type.cpp:
##########
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Interpreters/convert_field_to_type.cpp
+// and modified by Doris
+
+#include <vec/common/field_visitors.h>
+#include <vec/common/typeid_cast.h>
+#include <vec/core/accurate_comparison.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_date.h>
+#include <vec/data_types/data_type_date_time.h>
+#include <vec/data_types/data_type_decimal.h>
+#include <vec/data_types/data_type_nullable.h>
+#include <vec/data_types/data_type_number.h>
+#include <vec/data_types/data_type_object.h>
+#include <vec/data_types/data_type_string.h>
+// #include <vec/data_types/data_type_tuple.h>
+namespace doris::vectorized {
+/** Checking for a `Field from` of `From` type falls to a range of values of type `To`.
+ * `From` and `To` - numeric types. They can be floating-point types.
+ * `From` is one of UInt64, Int64, Float64,
+ * whereas `To` can also be 8, 16, 32 bit.
+ *
+ * If falls into a range, then `from` is converted to the `Field` closest to the `To` type.
+ * If not, return Field(Null).
+ */
+
+/** simple types of implementation of visitor to string*/
+// TODO support more types
+class FieldVisitorToStringSimple : public StaticVisitor<String> {
+public:
+ String operator()(const Null& x) const { return "NULL"; }
+ String operator()(const UInt64& x) const { return std::to_string(x); }
+ String operator()(const Int64& x) const { return std::to_string(x); }
+ String operator()(const Float64& x) const { return std::to_string(x); }
+ String operator()(const String& x) const { return x; }
+ [[noreturn]] String operator()(const UInt128& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const Array& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const Tuple& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const DecimalField<Decimal32>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal64>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal128>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal128I>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const AggregateFunctionStateData& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+};
+
+namespace {
+template <typename From, typename To>
+Field convert_numeric_type_impl(const Field& from) {
+ To result;
+ if (!accurate::convertNumeric(from.get<From>(), result)) return {};
+ return result;
+}
+template <typename To>
+Status convert_numric_type(const Field& from, const IDataType& type, Field* to) {
+ if (from.get_type() == Field::Types::UInt64)
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (from.get_type() == Field::Types::UInt64) {
```
be/src/vec/data_types/convert_field_to_type.cpp:83:
```diff
- else if (from.get_type() == Field::Types::Int64)
+ } else if (from.get_type() == Field::Types::Int64)
```
##########
be/src/vec/common/schema_util.cpp:
##########
@@ -0,0 +1,481 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+
+#include <vec/columns/column_array.h>
+#include <vec/columns/column_object.h>
+#include <vec/common/schema_util.h>
+#include <vec/core/field.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_object.h>
+#include <vec/functions/simple_function_factory.h>
+#include <vec/json/parse2column.h>
+
+#include <vec/data_types/data_type_factory.hpp>
+
+#include "gen_cpp/FrontendService.h"
+#include "gen_cpp/HeartbeatService_types.h"
+#include "olap/rowset/rowset_writer_context.h"
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "util/thrift_rpc_helper.h"
+
+namespace doris::vectorized::schema_util {
+
+size_t get_number_of_dimensions(const IDataType& type) {
+ if (const auto* type_array = typeid_cast<const DataTypeArray*>(&type))
+ return type_array->get_number_of_dimensions();
+ return 0;
+}
+size_t get_number_of_dimensions(const IColumn& column) {
+ if (const auto* column_array = check_and_get_column<ColumnArray>(column))
+ return column_array->get_number_of_dimensions();
+ return 0;
+}
+
+DataTypePtr get_base_type_of_array(const DataTypePtr& type) {
+ /// Get raw pointers to avoid extra copying of type pointers.
+ const DataTypeArray* last_array = nullptr;
+ const auto* current_type = type.get();
+ while (const auto* type_array = typeid_cast<const DataTypeArray*>(current_type)) {
+ current_type = type_array->get_nested_type().get();
+ last_array = type_array;
+ }
+ return last_array ? last_array->get_nested_type() : type;
+}
+
+Array create_empty_array_field(size_t num_dimensions) {
+ DCHECK(num_dimensions > 0);
+ Array array;
+ Array* current_array = &array;
+ for (size_t i = 1; i < num_dimensions; ++i) {
+ current_array->push_back(Array());
+ current_array = ¤t_array->back().get<Array&>();
+ }
+ return array;
+}
+
+FieldType get_field_type(const IDataType* data_type) {
+ switch (data_type->get_type_id()) {
+ case TypeIndex::UInt8:
+ return FieldType::OLAP_FIELD_TYPE_UNSIGNED_TINYINT;
+ case TypeIndex::UInt16:
+ return FieldType::OLAP_FIELD_TYPE_UNSIGNED_SMALLINT;
+ case TypeIndex::UInt32:
+ return FieldType::OLAP_FIELD_TYPE_UNSIGNED_INT;
+ case TypeIndex::UInt64:
+ return FieldType::OLAP_FIELD_TYPE_UNSIGNED_BIGINT;
+ case TypeIndex::Int8:
+ return FieldType::OLAP_FIELD_TYPE_TINYINT;
+ case TypeIndex::Int16:
+ return FieldType::OLAP_FIELD_TYPE_SMALLINT;
+ case TypeIndex::Int32:
+ return FieldType::OLAP_FIELD_TYPE_INT;
+ case TypeIndex::Int64:
+ return FieldType::OLAP_FIELD_TYPE_BIGINT;
+ case TypeIndex::Float32:
+ return FieldType::OLAP_FIELD_TYPE_FLOAT;
+ case TypeIndex::Float64:
+ return FieldType::OLAP_FIELD_TYPE_DOUBLE;
+ case TypeIndex::Decimal32:
+ return FieldType::OLAP_FIELD_TYPE_DECIMAL;
+ case TypeIndex::Array:
+ return FieldType::OLAP_FIELD_TYPE_ARRAY;
+ case TypeIndex::String:
+ return FieldType::OLAP_FIELD_TYPE_STRING;
+ case TypeIndex::Date:
+ return FieldType::OLAP_FIELD_TYPE_DATE;
+ case TypeIndex::DateTime:
+ return FieldType::OLAP_FIELD_TYPE_DATETIME;
+ case TypeIndex::Tuple:
+ return FieldType::OLAP_FIELD_TYPE_STRUCT;
+ // TODO add more types
+ default:
+ LOG(FATAL) << "unknow type";
+ return FieldType::OLAP_FIELD_TYPE_UNKNOWN;
+ }
+}
+
+Status parse_object_column(ColumnObject& dest, const IColumn& src, bool need_finalize,
+ const int* row_begin, const int* row_end) {
+ assert(src.is_column_string());
+ const ColumnString* parsing_column {nullptr};
+ if (!src.is_nullable()) {
+ parsing_column = reinterpret_cast<const ColumnString*>(src.get_ptr().get());
+ } else {
+ auto nullable_column = reinterpret_cast<const ColumnNullable*>(src.get_ptr().get());
+ parsing_column = reinterpret_cast<const ColumnString*>(
+ nullable_column->get_nested_column().get_ptr().get());
+ }
+ std::vector<StringRef> jsons;
+ if (row_begin != nullptr) {
+ assert(row_end);
+ for (auto x = row_begin; x != row_end; ++x) {
+ StringRef ref = parsing_column->get_data_at(*x);
+ jsons.push_back(ref);
+ }
+ } else {
+ for (size_t i = 0; i < parsing_column->size(); ++i) {
+ StringRef ref = parsing_column->get_data_at(i);
+ jsons.push_back(ref);
+ }
+ }
+ // batch parse
+ RETURN_IF_ERROR(parse_json_to_variant(dest, jsons));
+
+ if (need_finalize) {
+ dest.finalize();
+ }
+ return Status::OK();
+}
+
+Status parse_object_column(Block& block, size_t position) {
+ // parse variant column and rewrite column
+ auto col = block.get_by_position(position).column;
+ const std::string& col_name = block.get_by_position(position).name;
+ if (!col->is_column_string()) {
+ return Status::InvalidArgument("only ColumnString can be parsed to ColumnObject");
+ }
+ vectorized::DataTypePtr type(
+ std::make_shared<vectorized::DataTypeObject>("", col->is_nullable()));
+ auto column_object = type->create_column();
+ RETURN_IF_ERROR(
+ parse_object_column(assert_cast<ColumnObject&>(column_object->assume_mutable_ref()),
+ *col, true /*need finalize*/, nullptr, nullptr));
+ // replace by object
+ block.safe_get_by_position(position).column = column_object->get_ptr();
+ block.safe_get_by_position(position).type = type;
+ block.safe_get_by_position(position).name = col_name;
+ return Status::OK();
+}
+
+void flatten_object(Block& block, size_t pos, bool replace_if_duplicated) {
+ auto column_object_ptr =
+ assert_cast<ColumnObject*>(block.get_by_position(pos).column->assume_mutable().get());
+ if (column_object_ptr->empty()) {
+ block.erase(pos);
+ return;
+ }
+ size_t num_rows = column_object_ptr->size();
+ assert(block.rows() <= num_rows);
+ assert(column_object_ptr->is_finalized());
+ Columns subcolumns;
+ DataTypes types;
+ Names names;
+ for (auto& subcolumn : column_object_ptr->get_subcolumns()) {
+ subcolumns.push_back(subcolumn->data.get_finalized_column().get_ptr());
+ types.push_back(subcolumn->data.get_least_common_type());
+ names.push_back(subcolumn->path.get_path());
+ }
+ block.erase(pos);
+ for (size_t i = 0; i < subcolumns.size(); ++i) {
+ // block may already contains this column, eg. key columns, we should ignore
+ // or replcace the same column from object subcolumn
+ if (block.has(names[i])) {
+ if (replace_if_duplicated) {
+ auto& column_type_name = block.get_by_name(names[i]);
+ column_type_name.column = subcolumns[i];
+ column_type_name.type = types[i];
+ }
+ continue;
+ }
+ block.insert(ColumnWithTypeAndName {subcolumns[i], types[i], names[i]});
+ }
+
+ // fill default value
+ for (auto& [column, _1, _2] : block.get_columns_with_type_and_name()) {
+ if (column->size() < num_rows) {
+ column->assume_mutable()->insert_many_defaults(num_rows - column->size());
+ }
+ }
+}
+
+Status flatten_object(Block& block, bool replace_if_duplicated) {
+ auto object_pos =
+ std::find_if(block.begin(), block.end(), [](const ColumnWithTypeAndName& column) {
+ return column.type->get_type_id() == TypeIndex::VARIANT;
+ });
+ if (object_pos != block.end()) {
+ flatten_object(block, object_pos - block.begin(), replace_if_duplicated);
+ }
+ return Status::OK();
+}
+
+bool is_conversion_required_between_integers(const IDataType& lhs, const IDataType& rhs) {
+ WhichDataType which_lhs(lhs);
+ WhichDataType which_rhs(rhs);
+ bool is_native_int = which_lhs.is_native_int() && which_rhs.is_native_int();
+ bool is_native_uint = which_lhs.is_native_uint() && which_rhs.is_native_uint();
+ return (is_native_int || is_native_uint) &&
+ lhs.get_size_of_value_in_memory() <= rhs.get_size_of_value_in_memory();
+}
+
+bool is_conversion_required_between_integers(FieldType lhs, FieldType rhs) {
+ // We only support signed integers for semi-structure data at present
+ // TODO add unsigned integers
+ if (lhs == OLAP_FIELD_TYPE_BIGINT) {
+ return !(rhs == OLAP_FIELD_TYPE_TINYINT || rhs == OLAP_FIELD_TYPE_SMALLINT ||
+ rhs == OLAP_FIELD_TYPE_INT || rhs == OLAP_FIELD_TYPE_BIGINT);
+ }
+ if (lhs == OLAP_FIELD_TYPE_INT) {
+ return !(rhs == OLAP_FIELD_TYPE_TINYINT || rhs == OLAP_FIELD_TYPE_SMALLINT ||
+ rhs == OLAP_FIELD_TYPE_INT);
+ }
+ if (lhs == OLAP_FIELD_TYPE_SMALLINT) {
+ return !(rhs == OLAP_FIELD_TYPE_TINYINT || rhs == OLAP_FIELD_TYPE_SMALLINT);
+ }
+ if (lhs == OLAP_FIELD_TYPE_TINYINT) {
+ return !(rhs == OLAP_FIELD_TYPE_TINYINT);
+ }
+ return true;
+}
+
+Status cast_column(const ColumnWithTypeAndName& arg, const DataTypePtr& type, ColumnPtr* result) {
+ ColumnsWithTypeAndName arguments {arg,
+ {type->create_column_const_with_default_value(1), type, ""}};
+ auto function = SimpleFunctionFactory::instance().get_function("CAST", arguments, type);
+ Block tmp_block {arguments};
+ // the 0 position is input argument, the 1 position is to type argument, the 2 position is result argument
+ vectorized::ColumnNumbers argnum;
+ argnum.emplace_back(0);
+ argnum.emplace_back(1);
+ size_t result_column = tmp_block.columns();
+ tmp_block.insert({nullptr, type, arg.name});
+ RETURN_IF_ERROR(
+ function->execute(nullptr, tmp_block, argnum, result_column, arg.column->size()));
+ *result = std::move(tmp_block.get_by_position(result_column).column);
+ return Status::OK();
+}
+
+static void get_column_def(const vectorized::DataTypePtr& data_type, const std::string& name,
+ TColumnDef* column) {
+ if (!name.empty()) {
+ column->columnDesc.__set_columnName(name);
+ }
+ if (data_type->is_nullable()) {
+ const auto& real_type = static_cast<const DataTypeNullable&>(*data_type);
+ column->columnDesc.__set_isAllowNull(true);
+ get_column_def(real_type.get_nested_type(), "", column);
+ return;
+ }
+ column->columnDesc.__set_columnType(to_thrift(get_primitive_type(data_type->get_type_id())));
+ if (data_type->get_type_id() == TypeIndex::Array) {
+ TColumnDef child;
+ column->columnDesc.__set_children({});
+ get_column_def(assert_cast<const DataTypeArray*>(data_type.get())->get_nested_type(), "",
+ &child);
+ column->columnDesc.columnLength =
+ TabletColumn::get_field_length_by_type(column->columnDesc.columnType, 0);
+ column->columnDesc.children.push_back(child.columnDesc);
+ return;
+ }
+ if (data_type->get_type_id() == TypeIndex::Tuple) {
+ // TODO
+ // auto tuple_type = assert_cast<const DataTypeTuple*>(data_type.get());
+ // DCHECK_EQ(tuple_type->get_elements().size(), tuple_type->get_element_names().size());
+ // for (size_t i = 0; i < tuple_type->get_elements().size(); ++i) {
+ // TColumnDef child;
+ // get_column_def(tuple_type->get_element(i), tuple_type->get_element_names()[i], &child);
+ // column->columnDesc.children.push_back(child.columnDesc);
+ // }
+ // return;
+ }
+ if (data_type->get_type_id() == TypeIndex::String) {
+ return;
+ }
+ if (WhichDataType(*data_type).is_simple()) {
+ column->columnDesc.__set_columnLength(data_type->get_size_of_value_in_memory());
+ return;
+ }
+ return;
+}
Review Comment:
warning: redundant return statement at the end of a function with a void return type [readability-redundant-control-flow]
```suggestion
}
```
##########
be/src/vec/core/block.cpp:
##########
@@ -978,4 +999,33 @@ void MutableBlock::clear_column_data() noexcept {
}
}
+void MutableBlock::initialize_index_by_name() {
+ for (size_t i = 0, size = _names.size(); i < size; ++i) {
+ index_by_name[_names[i]] = i;
+ }
+}
+
+bool MutableBlock::has(const std::string& name) const {
+ return index_by_name.end() != index_by_name.find(name);
+}
+
+size_t MutableBlock::get_position_by_name(const std::string& name) const {
+ auto it = index_by_name.find(name);
+ if (index_by_name.end() == it) {
+ LOG(FATAL) << fmt::format("Not found column {} in block. There are only columns: {}", name,
+ dump_names());
+ }
+
+ return it->second;
+}
+
+std::string MutableBlock::dump_names() const {
+ std::stringstream out;
+ for (auto it = _names.begin(); it != _names.end(); ++it) {
+ if (it != _names.begin()) out << ", ";
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (it != _names.begin()) { out << ", ";
}
```
##########
be/src/vec/data_types/convert_field_to_type.cpp:
##########
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Interpreters/convert_field_to_type.cpp
+// and modified by Doris
+
+#include <vec/common/field_visitors.h>
+#include <vec/common/typeid_cast.h>
+#include <vec/core/accurate_comparison.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_date.h>
+#include <vec/data_types/data_type_date_time.h>
+#include <vec/data_types/data_type_decimal.h>
+#include <vec/data_types/data_type_nullable.h>
+#include <vec/data_types/data_type_number.h>
+#include <vec/data_types/data_type_object.h>
+#include <vec/data_types/data_type_string.h>
+// #include <vec/data_types/data_type_tuple.h>
+namespace doris::vectorized {
+/** Checking for a `Field from` of `From` type falls to a range of values of type `To`.
+ * `From` and `To` - numeric types. They can be floating-point types.
+ * `From` is one of UInt64, Int64, Float64,
+ * whereas `To` can also be 8, 16, 32 bit.
+ *
+ * If falls into a range, then `from` is converted to the `Field` closest to the `To` type.
+ * If not, return Field(Null).
+ */
+
+/** simple types of implementation of visitor to string*/
+// TODO support more types
+class FieldVisitorToStringSimple : public StaticVisitor<String> {
+public:
+ String operator()(const Null& x) const { return "NULL"; }
+ String operator()(const UInt64& x) const { return std::to_string(x); }
+ String operator()(const Int64& x) const { return std::to_string(x); }
+ String operator()(const Float64& x) const { return std::to_string(x); }
+ String operator()(const String& x) const { return x; }
+ [[noreturn]] String operator()(const UInt128& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const Array& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const Tuple& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const DecimalField<Decimal32>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal64>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal128>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal128I>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const AggregateFunctionStateData& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+};
+
+namespace {
+template <typename From, typename To>
+Field convert_numeric_type_impl(const Field& from) {
+ To result;
+ if (!accurate::convertNumeric(from.get<From>(), result)) return {};
+ return result;
+}
+template <typename To>
+Status convert_numric_type(const Field& from, const IDataType& type, Field* to) {
+ if (from.get_type() == Field::Types::UInt64)
+ *to = convert_numeric_type_impl<UInt64, To>(from);
+ else if (from.get_type() == Field::Types::Int64)
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
else if (from.get_type() == Field::Types::Int64) {
```
be/src/vec/data_types/convert_field_to_type.cpp:85:
```diff
- else if (from.get_type() == Field::Types::Float64)
+ } else if (from.get_type() == Field::Types::Float64)
```
##########
be/src/vec/data_types/convert_field_to_type.cpp:
##########
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Interpreters/convert_field_to_type.cpp
+// and modified by Doris
+
+#include <vec/common/field_visitors.h>
+#include <vec/common/typeid_cast.h>
+#include <vec/core/accurate_comparison.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_date.h>
+#include <vec/data_types/data_type_date_time.h>
+#include <vec/data_types/data_type_decimal.h>
+#include <vec/data_types/data_type_nullable.h>
+#include <vec/data_types/data_type_number.h>
+#include <vec/data_types/data_type_object.h>
+#include <vec/data_types/data_type_string.h>
+// #include <vec/data_types/data_type_tuple.h>
+namespace doris::vectorized {
+/** Checking for a `Field from` of `From` type falls to a range of values of type `To`.
+ * `From` and `To` - numeric types. They can be floating-point types.
+ * `From` is one of UInt64, Int64, Float64,
+ * whereas `To` can also be 8, 16, 32 bit.
+ *
+ * If falls into a range, then `from` is converted to the `Field` closest to the `To` type.
+ * If not, return Field(Null).
+ */
+
+/** simple types of implementation of visitor to string*/
+// TODO support more types
+class FieldVisitorToStringSimple : public StaticVisitor<String> {
+public:
+ String operator()(const Null& x) const { return "NULL"; }
+ String operator()(const UInt64& x) const { return std::to_string(x); }
+ String operator()(const Int64& x) const { return std::to_string(x); }
+ String operator()(const Float64& x) const { return std::to_string(x); }
+ String operator()(const String& x) const { return x; }
+ [[noreturn]] String operator()(const UInt128& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const Array& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const Tuple& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const DecimalField<Decimal32>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal64>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal128>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal128I>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const AggregateFunctionStateData& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+};
+
+namespace {
+template <typename From, typename To>
+Field convert_numeric_type_impl(const Field& from) {
+ To result;
+ if (!accurate::convertNumeric(from.get<From>(), result)) return {};
+ return result;
+}
+template <typename To>
+Status convert_numric_type(const Field& from, const IDataType& type, Field* to) {
+ if (from.get_type() == Field::Types::UInt64)
+ *to = convert_numeric_type_impl<UInt64, To>(from);
+ else if (from.get_type() == Field::Types::Int64)
+ *to = convert_numeric_type_impl<Int64, To>(from);
+ else if (from.get_type() == Field::Types::Float64)
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
else if (from.get_type() == Field::Types::Float64) {
```
be/src/vec/data_types/convert_field_to_type.cpp:87:
```diff
- else if (from.get_type() == Field::Types::UInt128)
+ } else if (from.get_type() == Field::Types::UInt128)
```
##########
be/src/vec/data_types/convert_field_to_type.cpp:
##########
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Interpreters/convert_field_to_type.cpp
+// and modified by Doris
+
+#include <vec/common/field_visitors.h>
+#include <vec/common/typeid_cast.h>
+#include <vec/core/accurate_comparison.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_date.h>
+#include <vec/data_types/data_type_date_time.h>
+#include <vec/data_types/data_type_decimal.h>
+#include <vec/data_types/data_type_nullable.h>
+#include <vec/data_types/data_type_number.h>
+#include <vec/data_types/data_type_object.h>
+#include <vec/data_types/data_type_string.h>
+// #include <vec/data_types/data_type_tuple.h>
+namespace doris::vectorized {
+/** Checking for a `Field from` of `From` type falls to a range of values of type `To`.
+ * `From` and `To` - numeric types. They can be floating-point types.
+ * `From` is one of UInt64, Int64, Float64,
+ * whereas `To` can also be 8, 16, 32 bit.
+ *
+ * If falls into a range, then `from` is converted to the `Field` closest to the `To` type.
+ * If not, return Field(Null).
+ */
+
+/** simple types of implementation of visitor to string*/
+// TODO support more types
+class FieldVisitorToStringSimple : public StaticVisitor<String> {
+public:
+ String operator()(const Null& x) const { return "NULL"; }
+ String operator()(const UInt64& x) const { return std::to_string(x); }
+ String operator()(const Int64& x) const { return std::to_string(x); }
+ String operator()(const Float64& x) const { return std::to_string(x); }
+ String operator()(const String& x) const { return x; }
+ [[noreturn]] String operator()(const UInt128& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const Array& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const Tuple& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const DecimalField<Decimal32>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal64>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal128>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal128I>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const AggregateFunctionStateData& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+};
+
+namespace {
+template <typename From, typename To>
+Field convert_numeric_type_impl(const Field& from) {
+ To result;
+ if (!accurate::convertNumeric(from.get<From>(), result)) return {};
+ return result;
+}
+template <typename To>
+Status convert_numric_type(const Field& from, const IDataType& type, Field* to) {
+ if (from.get_type() == Field::Types::UInt64)
+ *to = convert_numeric_type_impl<UInt64, To>(from);
+ else if (from.get_type() == Field::Types::Int64)
+ *to = convert_numeric_type_impl<Int64, To>(from);
+ else if (from.get_type() == Field::Types::Float64)
+ *to = convert_numeric_type_impl<Float64, To>(from);
+ else if (from.get_type() == Field::Types::UInt128)
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
else if (from.get_type() == Field::Types::UInt128) {
```
be/src/vec/data_types/convert_field_to_type.cpp:89:
```diff
- else if (from.get_type() == Field::Types::Int128)
+ } else if (from.get_type() == Field::Types::Int128)
```
##########
be/src/vec/data_types/convert_field_to_type.cpp:
##########
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Interpreters/convert_field_to_type.cpp
+// and modified by Doris
+
+#include <vec/common/field_visitors.h>
+#include <vec/common/typeid_cast.h>
+#include <vec/core/accurate_comparison.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_date.h>
+#include <vec/data_types/data_type_date_time.h>
+#include <vec/data_types/data_type_decimal.h>
+#include <vec/data_types/data_type_nullable.h>
+#include <vec/data_types/data_type_number.h>
+#include <vec/data_types/data_type_object.h>
+#include <vec/data_types/data_type_string.h>
+// #include <vec/data_types/data_type_tuple.h>
+namespace doris::vectorized {
+/** Checking for a `Field from` of `From` type falls to a range of values of type `To`.
+ * `From` and `To` - numeric types. They can be floating-point types.
+ * `From` is one of UInt64, Int64, Float64,
+ * whereas `To` can also be 8, 16, 32 bit.
+ *
+ * If falls into a range, then `from` is converted to the `Field` closest to the `To` type.
+ * If not, return Field(Null).
+ */
+
+/** simple types of implementation of visitor to string*/
+// TODO support more types
+class FieldVisitorToStringSimple : public StaticVisitor<String> {
+public:
+ String operator()(const Null& x) const { return "NULL"; }
+ String operator()(const UInt64& x) const { return std::to_string(x); }
+ String operator()(const Int64& x) const { return std::to_string(x); }
+ String operator()(const Float64& x) const { return std::to_string(x); }
+ String operator()(const String& x) const { return x; }
+ [[noreturn]] String operator()(const UInt128& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const Array& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const Tuple& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const DecimalField<Decimal32>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal64>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal128>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal128I>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const AggregateFunctionStateData& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+};
+
+namespace {
+template <typename From, typename To>
+Field convert_numeric_type_impl(const Field& from) {
+ To result;
+ if (!accurate::convertNumeric(from.get<From>(), result)) return {};
+ return result;
+}
+template <typename To>
+Status convert_numric_type(const Field& from, const IDataType& type, Field* to) {
+ if (from.get_type() == Field::Types::UInt64)
+ *to = convert_numeric_type_impl<UInt64, To>(from);
+ else if (from.get_type() == Field::Types::Int64)
+ *to = convert_numeric_type_impl<Int64, To>(from);
+ else if (from.get_type() == Field::Types::Float64)
+ *to = convert_numeric_type_impl<Float64, To>(from);
+ else if (from.get_type() == Field::Types::UInt128)
+ *to = convert_numeric_type_impl<UInt128, To>(from);
+ else if (from.get_type() == Field::Types::Int128)
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
else if (from.get_type() == Field::Types::Int128) {
```
be/src/vec/data_types/convert_field_to_type.cpp:91:
```diff
- else
+ } else
```
##########
be/src/vec/data_types/convert_field_to_type.cpp:
##########
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Interpreters/convert_field_to_type.cpp
+// and modified by Doris
+
+#include <vec/common/field_visitors.h>
+#include <vec/common/typeid_cast.h>
+#include <vec/core/accurate_comparison.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_date.h>
+#include <vec/data_types/data_type_date_time.h>
+#include <vec/data_types/data_type_decimal.h>
+#include <vec/data_types/data_type_nullable.h>
+#include <vec/data_types/data_type_number.h>
+#include <vec/data_types/data_type_object.h>
+#include <vec/data_types/data_type_string.h>
+// #include <vec/data_types/data_type_tuple.h>
+namespace doris::vectorized {
+/** Checking for a `Field from` of `From` type falls to a range of values of type `To`.
+ * `From` and `To` - numeric types. They can be floating-point types.
+ * `From` is one of UInt64, Int64, Float64,
+ * whereas `To` can also be 8, 16, 32 bit.
+ *
+ * If falls into a range, then `from` is converted to the `Field` closest to the `To` type.
+ * If not, return Field(Null).
+ */
+
+/** simple types of implementation of visitor to string*/
+// TODO support more types
+class FieldVisitorToStringSimple : public StaticVisitor<String> {
+public:
+ String operator()(const Null& x) const { return "NULL"; }
+ String operator()(const UInt64& x) const { return std::to_string(x); }
+ String operator()(const Int64& x) const { return std::to_string(x); }
+ String operator()(const Float64& x) const { return std::to_string(x); }
+ String operator()(const String& x) const { return x; }
+ [[noreturn]] String operator()(const UInt128& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const Array& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const Tuple& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const DecimalField<Decimal32>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal64>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal128>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal128I>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const AggregateFunctionStateData& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+};
+
+namespace {
+template <typename From, typename To>
+Field convert_numeric_type_impl(const Field& from) {
+ To result;
+ if (!accurate::convertNumeric(from.get<From>(), result)) return {};
+ return result;
+}
+template <typename To>
+Status convert_numric_type(const Field& from, const IDataType& type, Field* to) {
+ if (from.get_type() == Field::Types::UInt64)
+ *to = convert_numeric_type_impl<UInt64, To>(from);
+ else if (from.get_type() == Field::Types::Int64)
+ *to = convert_numeric_type_impl<Int64, To>(from);
+ else if (from.get_type() == Field::Types::Float64)
+ *to = convert_numeric_type_impl<Float64, To>(from);
+ else if (from.get_type() == Field::Types::UInt128)
+ *to = convert_numeric_type_impl<UInt128, To>(from);
+ else if (from.get_type() == Field::Types::Int128)
+ *to = convert_numeric_type_impl<Int128, To>(from);
+ else
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
else {
```
be/src/vec/data_types/convert_field_to_type.cpp:94:
```diff
- type.get_name(), from.get_type()));
+ type.get_name(), from.get_type()));
+ }
```
##########
be/src/vec/data_types/convert_field_to_type.cpp:
##########
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Interpreters/convert_field_to_type.cpp
+// and modified by Doris
+
+#include <vec/common/field_visitors.h>
+#include <vec/common/typeid_cast.h>
+#include <vec/core/accurate_comparison.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_date.h>
+#include <vec/data_types/data_type_date_time.h>
+#include <vec/data_types/data_type_decimal.h>
+#include <vec/data_types/data_type_nullable.h>
+#include <vec/data_types/data_type_number.h>
+#include <vec/data_types/data_type_object.h>
+#include <vec/data_types/data_type_string.h>
+// #include <vec/data_types/data_type_tuple.h>
+namespace doris::vectorized {
+/** Checking for a `Field from` of `From` type falls to a range of values of type `To`.
+ * `From` and `To` - numeric types. They can be floating-point types.
+ * `From` is one of UInt64, Int64, Float64,
+ * whereas `To` can also be 8, 16, 32 bit.
+ *
+ * If falls into a range, then `from` is converted to the `Field` closest to the `To` type.
+ * If not, return Field(Null).
+ */
+
+/** simple types of implementation of visitor to string*/
+// TODO support more types
+class FieldVisitorToStringSimple : public StaticVisitor<String> {
+public:
+ String operator()(const Null& x) const { return "NULL"; }
+ String operator()(const UInt64& x) const { return std::to_string(x); }
+ String operator()(const Int64& x) const { return std::to_string(x); }
+ String operator()(const Float64& x) const { return std::to_string(x); }
+ String operator()(const String& x) const { return x; }
+ [[noreturn]] String operator()(const UInt128& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const Array& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const Tuple& x) const { LOG(FATAL) << "not implemeted"; }
+ [[noreturn]] String operator()(const DecimalField<Decimal32>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal64>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal128>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const DecimalField<Decimal128I>& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+ [[noreturn]] String operator()(const AggregateFunctionStateData& x) const {
+ LOG(FATAL) << "not implemeted";
+ }
+};
+
+namespace {
+template <typename From, typename To>
+Field convert_numeric_type_impl(const Field& from) {
+ To result;
+ if (!accurate::convertNumeric(from.get<From>(), result)) return {};
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!accurate::convertNumeric(from.get<From>(), result)) { return {};
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] eldenmoon closed pull request #16319: [DRAFT] Dynamic table
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon closed pull request #16319: [DRAFT] Dynamic table
URL: https://github.com/apache/doris/pull/16319
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16319: [DRAFT] Dynamic table
Posted by github-actions.
github-actions[bot] commented on code in PR #16319:
URL: https://github.com/apache/doris/pull/16319#discussion_r1092931347
##########
be/src/vec/columns/column_array.h:
##########
@@ -177,6 +178,20 @@ class ColumnArray final : public COWHelper<IColumn, ColumnArray> {
}
Status filter_by_selector(const uint16_t* sel, size_t sel_size, IColumn* col_ptr) override;
+ size_t get_number_of_dimensions() const {
+ const auto* nested_array = check_and_get_column<ColumnArray>(*data);
+ if (!nested_array) return 1;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!nested_array) { return 1;
}
```
##########
be/src/vec/columns/column_object.cpp:
##########
@@ -0,0 +1,860 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.cpp
+// and modified by Doris
+
+#include <parallel_hashmap/phmap.h>
+#include <vec/columns/column_array.h>
+#include <vec/columns/column_nullable.h>
+#include <vec/columns/column_object.h>
+#include <vec/columns/columns_number.h>
+#include <vec/common/field_visitors.h>
+#include <vec/common/hash_table/hash_set.h>
+#include <vec/common/object_util.h>
+#include <vec/common/pod_array_fwd.h>
+#include <vec/core/field.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_nothing.h>
+#include <vec/data_types/get_least_supertype.h>
+
+#include <vec/data_types/data_type_factory.hpp>
+
+namespace doris::vectorized {
+namespace {
+
+DataTypePtr create_array_of_type(DataTypePtr type, size_t num_dimensions) {
+ for (size_t i = 0; i < num_dimensions; ++i)
+ type = std::make_shared<DataTypeArray>(std::move(type));
+ return type;
+}
+
+DataTypePtr getBaseTypeOfArray(const DataTypePtr& type) {
+ /// Get raw pointers to avoid extra copying of type pointers.
+ const DataTypeArray* last_array = nullptr;
+ const auto* current_type = type.get();
+ while (const auto* type_array = typeid_cast<const DataTypeArray*>(current_type)) {
+ current_type = type_array->get_nested_type().get();
+ last_array = type_array;
+ }
+ return last_array ? last_array->get_nested_type() : type;
+}
+
+size_t getNumberOfDimensions(const IDataType& type) {
+ if (const auto* type_array = typeid_cast<const DataTypeArray*>(&type)) {
+ return type_array->get_number_of_dimensions();
+ }
+ return 0;
+}
+
+DataTypePtr get_data_type_by_column(const IColumn& column) {
+ auto idx = column.get_data_type();
+ if (WhichDataType(idx).is_simple()) {
+ return DataTypeFactory::instance().get(String(getTypeName(idx)));
+ }
+ if (WhichDataType(idx).is_nothing()) {
+ return std::make_shared<DataTypeNothing>();
+ }
+ if (const auto* column_array = check_and_get_column<ColumnArray>(&column)) {
+ return std::make_shared<DataTypeArray>(get_data_type_by_column(column_array->get_data()));
+ }
+ if (const auto* column_nullable = check_and_get_column<ColumnNullable>(&column)) {
+ return make_nullable(get_data_type_by_column(column_nullable->get_nested_column()));
+ }
+ // TODO add more types
+ assert(false);
+ return nullptr;
+}
+
+/// Recreates column with default scalar values and keeps sizes of arrays.
+ColumnPtr recreate_column_with_default_value(const ColumnPtr& column,
+ const DataTypePtr& scalar_type,
+ size_t num_dimensions) {
+ const auto* column_array = check_and_get_column<ColumnArray>(column.get());
+ if (column_array && num_dimensions) {
+ return ColumnArray::create(
+ recreate_column_with_default_value(column_array->get_data_ptr(), scalar_type,
+ num_dimensions - 1),
+ IColumn::mutate(column_array->get_offsets_ptr()));
+ }
+ return create_array_of_type(scalar_type, num_dimensions)
+ ->create_column()
+ ->clone_resized(column->size());
+}
+
+Array create_empty_array_field(size_t num_dimensions) {
+ assert(num_dimensions != 0);
+ Array array;
+ Array* current_array = &array;
+ for (size_t i = 1; i < num_dimensions; ++i) {
+ current_array->push_back(Array());
+ current_array = ¤t_array->back().get<Array&>();
+ }
+ return array;
+}
+
+/// Replaces NULL fields to given field or empty array.
+class FieldVisitorReplaceNull : public StaticVisitor<Field> {
+public:
+ explicit FieldVisitorReplaceNull(const Field& replacement_, size_t num_dimensions_)
+ : replacement(replacement_), num_dimensions(num_dimensions_) {}
+ Field operator()(const Null&) const {
+ return num_dimensions ? create_empty_array_field(num_dimensions) : replacement;
+ }
+ Field operator()(const Array& x) const {
+ assert(num_dimensions > 0);
+ const size_t size = x.size();
+ Array res(size);
+ for (size_t i = 0; i < size; ++i) {
+ res[i] = apply_visitor(FieldVisitorReplaceNull(replacement, num_dimensions - 1), x[i]);
+ }
+ return res;
+ }
+ template <typename T>
+ Field operator()(const T& x) const {
+ return x;
+ }
+
+private:
+ const Field& replacement;
+ size_t num_dimensions;
+};
+
+/// Calculates number of dimensions in array field.
+/// Returns 0 for scalar fields.
+class FieldVisitorToNumberOfDimensions : public StaticVisitor<size_t> {
+public:
+ explicit FieldVisitorToNumberOfDimensions(Status* st) : _st(st) {}
+ size_t operator()(const Array& x) const {
+ const size_t size = x.size();
+ std::optional<size_t> dimensions;
+ for (size_t i = 0; i < size; ++i) {
+ /// Do not count Nulls, because they will be replaced by default
+ /// values with proper number of dimensions.
+ if (x[i].is_null()) continue;
+ size_t current_dimensions = apply_visitor(*this, x[i]);
+ if (!dimensions) {
+ dimensions = current_dimensions;
+ } else if (current_dimensions != *dimensions) {
+ *_st = Status::InvalidArgument(
+ "Number of dimensions mismatched among array elements");
+ return 0;
+ }
+ }
+ return 1 + dimensions.value_or(0);
+ }
+ template <typename T>
+ size_t operator()(const T&) const {
+ return 0;
+ }
+
+private:
+ mutable Status* _st;
+};
+
+/// Visitor that allows to get type of scalar field
+/// or least common type of scalars in array.
+/// More optimized version of FieldToDataType.
+class FieldVisitorToScalarType : public StaticVisitor<size_t> {
+public:
+ using FieldType = Field::Types::Which;
+ size_t operator()(const Array& x) {
+ size_t size = x.size();
+ for (size_t i = 0; i < size; ++i) apply_visitor(*this, x[i]);
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
for (size_t i = 0; i < size; ++i) { apply_visitor(*this, x[i]);
}
```
##########
be/src/vec/columns/column_object.cpp:
##########
@@ -0,0 +1,860 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.cpp
+// and modified by Doris
+
+#include <parallel_hashmap/phmap.h>
+#include <vec/columns/column_array.h>
+#include <vec/columns/column_nullable.h>
+#include <vec/columns/column_object.h>
+#include <vec/columns/columns_number.h>
+#include <vec/common/field_visitors.h>
+#include <vec/common/hash_table/hash_set.h>
+#include <vec/common/object_util.h>
+#include <vec/common/pod_array_fwd.h>
+#include <vec/core/field.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_nothing.h>
+#include <vec/data_types/get_least_supertype.h>
+
+#include <vec/data_types/data_type_factory.hpp>
+
+namespace doris::vectorized {
+namespace {
+
+DataTypePtr create_array_of_type(DataTypePtr type, size_t num_dimensions) {
+ for (size_t i = 0; i < num_dimensions; ++i)
+ type = std::make_shared<DataTypeArray>(std::move(type));
+ return type;
+}
+
+DataTypePtr getBaseTypeOfArray(const DataTypePtr& type) {
+ /// Get raw pointers to avoid extra copying of type pointers.
+ const DataTypeArray* last_array = nullptr;
+ const auto* current_type = type.get();
+ while (const auto* type_array = typeid_cast<const DataTypeArray*>(current_type)) {
+ current_type = type_array->get_nested_type().get();
+ last_array = type_array;
+ }
+ return last_array ? last_array->get_nested_type() : type;
+}
+
+size_t getNumberOfDimensions(const IDataType& type) {
+ if (const auto* type_array = typeid_cast<const DataTypeArray*>(&type)) {
+ return type_array->get_number_of_dimensions();
+ }
+ return 0;
+}
+
+DataTypePtr get_data_type_by_column(const IColumn& column) {
+ auto idx = column.get_data_type();
+ if (WhichDataType(idx).is_simple()) {
+ return DataTypeFactory::instance().get(String(getTypeName(idx)));
+ }
+ if (WhichDataType(idx).is_nothing()) {
+ return std::make_shared<DataTypeNothing>();
+ }
+ if (const auto* column_array = check_and_get_column<ColumnArray>(&column)) {
+ return std::make_shared<DataTypeArray>(get_data_type_by_column(column_array->get_data()));
+ }
+ if (const auto* column_nullable = check_and_get_column<ColumnNullable>(&column)) {
+ return make_nullable(get_data_type_by_column(column_nullable->get_nested_column()));
+ }
+ // TODO add more types
+ assert(false);
+ return nullptr;
+}
+
+/// Recreates column with default scalar values and keeps sizes of arrays.
+ColumnPtr recreate_column_with_default_value(const ColumnPtr& column,
+ const DataTypePtr& scalar_type,
+ size_t num_dimensions) {
+ const auto* column_array = check_and_get_column<ColumnArray>(column.get());
+ if (column_array && num_dimensions) {
+ return ColumnArray::create(
+ recreate_column_with_default_value(column_array->get_data_ptr(), scalar_type,
+ num_dimensions - 1),
+ IColumn::mutate(column_array->get_offsets_ptr()));
+ }
+ return create_array_of_type(scalar_type, num_dimensions)
+ ->create_column()
+ ->clone_resized(column->size());
+}
+
+Array create_empty_array_field(size_t num_dimensions) {
+ assert(num_dimensions != 0);
+ Array array;
+ Array* current_array = &array;
+ for (size_t i = 1; i < num_dimensions; ++i) {
+ current_array->push_back(Array());
+ current_array = ¤t_array->back().get<Array&>();
+ }
+ return array;
+}
+
+/// Replaces NULL fields to given field or empty array.
+class FieldVisitorReplaceNull : public StaticVisitor<Field> {
+public:
+ explicit FieldVisitorReplaceNull(const Field& replacement_, size_t num_dimensions_)
+ : replacement(replacement_), num_dimensions(num_dimensions_) {}
+ Field operator()(const Null&) const {
+ return num_dimensions ? create_empty_array_field(num_dimensions) : replacement;
+ }
+ Field operator()(const Array& x) const {
+ assert(num_dimensions > 0);
+ const size_t size = x.size();
+ Array res(size);
+ for (size_t i = 0; i < size; ++i) {
+ res[i] = apply_visitor(FieldVisitorReplaceNull(replacement, num_dimensions - 1), x[i]);
+ }
+ return res;
+ }
+ template <typename T>
+ Field operator()(const T& x) const {
+ return x;
+ }
+
+private:
+ const Field& replacement;
+ size_t num_dimensions;
+};
+
+/// Calculates number of dimensions in array field.
+/// Returns 0 for scalar fields.
+class FieldVisitorToNumberOfDimensions : public StaticVisitor<size_t> {
+public:
+ explicit FieldVisitorToNumberOfDimensions(Status* st) : _st(st) {}
+ size_t operator()(const Array& x) const {
+ const size_t size = x.size();
+ std::optional<size_t> dimensions;
+ for (size_t i = 0; i < size; ++i) {
+ /// Do not count Nulls, because they will be replaced by default
+ /// values with proper number of dimensions.
+ if (x[i].is_null()) continue;
+ size_t current_dimensions = apply_visitor(*this, x[i]);
+ if (!dimensions) {
+ dimensions = current_dimensions;
+ } else if (current_dimensions != *dimensions) {
+ *_st = Status::InvalidArgument(
+ "Number of dimensions mismatched among array elements");
+ return 0;
+ }
+ }
+ return 1 + dimensions.value_or(0);
+ }
+ template <typename T>
+ size_t operator()(const T&) const {
+ return 0;
+ }
+
+private:
+ mutable Status* _st;
+};
+
+/// Visitor that allows to get type of scalar field
+/// or least common type of scalars in array.
+/// More optimized version of FieldToDataType.
+class FieldVisitorToScalarType : public StaticVisitor<size_t> {
+public:
+ using FieldType = Field::Types::Which;
+ size_t operator()(const Array& x) {
+ size_t size = x.size();
+ for (size_t i = 0; i < size; ++i) apply_visitor(*this, x[i]);
+ return 0;
+ }
+ // TODO doris not support unsigned integers for now
+ // treat as signed integers
+ size_t operator()(const UInt64& x) {
+ field_types.insert(FieldType::UInt64);
+ if (x <= std::numeric_limits<Int8>::max()) {
+ type_indexes.insert(TypeIndex::Int8);
+ } else if (x <= std::numeric_limits<Int16>::max()) {
+ type_indexes.insert(TypeIndex::Int16);
+ } else if (x <= std::numeric_limits<Int32>::max()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Int64& x) {
+ // Only support Int32 and Int64
+ field_types.insert(FieldType::Int64);
+ if (x <= std::numeric_limits<Int32>::max() && x >= std::numeric_limits<Int32>::min()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Null&) {
+ have_nulls = true;
+ return 0;
+ }
+ size_t operator()(const Int128I& x) { LOG(FATAL) << "not implemented"; }
+ template <typename T>
+ size_t operator()(const T&) {
+ Field::EnumToType<Field::Types::Array>::Type a;
+ field_types.insert(Field::TypeToEnum<NearestFieldType<T>>::value);
+ type_indexes.insert(TypeId<NearestFieldType<T>>::value);
+ return 0;
+ }
+ Status get_scalar_type(DataTypePtr* type) const {
+ return get_least_supertype(type_indexes, type, true /*compatible with string type*/);
+ }
+ bool contain_nulls() const { return have_nulls; }
+ bool need_convert_field() const { return field_types.size() > 1; }
+
+private:
+ phmap::flat_hash_set<TypeIndex> type_indexes;
+ phmap::flat_hash_set<FieldType> field_types;
+ bool have_nulls = false;
+};
+
+} // namespace
+Status get_field_info(const Field& field, FieldInfo* info) {
+ FieldVisitorToScalarType to_scalar_type_visitor;
+ apply_visitor(to_scalar_type_visitor, field);
+ DataTypePtr type = nullptr;
+ RETURN_IF_ERROR(to_scalar_type_visitor.get_scalar_type(&type));
+ // array item's dimension may missmatch, eg. [1, 2, [1, 2, 3]]
+ Status num_to_dimensions_status;
+ *info = {
+ type,
+ to_scalar_type_visitor.contain_nulls(),
+ to_scalar_type_visitor.need_convert_field(),
+ apply_visitor(FieldVisitorToNumberOfDimensions(&num_to_dimensions_status), field),
+ };
+ RETURN_IF_ERROR(num_to_dimensions_status);
+ return Status::OK();
+}
+
+ColumnObject::Subcolumn::Subcolumn(MutableColumnPtr&& data_, bool is_nullable_)
+ : least_common_type(get_data_type_by_column(*data_)), is_nullable(is_nullable_) {
+ data.push_back(std::move(data_));
+}
+
+ColumnObject::Subcolumn::Subcolumn(size_t size_, bool is_nullable_)
+ : least_common_type(std::make_shared<DataTypeNothing>()),
+ is_nullable(is_nullable_),
+ num_of_defaults_in_prefix(size_) {}
+
+size_t ColumnObject::Subcolumn::Subcolumn::size() const {
+ size_t res = num_of_defaults_in_prefix;
+ for (const auto& part : data) {
+ res += part->size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::byteSize() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->byte_size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::allocatedBytes() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->allocated_bytes();
+ }
+ return res;
+}
+
+Status ColumnObject::Subcolumn::insert(Field field) {
+ FieldInfo info;
+ RETURN_IF_ERROR(get_field_info(field, &info));
+ return insert(std::move(field), std::move(info));
+}
+
+void ColumnObject::Subcolumn::add_new_column_part(DataTypePtr type) {
+ data.push_back(type->create_column());
+ least_common_type = LeastCommonType {std::move(type)};
+}
+
+Status ColumnObject::Subcolumn::insert(Field field, FieldInfo info) {
+ auto base_type = std::move(info.scalar_type);
+ if (is_nothing(base_type)) {
+ insertDefault();
+ return Status::OK();
+ }
+ auto column_dim = least_common_type.get_dimensions();
+ auto value_dim = info.num_dimensions;
+ if (is_nothing(least_common_type.getBase())) {
+ column_dim = value_dim;
+ }
+ if (is_nothing(base_type)) {
+ value_dim = column_dim;
+ }
+ if (value_dim != column_dim) {
+ return Status::InvalidArgument(
+ "Dimension of types mismatched between inserted value and column.");
+ }
+ if (is_nullable && !is_nothing(base_type)) {
+ base_type = make_nullable(base_type);
+ }
+ // alawys nullable at present
+ if (!is_nullable && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+ // need replace muli dimensions array which contains null. eg. [[1, 2, 3], null] -> [[1, 2, 3], []]
+ // since column array doesnt known null's dimension
+ if (info.num_dimensions >= 2 && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+
+ bool type_changed = false;
+ const auto& least_common_base_type = least_common_type.getBase();
+ if (data.empty()) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ } else if (!least_common_base_type->equals(*base_type) && !is_nothing(base_type)) {
+ if (!object_util::is_conversion_required_between_integers(*base_type,
+ *least_common_base_type)) {
+ RETURN_IF_ERROR(
+ get_least_supertype(DataTypes {std::move(base_type), least_common_base_type},
+ &base_type, true /*compatible with string type*/));
+ type_changed = true;
+ if (!least_common_base_type->equals(*base_type)) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ }
+ }
+ }
+
+ if (type_changed || info.need_convert) {
+ Field new_field;
+ RETURN_IF_ERROR(convert_field_to_type(field, *least_common_type.get(), &new_field));
+ field = new_field;
+ }
+
+ data.back()->insert(field);
+ return Status::OK();
+}
+
+Status ColumnObject::Subcolumn::insertRangeFrom(const Subcolumn& src, size_t start, size_t length) {
+ assert(src.is_finalized());
+ const auto& src_column = src.data.back();
+ const auto& src_type = src.least_common_type.get();
+ if (data.empty()) {
+ add_new_column_part(src.least_common_type.get());
+ data.back()->insert_range_from(*src_column, start, length);
+ } else if (least_common_type.get()->equals(*src_type)) {
+ data.back()->insert_range_from(*src_column, start, length);
+ } else {
+ DataTypePtr new_least_common_type = nullptr;
+ RETURN_IF_ERROR(get_least_supertype(DataTypes {least_common_type.get(), src_type},
+ &new_least_common_type,
+ true /*compatible with string type*/));
+ ColumnPtr casted_column;
+ RETURN_IF_ERROR(object_util::cast_column({src_column, src_type, ""}, new_least_common_type,
+ &casted_column));
+ if (!least_common_type.get()->equals(*new_least_common_type))
+ add_new_column_part(std::move(new_least_common_type));
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!least_common_type.get()->equals(*new_least_common_type)) {
add_new_column_part(std::move(new_least_common_type));
}
```
##########
be/src/vec/columns/column_const.cpp:
##########
@@ -163,4 +163,23 @@ void ColumnConst::get_permutation(bool /*reverse*/, size_t /*limit*/, int /*nan_
}
}
+void ColumnConst::get_indices_of_non_default_rows(Offsets64& indices, size_t from,
+ size_t limit) const {
+ if (!data->is_default_at(0)) {
+ size_t to = limit && from + limit < size() ? from + limit : size();
+ indices.reserve(indices.size() + to - from);
+ for (size_t i = from; i < to; ++i) indices.push_back(i);
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
for (size_t i = from; i < to; ++i) { indices.push_back(i);
}
```
##########
be/src/vec/columns/column_object.h:
##########
@@ -0,0 +1,364 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.h
+// and modified by Doris
+
+#pragma once
+#include <vec/columns/column.h>
+#include <vec/columns/subcolumn_tree.h>
+#include <vec/common/hash_table/hash_map.h>
+#include <vec/common/pod_array.h>
+#include <vec/core/field.h>
+#include <vec/core/names.h>
+#include <vec/data_types/data_type.h>
+#include <vec/json/json_parser.h>
+
+#include "common/status.h"
+namespace doris::vectorized {
+
+/// Info that represents a scalar or array field in a decomposed view.
+/// It allows to recreate field with different number
+/// of dimensions or nullability.
+struct FieldInfo {
+ /// The common type of of all scalars in field.
+ DataTypePtr scalar_type;
+ /// Do we have NULL scalar in field.
+ bool have_nulls;
+ /// If true then we have scalars with different types in array and
+ /// we need to convert scalars to the common type.
+ bool need_convert;
+ /// Number of dimension in array. 0 if field is scalar.
+ size_t num_dimensions;
+};
+Status get_field_info(const Field& field, FieldInfo* info);
+/** A column that represents object with dynamic set of subcolumns.
+ * Subcolumns are identified by paths in document and are stored in
+ * a trie-like structure. ColumnObject is not suitable for writing into tables
+ * and it should be converted to Tuple with fixed set of subcolumns before that.
+ */
+class ColumnObject final : public COWHelper<IColumn, ColumnObject> {
+public:
+ /** Class that represents one subcolumn.
+ * It stores values in several parts of column
+ * and keeps current common type of all parts.
+ * We add a new column part with a new type, when we insert a field,
+ * which can't be converted to the current common type.
+ * After insertion of all values subcolumn should be finalized
+ * for writing and other operations.
+ */
+ class Subcolumn {
+ public:
+ Subcolumn() = default;
+
+ Subcolumn(size_t size_, bool is_nullable_);
+
+ Subcolumn(MutableColumnPtr&& data_, bool is_nullable_);
+
+ size_t size() const;
+
+ size_t byteSize() const;
+
+ size_t allocatedBytes() const;
+
+ bool is_finalized() const;
+
+ const DataTypePtr& get_least_common_type() const { return least_common_type.get(); }
+
+ const DataTypePtr& get_least_common_typeBase() const { return least_common_type.getBase(); }
+
+ size_t get_dimensions() const { return least_common_type.get_dimensions(); }
+
+ /// Checks the consistency of column's parts stored in @data.
+ void checkTypes() const;
+
+ /// Inserts a field, which scalars can be arbitrary, but number of
+ /// dimensions should be consistent with current common type.
+ /// return Status::InvalidArgument when meet conflict types
+ Status insert(Field field);
+
+ Status insert(Field field, FieldInfo info);
+
+ void insertDefault();
+
+ void insertManyDefaults(size_t length);
+
+ Status insertRangeFrom(const Subcolumn& src, size_t start, size_t length);
+
+ void pop_back(size_t n);
+
+ /// Converts all column's parts to the common type and
+ /// creates a single column that stores all values.
+ void finalize();
+
+ /// Returns last inserted field.
+ Field getLastField() const;
+
+ /// Recreates subcolumn with default scalar values and keeps sizes of arrays.
+ /// Used to create columns of type Nested with consistent array sizes.
+ Subcolumn recreate_with_default_values(const FieldInfo& field_info) const;
+
+ /// Returns single column if subcolumn in finalizes.
+ /// Otherwise -- undefined behaviour.
+ IColumn& get_finalized_column();
+
+ const IColumn& get_finalized_column() const;
+
+ const ColumnPtr& get_finalized_column_ptr() const;
+
+ friend class ColumnObject;
+
+ private:
+ class LeastCommonType {
+ public:
+ LeastCommonType() = default;
+
+ explicit LeastCommonType(DataTypePtr type_);
+
+ const DataTypePtr& get() const { return type; }
+
+ const DataTypePtr& getBase() const { return base_type; }
+
+ size_t get_dimensions() const { return num_dimensions; }
+
+ private:
+ DataTypePtr type;
+ DataTypePtr base_type;
+ size_t num_dimensions = 0;
+ };
+ void add_new_column_part(DataTypePtr type);
+
+ /// Current least common type of all values inserted to this subcolumn.
+ LeastCommonType least_common_type;
+ /// If true then common type type of subcolumn is Nullable
+ /// and default values are NULLs.
+ bool is_nullable = false;
+ /// Parts of column. Parts should be in increasing order in terms of subtypes/supertypes.
+ /// That means that the least common type for i-th prefix is the type of i-th part
+ /// and it's the supertype for all type of column from 0 to i-1.
+ std::vector<WrappedPtr> data;
+ /// Until we insert any non-default field we don't know further
+ /// least common type and we count number of defaults in prefix,
+ /// which will be converted to the default type of final common type.
+ size_t num_of_defaults_in_prefix = 0;
+ };
+ using Subcolumns = SubcolumnsTree<Subcolumn>;
+
+private:
+ /// If true then all subcolumns are nullable.
+ const bool is_nullable;
+ Subcolumns subcolumns;
+ size_t num_rows;
+
+public:
+ static constexpr auto COLUMN_NAME_DUMMY = "_dummy";
+
+ explicit ColumnObject(bool is_nullable_);
+
+ ColumnObject(Subcolumns&& subcolumns_, bool is_nullable_);
+
+ ~ColumnObject() = default;
Review Comment:
warning: annotate this function with 'override' or (rarely) 'final' [modernize-use-override]
```suggestion
~ColumnObject() override = default;
```
##########
be/src/vec/columns/column_object.h:
##########
@@ -0,0 +1,364 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.h
+// and modified by Doris
+
+#pragma once
+#include <vec/columns/column.h>
+#include <vec/columns/subcolumn_tree.h>
+#include <vec/common/hash_table/hash_map.h>
+#include <vec/common/pod_array.h>
+#include <vec/core/field.h>
+#include <vec/core/names.h>
+#include <vec/data_types/data_type.h>
+#include <vec/json/json_parser.h>
+
+#include "common/status.h"
+namespace doris::vectorized {
+
+/// Info that represents a scalar or array field in a decomposed view.
+/// It allows to recreate field with different number
+/// of dimensions or nullability.
+struct FieldInfo {
+ /// The common type of of all scalars in field.
+ DataTypePtr scalar_type;
+ /// Do we have NULL scalar in field.
+ bool have_nulls;
+ /// If true then we have scalars with different types in array and
+ /// we need to convert scalars to the common type.
+ bool need_convert;
+ /// Number of dimension in array. 0 if field is scalar.
+ size_t num_dimensions;
+};
+Status get_field_info(const Field& field, FieldInfo* info);
+/** A column that represents object with dynamic set of subcolumns.
+ * Subcolumns are identified by paths in document and are stored in
+ * a trie-like structure. ColumnObject is not suitable for writing into tables
+ * and it should be converted to Tuple with fixed set of subcolumns before that.
+ */
+class ColumnObject final : public COWHelper<IColumn, ColumnObject> {
+public:
+ /** Class that represents one subcolumn.
+ * It stores values in several parts of column
+ * and keeps current common type of all parts.
+ * We add a new column part with a new type, when we insert a field,
+ * which can't be converted to the current common type.
+ * After insertion of all values subcolumn should be finalized
+ * for writing and other operations.
+ */
+ class Subcolumn {
+ public:
+ Subcolumn() = default;
+
+ Subcolumn(size_t size_, bool is_nullable_);
+
+ Subcolumn(MutableColumnPtr&& data_, bool is_nullable_);
+
+ size_t size() const;
+
+ size_t byteSize() const;
+
+ size_t allocatedBytes() const;
+
+ bool is_finalized() const;
+
+ const DataTypePtr& get_least_common_type() const { return least_common_type.get(); }
+
+ const DataTypePtr& get_least_common_typeBase() const { return least_common_type.getBase(); }
+
+ size_t get_dimensions() const { return least_common_type.get_dimensions(); }
+
+ /// Checks the consistency of column's parts stored in @data.
+ void checkTypes() const;
+
+ /// Inserts a field, which scalars can be arbitrary, but number of
+ /// dimensions should be consistent with current common type.
+ /// return Status::InvalidArgument when meet conflict types
+ Status insert(Field field);
+
+ Status insert(Field field, FieldInfo info);
+
+ void insertDefault();
+
+ void insertManyDefaults(size_t length);
+
+ Status insertRangeFrom(const Subcolumn& src, size_t start, size_t length);
+
+ void pop_back(size_t n);
+
+ /// Converts all column's parts to the common type and
+ /// creates a single column that stores all values.
+ void finalize();
+
+ /// Returns last inserted field.
+ Field getLastField() const;
+
+ /// Recreates subcolumn with default scalar values and keeps sizes of arrays.
+ /// Used to create columns of type Nested with consistent array sizes.
+ Subcolumn recreate_with_default_values(const FieldInfo& field_info) const;
+
+ /// Returns single column if subcolumn in finalizes.
+ /// Otherwise -- undefined behaviour.
+ IColumn& get_finalized_column();
+
+ const IColumn& get_finalized_column() const;
+
+ const ColumnPtr& get_finalized_column_ptr() const;
+
+ friend class ColumnObject;
+
+ private:
+ class LeastCommonType {
+ public:
+ LeastCommonType() = default;
+
+ explicit LeastCommonType(DataTypePtr type_);
+
+ const DataTypePtr& get() const { return type; }
+
+ const DataTypePtr& getBase() const { return base_type; }
+
+ size_t get_dimensions() const { return num_dimensions; }
+
+ private:
+ DataTypePtr type;
+ DataTypePtr base_type;
+ size_t num_dimensions = 0;
+ };
+ void add_new_column_part(DataTypePtr type);
+
+ /// Current least common type of all values inserted to this subcolumn.
+ LeastCommonType least_common_type;
+ /// If true then common type type of subcolumn is Nullable
+ /// and default values are NULLs.
+ bool is_nullable = false;
+ /// Parts of column. Parts should be in increasing order in terms of subtypes/supertypes.
+ /// That means that the least common type for i-th prefix is the type of i-th part
+ /// and it's the supertype for all type of column from 0 to i-1.
+ std::vector<WrappedPtr> data;
+ /// Until we insert any non-default field we don't know further
+ /// least common type and we count number of defaults in prefix,
+ /// which will be converted to the default type of final common type.
+ size_t num_of_defaults_in_prefix = 0;
+ };
+ using Subcolumns = SubcolumnsTree<Subcolumn>;
+
+private:
+ /// If true then all subcolumns are nullable.
+ const bool is_nullable;
+ Subcolumns subcolumns;
+ size_t num_rows;
+
+public:
+ static constexpr auto COLUMN_NAME_DUMMY = "_dummy";
+
+ explicit ColumnObject(bool is_nullable_);
+
+ ColumnObject(Subcolumns&& subcolumns_, bool is_nullable_);
+
+ ~ColumnObject() = default;
+
+ bool can_be_inside_nullable() const override { return true; }
+
+ /// Checks that all subcolumns have consistent sizes.
+ void check_consistency() const;
+
+ bool has_subcolumn(const PathInData& key) const;
+
+ // return null if not found
+ const Subcolumn* get_subcolumn(const PathInData& key) const;
+
+ // return null if not found
+ Subcolumn* get_subcolumn(const PathInData& key);
+
+ void incr_num_rows() { ++num_rows; }
+
+ /// Adds a subcolumn from existing IColumn.
+ bool add_sub_column(const PathInData& key, MutableColumnPtr&& subcolumn);
+
+ /// Adds a subcolumn of specific size with default values.
+ bool add_sub_column(const PathInData& key, size_t new_size);
+
+ /// Adds a subcolumn of type Nested of specific size with default values.
+ /// It cares about consistency of sizes of Nested arrays.
+ bool add_nested_subcolumn(const PathInData& key, const FieldInfo& field_info, size_t new_size);
+
+ const Subcolumns& get_subcolumns() const { return subcolumns; }
+
+ Subcolumns& get_subcolumns() { return subcolumns; }
+
+ PathsInData getKeys() const;
+
+ std::string get_keys_str() const {
+ std::stringstream ss;
+ bool first = true;
+ for (auto& k : getKeys()) {
+ if (first) {
+ first = false;
+ } else {
+ ss << ", ";
+ }
+ ss << k.get_path();
+ }
+
+ return ss.str();
+ }
+
+ void remove_subcolumns(const std::unordered_set<std::string>& keys);
+
+ /// Finalizes all subcolumns.
+ void finalize();
+
+ bool is_finalized() const;
+
+ /// Part of interface
+ const char* get_family_name() const override { return "Variant"; }
+
+ TypeIndex get_data_type() const override { return TypeIndex::VARIANT; }
+
+ size_t size() const override;
+
+ MutableColumnPtr clone_resized(size_t new_size) const override;
+
+ size_t byte_size() const override;
+
+ size_t allocated_bytes() const override;
+
+ void for_each_subcolumn(ColumnCallback callback) override;
+
+ // Do nothing, call try_insert instead
+ void insert(const Field& field) override {
+ Status st = try_insert(field);
+ if (!st.ok()) LOG(FATAL) << "insert return ERROR status: " << st;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!st.ok()) { LOG(FATAL) << "insert return ERROR status: " << st;
}
```
##########
be/src/vec/columns/columns_common.h:
##########
@@ -51,4 +51,43 @@
PaddedPODArray<T>& res_elems, const IColumn::Filter& filt,
ssize_t result_size_hint);
+namespace detail {
+template <typename T>
+const PaddedPODArray<T>* get_indexes_data(const IColumn& indexes);
+}
+
+/// Check limit <= indexes->size() and call column.index_impl(const PaddedPodArray<Type> & indexes, UInt64 limit).
+template <typename Column>
+ColumnPtr select_index_impl(const Column& column, const IColumn& indexes, size_t limit) {
+ if (limit == 0) limit = indexes.size();
+
+ if (indexes.size() < limit) {
+ LOG(FATAL) << "Size of indexes is less than required.";
+ }
+
+ if (auto* data_uint8 = detail::get_indexes_data<UInt8>(indexes))
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (auto* data_uint8 = detail::get_indexes_data<UInt8>(indexes)) {
```
be/src/vec/columns/columns_common.h:69:
```diff
- else if (auto* data_uint16 = detail::get_indexes_data<UInt16>(indexes))
+ } else if (auto* data_uint16 = detail::get_indexes_data<UInt16>(indexes))
```
##########
be/src/vec/columns/column_impl.h:
##########
@@ -74,4 +74,14 @@ void IColumn::append_data_by_selector_impl(MutablePtr& res, const Selector& sele
static_cast<Derived&>(*res).insert_from(*this, selector[i]);
}
+template <typename Derived>
+void IColumn::get_indices_of_non_default_rows_impl(IColumn::Offsets64& indices, size_t from,
+ size_t limit) const {
+ size_t to = limit && from + limit < size() ? from + limit : size();
+ indices.reserve(indices.size() + to - from);
+ for (size_t i = from; i < to; ++i) {
+ if (!static_cast<const Derived&>(*this).is_default_at(i)) indices.push_back(i);
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!static_cast<const Derived&>(*this).is_default_at(i)) { indices.push_back(i);
}
```
##########
be/src/vec/columns/column_object.cpp:
##########
@@ -0,0 +1,860 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.cpp
+// and modified by Doris
+
+#include <parallel_hashmap/phmap.h>
+#include <vec/columns/column_array.h>
+#include <vec/columns/column_nullable.h>
+#include <vec/columns/column_object.h>
+#include <vec/columns/columns_number.h>
+#include <vec/common/field_visitors.h>
+#include <vec/common/hash_table/hash_set.h>
+#include <vec/common/object_util.h>
+#include <vec/common/pod_array_fwd.h>
+#include <vec/core/field.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_nothing.h>
+#include <vec/data_types/get_least_supertype.h>
+
+#include <vec/data_types/data_type_factory.hpp>
+
+namespace doris::vectorized {
+namespace {
+
+DataTypePtr create_array_of_type(DataTypePtr type, size_t num_dimensions) {
+ for (size_t i = 0; i < num_dimensions; ++i)
+ type = std::make_shared<DataTypeArray>(std::move(type));
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
for (size_t i = 0; i < num_dimensions; ++i) {
type = std::make_shared<DataTypeArray>(std::move(type));
}
```
##########
be/src/vec/columns/column_object.cpp:
##########
@@ -0,0 +1,860 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.cpp
+// and modified by Doris
+
+#include <parallel_hashmap/phmap.h>
+#include <vec/columns/column_array.h>
+#include <vec/columns/column_nullable.h>
+#include <vec/columns/column_object.h>
+#include <vec/columns/columns_number.h>
+#include <vec/common/field_visitors.h>
+#include <vec/common/hash_table/hash_set.h>
+#include <vec/common/object_util.h>
+#include <vec/common/pod_array_fwd.h>
+#include <vec/core/field.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_nothing.h>
+#include <vec/data_types/get_least_supertype.h>
+
+#include <vec/data_types/data_type_factory.hpp>
+
+namespace doris::vectorized {
+namespace {
+
+DataTypePtr create_array_of_type(DataTypePtr type, size_t num_dimensions) {
+ for (size_t i = 0; i < num_dimensions; ++i)
+ type = std::make_shared<DataTypeArray>(std::move(type));
+ return type;
+}
+
+DataTypePtr getBaseTypeOfArray(const DataTypePtr& type) {
+ /// Get raw pointers to avoid extra copying of type pointers.
+ const DataTypeArray* last_array = nullptr;
+ const auto* current_type = type.get();
+ while (const auto* type_array = typeid_cast<const DataTypeArray*>(current_type)) {
+ current_type = type_array->get_nested_type().get();
+ last_array = type_array;
+ }
+ return last_array ? last_array->get_nested_type() : type;
+}
+
+size_t getNumberOfDimensions(const IDataType& type) {
+ if (const auto* type_array = typeid_cast<const DataTypeArray*>(&type)) {
+ return type_array->get_number_of_dimensions();
+ }
+ return 0;
+}
+
+DataTypePtr get_data_type_by_column(const IColumn& column) {
+ auto idx = column.get_data_type();
+ if (WhichDataType(idx).is_simple()) {
+ return DataTypeFactory::instance().get(String(getTypeName(idx)));
+ }
+ if (WhichDataType(idx).is_nothing()) {
+ return std::make_shared<DataTypeNothing>();
+ }
+ if (const auto* column_array = check_and_get_column<ColumnArray>(&column)) {
+ return std::make_shared<DataTypeArray>(get_data_type_by_column(column_array->get_data()));
+ }
+ if (const auto* column_nullable = check_and_get_column<ColumnNullable>(&column)) {
+ return make_nullable(get_data_type_by_column(column_nullable->get_nested_column()));
+ }
+ // TODO add more types
+ assert(false);
+ return nullptr;
+}
+
+/// Recreates column with default scalar values and keeps sizes of arrays.
+ColumnPtr recreate_column_with_default_value(const ColumnPtr& column,
+ const DataTypePtr& scalar_type,
+ size_t num_dimensions) {
+ const auto* column_array = check_and_get_column<ColumnArray>(column.get());
+ if (column_array && num_dimensions) {
+ return ColumnArray::create(
+ recreate_column_with_default_value(column_array->get_data_ptr(), scalar_type,
+ num_dimensions - 1),
+ IColumn::mutate(column_array->get_offsets_ptr()));
+ }
+ return create_array_of_type(scalar_type, num_dimensions)
+ ->create_column()
+ ->clone_resized(column->size());
+}
+
+Array create_empty_array_field(size_t num_dimensions) {
+ assert(num_dimensions != 0);
+ Array array;
+ Array* current_array = &array;
+ for (size_t i = 1; i < num_dimensions; ++i) {
+ current_array->push_back(Array());
+ current_array = ¤t_array->back().get<Array&>();
+ }
+ return array;
+}
+
+/// Replaces NULL fields to given field or empty array.
+class FieldVisitorReplaceNull : public StaticVisitor<Field> {
+public:
+ explicit FieldVisitorReplaceNull(const Field& replacement_, size_t num_dimensions_)
+ : replacement(replacement_), num_dimensions(num_dimensions_) {}
+ Field operator()(const Null&) const {
+ return num_dimensions ? create_empty_array_field(num_dimensions) : replacement;
+ }
+ Field operator()(const Array& x) const {
+ assert(num_dimensions > 0);
+ const size_t size = x.size();
+ Array res(size);
+ for (size_t i = 0; i < size; ++i) {
+ res[i] = apply_visitor(FieldVisitorReplaceNull(replacement, num_dimensions - 1), x[i]);
+ }
+ return res;
+ }
+ template <typename T>
+ Field operator()(const T& x) const {
+ return x;
+ }
+
+private:
+ const Field& replacement;
+ size_t num_dimensions;
+};
+
+/// Calculates number of dimensions in array field.
+/// Returns 0 for scalar fields.
+class FieldVisitorToNumberOfDimensions : public StaticVisitor<size_t> {
+public:
+ explicit FieldVisitorToNumberOfDimensions(Status* st) : _st(st) {}
+ size_t operator()(const Array& x) const {
+ const size_t size = x.size();
+ std::optional<size_t> dimensions;
+ for (size_t i = 0; i < size; ++i) {
+ /// Do not count Nulls, because they will be replaced by default
+ /// values with proper number of dimensions.
+ if (x[i].is_null()) continue;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (x[i].is_null()) { continue;
}
```
##########
be/src/vec/columns/column_object.h:
##########
@@ -0,0 +1,364 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.h
+// and modified by Doris
+
+#pragma once
+#include <vec/columns/column.h>
+#include <vec/columns/subcolumn_tree.h>
+#include <vec/common/hash_table/hash_map.h>
+#include <vec/common/pod_array.h>
+#include <vec/core/field.h>
+#include <vec/core/names.h>
+#include <vec/data_types/data_type.h>
+#include <vec/json/json_parser.h>
+
+#include "common/status.h"
+namespace doris::vectorized {
+
+/// Info that represents a scalar or array field in a decomposed view.
+/// It allows to recreate field with different number
+/// of dimensions or nullability.
+struct FieldInfo {
+ /// The common type of of all scalars in field.
+ DataTypePtr scalar_type;
+ /// Do we have NULL scalar in field.
+ bool have_nulls;
+ /// If true then we have scalars with different types in array and
+ /// we need to convert scalars to the common type.
+ bool need_convert;
+ /// Number of dimension in array. 0 if field is scalar.
+ size_t num_dimensions;
+};
+Status get_field_info(const Field& field, FieldInfo* info);
+/** A column that represents object with dynamic set of subcolumns.
+ * Subcolumns are identified by paths in document and are stored in
+ * a trie-like structure. ColumnObject is not suitable for writing into tables
+ * and it should be converted to Tuple with fixed set of subcolumns before that.
+ */
+class ColumnObject final : public COWHelper<IColumn, ColumnObject> {
+public:
+ /** Class that represents one subcolumn.
+ * It stores values in several parts of column
+ * and keeps current common type of all parts.
+ * We add a new column part with a new type, when we insert a field,
+ * which can't be converted to the current common type.
+ * After insertion of all values subcolumn should be finalized
+ * for writing and other operations.
+ */
+ class Subcolumn {
+ public:
+ Subcolumn() = default;
+
+ Subcolumn(size_t size_, bool is_nullable_);
+
+ Subcolumn(MutableColumnPtr&& data_, bool is_nullable_);
+
+ size_t size() const;
+
+ size_t byteSize() const;
+
+ size_t allocatedBytes() const;
+
+ bool is_finalized() const;
+
+ const DataTypePtr& get_least_common_type() const { return least_common_type.get(); }
+
+ const DataTypePtr& get_least_common_typeBase() const { return least_common_type.getBase(); }
+
+ size_t get_dimensions() const { return least_common_type.get_dimensions(); }
+
+ /// Checks the consistency of column's parts stored in @data.
+ void checkTypes() const;
+
+ /// Inserts a field, which scalars can be arbitrary, but number of
+ /// dimensions should be consistent with current common type.
+ /// return Status::InvalidArgument when meet conflict types
+ Status insert(Field field);
+
+ Status insert(Field field, FieldInfo info);
+
+ void insertDefault();
+
+ void insertManyDefaults(size_t length);
+
+ Status insertRangeFrom(const Subcolumn& src, size_t start, size_t length);
+
+ void pop_back(size_t n);
+
+ /// Converts all column's parts to the common type and
+ /// creates a single column that stores all values.
+ void finalize();
+
+ /// Returns last inserted field.
+ Field getLastField() const;
+
+ /// Recreates subcolumn with default scalar values and keeps sizes of arrays.
+ /// Used to create columns of type Nested with consistent array sizes.
+ Subcolumn recreate_with_default_values(const FieldInfo& field_info) const;
+
+ /// Returns single column if subcolumn in finalizes.
+ /// Otherwise -- undefined behaviour.
+ IColumn& get_finalized_column();
+
+ const IColumn& get_finalized_column() const;
+
+ const ColumnPtr& get_finalized_column_ptr() const;
+
+ friend class ColumnObject;
+
+ private:
+ class LeastCommonType {
+ public:
+ LeastCommonType() = default;
+
+ explicit LeastCommonType(DataTypePtr type_);
+
+ const DataTypePtr& get() const { return type; }
+
+ const DataTypePtr& getBase() const { return base_type; }
+
+ size_t get_dimensions() const { return num_dimensions; }
+
+ private:
+ DataTypePtr type;
+ DataTypePtr base_type;
+ size_t num_dimensions = 0;
+ };
+ void add_new_column_part(DataTypePtr type);
+
+ /// Current least common type of all values inserted to this subcolumn.
+ LeastCommonType least_common_type;
+ /// If true then common type type of subcolumn is Nullable
+ /// and default values are NULLs.
+ bool is_nullable = false;
+ /// Parts of column. Parts should be in increasing order in terms of subtypes/supertypes.
+ /// That means that the least common type for i-th prefix is the type of i-th part
+ /// and it's the supertype for all type of column from 0 to i-1.
+ std::vector<WrappedPtr> data;
+ /// Until we insert any non-default field we don't know further
+ /// least common type and we count number of defaults in prefix,
+ /// which will be converted to the default type of final common type.
+ size_t num_of_defaults_in_prefix = 0;
+ };
+ using Subcolumns = SubcolumnsTree<Subcolumn>;
+
+private:
+ /// If true then all subcolumns are nullable.
+ const bool is_nullable;
+ Subcolumns subcolumns;
+ size_t num_rows;
+
+public:
+ static constexpr auto COLUMN_NAME_DUMMY = "_dummy";
+
+ explicit ColumnObject(bool is_nullable_);
+
+ ColumnObject(Subcolumns&& subcolumns_, bool is_nullable_);
+
+ ~ColumnObject() = default;
+
+ bool can_be_inside_nullable() const override { return true; }
+
+ /// Checks that all subcolumns have consistent sizes.
+ void check_consistency() const;
+
+ bool has_subcolumn(const PathInData& key) const;
+
+ // return null if not found
+ const Subcolumn* get_subcolumn(const PathInData& key) const;
+
+ // return null if not found
+ Subcolumn* get_subcolumn(const PathInData& key);
+
+ void incr_num_rows() { ++num_rows; }
+
+ /// Adds a subcolumn from existing IColumn.
+ bool add_sub_column(const PathInData& key, MutableColumnPtr&& subcolumn);
+
+ /// Adds a subcolumn of specific size with default values.
+ bool add_sub_column(const PathInData& key, size_t new_size);
+
+ /// Adds a subcolumn of type Nested of specific size with default values.
+ /// It cares about consistency of sizes of Nested arrays.
+ bool add_nested_subcolumn(const PathInData& key, const FieldInfo& field_info, size_t new_size);
+
+ const Subcolumns& get_subcolumns() const { return subcolumns; }
+
+ Subcolumns& get_subcolumns() { return subcolumns; }
+
+ PathsInData getKeys() const;
+
+ std::string get_keys_str() const {
+ std::stringstream ss;
+ bool first = true;
+ for (auto& k : getKeys()) {
+ if (first) {
+ first = false;
+ } else {
+ ss << ", ";
+ }
+ ss << k.get_path();
+ }
+
+ return ss.str();
+ }
+
+ void remove_subcolumns(const std::unordered_set<std::string>& keys);
+
+ /// Finalizes all subcolumns.
+ void finalize();
+
+ bool is_finalized() const;
+
+ /// Part of interface
+ const char* get_family_name() const override { return "Variant"; }
+
+ TypeIndex get_data_type() const override { return TypeIndex::VARIANT; }
+
+ size_t size() const override;
+
+ MutableColumnPtr clone_resized(size_t new_size) const override;
+
+ size_t byte_size() const override;
+
+ size_t allocated_bytes() const override;
+
+ void for_each_subcolumn(ColumnCallback callback) override;
+
+ // Do nothing, call try_insert instead
+ void insert(const Field& field) override {
+ Status st = try_insert(field);
+ if (!st.ok()) LOG(FATAL) << "insert return ERROR status: " << st;
+ }
+
+ // Do nothing, call try_insert_range_from instead
+ void insert_range_from(const IColumn& src, size_t start, size_t length) override {
+ Status st = try_insert_range_from(src, start, length);
+ if (!st.ok()) LOG(FATAL) << "insert_range_from return ERROR status: " << st;
+ }
+
+ // Only called in Block::add_row
+ Status try_insert(const Field& field);
+
+ Status try_insert_from(const IColumn& src, size_t n);
+
+ // Only called in Block::add_row
+ Status try_insert_range_from(const IColumn& src, size_t start, size_t length);
+
+ void insert_default() override;
+
+ [[noreturn]] ColumnPtr replicate(const Offsets& offsets) const override {
+ LOG(FATAL) << "should not call the method replicate in column object";
+ }
+
+ void pop_back(size_t length) override;
+
+ Field operator[](size_t n) const override;
+
+ void get(size_t n, Field& res) const override;
+
+ /// All other methods throw exception.
+ StringRef get_data_at(size_t) const override {
+ LOG(FATAL) << "should not call the method in column object";
+ return StringRef();
+ }
+
+ void insert_indices_from(const IColumn& src, const int* indices_begin,
+ const int* indices_end) override {
+ LOG(FATAL) << "should not call the method in column object";
+ }
+
+ Status try_insert_indices_from(const IColumn& src, const int* indices_begin,
+ const int* indices_end);
+
+ StringRef serialize_value_into_arena(size_t n, Arena& arena,
+ char const*& begin) const override {
+ LOG(FATAL) << "should not call the method in column object";
+ return StringRef();
+ }
+
+ const char* deserialize_and_insert_from_arena(const char* pos) override {
+ LOG(FATAL) << "should not call the method in column object";
+ return nullptr;
+ }
+
+ void update_hash_with_value(size_t n, SipHash& hash) const override {
+ LOG(FATAL) << "should not call the method in column object";
+ }
+
+ void insert_data(const char* pos, size_t length) override {
+ LOG(FATAL) << "should not call the method in column object";
+ }
+
+ ColumnPtr filter(const Filter&, ssize_t) const override {
+ LOG(FATAL) << "should not call the method in column object";
+ return nullptr;
+ }
+
+ ColumnPtr permute(const Permutation&, size_t) const override {
+ LOG(FATAL) << "should not call the method in column object";
+ return nullptr;
+ }
+
+ int compare_at(size_t n, size_t m, const IColumn& rhs, int nan_direction_hint) const override {
+ LOG(FATAL) << "should not call the method in column object";
+ return 0;
+ }
+
+ void get_permutation(bool reverse, size_t limit, int nan_direction_hint,
+ Permutation& res) const override {
+ LOG(FATAL) << "should not call the method in column object";
+ }
+
+ MutableColumns scatter(ColumnIndex, const Selector&) const override {
+ LOG(FATAL) << "should not call the method in column object";
+ return {};
+ }
+
+ void replace_column_data(const IColumn&, size_t row, size_t self_row) override {
+ LOG(FATAL) << "should not call the method in column object";
+ }
+
+ void replace_column_data_default(size_t self_row) override {
+ LOG(FATAL) << "should not call the method in column object";
+ }
+
+ virtual void get_extremes(Field& min, Field& max) const override {
Review Comment:
warning: 'virtual' is redundant since the function is already declared 'override' [modernize-use-override]
```suggestion
void get_extremes(Field& min, Field& max) const override {
```
##########
be/src/vec/columns/column_object.cpp:
##########
@@ -0,0 +1,860 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.cpp
+// and modified by Doris
+
+#include <parallel_hashmap/phmap.h>
+#include <vec/columns/column_array.h>
+#include <vec/columns/column_nullable.h>
+#include <vec/columns/column_object.h>
+#include <vec/columns/columns_number.h>
+#include <vec/common/field_visitors.h>
+#include <vec/common/hash_table/hash_set.h>
+#include <vec/common/object_util.h>
+#include <vec/common/pod_array_fwd.h>
+#include <vec/core/field.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_nothing.h>
+#include <vec/data_types/get_least_supertype.h>
+
+#include <vec/data_types/data_type_factory.hpp>
+
+namespace doris::vectorized {
+namespace {
+
+DataTypePtr create_array_of_type(DataTypePtr type, size_t num_dimensions) {
+ for (size_t i = 0; i < num_dimensions; ++i)
+ type = std::make_shared<DataTypeArray>(std::move(type));
+ return type;
+}
+
+DataTypePtr getBaseTypeOfArray(const DataTypePtr& type) {
+ /// Get raw pointers to avoid extra copying of type pointers.
+ const DataTypeArray* last_array = nullptr;
+ const auto* current_type = type.get();
+ while (const auto* type_array = typeid_cast<const DataTypeArray*>(current_type)) {
+ current_type = type_array->get_nested_type().get();
+ last_array = type_array;
+ }
+ return last_array ? last_array->get_nested_type() : type;
+}
+
+size_t getNumberOfDimensions(const IDataType& type) {
+ if (const auto* type_array = typeid_cast<const DataTypeArray*>(&type)) {
+ return type_array->get_number_of_dimensions();
+ }
+ return 0;
+}
+
+DataTypePtr get_data_type_by_column(const IColumn& column) {
+ auto idx = column.get_data_type();
+ if (WhichDataType(idx).is_simple()) {
+ return DataTypeFactory::instance().get(String(getTypeName(idx)));
+ }
+ if (WhichDataType(idx).is_nothing()) {
+ return std::make_shared<DataTypeNothing>();
+ }
+ if (const auto* column_array = check_and_get_column<ColumnArray>(&column)) {
+ return std::make_shared<DataTypeArray>(get_data_type_by_column(column_array->get_data()));
+ }
+ if (const auto* column_nullable = check_and_get_column<ColumnNullable>(&column)) {
+ return make_nullable(get_data_type_by_column(column_nullable->get_nested_column()));
+ }
+ // TODO add more types
+ assert(false);
+ return nullptr;
+}
+
+/// Recreates column with default scalar values and keeps sizes of arrays.
+ColumnPtr recreate_column_with_default_value(const ColumnPtr& column,
+ const DataTypePtr& scalar_type,
+ size_t num_dimensions) {
+ const auto* column_array = check_and_get_column<ColumnArray>(column.get());
+ if (column_array && num_dimensions) {
+ return ColumnArray::create(
+ recreate_column_with_default_value(column_array->get_data_ptr(), scalar_type,
+ num_dimensions - 1),
+ IColumn::mutate(column_array->get_offsets_ptr()));
+ }
+ return create_array_of_type(scalar_type, num_dimensions)
+ ->create_column()
+ ->clone_resized(column->size());
+}
+
+Array create_empty_array_field(size_t num_dimensions) {
+ assert(num_dimensions != 0);
+ Array array;
+ Array* current_array = &array;
+ for (size_t i = 1; i < num_dimensions; ++i) {
+ current_array->push_back(Array());
+ current_array = ¤t_array->back().get<Array&>();
+ }
+ return array;
+}
+
+/// Replaces NULL fields to given field or empty array.
+class FieldVisitorReplaceNull : public StaticVisitor<Field> {
+public:
+ explicit FieldVisitorReplaceNull(const Field& replacement_, size_t num_dimensions_)
+ : replacement(replacement_), num_dimensions(num_dimensions_) {}
+ Field operator()(const Null&) const {
+ return num_dimensions ? create_empty_array_field(num_dimensions) : replacement;
+ }
+ Field operator()(const Array& x) const {
+ assert(num_dimensions > 0);
+ const size_t size = x.size();
+ Array res(size);
+ for (size_t i = 0; i < size; ++i) {
+ res[i] = apply_visitor(FieldVisitorReplaceNull(replacement, num_dimensions - 1), x[i]);
+ }
+ return res;
+ }
+ template <typename T>
+ Field operator()(const T& x) const {
+ return x;
+ }
+
+private:
+ const Field& replacement;
+ size_t num_dimensions;
+};
+
+/// Calculates number of dimensions in array field.
+/// Returns 0 for scalar fields.
+class FieldVisitorToNumberOfDimensions : public StaticVisitor<size_t> {
+public:
+ explicit FieldVisitorToNumberOfDimensions(Status* st) : _st(st) {}
+ size_t operator()(const Array& x) const {
+ const size_t size = x.size();
+ std::optional<size_t> dimensions;
+ for (size_t i = 0; i < size; ++i) {
+ /// Do not count Nulls, because they will be replaced by default
+ /// values with proper number of dimensions.
+ if (x[i].is_null()) continue;
+ size_t current_dimensions = apply_visitor(*this, x[i]);
+ if (!dimensions) {
+ dimensions = current_dimensions;
+ } else if (current_dimensions != *dimensions) {
+ *_st = Status::InvalidArgument(
+ "Number of dimensions mismatched among array elements");
+ return 0;
+ }
+ }
+ return 1 + dimensions.value_or(0);
+ }
+ template <typename T>
+ size_t operator()(const T&) const {
+ return 0;
+ }
+
+private:
+ mutable Status* _st;
+};
+
+/// Visitor that allows to get type of scalar field
+/// or least common type of scalars in array.
+/// More optimized version of FieldToDataType.
+class FieldVisitorToScalarType : public StaticVisitor<size_t> {
+public:
+ using FieldType = Field::Types::Which;
+ size_t operator()(const Array& x) {
+ size_t size = x.size();
+ for (size_t i = 0; i < size; ++i) apply_visitor(*this, x[i]);
+ return 0;
+ }
+ // TODO doris not support unsigned integers for now
+ // treat as signed integers
+ size_t operator()(const UInt64& x) {
+ field_types.insert(FieldType::UInt64);
+ if (x <= std::numeric_limits<Int8>::max()) {
+ type_indexes.insert(TypeIndex::Int8);
+ } else if (x <= std::numeric_limits<Int16>::max()) {
+ type_indexes.insert(TypeIndex::Int16);
+ } else if (x <= std::numeric_limits<Int32>::max()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Int64& x) {
+ // Only support Int32 and Int64
+ field_types.insert(FieldType::Int64);
+ if (x <= std::numeric_limits<Int32>::max() && x >= std::numeric_limits<Int32>::min()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Null&) {
+ have_nulls = true;
+ return 0;
+ }
+ size_t operator()(const Int128I& x) { LOG(FATAL) << "not implemented"; }
+ template <typename T>
+ size_t operator()(const T&) {
+ Field::EnumToType<Field::Types::Array>::Type a;
+ field_types.insert(Field::TypeToEnum<NearestFieldType<T>>::value);
+ type_indexes.insert(TypeId<NearestFieldType<T>>::value);
+ return 0;
+ }
+ Status get_scalar_type(DataTypePtr* type) const {
+ return get_least_supertype(type_indexes, type, true /*compatible with string type*/);
+ }
+ bool contain_nulls() const { return have_nulls; }
+ bool need_convert_field() const { return field_types.size() > 1; }
+
+private:
+ phmap::flat_hash_set<TypeIndex> type_indexes;
+ phmap::flat_hash_set<FieldType> field_types;
+ bool have_nulls = false;
+};
+
+} // namespace
+Status get_field_info(const Field& field, FieldInfo* info) {
+ FieldVisitorToScalarType to_scalar_type_visitor;
+ apply_visitor(to_scalar_type_visitor, field);
+ DataTypePtr type = nullptr;
+ RETURN_IF_ERROR(to_scalar_type_visitor.get_scalar_type(&type));
+ // array item's dimension may missmatch, eg. [1, 2, [1, 2, 3]]
+ Status num_to_dimensions_status;
+ *info = {
+ type,
+ to_scalar_type_visitor.contain_nulls(),
+ to_scalar_type_visitor.need_convert_field(),
+ apply_visitor(FieldVisitorToNumberOfDimensions(&num_to_dimensions_status), field),
+ };
+ RETURN_IF_ERROR(num_to_dimensions_status);
+ return Status::OK();
+}
+
+ColumnObject::Subcolumn::Subcolumn(MutableColumnPtr&& data_, bool is_nullable_)
+ : least_common_type(get_data_type_by_column(*data_)), is_nullable(is_nullable_) {
+ data.push_back(std::move(data_));
+}
+
+ColumnObject::Subcolumn::Subcolumn(size_t size_, bool is_nullable_)
+ : least_common_type(std::make_shared<DataTypeNothing>()),
+ is_nullable(is_nullable_),
+ num_of_defaults_in_prefix(size_) {}
+
+size_t ColumnObject::Subcolumn::Subcolumn::size() const {
+ size_t res = num_of_defaults_in_prefix;
+ for (const auto& part : data) {
+ res += part->size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::byteSize() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->byte_size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::allocatedBytes() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->allocated_bytes();
+ }
+ return res;
+}
+
+Status ColumnObject::Subcolumn::insert(Field field) {
+ FieldInfo info;
+ RETURN_IF_ERROR(get_field_info(field, &info));
+ return insert(std::move(field), std::move(info));
+}
+
+void ColumnObject::Subcolumn::add_new_column_part(DataTypePtr type) {
+ data.push_back(type->create_column());
+ least_common_type = LeastCommonType {std::move(type)};
+}
+
+Status ColumnObject::Subcolumn::insert(Field field, FieldInfo info) {
+ auto base_type = std::move(info.scalar_type);
+ if (is_nothing(base_type)) {
+ insertDefault();
+ return Status::OK();
+ }
+ auto column_dim = least_common_type.get_dimensions();
+ auto value_dim = info.num_dimensions;
+ if (is_nothing(least_common_type.getBase())) {
+ column_dim = value_dim;
+ }
+ if (is_nothing(base_type)) {
+ value_dim = column_dim;
+ }
+ if (value_dim != column_dim) {
+ return Status::InvalidArgument(
+ "Dimension of types mismatched between inserted value and column.");
+ }
+ if (is_nullable && !is_nothing(base_type)) {
+ base_type = make_nullable(base_type);
+ }
+ // alawys nullable at present
+ if (!is_nullable && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+ // need replace muli dimensions array which contains null. eg. [[1, 2, 3], null] -> [[1, 2, 3], []]
+ // since column array doesnt known null's dimension
+ if (info.num_dimensions >= 2 && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+
+ bool type_changed = false;
+ const auto& least_common_base_type = least_common_type.getBase();
+ if (data.empty()) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ } else if (!least_common_base_type->equals(*base_type) && !is_nothing(base_type)) {
+ if (!object_util::is_conversion_required_between_integers(*base_type,
+ *least_common_base_type)) {
+ RETURN_IF_ERROR(
+ get_least_supertype(DataTypes {std::move(base_type), least_common_base_type},
+ &base_type, true /*compatible with string type*/));
+ type_changed = true;
+ if (!least_common_base_type->equals(*base_type)) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ }
+ }
+ }
+
+ if (type_changed || info.need_convert) {
+ Field new_field;
+ RETURN_IF_ERROR(convert_field_to_type(field, *least_common_type.get(), &new_field));
+ field = new_field;
+ }
+
+ data.back()->insert(field);
+ return Status::OK();
+}
+
+Status ColumnObject::Subcolumn::insertRangeFrom(const Subcolumn& src, size_t start, size_t length) {
+ assert(src.is_finalized());
+ const auto& src_column = src.data.back();
+ const auto& src_type = src.least_common_type.get();
+ if (data.empty()) {
+ add_new_column_part(src.least_common_type.get());
+ data.back()->insert_range_from(*src_column, start, length);
+ } else if (least_common_type.get()->equals(*src_type)) {
+ data.back()->insert_range_from(*src_column, start, length);
+ } else {
+ DataTypePtr new_least_common_type = nullptr;
+ RETURN_IF_ERROR(get_least_supertype(DataTypes {least_common_type.get(), src_type},
+ &new_least_common_type,
+ true /*compatible with string type*/));
+ ColumnPtr casted_column;
+ RETURN_IF_ERROR(object_util::cast_column({src_column, src_type, ""}, new_least_common_type,
+ &casted_column));
+ if (!least_common_type.get()->equals(*new_least_common_type))
+ add_new_column_part(std::move(new_least_common_type));
+ data.back()->insert_range_from(*casted_column, start, length);
+ }
+ return Status::OK();
+}
+
+bool ColumnObject::Subcolumn::is_finalized() const {
+ return data.empty() || (data.size() == 1 && num_of_defaults_in_prefix == 0);
+}
+
+template <typename Func>
+ColumnPtr ColumnObject::apply_for_subcolumns(Func&& func, std::string_view func_name) const {
+ if (!is_finalized()) {
+ LOG(FATAL) << "Cannot " << func_name << " non-finalized ColumnObject";
+ }
+ auto res = ColumnObject::create(is_nullable);
+ for (const auto& subcolumn : subcolumns) {
+ auto new_subcolumn = func(subcolumn->data.get_finalized_column());
+ res->add_sub_column(subcolumn->path, new_subcolumn->assume_mutable());
+ }
+ return res;
+}
+ColumnPtr ColumnObject::index(const IColumn& indexes, size_t limit) const {
+ return apply_for_subcolumns(
+ [&](const auto& subcolumn) { return subcolumn.index(indexes, limit); }, "index");
+}
+
+void ColumnObject::Subcolumn::finalize() {
+ if (is_finalized()) {
+ return;
+ }
+ if (data.size() == 1 && num_of_defaults_in_prefix == 0) {
+ data[0] = data[0]->convert_to_full_column_if_const();
+ return;
+ }
+ const auto& to_type = least_common_type.get();
+ auto result_column = to_type->create_column();
+ if (num_of_defaults_in_prefix) {
+ result_column->insert_many_defaults(num_of_defaults_in_prefix);
+ }
+ for (auto& part : data) {
+ part = part->convert_to_full_column_if_const();
+ auto from_type = get_data_type_by_column(*part);
+ size_t part_size = part->size();
+ if (!from_type->equals(*to_type)) {
+ auto offsets = ColumnUInt64::create();
+ auto& offsets_data = offsets->get_data();
+ /// We need to convert only non-default values and then recreate column
+ /// with default value of new type, because default values (which represents misses in data)
+ /// may be inconsistent between types (e.g "0" in UInt64 and empty string in String).
+ part->get_indices_of_non_default_rows(offsets_data, 0, part_size);
+ if (offsets->size() == part_size) {
+ ColumnPtr ptr;
+ object_util::cast_column({part, from_type, ""}, to_type, &ptr);
+ part = ptr;
+ } else {
+ auto values = part->index(*offsets, offsets->size());
+ object_util::cast_column({values, from_type, ""}, to_type, &values);
+ part = values->create_with_offsets(offsets_data, to_type->get_default(), part_size,
+ /*shift=*/0);
+ }
+ }
+ result_column->insert_range_from(*part, 0, part_size);
+ }
+ data = {std::move(result_column)};
+ num_of_defaults_in_prefix = 0;
+}
+
+void ColumnObject::Subcolumn::insertDefault() {
+ if (data.empty()) {
+ ++num_of_defaults_in_prefix;
+ } else {
+ data.back()->insert_default();
+ }
+}
+
+void ColumnObject::Subcolumn::insertManyDefaults(size_t length) {
+ if (data.empty()) {
+ num_of_defaults_in_prefix += length;
+ } else {
+ data.back()->insert_many_defaults(length);
+ }
+}
+
+void ColumnObject::Subcolumn::pop_back(size_t n) {
+ assert(n <= size());
+ size_t num_removed = 0;
+ for (auto it = data.rbegin(); it != data.rend(); ++it) {
+ if (n == 0) break;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (n == 0) { break;
}
```
##########
be/src/vec/columns/column_object.cpp:
##########
@@ -0,0 +1,860 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.cpp
+// and modified by Doris
+
+#include <parallel_hashmap/phmap.h>
+#include <vec/columns/column_array.h>
+#include <vec/columns/column_nullable.h>
+#include <vec/columns/column_object.h>
+#include <vec/columns/columns_number.h>
+#include <vec/common/field_visitors.h>
+#include <vec/common/hash_table/hash_set.h>
+#include <vec/common/object_util.h>
+#include <vec/common/pod_array_fwd.h>
+#include <vec/core/field.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_nothing.h>
+#include <vec/data_types/get_least_supertype.h>
+
+#include <vec/data_types/data_type_factory.hpp>
+
+namespace doris::vectorized {
+namespace {
+
+DataTypePtr create_array_of_type(DataTypePtr type, size_t num_dimensions) {
+ for (size_t i = 0; i < num_dimensions; ++i)
+ type = std::make_shared<DataTypeArray>(std::move(type));
+ return type;
+}
+
+DataTypePtr getBaseTypeOfArray(const DataTypePtr& type) {
+ /// Get raw pointers to avoid extra copying of type pointers.
+ const DataTypeArray* last_array = nullptr;
+ const auto* current_type = type.get();
+ while (const auto* type_array = typeid_cast<const DataTypeArray*>(current_type)) {
+ current_type = type_array->get_nested_type().get();
+ last_array = type_array;
+ }
+ return last_array ? last_array->get_nested_type() : type;
+}
+
+size_t getNumberOfDimensions(const IDataType& type) {
+ if (const auto* type_array = typeid_cast<const DataTypeArray*>(&type)) {
+ return type_array->get_number_of_dimensions();
+ }
+ return 0;
+}
+
+DataTypePtr get_data_type_by_column(const IColumn& column) {
+ auto idx = column.get_data_type();
+ if (WhichDataType(idx).is_simple()) {
+ return DataTypeFactory::instance().get(String(getTypeName(idx)));
+ }
+ if (WhichDataType(idx).is_nothing()) {
+ return std::make_shared<DataTypeNothing>();
+ }
+ if (const auto* column_array = check_and_get_column<ColumnArray>(&column)) {
+ return std::make_shared<DataTypeArray>(get_data_type_by_column(column_array->get_data()));
+ }
+ if (const auto* column_nullable = check_and_get_column<ColumnNullable>(&column)) {
+ return make_nullable(get_data_type_by_column(column_nullable->get_nested_column()));
+ }
+ // TODO add more types
+ assert(false);
+ return nullptr;
+}
+
+/// Recreates column with default scalar values and keeps sizes of arrays.
+ColumnPtr recreate_column_with_default_value(const ColumnPtr& column,
+ const DataTypePtr& scalar_type,
+ size_t num_dimensions) {
+ const auto* column_array = check_and_get_column<ColumnArray>(column.get());
+ if (column_array && num_dimensions) {
+ return ColumnArray::create(
+ recreate_column_with_default_value(column_array->get_data_ptr(), scalar_type,
+ num_dimensions - 1),
+ IColumn::mutate(column_array->get_offsets_ptr()));
+ }
+ return create_array_of_type(scalar_type, num_dimensions)
+ ->create_column()
+ ->clone_resized(column->size());
+}
+
+Array create_empty_array_field(size_t num_dimensions) {
+ assert(num_dimensions != 0);
+ Array array;
+ Array* current_array = &array;
+ for (size_t i = 1; i < num_dimensions; ++i) {
+ current_array->push_back(Array());
+ current_array = ¤t_array->back().get<Array&>();
+ }
+ return array;
+}
+
+/// Replaces NULL fields to given field or empty array.
+class FieldVisitorReplaceNull : public StaticVisitor<Field> {
+public:
+ explicit FieldVisitorReplaceNull(const Field& replacement_, size_t num_dimensions_)
+ : replacement(replacement_), num_dimensions(num_dimensions_) {}
+ Field operator()(const Null&) const {
+ return num_dimensions ? create_empty_array_field(num_dimensions) : replacement;
+ }
+ Field operator()(const Array& x) const {
+ assert(num_dimensions > 0);
+ const size_t size = x.size();
+ Array res(size);
+ for (size_t i = 0; i < size; ++i) {
+ res[i] = apply_visitor(FieldVisitorReplaceNull(replacement, num_dimensions - 1), x[i]);
+ }
+ return res;
+ }
+ template <typename T>
+ Field operator()(const T& x) const {
+ return x;
+ }
+
+private:
+ const Field& replacement;
+ size_t num_dimensions;
+};
+
+/// Calculates number of dimensions in array field.
+/// Returns 0 for scalar fields.
+class FieldVisitorToNumberOfDimensions : public StaticVisitor<size_t> {
+public:
+ explicit FieldVisitorToNumberOfDimensions(Status* st) : _st(st) {}
+ size_t operator()(const Array& x) const {
+ const size_t size = x.size();
+ std::optional<size_t> dimensions;
+ for (size_t i = 0; i < size; ++i) {
+ /// Do not count Nulls, because they will be replaced by default
+ /// values with proper number of dimensions.
+ if (x[i].is_null()) continue;
+ size_t current_dimensions = apply_visitor(*this, x[i]);
+ if (!dimensions) {
+ dimensions = current_dimensions;
+ } else if (current_dimensions != *dimensions) {
+ *_st = Status::InvalidArgument(
+ "Number of dimensions mismatched among array elements");
+ return 0;
+ }
+ }
+ return 1 + dimensions.value_or(0);
+ }
+ template <typename T>
+ size_t operator()(const T&) const {
+ return 0;
+ }
+
+private:
+ mutable Status* _st;
+};
+
+/// Visitor that allows to get type of scalar field
+/// or least common type of scalars in array.
+/// More optimized version of FieldToDataType.
+class FieldVisitorToScalarType : public StaticVisitor<size_t> {
+public:
+ using FieldType = Field::Types::Which;
+ size_t operator()(const Array& x) {
+ size_t size = x.size();
+ for (size_t i = 0; i < size; ++i) apply_visitor(*this, x[i]);
+ return 0;
+ }
+ // TODO doris not support unsigned integers for now
+ // treat as signed integers
+ size_t operator()(const UInt64& x) {
+ field_types.insert(FieldType::UInt64);
+ if (x <= std::numeric_limits<Int8>::max()) {
+ type_indexes.insert(TypeIndex::Int8);
+ } else if (x <= std::numeric_limits<Int16>::max()) {
+ type_indexes.insert(TypeIndex::Int16);
+ } else if (x <= std::numeric_limits<Int32>::max()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Int64& x) {
+ // Only support Int32 and Int64
+ field_types.insert(FieldType::Int64);
+ if (x <= std::numeric_limits<Int32>::max() && x >= std::numeric_limits<Int32>::min()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Null&) {
+ have_nulls = true;
+ return 0;
+ }
+ size_t operator()(const Int128I& x) { LOG(FATAL) << "not implemented"; }
+ template <typename T>
+ size_t operator()(const T&) {
+ Field::EnumToType<Field::Types::Array>::Type a;
+ field_types.insert(Field::TypeToEnum<NearestFieldType<T>>::value);
+ type_indexes.insert(TypeId<NearestFieldType<T>>::value);
+ return 0;
+ }
+ Status get_scalar_type(DataTypePtr* type) const {
+ return get_least_supertype(type_indexes, type, true /*compatible with string type*/);
+ }
+ bool contain_nulls() const { return have_nulls; }
+ bool need_convert_field() const { return field_types.size() > 1; }
+
+private:
+ phmap::flat_hash_set<TypeIndex> type_indexes;
+ phmap::flat_hash_set<FieldType> field_types;
+ bool have_nulls = false;
+};
+
+} // namespace
+Status get_field_info(const Field& field, FieldInfo* info) {
+ FieldVisitorToScalarType to_scalar_type_visitor;
+ apply_visitor(to_scalar_type_visitor, field);
+ DataTypePtr type = nullptr;
+ RETURN_IF_ERROR(to_scalar_type_visitor.get_scalar_type(&type));
+ // array item's dimension may missmatch, eg. [1, 2, [1, 2, 3]]
+ Status num_to_dimensions_status;
+ *info = {
+ type,
+ to_scalar_type_visitor.contain_nulls(),
+ to_scalar_type_visitor.need_convert_field(),
+ apply_visitor(FieldVisitorToNumberOfDimensions(&num_to_dimensions_status), field),
+ };
+ RETURN_IF_ERROR(num_to_dimensions_status);
+ return Status::OK();
+}
+
+ColumnObject::Subcolumn::Subcolumn(MutableColumnPtr&& data_, bool is_nullable_)
+ : least_common_type(get_data_type_by_column(*data_)), is_nullable(is_nullable_) {
+ data.push_back(std::move(data_));
+}
+
+ColumnObject::Subcolumn::Subcolumn(size_t size_, bool is_nullable_)
+ : least_common_type(std::make_shared<DataTypeNothing>()),
+ is_nullable(is_nullable_),
+ num_of_defaults_in_prefix(size_) {}
+
+size_t ColumnObject::Subcolumn::Subcolumn::size() const {
+ size_t res = num_of_defaults_in_prefix;
+ for (const auto& part : data) {
+ res += part->size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::byteSize() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->byte_size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::allocatedBytes() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->allocated_bytes();
+ }
+ return res;
+}
+
+Status ColumnObject::Subcolumn::insert(Field field) {
+ FieldInfo info;
+ RETURN_IF_ERROR(get_field_info(field, &info));
+ return insert(std::move(field), std::move(info));
+}
+
+void ColumnObject::Subcolumn::add_new_column_part(DataTypePtr type) {
+ data.push_back(type->create_column());
+ least_common_type = LeastCommonType {std::move(type)};
+}
+
+Status ColumnObject::Subcolumn::insert(Field field, FieldInfo info) {
+ auto base_type = std::move(info.scalar_type);
+ if (is_nothing(base_type)) {
+ insertDefault();
+ return Status::OK();
+ }
+ auto column_dim = least_common_type.get_dimensions();
+ auto value_dim = info.num_dimensions;
+ if (is_nothing(least_common_type.getBase())) {
+ column_dim = value_dim;
+ }
+ if (is_nothing(base_type)) {
+ value_dim = column_dim;
+ }
+ if (value_dim != column_dim) {
+ return Status::InvalidArgument(
+ "Dimension of types mismatched between inserted value and column.");
+ }
+ if (is_nullable && !is_nothing(base_type)) {
+ base_type = make_nullable(base_type);
+ }
+ // alawys nullable at present
+ if (!is_nullable && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+ // need replace muli dimensions array which contains null. eg. [[1, 2, 3], null] -> [[1, 2, 3], []]
+ // since column array doesnt known null's dimension
+ if (info.num_dimensions >= 2 && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+
+ bool type_changed = false;
+ const auto& least_common_base_type = least_common_type.getBase();
+ if (data.empty()) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ } else if (!least_common_base_type->equals(*base_type) && !is_nothing(base_type)) {
+ if (!object_util::is_conversion_required_between_integers(*base_type,
+ *least_common_base_type)) {
+ RETURN_IF_ERROR(
+ get_least_supertype(DataTypes {std::move(base_type), least_common_base_type},
+ &base_type, true /*compatible with string type*/));
+ type_changed = true;
+ if (!least_common_base_type->equals(*base_type)) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ }
+ }
+ }
+
+ if (type_changed || info.need_convert) {
+ Field new_field;
+ RETURN_IF_ERROR(convert_field_to_type(field, *least_common_type.get(), &new_field));
+ field = new_field;
+ }
+
+ data.back()->insert(field);
+ return Status::OK();
+}
+
+Status ColumnObject::Subcolumn::insertRangeFrom(const Subcolumn& src, size_t start, size_t length) {
+ assert(src.is_finalized());
+ const auto& src_column = src.data.back();
+ const auto& src_type = src.least_common_type.get();
+ if (data.empty()) {
+ add_new_column_part(src.least_common_type.get());
+ data.back()->insert_range_from(*src_column, start, length);
+ } else if (least_common_type.get()->equals(*src_type)) {
+ data.back()->insert_range_from(*src_column, start, length);
+ } else {
+ DataTypePtr new_least_common_type = nullptr;
+ RETURN_IF_ERROR(get_least_supertype(DataTypes {least_common_type.get(), src_type},
+ &new_least_common_type,
+ true /*compatible with string type*/));
+ ColumnPtr casted_column;
+ RETURN_IF_ERROR(object_util::cast_column({src_column, src_type, ""}, new_least_common_type,
+ &casted_column));
+ if (!least_common_type.get()->equals(*new_least_common_type))
+ add_new_column_part(std::move(new_least_common_type));
+ data.back()->insert_range_from(*casted_column, start, length);
+ }
+ return Status::OK();
+}
+
+bool ColumnObject::Subcolumn::is_finalized() const {
+ return data.empty() || (data.size() == 1 && num_of_defaults_in_prefix == 0);
+}
+
+template <typename Func>
+ColumnPtr ColumnObject::apply_for_subcolumns(Func&& func, std::string_view func_name) const {
+ if (!is_finalized()) {
+ LOG(FATAL) << "Cannot " << func_name << " non-finalized ColumnObject";
+ }
+ auto res = ColumnObject::create(is_nullable);
+ for (const auto& subcolumn : subcolumns) {
+ auto new_subcolumn = func(subcolumn->data.get_finalized_column());
+ res->add_sub_column(subcolumn->path, new_subcolumn->assume_mutable());
+ }
+ return res;
+}
+ColumnPtr ColumnObject::index(const IColumn& indexes, size_t limit) const {
+ return apply_for_subcolumns(
+ [&](const auto& subcolumn) { return subcolumn.index(indexes, limit); }, "index");
+}
+
+void ColumnObject::Subcolumn::finalize() {
+ if (is_finalized()) {
+ return;
+ }
+ if (data.size() == 1 && num_of_defaults_in_prefix == 0) {
+ data[0] = data[0]->convert_to_full_column_if_const();
+ return;
+ }
+ const auto& to_type = least_common_type.get();
+ auto result_column = to_type->create_column();
+ if (num_of_defaults_in_prefix) {
+ result_column->insert_many_defaults(num_of_defaults_in_prefix);
+ }
+ for (auto& part : data) {
+ part = part->convert_to_full_column_if_const();
+ auto from_type = get_data_type_by_column(*part);
+ size_t part_size = part->size();
+ if (!from_type->equals(*to_type)) {
+ auto offsets = ColumnUInt64::create();
+ auto& offsets_data = offsets->get_data();
+ /// We need to convert only non-default values and then recreate column
+ /// with default value of new type, because default values (which represents misses in data)
+ /// may be inconsistent between types (e.g "0" in UInt64 and empty string in String).
+ part->get_indices_of_non_default_rows(offsets_data, 0, part_size);
+ if (offsets->size() == part_size) {
+ ColumnPtr ptr;
+ object_util::cast_column({part, from_type, ""}, to_type, &ptr);
+ part = ptr;
+ } else {
+ auto values = part->index(*offsets, offsets->size());
+ object_util::cast_column({values, from_type, ""}, to_type, &values);
+ part = values->create_with_offsets(offsets_data, to_type->get_default(), part_size,
+ /*shift=*/0);
+ }
+ }
+ result_column->insert_range_from(*part, 0, part_size);
+ }
+ data = {std::move(result_column)};
+ num_of_defaults_in_prefix = 0;
+}
+
+void ColumnObject::Subcolumn::insertDefault() {
+ if (data.empty()) {
+ ++num_of_defaults_in_prefix;
+ } else {
+ data.back()->insert_default();
+ }
+}
+
+void ColumnObject::Subcolumn::insertManyDefaults(size_t length) {
+ if (data.empty()) {
+ num_of_defaults_in_prefix += length;
+ } else {
+ data.back()->insert_many_defaults(length);
+ }
+}
+
+void ColumnObject::Subcolumn::pop_back(size_t n) {
+ assert(n <= size());
+ size_t num_removed = 0;
+ for (auto it = data.rbegin(); it != data.rend(); ++it) {
+ if (n == 0) break;
+ auto& column = *it;
+ if (n < column->size()) {
+ column->pop_back(n);
+ n = 0;
+ } else {
+ ++num_removed;
+ n -= column->size();
+ }
+ }
+ data.resize(data.size() - num_removed);
+ num_of_defaults_in_prefix -= n;
+}
+
+Field ColumnObject::Subcolumn::getLastField() const {
+ if (data.empty()) return Field();
+ const auto& last_part = data.back();
+ assert(!last_part->empty());
+ return (*last_part)[last_part->size() - 1];
+}
+
+ColumnObject::Subcolumn ColumnObject::Subcolumn::recreate_with_default_values(
+ const FieldInfo& field_info) const {
+ auto scalar_type = field_info.scalar_type;
+ if (is_nullable) {
+ scalar_type = make_nullable(scalar_type);
+ }
+ Subcolumn new_subcolumn;
+ new_subcolumn.least_common_type =
+ LeastCommonType {create_array_of_type(scalar_type, field_info.num_dimensions)};
+ new_subcolumn.is_nullable = is_nullable;
+ new_subcolumn.num_of_defaults_in_prefix = num_of_defaults_in_prefix;
+ new_subcolumn.data.reserve(data.size());
+ for (const auto& part : data) {
+ new_subcolumn.data.push_back(
+ recreate_column_with_default_value(part, scalar_type, field_info.num_dimensions));
+ }
+ return new_subcolumn;
+}
+
+IColumn& ColumnObject::Subcolumn::get_finalized_column() {
+ assert(is_finalized());
+ return *data[0];
+}
+
+const IColumn& ColumnObject::Subcolumn::get_finalized_column() const {
+ assert(is_finalized());
+ return *data[0];
+}
+
+const ColumnPtr& ColumnObject::Subcolumn::get_finalized_column_ptr() const {
+ assert(is_finalized());
+ return data[0];
+}
+
+ColumnObject::Subcolumn::LeastCommonType::LeastCommonType(DataTypePtr type_)
+ : type(std::move(type_)),
+ base_type(getBaseTypeOfArray(type)),
+ num_dimensions(getNumberOfDimensions(*type)) {}
+
+ColumnObject::ColumnObject(bool is_nullable_) : is_nullable(is_nullable_), num_rows(0) {}
+
+ColumnObject::ColumnObject(Subcolumns&& subcolumns_, bool is_nullable_)
+ : is_nullable(is_nullable_),
+ subcolumns(std::move(subcolumns_)),
+ num_rows(subcolumns.empty() ? 0 : (*subcolumns.begin())->data.size()) {
+ check_consistency();
+}
+
+void ColumnObject::check_consistency() const {
+ if (subcolumns.empty()) return;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (subcolumns.empty()) { return;
}
```
##########
be/src/vec/columns/column_object.cpp:
##########
@@ -0,0 +1,860 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.cpp
+// and modified by Doris
+
+#include <parallel_hashmap/phmap.h>
+#include <vec/columns/column_array.h>
+#include <vec/columns/column_nullable.h>
+#include <vec/columns/column_object.h>
+#include <vec/columns/columns_number.h>
+#include <vec/common/field_visitors.h>
+#include <vec/common/hash_table/hash_set.h>
+#include <vec/common/object_util.h>
+#include <vec/common/pod_array_fwd.h>
+#include <vec/core/field.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_nothing.h>
+#include <vec/data_types/get_least_supertype.h>
+
+#include <vec/data_types/data_type_factory.hpp>
+
+namespace doris::vectorized {
+namespace {
+
+DataTypePtr create_array_of_type(DataTypePtr type, size_t num_dimensions) {
+ for (size_t i = 0; i < num_dimensions; ++i)
+ type = std::make_shared<DataTypeArray>(std::move(type));
+ return type;
+}
+
+DataTypePtr getBaseTypeOfArray(const DataTypePtr& type) {
+ /// Get raw pointers to avoid extra copying of type pointers.
+ const DataTypeArray* last_array = nullptr;
+ const auto* current_type = type.get();
+ while (const auto* type_array = typeid_cast<const DataTypeArray*>(current_type)) {
+ current_type = type_array->get_nested_type().get();
+ last_array = type_array;
+ }
+ return last_array ? last_array->get_nested_type() : type;
+}
+
+size_t getNumberOfDimensions(const IDataType& type) {
+ if (const auto* type_array = typeid_cast<const DataTypeArray*>(&type)) {
+ return type_array->get_number_of_dimensions();
+ }
+ return 0;
+}
+
+DataTypePtr get_data_type_by_column(const IColumn& column) {
+ auto idx = column.get_data_type();
+ if (WhichDataType(idx).is_simple()) {
+ return DataTypeFactory::instance().get(String(getTypeName(idx)));
+ }
+ if (WhichDataType(idx).is_nothing()) {
+ return std::make_shared<DataTypeNothing>();
+ }
+ if (const auto* column_array = check_and_get_column<ColumnArray>(&column)) {
+ return std::make_shared<DataTypeArray>(get_data_type_by_column(column_array->get_data()));
+ }
+ if (const auto* column_nullable = check_and_get_column<ColumnNullable>(&column)) {
+ return make_nullable(get_data_type_by_column(column_nullable->get_nested_column()));
+ }
+ // TODO add more types
+ assert(false);
+ return nullptr;
+}
+
+/// Recreates column with default scalar values and keeps sizes of arrays.
+ColumnPtr recreate_column_with_default_value(const ColumnPtr& column,
+ const DataTypePtr& scalar_type,
+ size_t num_dimensions) {
+ const auto* column_array = check_and_get_column<ColumnArray>(column.get());
+ if (column_array && num_dimensions) {
+ return ColumnArray::create(
+ recreate_column_with_default_value(column_array->get_data_ptr(), scalar_type,
+ num_dimensions - 1),
+ IColumn::mutate(column_array->get_offsets_ptr()));
+ }
+ return create_array_of_type(scalar_type, num_dimensions)
+ ->create_column()
+ ->clone_resized(column->size());
+}
+
+Array create_empty_array_field(size_t num_dimensions) {
+ assert(num_dimensions != 0);
+ Array array;
+ Array* current_array = &array;
+ for (size_t i = 1; i < num_dimensions; ++i) {
+ current_array->push_back(Array());
+ current_array = ¤t_array->back().get<Array&>();
+ }
+ return array;
+}
+
+/// Replaces NULL fields to given field or empty array.
+class FieldVisitorReplaceNull : public StaticVisitor<Field> {
+public:
+ explicit FieldVisitorReplaceNull(const Field& replacement_, size_t num_dimensions_)
+ : replacement(replacement_), num_dimensions(num_dimensions_) {}
+ Field operator()(const Null&) const {
+ return num_dimensions ? create_empty_array_field(num_dimensions) : replacement;
+ }
+ Field operator()(const Array& x) const {
+ assert(num_dimensions > 0);
+ const size_t size = x.size();
+ Array res(size);
+ for (size_t i = 0; i < size; ++i) {
+ res[i] = apply_visitor(FieldVisitorReplaceNull(replacement, num_dimensions - 1), x[i]);
+ }
+ return res;
+ }
+ template <typename T>
+ Field operator()(const T& x) const {
+ return x;
+ }
+
+private:
+ const Field& replacement;
+ size_t num_dimensions;
+};
+
+/// Calculates number of dimensions in array field.
+/// Returns 0 for scalar fields.
+class FieldVisitorToNumberOfDimensions : public StaticVisitor<size_t> {
+public:
+ explicit FieldVisitorToNumberOfDimensions(Status* st) : _st(st) {}
+ size_t operator()(const Array& x) const {
+ const size_t size = x.size();
+ std::optional<size_t> dimensions;
+ for (size_t i = 0; i < size; ++i) {
+ /// Do not count Nulls, because they will be replaced by default
+ /// values with proper number of dimensions.
+ if (x[i].is_null()) continue;
+ size_t current_dimensions = apply_visitor(*this, x[i]);
+ if (!dimensions) {
+ dimensions = current_dimensions;
+ } else if (current_dimensions != *dimensions) {
+ *_st = Status::InvalidArgument(
+ "Number of dimensions mismatched among array elements");
+ return 0;
+ }
+ }
+ return 1 + dimensions.value_or(0);
+ }
+ template <typename T>
+ size_t operator()(const T&) const {
+ return 0;
+ }
+
+private:
+ mutable Status* _st;
+};
+
+/// Visitor that allows to get type of scalar field
+/// or least common type of scalars in array.
+/// More optimized version of FieldToDataType.
+class FieldVisitorToScalarType : public StaticVisitor<size_t> {
+public:
+ using FieldType = Field::Types::Which;
+ size_t operator()(const Array& x) {
+ size_t size = x.size();
+ for (size_t i = 0; i < size; ++i) apply_visitor(*this, x[i]);
+ return 0;
+ }
+ // TODO doris not support unsigned integers for now
+ // treat as signed integers
+ size_t operator()(const UInt64& x) {
+ field_types.insert(FieldType::UInt64);
+ if (x <= std::numeric_limits<Int8>::max()) {
+ type_indexes.insert(TypeIndex::Int8);
+ } else if (x <= std::numeric_limits<Int16>::max()) {
+ type_indexes.insert(TypeIndex::Int16);
+ } else if (x <= std::numeric_limits<Int32>::max()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Int64& x) {
+ // Only support Int32 and Int64
+ field_types.insert(FieldType::Int64);
+ if (x <= std::numeric_limits<Int32>::max() && x >= std::numeric_limits<Int32>::min()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Null&) {
+ have_nulls = true;
+ return 0;
+ }
+ size_t operator()(const Int128I& x) { LOG(FATAL) << "not implemented"; }
+ template <typename T>
+ size_t operator()(const T&) {
+ Field::EnumToType<Field::Types::Array>::Type a;
+ field_types.insert(Field::TypeToEnum<NearestFieldType<T>>::value);
+ type_indexes.insert(TypeId<NearestFieldType<T>>::value);
+ return 0;
+ }
+ Status get_scalar_type(DataTypePtr* type) const {
+ return get_least_supertype(type_indexes, type, true /*compatible with string type*/);
+ }
+ bool contain_nulls() const { return have_nulls; }
+ bool need_convert_field() const { return field_types.size() > 1; }
+
+private:
+ phmap::flat_hash_set<TypeIndex> type_indexes;
+ phmap::flat_hash_set<FieldType> field_types;
+ bool have_nulls = false;
+};
+
+} // namespace
+Status get_field_info(const Field& field, FieldInfo* info) {
+ FieldVisitorToScalarType to_scalar_type_visitor;
+ apply_visitor(to_scalar_type_visitor, field);
+ DataTypePtr type = nullptr;
+ RETURN_IF_ERROR(to_scalar_type_visitor.get_scalar_type(&type));
+ // array item's dimension may missmatch, eg. [1, 2, [1, 2, 3]]
+ Status num_to_dimensions_status;
+ *info = {
+ type,
+ to_scalar_type_visitor.contain_nulls(),
+ to_scalar_type_visitor.need_convert_field(),
+ apply_visitor(FieldVisitorToNumberOfDimensions(&num_to_dimensions_status), field),
+ };
+ RETURN_IF_ERROR(num_to_dimensions_status);
+ return Status::OK();
+}
+
+ColumnObject::Subcolumn::Subcolumn(MutableColumnPtr&& data_, bool is_nullable_)
+ : least_common_type(get_data_type_by_column(*data_)), is_nullable(is_nullable_) {
+ data.push_back(std::move(data_));
+}
+
+ColumnObject::Subcolumn::Subcolumn(size_t size_, bool is_nullable_)
+ : least_common_type(std::make_shared<DataTypeNothing>()),
+ is_nullable(is_nullable_),
+ num_of_defaults_in_prefix(size_) {}
+
+size_t ColumnObject::Subcolumn::Subcolumn::size() const {
+ size_t res = num_of_defaults_in_prefix;
+ for (const auto& part : data) {
+ res += part->size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::byteSize() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->byte_size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::allocatedBytes() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->allocated_bytes();
+ }
+ return res;
+}
+
+Status ColumnObject::Subcolumn::insert(Field field) {
+ FieldInfo info;
+ RETURN_IF_ERROR(get_field_info(field, &info));
+ return insert(std::move(field), std::move(info));
+}
+
+void ColumnObject::Subcolumn::add_new_column_part(DataTypePtr type) {
+ data.push_back(type->create_column());
+ least_common_type = LeastCommonType {std::move(type)};
+}
+
+Status ColumnObject::Subcolumn::insert(Field field, FieldInfo info) {
+ auto base_type = std::move(info.scalar_type);
+ if (is_nothing(base_type)) {
+ insertDefault();
+ return Status::OK();
+ }
+ auto column_dim = least_common_type.get_dimensions();
+ auto value_dim = info.num_dimensions;
+ if (is_nothing(least_common_type.getBase())) {
+ column_dim = value_dim;
+ }
+ if (is_nothing(base_type)) {
+ value_dim = column_dim;
+ }
+ if (value_dim != column_dim) {
+ return Status::InvalidArgument(
+ "Dimension of types mismatched between inserted value and column.");
+ }
+ if (is_nullable && !is_nothing(base_type)) {
+ base_type = make_nullable(base_type);
+ }
+ // alawys nullable at present
+ if (!is_nullable && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+ // need replace muli dimensions array which contains null. eg. [[1, 2, 3], null] -> [[1, 2, 3], []]
+ // since column array doesnt known null's dimension
+ if (info.num_dimensions >= 2 && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+
+ bool type_changed = false;
+ const auto& least_common_base_type = least_common_type.getBase();
+ if (data.empty()) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ } else if (!least_common_base_type->equals(*base_type) && !is_nothing(base_type)) {
+ if (!object_util::is_conversion_required_between_integers(*base_type,
+ *least_common_base_type)) {
+ RETURN_IF_ERROR(
+ get_least_supertype(DataTypes {std::move(base_type), least_common_base_type},
+ &base_type, true /*compatible with string type*/));
+ type_changed = true;
+ if (!least_common_base_type->equals(*base_type)) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ }
+ }
+ }
+
+ if (type_changed || info.need_convert) {
+ Field new_field;
+ RETURN_IF_ERROR(convert_field_to_type(field, *least_common_type.get(), &new_field));
+ field = new_field;
+ }
+
+ data.back()->insert(field);
+ return Status::OK();
+}
+
+Status ColumnObject::Subcolumn::insertRangeFrom(const Subcolumn& src, size_t start, size_t length) {
+ assert(src.is_finalized());
+ const auto& src_column = src.data.back();
+ const auto& src_type = src.least_common_type.get();
+ if (data.empty()) {
+ add_new_column_part(src.least_common_type.get());
+ data.back()->insert_range_from(*src_column, start, length);
+ } else if (least_common_type.get()->equals(*src_type)) {
+ data.back()->insert_range_from(*src_column, start, length);
+ } else {
+ DataTypePtr new_least_common_type = nullptr;
+ RETURN_IF_ERROR(get_least_supertype(DataTypes {least_common_type.get(), src_type},
+ &new_least_common_type,
+ true /*compatible with string type*/));
+ ColumnPtr casted_column;
+ RETURN_IF_ERROR(object_util::cast_column({src_column, src_type, ""}, new_least_common_type,
+ &casted_column));
+ if (!least_common_type.get()->equals(*new_least_common_type))
+ add_new_column_part(std::move(new_least_common_type));
+ data.back()->insert_range_from(*casted_column, start, length);
+ }
+ return Status::OK();
+}
+
+bool ColumnObject::Subcolumn::is_finalized() const {
+ return data.empty() || (data.size() == 1 && num_of_defaults_in_prefix == 0);
+}
+
+template <typename Func>
+ColumnPtr ColumnObject::apply_for_subcolumns(Func&& func, std::string_view func_name) const {
+ if (!is_finalized()) {
+ LOG(FATAL) << "Cannot " << func_name << " non-finalized ColumnObject";
+ }
+ auto res = ColumnObject::create(is_nullable);
+ for (const auto& subcolumn : subcolumns) {
+ auto new_subcolumn = func(subcolumn->data.get_finalized_column());
+ res->add_sub_column(subcolumn->path, new_subcolumn->assume_mutable());
+ }
+ return res;
+}
+ColumnPtr ColumnObject::index(const IColumn& indexes, size_t limit) const {
+ return apply_for_subcolumns(
+ [&](const auto& subcolumn) { return subcolumn.index(indexes, limit); }, "index");
+}
+
+void ColumnObject::Subcolumn::finalize() {
+ if (is_finalized()) {
+ return;
+ }
+ if (data.size() == 1 && num_of_defaults_in_prefix == 0) {
+ data[0] = data[0]->convert_to_full_column_if_const();
+ return;
+ }
+ const auto& to_type = least_common_type.get();
+ auto result_column = to_type->create_column();
+ if (num_of_defaults_in_prefix) {
+ result_column->insert_many_defaults(num_of_defaults_in_prefix);
+ }
+ for (auto& part : data) {
+ part = part->convert_to_full_column_if_const();
+ auto from_type = get_data_type_by_column(*part);
+ size_t part_size = part->size();
+ if (!from_type->equals(*to_type)) {
+ auto offsets = ColumnUInt64::create();
+ auto& offsets_data = offsets->get_data();
+ /// We need to convert only non-default values and then recreate column
+ /// with default value of new type, because default values (which represents misses in data)
+ /// may be inconsistent between types (e.g "0" in UInt64 and empty string in String).
+ part->get_indices_of_non_default_rows(offsets_data, 0, part_size);
+ if (offsets->size() == part_size) {
+ ColumnPtr ptr;
+ object_util::cast_column({part, from_type, ""}, to_type, &ptr);
+ part = ptr;
+ } else {
+ auto values = part->index(*offsets, offsets->size());
+ object_util::cast_column({values, from_type, ""}, to_type, &values);
+ part = values->create_with_offsets(offsets_data, to_type->get_default(), part_size,
+ /*shift=*/0);
+ }
+ }
+ result_column->insert_range_from(*part, 0, part_size);
+ }
+ data = {std::move(result_column)};
+ num_of_defaults_in_prefix = 0;
+}
+
+void ColumnObject::Subcolumn::insertDefault() {
+ if (data.empty()) {
+ ++num_of_defaults_in_prefix;
+ } else {
+ data.back()->insert_default();
+ }
+}
+
+void ColumnObject::Subcolumn::insertManyDefaults(size_t length) {
+ if (data.empty()) {
+ num_of_defaults_in_prefix += length;
+ } else {
+ data.back()->insert_many_defaults(length);
+ }
+}
+
+void ColumnObject::Subcolumn::pop_back(size_t n) {
+ assert(n <= size());
+ size_t num_removed = 0;
+ for (auto it = data.rbegin(); it != data.rend(); ++it) {
+ if (n == 0) break;
+ auto& column = *it;
+ if (n < column->size()) {
+ column->pop_back(n);
+ n = 0;
+ } else {
+ ++num_removed;
+ n -= column->size();
+ }
+ }
+ data.resize(data.size() - num_removed);
+ num_of_defaults_in_prefix -= n;
+}
+
+Field ColumnObject::Subcolumn::getLastField() const {
+ if (data.empty()) return Field();
+ const auto& last_part = data.back();
+ assert(!last_part->empty());
+ return (*last_part)[last_part->size() - 1];
+}
+
+ColumnObject::Subcolumn ColumnObject::Subcolumn::recreate_with_default_values(
+ const FieldInfo& field_info) const {
+ auto scalar_type = field_info.scalar_type;
+ if (is_nullable) {
+ scalar_type = make_nullable(scalar_type);
+ }
+ Subcolumn new_subcolumn;
+ new_subcolumn.least_common_type =
+ LeastCommonType {create_array_of_type(scalar_type, field_info.num_dimensions)};
+ new_subcolumn.is_nullable = is_nullable;
+ new_subcolumn.num_of_defaults_in_prefix = num_of_defaults_in_prefix;
+ new_subcolumn.data.reserve(data.size());
+ for (const auto& part : data) {
+ new_subcolumn.data.push_back(
+ recreate_column_with_default_value(part, scalar_type, field_info.num_dimensions));
+ }
+ return new_subcolumn;
+}
+
+IColumn& ColumnObject::Subcolumn::get_finalized_column() {
+ assert(is_finalized());
+ return *data[0];
+}
+
+const IColumn& ColumnObject::Subcolumn::get_finalized_column() const {
+ assert(is_finalized());
+ return *data[0];
+}
+
+const ColumnPtr& ColumnObject::Subcolumn::get_finalized_column_ptr() const {
+ assert(is_finalized());
+ return data[0];
+}
+
+ColumnObject::Subcolumn::LeastCommonType::LeastCommonType(DataTypePtr type_)
+ : type(std::move(type_)),
+ base_type(getBaseTypeOfArray(type)),
+ num_dimensions(getNumberOfDimensions(*type)) {}
+
+ColumnObject::ColumnObject(bool is_nullable_) : is_nullable(is_nullable_), num_rows(0) {}
+
+ColumnObject::ColumnObject(Subcolumns&& subcolumns_, bool is_nullable_)
+ : is_nullable(is_nullable_),
+ subcolumns(std::move(subcolumns_)),
+ num_rows(subcolumns.empty() ? 0 : (*subcolumns.begin())->data.size()) {
+ check_consistency();
+}
+
+void ColumnObject::check_consistency() const {
+ if (subcolumns.empty()) return;
+ for (const auto& leaf : subcolumns) {
+ if (num_rows != leaf->data.size()) {
+ assert(false);
+ }
+ }
+}
+
+size_t ColumnObject::size() const {
+#ifndef NDEBUG
+ check_consistency();
+#endif
+ return num_rows;
+}
+
+MutableColumnPtr ColumnObject::clone_resized(size_t new_size) const {
+ /// cloneResized with new_size == 0 is used for cloneEmpty().
+ if (new_size != 0) {
+ LOG(FATAL) << "ColumnObject doesn't support resize to non-zero length";
+ }
+ return ColumnObject::create(is_nullable);
+}
+
+size_t ColumnObject::byte_size() const {
+ size_t res = 0;
+ for (const auto& entry : subcolumns) {
+ res += entry->data.byteSize();
+ }
+ return res;
+}
+
+size_t ColumnObject::allocated_bytes() const {
+ size_t res = 0;
+ for (const auto& entry : subcolumns) {
+ res += entry->data.allocatedBytes();
+ }
+ return res;
+}
+
+void ColumnObject::for_each_subcolumn(ColumnCallback callback) {
+ if (!is_finalized()) assert(false);
+ for (auto& entry : subcolumns) {
+ callback(entry->data.data.back());
+ }
+}
+
+Status ColumnObject::try_insert_from(const IColumn& src, size_t n) {
+ return try_insert(src[n]);
+}
+
+Status ColumnObject::try_insert(const Field& field) {
+ const auto& object = field.get<const Object&>();
+ phmap::flat_hash_set<StringRef, StringRefHash> inserted;
+ size_t old_size = size();
+ for (const auto& [key_str, value] : object) {
+ PathInData key(key_str);
+ inserted.insert(key_str);
+ if (!has_subcolumn(key)) {
+ bool succ = add_sub_column(key, old_size);
+ if (!succ) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to add sub column {}", key.get_path()));
+ }
+ }
+ auto* subcolumn = get_subcolumn(key);
+ if (!subcolumn) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to find sub column {}", key.get_path()));
+ }
+ RETURN_IF_ERROR(subcolumn->insert(value));
+ }
+ for (auto& entry : subcolumns) {
+ if (!inserted.contains(entry->path.get_path())) {
+ entry->data.insertDefault();
+ }
+ }
+ ++num_rows;
+ return Status::OK();
+}
+
+void ColumnObject::insert_default() {
+ for (auto& entry : subcolumns) {
+ entry->data.insertDefault();
+ }
+ ++num_rows;
+}
+
+Field ColumnObject::operator[](size_t n) const {
+ if (!is_finalized()) assert(false);
+ Object object;
+ for (const auto& entry : subcolumns) {
+ object[entry->path.get_path()] = (*entry->data.data.back())[n];
+ }
+ return object;
+}
+
+void ColumnObject::get(size_t n, Field& res) const {
+ if (!is_finalized()) assert(false);
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!is_finalized()) { assert(false);
}
```
##########
be/src/vec/columns/column_object.cpp:
##########
@@ -0,0 +1,860 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.cpp
+// and modified by Doris
+
+#include <parallel_hashmap/phmap.h>
+#include <vec/columns/column_array.h>
+#include <vec/columns/column_nullable.h>
+#include <vec/columns/column_object.h>
+#include <vec/columns/columns_number.h>
+#include <vec/common/field_visitors.h>
+#include <vec/common/hash_table/hash_set.h>
+#include <vec/common/object_util.h>
+#include <vec/common/pod_array_fwd.h>
+#include <vec/core/field.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_nothing.h>
+#include <vec/data_types/get_least_supertype.h>
+
+#include <vec/data_types/data_type_factory.hpp>
+
+namespace doris::vectorized {
+namespace {
+
+DataTypePtr create_array_of_type(DataTypePtr type, size_t num_dimensions) {
+ for (size_t i = 0; i < num_dimensions; ++i)
+ type = std::make_shared<DataTypeArray>(std::move(type));
+ return type;
+}
+
+DataTypePtr getBaseTypeOfArray(const DataTypePtr& type) {
+ /// Get raw pointers to avoid extra copying of type pointers.
+ const DataTypeArray* last_array = nullptr;
+ const auto* current_type = type.get();
+ while (const auto* type_array = typeid_cast<const DataTypeArray*>(current_type)) {
+ current_type = type_array->get_nested_type().get();
+ last_array = type_array;
+ }
+ return last_array ? last_array->get_nested_type() : type;
+}
+
+size_t getNumberOfDimensions(const IDataType& type) {
+ if (const auto* type_array = typeid_cast<const DataTypeArray*>(&type)) {
+ return type_array->get_number_of_dimensions();
+ }
+ return 0;
+}
+
+DataTypePtr get_data_type_by_column(const IColumn& column) {
+ auto idx = column.get_data_type();
+ if (WhichDataType(idx).is_simple()) {
+ return DataTypeFactory::instance().get(String(getTypeName(idx)));
+ }
+ if (WhichDataType(idx).is_nothing()) {
+ return std::make_shared<DataTypeNothing>();
+ }
+ if (const auto* column_array = check_and_get_column<ColumnArray>(&column)) {
+ return std::make_shared<DataTypeArray>(get_data_type_by_column(column_array->get_data()));
+ }
+ if (const auto* column_nullable = check_and_get_column<ColumnNullable>(&column)) {
+ return make_nullable(get_data_type_by_column(column_nullable->get_nested_column()));
+ }
+ // TODO add more types
+ assert(false);
+ return nullptr;
+}
+
+/// Recreates column with default scalar values and keeps sizes of arrays.
+ColumnPtr recreate_column_with_default_value(const ColumnPtr& column,
+ const DataTypePtr& scalar_type,
+ size_t num_dimensions) {
+ const auto* column_array = check_and_get_column<ColumnArray>(column.get());
+ if (column_array && num_dimensions) {
+ return ColumnArray::create(
+ recreate_column_with_default_value(column_array->get_data_ptr(), scalar_type,
+ num_dimensions - 1),
+ IColumn::mutate(column_array->get_offsets_ptr()));
+ }
+ return create_array_of_type(scalar_type, num_dimensions)
+ ->create_column()
+ ->clone_resized(column->size());
+}
+
+Array create_empty_array_field(size_t num_dimensions) {
+ assert(num_dimensions != 0);
+ Array array;
+ Array* current_array = &array;
+ for (size_t i = 1; i < num_dimensions; ++i) {
+ current_array->push_back(Array());
+ current_array = ¤t_array->back().get<Array&>();
+ }
+ return array;
+}
+
+/// Replaces NULL fields to given field or empty array.
+class FieldVisitorReplaceNull : public StaticVisitor<Field> {
+public:
+ explicit FieldVisitorReplaceNull(const Field& replacement_, size_t num_dimensions_)
+ : replacement(replacement_), num_dimensions(num_dimensions_) {}
+ Field operator()(const Null&) const {
+ return num_dimensions ? create_empty_array_field(num_dimensions) : replacement;
+ }
+ Field operator()(const Array& x) const {
+ assert(num_dimensions > 0);
+ const size_t size = x.size();
+ Array res(size);
+ for (size_t i = 0; i < size; ++i) {
+ res[i] = apply_visitor(FieldVisitorReplaceNull(replacement, num_dimensions - 1), x[i]);
+ }
+ return res;
+ }
+ template <typename T>
+ Field operator()(const T& x) const {
+ return x;
+ }
+
+private:
+ const Field& replacement;
+ size_t num_dimensions;
+};
+
+/// Calculates number of dimensions in array field.
+/// Returns 0 for scalar fields.
+class FieldVisitorToNumberOfDimensions : public StaticVisitor<size_t> {
+public:
+ explicit FieldVisitorToNumberOfDimensions(Status* st) : _st(st) {}
+ size_t operator()(const Array& x) const {
+ const size_t size = x.size();
+ std::optional<size_t> dimensions;
+ for (size_t i = 0; i < size; ++i) {
+ /// Do not count Nulls, because they will be replaced by default
+ /// values with proper number of dimensions.
+ if (x[i].is_null()) continue;
+ size_t current_dimensions = apply_visitor(*this, x[i]);
+ if (!dimensions) {
+ dimensions = current_dimensions;
+ } else if (current_dimensions != *dimensions) {
+ *_st = Status::InvalidArgument(
+ "Number of dimensions mismatched among array elements");
+ return 0;
+ }
+ }
+ return 1 + dimensions.value_or(0);
+ }
+ template <typename T>
+ size_t operator()(const T&) const {
+ return 0;
+ }
+
+private:
+ mutable Status* _st;
+};
+
+/// Visitor that allows to get type of scalar field
+/// or least common type of scalars in array.
+/// More optimized version of FieldToDataType.
+class FieldVisitorToScalarType : public StaticVisitor<size_t> {
+public:
+ using FieldType = Field::Types::Which;
+ size_t operator()(const Array& x) {
+ size_t size = x.size();
+ for (size_t i = 0; i < size; ++i) apply_visitor(*this, x[i]);
+ return 0;
+ }
+ // TODO doris not support unsigned integers for now
+ // treat as signed integers
+ size_t operator()(const UInt64& x) {
+ field_types.insert(FieldType::UInt64);
+ if (x <= std::numeric_limits<Int8>::max()) {
+ type_indexes.insert(TypeIndex::Int8);
+ } else if (x <= std::numeric_limits<Int16>::max()) {
+ type_indexes.insert(TypeIndex::Int16);
+ } else if (x <= std::numeric_limits<Int32>::max()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Int64& x) {
+ // Only support Int32 and Int64
+ field_types.insert(FieldType::Int64);
+ if (x <= std::numeric_limits<Int32>::max() && x >= std::numeric_limits<Int32>::min()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Null&) {
+ have_nulls = true;
+ return 0;
+ }
+ size_t operator()(const Int128I& x) { LOG(FATAL) << "not implemented"; }
+ template <typename T>
+ size_t operator()(const T&) {
+ Field::EnumToType<Field::Types::Array>::Type a;
+ field_types.insert(Field::TypeToEnum<NearestFieldType<T>>::value);
+ type_indexes.insert(TypeId<NearestFieldType<T>>::value);
+ return 0;
+ }
+ Status get_scalar_type(DataTypePtr* type) const {
+ return get_least_supertype(type_indexes, type, true /*compatible with string type*/);
+ }
+ bool contain_nulls() const { return have_nulls; }
+ bool need_convert_field() const { return field_types.size() > 1; }
+
+private:
+ phmap::flat_hash_set<TypeIndex> type_indexes;
+ phmap::flat_hash_set<FieldType> field_types;
+ bool have_nulls = false;
+};
+
+} // namespace
+Status get_field_info(const Field& field, FieldInfo* info) {
+ FieldVisitorToScalarType to_scalar_type_visitor;
+ apply_visitor(to_scalar_type_visitor, field);
+ DataTypePtr type = nullptr;
+ RETURN_IF_ERROR(to_scalar_type_visitor.get_scalar_type(&type));
+ // array item's dimension may missmatch, eg. [1, 2, [1, 2, 3]]
+ Status num_to_dimensions_status;
+ *info = {
+ type,
+ to_scalar_type_visitor.contain_nulls(),
+ to_scalar_type_visitor.need_convert_field(),
+ apply_visitor(FieldVisitorToNumberOfDimensions(&num_to_dimensions_status), field),
+ };
+ RETURN_IF_ERROR(num_to_dimensions_status);
+ return Status::OK();
+}
+
+ColumnObject::Subcolumn::Subcolumn(MutableColumnPtr&& data_, bool is_nullable_)
+ : least_common_type(get_data_type_by_column(*data_)), is_nullable(is_nullable_) {
+ data.push_back(std::move(data_));
+}
+
+ColumnObject::Subcolumn::Subcolumn(size_t size_, bool is_nullable_)
+ : least_common_type(std::make_shared<DataTypeNothing>()),
+ is_nullable(is_nullable_),
+ num_of_defaults_in_prefix(size_) {}
+
+size_t ColumnObject::Subcolumn::Subcolumn::size() const {
+ size_t res = num_of_defaults_in_prefix;
+ for (const auto& part : data) {
+ res += part->size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::byteSize() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->byte_size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::allocatedBytes() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->allocated_bytes();
+ }
+ return res;
+}
+
+Status ColumnObject::Subcolumn::insert(Field field) {
+ FieldInfo info;
+ RETURN_IF_ERROR(get_field_info(field, &info));
+ return insert(std::move(field), std::move(info));
+}
+
+void ColumnObject::Subcolumn::add_new_column_part(DataTypePtr type) {
+ data.push_back(type->create_column());
+ least_common_type = LeastCommonType {std::move(type)};
+}
+
+Status ColumnObject::Subcolumn::insert(Field field, FieldInfo info) {
+ auto base_type = std::move(info.scalar_type);
+ if (is_nothing(base_type)) {
+ insertDefault();
+ return Status::OK();
+ }
+ auto column_dim = least_common_type.get_dimensions();
+ auto value_dim = info.num_dimensions;
+ if (is_nothing(least_common_type.getBase())) {
+ column_dim = value_dim;
+ }
+ if (is_nothing(base_type)) {
+ value_dim = column_dim;
+ }
+ if (value_dim != column_dim) {
+ return Status::InvalidArgument(
+ "Dimension of types mismatched between inserted value and column.");
+ }
+ if (is_nullable && !is_nothing(base_type)) {
+ base_type = make_nullable(base_type);
+ }
+ // alawys nullable at present
+ if (!is_nullable && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+ // need replace muli dimensions array which contains null. eg. [[1, 2, 3], null] -> [[1, 2, 3], []]
+ // since column array doesnt known null's dimension
+ if (info.num_dimensions >= 2 && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+
+ bool type_changed = false;
+ const auto& least_common_base_type = least_common_type.getBase();
+ if (data.empty()) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ } else if (!least_common_base_type->equals(*base_type) && !is_nothing(base_type)) {
+ if (!object_util::is_conversion_required_between_integers(*base_type,
+ *least_common_base_type)) {
+ RETURN_IF_ERROR(
+ get_least_supertype(DataTypes {std::move(base_type), least_common_base_type},
+ &base_type, true /*compatible with string type*/));
+ type_changed = true;
+ if (!least_common_base_type->equals(*base_type)) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ }
+ }
+ }
+
+ if (type_changed || info.need_convert) {
+ Field new_field;
+ RETURN_IF_ERROR(convert_field_to_type(field, *least_common_type.get(), &new_field));
+ field = new_field;
+ }
+
+ data.back()->insert(field);
+ return Status::OK();
+}
+
+Status ColumnObject::Subcolumn::insertRangeFrom(const Subcolumn& src, size_t start, size_t length) {
+ assert(src.is_finalized());
+ const auto& src_column = src.data.back();
+ const auto& src_type = src.least_common_type.get();
+ if (data.empty()) {
+ add_new_column_part(src.least_common_type.get());
+ data.back()->insert_range_from(*src_column, start, length);
+ } else if (least_common_type.get()->equals(*src_type)) {
+ data.back()->insert_range_from(*src_column, start, length);
+ } else {
+ DataTypePtr new_least_common_type = nullptr;
+ RETURN_IF_ERROR(get_least_supertype(DataTypes {least_common_type.get(), src_type},
+ &new_least_common_type,
+ true /*compatible with string type*/));
+ ColumnPtr casted_column;
+ RETURN_IF_ERROR(object_util::cast_column({src_column, src_type, ""}, new_least_common_type,
+ &casted_column));
+ if (!least_common_type.get()->equals(*new_least_common_type))
+ add_new_column_part(std::move(new_least_common_type));
+ data.back()->insert_range_from(*casted_column, start, length);
+ }
+ return Status::OK();
+}
+
+bool ColumnObject::Subcolumn::is_finalized() const {
+ return data.empty() || (data.size() == 1 && num_of_defaults_in_prefix == 0);
+}
+
+template <typename Func>
+ColumnPtr ColumnObject::apply_for_subcolumns(Func&& func, std::string_view func_name) const {
+ if (!is_finalized()) {
+ LOG(FATAL) << "Cannot " << func_name << " non-finalized ColumnObject";
+ }
+ auto res = ColumnObject::create(is_nullable);
+ for (const auto& subcolumn : subcolumns) {
+ auto new_subcolumn = func(subcolumn->data.get_finalized_column());
+ res->add_sub_column(subcolumn->path, new_subcolumn->assume_mutable());
+ }
+ return res;
+}
+ColumnPtr ColumnObject::index(const IColumn& indexes, size_t limit) const {
+ return apply_for_subcolumns(
+ [&](const auto& subcolumn) { return subcolumn.index(indexes, limit); }, "index");
+}
+
+void ColumnObject::Subcolumn::finalize() {
+ if (is_finalized()) {
+ return;
+ }
+ if (data.size() == 1 && num_of_defaults_in_prefix == 0) {
+ data[0] = data[0]->convert_to_full_column_if_const();
+ return;
+ }
+ const auto& to_type = least_common_type.get();
+ auto result_column = to_type->create_column();
+ if (num_of_defaults_in_prefix) {
+ result_column->insert_many_defaults(num_of_defaults_in_prefix);
+ }
+ for (auto& part : data) {
+ part = part->convert_to_full_column_if_const();
+ auto from_type = get_data_type_by_column(*part);
+ size_t part_size = part->size();
+ if (!from_type->equals(*to_type)) {
+ auto offsets = ColumnUInt64::create();
+ auto& offsets_data = offsets->get_data();
+ /// We need to convert only non-default values and then recreate column
+ /// with default value of new type, because default values (which represents misses in data)
+ /// may be inconsistent between types (e.g "0" in UInt64 and empty string in String).
+ part->get_indices_of_non_default_rows(offsets_data, 0, part_size);
+ if (offsets->size() == part_size) {
+ ColumnPtr ptr;
+ object_util::cast_column({part, from_type, ""}, to_type, &ptr);
+ part = ptr;
+ } else {
+ auto values = part->index(*offsets, offsets->size());
+ object_util::cast_column({values, from_type, ""}, to_type, &values);
+ part = values->create_with_offsets(offsets_data, to_type->get_default(), part_size,
+ /*shift=*/0);
+ }
+ }
+ result_column->insert_range_from(*part, 0, part_size);
+ }
+ data = {std::move(result_column)};
+ num_of_defaults_in_prefix = 0;
+}
+
+void ColumnObject::Subcolumn::insertDefault() {
+ if (data.empty()) {
+ ++num_of_defaults_in_prefix;
+ } else {
+ data.back()->insert_default();
+ }
+}
+
+void ColumnObject::Subcolumn::insertManyDefaults(size_t length) {
+ if (data.empty()) {
+ num_of_defaults_in_prefix += length;
+ } else {
+ data.back()->insert_many_defaults(length);
+ }
+}
+
+void ColumnObject::Subcolumn::pop_back(size_t n) {
+ assert(n <= size());
+ size_t num_removed = 0;
+ for (auto it = data.rbegin(); it != data.rend(); ++it) {
+ if (n == 0) break;
+ auto& column = *it;
+ if (n < column->size()) {
+ column->pop_back(n);
+ n = 0;
+ } else {
+ ++num_removed;
+ n -= column->size();
+ }
+ }
+ data.resize(data.size() - num_removed);
+ num_of_defaults_in_prefix -= n;
+}
+
+Field ColumnObject::Subcolumn::getLastField() const {
+ if (data.empty()) return Field();
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (data.empty()) { return Field();
}
```
##########
be/src/vec/columns/column_object.cpp:
##########
@@ -0,0 +1,860 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.cpp
+// and modified by Doris
+
+#include <parallel_hashmap/phmap.h>
+#include <vec/columns/column_array.h>
+#include <vec/columns/column_nullable.h>
+#include <vec/columns/column_object.h>
+#include <vec/columns/columns_number.h>
+#include <vec/common/field_visitors.h>
+#include <vec/common/hash_table/hash_set.h>
+#include <vec/common/object_util.h>
+#include <vec/common/pod_array_fwd.h>
+#include <vec/core/field.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_nothing.h>
+#include <vec/data_types/get_least_supertype.h>
+
+#include <vec/data_types/data_type_factory.hpp>
+
+namespace doris::vectorized {
+namespace {
+
+DataTypePtr create_array_of_type(DataTypePtr type, size_t num_dimensions) {
+ for (size_t i = 0; i < num_dimensions; ++i)
+ type = std::make_shared<DataTypeArray>(std::move(type));
+ return type;
+}
+
+DataTypePtr getBaseTypeOfArray(const DataTypePtr& type) {
+ /// Get raw pointers to avoid extra copying of type pointers.
+ const DataTypeArray* last_array = nullptr;
+ const auto* current_type = type.get();
+ while (const auto* type_array = typeid_cast<const DataTypeArray*>(current_type)) {
+ current_type = type_array->get_nested_type().get();
+ last_array = type_array;
+ }
+ return last_array ? last_array->get_nested_type() : type;
+}
+
+size_t getNumberOfDimensions(const IDataType& type) {
+ if (const auto* type_array = typeid_cast<const DataTypeArray*>(&type)) {
+ return type_array->get_number_of_dimensions();
+ }
+ return 0;
+}
+
+DataTypePtr get_data_type_by_column(const IColumn& column) {
+ auto idx = column.get_data_type();
+ if (WhichDataType(idx).is_simple()) {
+ return DataTypeFactory::instance().get(String(getTypeName(idx)));
+ }
+ if (WhichDataType(idx).is_nothing()) {
+ return std::make_shared<DataTypeNothing>();
+ }
+ if (const auto* column_array = check_and_get_column<ColumnArray>(&column)) {
+ return std::make_shared<DataTypeArray>(get_data_type_by_column(column_array->get_data()));
+ }
+ if (const auto* column_nullable = check_and_get_column<ColumnNullable>(&column)) {
+ return make_nullable(get_data_type_by_column(column_nullable->get_nested_column()));
+ }
+ // TODO add more types
+ assert(false);
+ return nullptr;
+}
+
+/// Recreates column with default scalar values and keeps sizes of arrays.
+ColumnPtr recreate_column_with_default_value(const ColumnPtr& column,
+ const DataTypePtr& scalar_type,
+ size_t num_dimensions) {
+ const auto* column_array = check_and_get_column<ColumnArray>(column.get());
+ if (column_array && num_dimensions) {
+ return ColumnArray::create(
+ recreate_column_with_default_value(column_array->get_data_ptr(), scalar_type,
+ num_dimensions - 1),
+ IColumn::mutate(column_array->get_offsets_ptr()));
+ }
+ return create_array_of_type(scalar_type, num_dimensions)
+ ->create_column()
+ ->clone_resized(column->size());
+}
+
+Array create_empty_array_field(size_t num_dimensions) {
+ assert(num_dimensions != 0);
+ Array array;
+ Array* current_array = &array;
+ for (size_t i = 1; i < num_dimensions; ++i) {
+ current_array->push_back(Array());
+ current_array = ¤t_array->back().get<Array&>();
+ }
+ return array;
+}
+
+/// Replaces NULL fields to given field or empty array.
+class FieldVisitorReplaceNull : public StaticVisitor<Field> {
+public:
+ explicit FieldVisitorReplaceNull(const Field& replacement_, size_t num_dimensions_)
+ : replacement(replacement_), num_dimensions(num_dimensions_) {}
+ Field operator()(const Null&) const {
+ return num_dimensions ? create_empty_array_field(num_dimensions) : replacement;
+ }
+ Field operator()(const Array& x) const {
+ assert(num_dimensions > 0);
+ const size_t size = x.size();
+ Array res(size);
+ for (size_t i = 0; i < size; ++i) {
+ res[i] = apply_visitor(FieldVisitorReplaceNull(replacement, num_dimensions - 1), x[i]);
+ }
+ return res;
+ }
+ template <typename T>
+ Field operator()(const T& x) const {
+ return x;
+ }
+
+private:
+ const Field& replacement;
+ size_t num_dimensions;
+};
+
+/// Calculates number of dimensions in array field.
+/// Returns 0 for scalar fields.
+class FieldVisitorToNumberOfDimensions : public StaticVisitor<size_t> {
+public:
+ explicit FieldVisitorToNumberOfDimensions(Status* st) : _st(st) {}
+ size_t operator()(const Array& x) const {
+ const size_t size = x.size();
+ std::optional<size_t> dimensions;
+ for (size_t i = 0; i < size; ++i) {
+ /// Do not count Nulls, because they will be replaced by default
+ /// values with proper number of dimensions.
+ if (x[i].is_null()) continue;
+ size_t current_dimensions = apply_visitor(*this, x[i]);
+ if (!dimensions) {
+ dimensions = current_dimensions;
+ } else if (current_dimensions != *dimensions) {
+ *_st = Status::InvalidArgument(
+ "Number of dimensions mismatched among array elements");
+ return 0;
+ }
+ }
+ return 1 + dimensions.value_or(0);
+ }
+ template <typename T>
+ size_t operator()(const T&) const {
+ return 0;
+ }
+
+private:
+ mutable Status* _st;
+};
+
+/// Visitor that allows to get type of scalar field
+/// or least common type of scalars in array.
+/// More optimized version of FieldToDataType.
+class FieldVisitorToScalarType : public StaticVisitor<size_t> {
+public:
+ using FieldType = Field::Types::Which;
+ size_t operator()(const Array& x) {
+ size_t size = x.size();
+ for (size_t i = 0; i < size; ++i) apply_visitor(*this, x[i]);
+ return 0;
+ }
+ // TODO doris not support unsigned integers for now
+ // treat as signed integers
+ size_t operator()(const UInt64& x) {
+ field_types.insert(FieldType::UInt64);
+ if (x <= std::numeric_limits<Int8>::max()) {
+ type_indexes.insert(TypeIndex::Int8);
+ } else if (x <= std::numeric_limits<Int16>::max()) {
+ type_indexes.insert(TypeIndex::Int16);
+ } else if (x <= std::numeric_limits<Int32>::max()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Int64& x) {
+ // Only support Int32 and Int64
+ field_types.insert(FieldType::Int64);
+ if (x <= std::numeric_limits<Int32>::max() && x >= std::numeric_limits<Int32>::min()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Null&) {
+ have_nulls = true;
+ return 0;
+ }
+ size_t operator()(const Int128I& x) { LOG(FATAL) << "not implemented"; }
+ template <typename T>
+ size_t operator()(const T&) {
+ Field::EnumToType<Field::Types::Array>::Type a;
+ field_types.insert(Field::TypeToEnum<NearestFieldType<T>>::value);
+ type_indexes.insert(TypeId<NearestFieldType<T>>::value);
+ return 0;
+ }
+ Status get_scalar_type(DataTypePtr* type) const {
+ return get_least_supertype(type_indexes, type, true /*compatible with string type*/);
+ }
+ bool contain_nulls() const { return have_nulls; }
+ bool need_convert_field() const { return field_types.size() > 1; }
+
+private:
+ phmap::flat_hash_set<TypeIndex> type_indexes;
+ phmap::flat_hash_set<FieldType> field_types;
+ bool have_nulls = false;
+};
+
+} // namespace
+Status get_field_info(const Field& field, FieldInfo* info) {
+ FieldVisitorToScalarType to_scalar_type_visitor;
+ apply_visitor(to_scalar_type_visitor, field);
+ DataTypePtr type = nullptr;
+ RETURN_IF_ERROR(to_scalar_type_visitor.get_scalar_type(&type));
+ // array item's dimension may missmatch, eg. [1, 2, [1, 2, 3]]
+ Status num_to_dimensions_status;
+ *info = {
+ type,
+ to_scalar_type_visitor.contain_nulls(),
+ to_scalar_type_visitor.need_convert_field(),
+ apply_visitor(FieldVisitorToNumberOfDimensions(&num_to_dimensions_status), field),
+ };
+ RETURN_IF_ERROR(num_to_dimensions_status);
+ return Status::OK();
+}
+
+ColumnObject::Subcolumn::Subcolumn(MutableColumnPtr&& data_, bool is_nullable_)
+ : least_common_type(get_data_type_by_column(*data_)), is_nullable(is_nullable_) {
+ data.push_back(std::move(data_));
+}
+
+ColumnObject::Subcolumn::Subcolumn(size_t size_, bool is_nullable_)
+ : least_common_type(std::make_shared<DataTypeNothing>()),
+ is_nullable(is_nullable_),
+ num_of_defaults_in_prefix(size_) {}
+
+size_t ColumnObject::Subcolumn::Subcolumn::size() const {
+ size_t res = num_of_defaults_in_prefix;
+ for (const auto& part : data) {
+ res += part->size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::byteSize() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->byte_size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::allocatedBytes() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->allocated_bytes();
+ }
+ return res;
+}
+
+Status ColumnObject::Subcolumn::insert(Field field) {
+ FieldInfo info;
+ RETURN_IF_ERROR(get_field_info(field, &info));
+ return insert(std::move(field), std::move(info));
+}
+
+void ColumnObject::Subcolumn::add_new_column_part(DataTypePtr type) {
+ data.push_back(type->create_column());
+ least_common_type = LeastCommonType {std::move(type)};
+}
+
+Status ColumnObject::Subcolumn::insert(Field field, FieldInfo info) {
+ auto base_type = std::move(info.scalar_type);
+ if (is_nothing(base_type)) {
+ insertDefault();
+ return Status::OK();
+ }
+ auto column_dim = least_common_type.get_dimensions();
+ auto value_dim = info.num_dimensions;
+ if (is_nothing(least_common_type.getBase())) {
+ column_dim = value_dim;
+ }
+ if (is_nothing(base_type)) {
+ value_dim = column_dim;
+ }
+ if (value_dim != column_dim) {
+ return Status::InvalidArgument(
+ "Dimension of types mismatched between inserted value and column.");
+ }
+ if (is_nullable && !is_nothing(base_type)) {
+ base_type = make_nullable(base_type);
+ }
+ // alawys nullable at present
+ if (!is_nullable && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+ // need replace muli dimensions array which contains null. eg. [[1, 2, 3], null] -> [[1, 2, 3], []]
+ // since column array doesnt known null's dimension
+ if (info.num_dimensions >= 2 && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+
+ bool type_changed = false;
+ const auto& least_common_base_type = least_common_type.getBase();
+ if (data.empty()) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ } else if (!least_common_base_type->equals(*base_type) && !is_nothing(base_type)) {
+ if (!object_util::is_conversion_required_between_integers(*base_type,
+ *least_common_base_type)) {
+ RETURN_IF_ERROR(
+ get_least_supertype(DataTypes {std::move(base_type), least_common_base_type},
+ &base_type, true /*compatible with string type*/));
+ type_changed = true;
+ if (!least_common_base_type->equals(*base_type)) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ }
+ }
+ }
+
+ if (type_changed || info.need_convert) {
+ Field new_field;
+ RETURN_IF_ERROR(convert_field_to_type(field, *least_common_type.get(), &new_field));
+ field = new_field;
+ }
+
+ data.back()->insert(field);
+ return Status::OK();
+}
+
+Status ColumnObject::Subcolumn::insertRangeFrom(const Subcolumn& src, size_t start, size_t length) {
+ assert(src.is_finalized());
+ const auto& src_column = src.data.back();
+ const auto& src_type = src.least_common_type.get();
+ if (data.empty()) {
+ add_new_column_part(src.least_common_type.get());
+ data.back()->insert_range_from(*src_column, start, length);
+ } else if (least_common_type.get()->equals(*src_type)) {
+ data.back()->insert_range_from(*src_column, start, length);
+ } else {
+ DataTypePtr new_least_common_type = nullptr;
+ RETURN_IF_ERROR(get_least_supertype(DataTypes {least_common_type.get(), src_type},
+ &new_least_common_type,
+ true /*compatible with string type*/));
+ ColumnPtr casted_column;
+ RETURN_IF_ERROR(object_util::cast_column({src_column, src_type, ""}, new_least_common_type,
+ &casted_column));
+ if (!least_common_type.get()->equals(*new_least_common_type))
+ add_new_column_part(std::move(new_least_common_type));
+ data.back()->insert_range_from(*casted_column, start, length);
+ }
+ return Status::OK();
+}
+
+bool ColumnObject::Subcolumn::is_finalized() const {
+ return data.empty() || (data.size() == 1 && num_of_defaults_in_prefix == 0);
+}
+
+template <typename Func>
+ColumnPtr ColumnObject::apply_for_subcolumns(Func&& func, std::string_view func_name) const {
+ if (!is_finalized()) {
+ LOG(FATAL) << "Cannot " << func_name << " non-finalized ColumnObject";
+ }
+ auto res = ColumnObject::create(is_nullable);
+ for (const auto& subcolumn : subcolumns) {
+ auto new_subcolumn = func(subcolumn->data.get_finalized_column());
+ res->add_sub_column(subcolumn->path, new_subcolumn->assume_mutable());
+ }
+ return res;
+}
+ColumnPtr ColumnObject::index(const IColumn& indexes, size_t limit) const {
+ return apply_for_subcolumns(
+ [&](const auto& subcolumn) { return subcolumn.index(indexes, limit); }, "index");
+}
+
+void ColumnObject::Subcolumn::finalize() {
+ if (is_finalized()) {
+ return;
+ }
+ if (data.size() == 1 && num_of_defaults_in_prefix == 0) {
+ data[0] = data[0]->convert_to_full_column_if_const();
+ return;
+ }
+ const auto& to_type = least_common_type.get();
+ auto result_column = to_type->create_column();
+ if (num_of_defaults_in_prefix) {
+ result_column->insert_many_defaults(num_of_defaults_in_prefix);
+ }
+ for (auto& part : data) {
+ part = part->convert_to_full_column_if_const();
+ auto from_type = get_data_type_by_column(*part);
+ size_t part_size = part->size();
+ if (!from_type->equals(*to_type)) {
+ auto offsets = ColumnUInt64::create();
+ auto& offsets_data = offsets->get_data();
+ /// We need to convert only non-default values and then recreate column
+ /// with default value of new type, because default values (which represents misses in data)
+ /// may be inconsistent between types (e.g "0" in UInt64 and empty string in String).
+ part->get_indices_of_non_default_rows(offsets_data, 0, part_size);
+ if (offsets->size() == part_size) {
+ ColumnPtr ptr;
+ object_util::cast_column({part, from_type, ""}, to_type, &ptr);
+ part = ptr;
+ } else {
+ auto values = part->index(*offsets, offsets->size());
+ object_util::cast_column({values, from_type, ""}, to_type, &values);
+ part = values->create_with_offsets(offsets_data, to_type->get_default(), part_size,
+ /*shift=*/0);
+ }
+ }
+ result_column->insert_range_from(*part, 0, part_size);
+ }
+ data = {std::move(result_column)};
+ num_of_defaults_in_prefix = 0;
+}
+
+void ColumnObject::Subcolumn::insertDefault() {
+ if (data.empty()) {
+ ++num_of_defaults_in_prefix;
+ } else {
+ data.back()->insert_default();
+ }
+}
+
+void ColumnObject::Subcolumn::insertManyDefaults(size_t length) {
+ if (data.empty()) {
+ num_of_defaults_in_prefix += length;
+ } else {
+ data.back()->insert_many_defaults(length);
+ }
+}
+
+void ColumnObject::Subcolumn::pop_back(size_t n) {
+ assert(n <= size());
+ size_t num_removed = 0;
+ for (auto it = data.rbegin(); it != data.rend(); ++it) {
+ if (n == 0) break;
+ auto& column = *it;
+ if (n < column->size()) {
+ column->pop_back(n);
+ n = 0;
+ } else {
+ ++num_removed;
+ n -= column->size();
+ }
+ }
+ data.resize(data.size() - num_removed);
+ num_of_defaults_in_prefix -= n;
+}
+
+Field ColumnObject::Subcolumn::getLastField() const {
+ if (data.empty()) return Field();
+ const auto& last_part = data.back();
+ assert(!last_part->empty());
+ return (*last_part)[last_part->size() - 1];
+}
+
+ColumnObject::Subcolumn ColumnObject::Subcolumn::recreate_with_default_values(
+ const FieldInfo& field_info) const {
+ auto scalar_type = field_info.scalar_type;
+ if (is_nullable) {
+ scalar_type = make_nullable(scalar_type);
+ }
+ Subcolumn new_subcolumn;
+ new_subcolumn.least_common_type =
+ LeastCommonType {create_array_of_type(scalar_type, field_info.num_dimensions)};
+ new_subcolumn.is_nullable = is_nullable;
+ new_subcolumn.num_of_defaults_in_prefix = num_of_defaults_in_prefix;
+ new_subcolumn.data.reserve(data.size());
+ for (const auto& part : data) {
+ new_subcolumn.data.push_back(
+ recreate_column_with_default_value(part, scalar_type, field_info.num_dimensions));
+ }
+ return new_subcolumn;
+}
+
+IColumn& ColumnObject::Subcolumn::get_finalized_column() {
+ assert(is_finalized());
+ return *data[0];
+}
+
+const IColumn& ColumnObject::Subcolumn::get_finalized_column() const {
+ assert(is_finalized());
+ return *data[0];
+}
+
+const ColumnPtr& ColumnObject::Subcolumn::get_finalized_column_ptr() const {
+ assert(is_finalized());
+ return data[0];
+}
+
+ColumnObject::Subcolumn::LeastCommonType::LeastCommonType(DataTypePtr type_)
+ : type(std::move(type_)),
+ base_type(getBaseTypeOfArray(type)),
+ num_dimensions(getNumberOfDimensions(*type)) {}
+
+ColumnObject::ColumnObject(bool is_nullable_) : is_nullable(is_nullable_), num_rows(0) {}
+
+ColumnObject::ColumnObject(Subcolumns&& subcolumns_, bool is_nullable_)
+ : is_nullable(is_nullable_),
+ subcolumns(std::move(subcolumns_)),
+ num_rows(subcolumns.empty() ? 0 : (*subcolumns.begin())->data.size()) {
+ check_consistency();
+}
+
+void ColumnObject::check_consistency() const {
+ if (subcolumns.empty()) return;
+ for (const auto& leaf : subcolumns) {
+ if (num_rows != leaf->data.size()) {
+ assert(false);
+ }
+ }
+}
+
+size_t ColumnObject::size() const {
+#ifndef NDEBUG
+ check_consistency();
+#endif
+ return num_rows;
+}
+
+MutableColumnPtr ColumnObject::clone_resized(size_t new_size) const {
+ /// cloneResized with new_size == 0 is used for cloneEmpty().
+ if (new_size != 0) {
+ LOG(FATAL) << "ColumnObject doesn't support resize to non-zero length";
+ }
+ return ColumnObject::create(is_nullable);
+}
+
+size_t ColumnObject::byte_size() const {
+ size_t res = 0;
+ for (const auto& entry : subcolumns) {
+ res += entry->data.byteSize();
+ }
+ return res;
+}
+
+size_t ColumnObject::allocated_bytes() const {
+ size_t res = 0;
+ for (const auto& entry : subcolumns) {
+ res += entry->data.allocatedBytes();
+ }
+ return res;
+}
+
+void ColumnObject::for_each_subcolumn(ColumnCallback callback) {
+ if (!is_finalized()) assert(false);
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!is_finalized()) { assert(false);
}
```
##########
be/src/vec/columns/column_object.cpp:
##########
@@ -0,0 +1,860 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.cpp
+// and modified by Doris
+
+#include <parallel_hashmap/phmap.h>
+#include <vec/columns/column_array.h>
+#include <vec/columns/column_nullable.h>
+#include <vec/columns/column_object.h>
+#include <vec/columns/columns_number.h>
+#include <vec/common/field_visitors.h>
+#include <vec/common/hash_table/hash_set.h>
+#include <vec/common/object_util.h>
+#include <vec/common/pod_array_fwd.h>
+#include <vec/core/field.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_nothing.h>
+#include <vec/data_types/get_least_supertype.h>
+
+#include <vec/data_types/data_type_factory.hpp>
+
+namespace doris::vectorized {
+namespace {
+
+DataTypePtr create_array_of_type(DataTypePtr type, size_t num_dimensions) {
+ for (size_t i = 0; i < num_dimensions; ++i)
+ type = std::make_shared<DataTypeArray>(std::move(type));
+ return type;
+}
+
+DataTypePtr getBaseTypeOfArray(const DataTypePtr& type) {
+ /// Get raw pointers to avoid extra copying of type pointers.
+ const DataTypeArray* last_array = nullptr;
+ const auto* current_type = type.get();
+ while (const auto* type_array = typeid_cast<const DataTypeArray*>(current_type)) {
+ current_type = type_array->get_nested_type().get();
+ last_array = type_array;
+ }
+ return last_array ? last_array->get_nested_type() : type;
+}
+
+size_t getNumberOfDimensions(const IDataType& type) {
+ if (const auto* type_array = typeid_cast<const DataTypeArray*>(&type)) {
+ return type_array->get_number_of_dimensions();
+ }
+ return 0;
+}
+
+DataTypePtr get_data_type_by_column(const IColumn& column) {
+ auto idx = column.get_data_type();
+ if (WhichDataType(idx).is_simple()) {
+ return DataTypeFactory::instance().get(String(getTypeName(idx)));
+ }
+ if (WhichDataType(idx).is_nothing()) {
+ return std::make_shared<DataTypeNothing>();
+ }
+ if (const auto* column_array = check_and_get_column<ColumnArray>(&column)) {
+ return std::make_shared<DataTypeArray>(get_data_type_by_column(column_array->get_data()));
+ }
+ if (const auto* column_nullable = check_and_get_column<ColumnNullable>(&column)) {
+ return make_nullable(get_data_type_by_column(column_nullable->get_nested_column()));
+ }
+ // TODO add more types
+ assert(false);
+ return nullptr;
+}
+
+/// Recreates column with default scalar values and keeps sizes of arrays.
+ColumnPtr recreate_column_with_default_value(const ColumnPtr& column,
+ const DataTypePtr& scalar_type,
+ size_t num_dimensions) {
+ const auto* column_array = check_and_get_column<ColumnArray>(column.get());
+ if (column_array && num_dimensions) {
+ return ColumnArray::create(
+ recreate_column_with_default_value(column_array->get_data_ptr(), scalar_type,
+ num_dimensions - 1),
+ IColumn::mutate(column_array->get_offsets_ptr()));
+ }
+ return create_array_of_type(scalar_type, num_dimensions)
+ ->create_column()
+ ->clone_resized(column->size());
+}
+
+Array create_empty_array_field(size_t num_dimensions) {
+ assert(num_dimensions != 0);
+ Array array;
+ Array* current_array = &array;
+ for (size_t i = 1; i < num_dimensions; ++i) {
+ current_array->push_back(Array());
+ current_array = ¤t_array->back().get<Array&>();
+ }
+ return array;
+}
+
+/// Replaces NULL fields to given field or empty array.
+class FieldVisitorReplaceNull : public StaticVisitor<Field> {
+public:
+ explicit FieldVisitorReplaceNull(const Field& replacement_, size_t num_dimensions_)
+ : replacement(replacement_), num_dimensions(num_dimensions_) {}
+ Field operator()(const Null&) const {
+ return num_dimensions ? create_empty_array_field(num_dimensions) : replacement;
+ }
+ Field operator()(const Array& x) const {
+ assert(num_dimensions > 0);
+ const size_t size = x.size();
+ Array res(size);
+ for (size_t i = 0; i < size; ++i) {
+ res[i] = apply_visitor(FieldVisitorReplaceNull(replacement, num_dimensions - 1), x[i]);
+ }
+ return res;
+ }
+ template <typename T>
+ Field operator()(const T& x) const {
+ return x;
+ }
+
+private:
+ const Field& replacement;
+ size_t num_dimensions;
+};
+
+/// Calculates number of dimensions in array field.
+/// Returns 0 for scalar fields.
+class FieldVisitorToNumberOfDimensions : public StaticVisitor<size_t> {
+public:
+ explicit FieldVisitorToNumberOfDimensions(Status* st) : _st(st) {}
+ size_t operator()(const Array& x) const {
+ const size_t size = x.size();
+ std::optional<size_t> dimensions;
+ for (size_t i = 0; i < size; ++i) {
+ /// Do not count Nulls, because they will be replaced by default
+ /// values with proper number of dimensions.
+ if (x[i].is_null()) continue;
+ size_t current_dimensions = apply_visitor(*this, x[i]);
+ if (!dimensions) {
+ dimensions = current_dimensions;
+ } else if (current_dimensions != *dimensions) {
+ *_st = Status::InvalidArgument(
+ "Number of dimensions mismatched among array elements");
+ return 0;
+ }
+ }
+ return 1 + dimensions.value_or(0);
+ }
+ template <typename T>
+ size_t operator()(const T&) const {
+ return 0;
+ }
+
+private:
+ mutable Status* _st;
+};
+
+/// Visitor that allows to get type of scalar field
+/// or least common type of scalars in array.
+/// More optimized version of FieldToDataType.
+class FieldVisitorToScalarType : public StaticVisitor<size_t> {
+public:
+ using FieldType = Field::Types::Which;
+ size_t operator()(const Array& x) {
+ size_t size = x.size();
+ for (size_t i = 0; i < size; ++i) apply_visitor(*this, x[i]);
+ return 0;
+ }
+ // TODO doris not support unsigned integers for now
+ // treat as signed integers
+ size_t operator()(const UInt64& x) {
+ field_types.insert(FieldType::UInt64);
+ if (x <= std::numeric_limits<Int8>::max()) {
+ type_indexes.insert(TypeIndex::Int8);
+ } else if (x <= std::numeric_limits<Int16>::max()) {
+ type_indexes.insert(TypeIndex::Int16);
+ } else if (x <= std::numeric_limits<Int32>::max()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Int64& x) {
+ // Only support Int32 and Int64
+ field_types.insert(FieldType::Int64);
+ if (x <= std::numeric_limits<Int32>::max() && x >= std::numeric_limits<Int32>::min()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Null&) {
+ have_nulls = true;
+ return 0;
+ }
+ size_t operator()(const Int128I& x) { LOG(FATAL) << "not implemented"; }
+ template <typename T>
+ size_t operator()(const T&) {
+ Field::EnumToType<Field::Types::Array>::Type a;
+ field_types.insert(Field::TypeToEnum<NearestFieldType<T>>::value);
+ type_indexes.insert(TypeId<NearestFieldType<T>>::value);
+ return 0;
+ }
+ Status get_scalar_type(DataTypePtr* type) const {
+ return get_least_supertype(type_indexes, type, true /*compatible with string type*/);
+ }
+ bool contain_nulls() const { return have_nulls; }
+ bool need_convert_field() const { return field_types.size() > 1; }
+
+private:
+ phmap::flat_hash_set<TypeIndex> type_indexes;
+ phmap::flat_hash_set<FieldType> field_types;
+ bool have_nulls = false;
+};
+
+} // namespace
+Status get_field_info(const Field& field, FieldInfo* info) {
+ FieldVisitorToScalarType to_scalar_type_visitor;
+ apply_visitor(to_scalar_type_visitor, field);
+ DataTypePtr type = nullptr;
+ RETURN_IF_ERROR(to_scalar_type_visitor.get_scalar_type(&type));
+ // array item's dimension may missmatch, eg. [1, 2, [1, 2, 3]]
+ Status num_to_dimensions_status;
+ *info = {
+ type,
+ to_scalar_type_visitor.contain_nulls(),
+ to_scalar_type_visitor.need_convert_field(),
+ apply_visitor(FieldVisitorToNumberOfDimensions(&num_to_dimensions_status), field),
+ };
+ RETURN_IF_ERROR(num_to_dimensions_status);
+ return Status::OK();
+}
+
+ColumnObject::Subcolumn::Subcolumn(MutableColumnPtr&& data_, bool is_nullable_)
+ : least_common_type(get_data_type_by_column(*data_)), is_nullable(is_nullable_) {
+ data.push_back(std::move(data_));
+}
+
+ColumnObject::Subcolumn::Subcolumn(size_t size_, bool is_nullable_)
+ : least_common_type(std::make_shared<DataTypeNothing>()),
+ is_nullable(is_nullable_),
+ num_of_defaults_in_prefix(size_) {}
+
+size_t ColumnObject::Subcolumn::Subcolumn::size() const {
+ size_t res = num_of_defaults_in_prefix;
+ for (const auto& part : data) {
+ res += part->size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::byteSize() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->byte_size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::allocatedBytes() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->allocated_bytes();
+ }
+ return res;
+}
+
+Status ColumnObject::Subcolumn::insert(Field field) {
+ FieldInfo info;
+ RETURN_IF_ERROR(get_field_info(field, &info));
+ return insert(std::move(field), std::move(info));
+}
+
+void ColumnObject::Subcolumn::add_new_column_part(DataTypePtr type) {
+ data.push_back(type->create_column());
+ least_common_type = LeastCommonType {std::move(type)};
+}
+
+Status ColumnObject::Subcolumn::insert(Field field, FieldInfo info) {
+ auto base_type = std::move(info.scalar_type);
+ if (is_nothing(base_type)) {
+ insertDefault();
+ return Status::OK();
+ }
+ auto column_dim = least_common_type.get_dimensions();
+ auto value_dim = info.num_dimensions;
+ if (is_nothing(least_common_type.getBase())) {
+ column_dim = value_dim;
+ }
+ if (is_nothing(base_type)) {
+ value_dim = column_dim;
+ }
+ if (value_dim != column_dim) {
+ return Status::InvalidArgument(
+ "Dimension of types mismatched between inserted value and column.");
+ }
+ if (is_nullable && !is_nothing(base_type)) {
+ base_type = make_nullable(base_type);
+ }
+ // alawys nullable at present
+ if (!is_nullable && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+ // need replace muli dimensions array which contains null. eg. [[1, 2, 3], null] -> [[1, 2, 3], []]
+ // since column array doesnt known null's dimension
+ if (info.num_dimensions >= 2 && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+
+ bool type_changed = false;
+ const auto& least_common_base_type = least_common_type.getBase();
+ if (data.empty()) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ } else if (!least_common_base_type->equals(*base_type) && !is_nothing(base_type)) {
+ if (!object_util::is_conversion_required_between_integers(*base_type,
+ *least_common_base_type)) {
+ RETURN_IF_ERROR(
+ get_least_supertype(DataTypes {std::move(base_type), least_common_base_type},
+ &base_type, true /*compatible with string type*/));
+ type_changed = true;
+ if (!least_common_base_type->equals(*base_type)) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ }
+ }
+ }
+
+ if (type_changed || info.need_convert) {
+ Field new_field;
+ RETURN_IF_ERROR(convert_field_to_type(field, *least_common_type.get(), &new_field));
+ field = new_field;
+ }
+
+ data.back()->insert(field);
+ return Status::OK();
+}
+
+Status ColumnObject::Subcolumn::insertRangeFrom(const Subcolumn& src, size_t start, size_t length) {
+ assert(src.is_finalized());
+ const auto& src_column = src.data.back();
+ const auto& src_type = src.least_common_type.get();
+ if (data.empty()) {
+ add_new_column_part(src.least_common_type.get());
+ data.back()->insert_range_from(*src_column, start, length);
+ } else if (least_common_type.get()->equals(*src_type)) {
+ data.back()->insert_range_from(*src_column, start, length);
+ } else {
+ DataTypePtr new_least_common_type = nullptr;
+ RETURN_IF_ERROR(get_least_supertype(DataTypes {least_common_type.get(), src_type},
+ &new_least_common_type,
+ true /*compatible with string type*/));
+ ColumnPtr casted_column;
+ RETURN_IF_ERROR(object_util::cast_column({src_column, src_type, ""}, new_least_common_type,
+ &casted_column));
+ if (!least_common_type.get()->equals(*new_least_common_type))
+ add_new_column_part(std::move(new_least_common_type));
+ data.back()->insert_range_from(*casted_column, start, length);
+ }
+ return Status::OK();
+}
+
+bool ColumnObject::Subcolumn::is_finalized() const {
+ return data.empty() || (data.size() == 1 && num_of_defaults_in_prefix == 0);
+}
+
+template <typename Func>
+ColumnPtr ColumnObject::apply_for_subcolumns(Func&& func, std::string_view func_name) const {
+ if (!is_finalized()) {
+ LOG(FATAL) << "Cannot " << func_name << " non-finalized ColumnObject";
+ }
+ auto res = ColumnObject::create(is_nullable);
+ for (const auto& subcolumn : subcolumns) {
+ auto new_subcolumn = func(subcolumn->data.get_finalized_column());
+ res->add_sub_column(subcolumn->path, new_subcolumn->assume_mutable());
+ }
+ return res;
+}
+ColumnPtr ColumnObject::index(const IColumn& indexes, size_t limit) const {
+ return apply_for_subcolumns(
+ [&](const auto& subcolumn) { return subcolumn.index(indexes, limit); }, "index");
+}
+
+void ColumnObject::Subcolumn::finalize() {
+ if (is_finalized()) {
+ return;
+ }
+ if (data.size() == 1 && num_of_defaults_in_prefix == 0) {
+ data[0] = data[0]->convert_to_full_column_if_const();
+ return;
+ }
+ const auto& to_type = least_common_type.get();
+ auto result_column = to_type->create_column();
+ if (num_of_defaults_in_prefix) {
+ result_column->insert_many_defaults(num_of_defaults_in_prefix);
+ }
+ for (auto& part : data) {
+ part = part->convert_to_full_column_if_const();
+ auto from_type = get_data_type_by_column(*part);
+ size_t part_size = part->size();
+ if (!from_type->equals(*to_type)) {
+ auto offsets = ColumnUInt64::create();
+ auto& offsets_data = offsets->get_data();
+ /// We need to convert only non-default values and then recreate column
+ /// with default value of new type, because default values (which represents misses in data)
+ /// may be inconsistent between types (e.g "0" in UInt64 and empty string in String).
+ part->get_indices_of_non_default_rows(offsets_data, 0, part_size);
+ if (offsets->size() == part_size) {
+ ColumnPtr ptr;
+ object_util::cast_column({part, from_type, ""}, to_type, &ptr);
+ part = ptr;
+ } else {
+ auto values = part->index(*offsets, offsets->size());
+ object_util::cast_column({values, from_type, ""}, to_type, &values);
+ part = values->create_with_offsets(offsets_data, to_type->get_default(), part_size,
+ /*shift=*/0);
+ }
+ }
+ result_column->insert_range_from(*part, 0, part_size);
+ }
+ data = {std::move(result_column)};
+ num_of_defaults_in_prefix = 0;
+}
+
+void ColumnObject::Subcolumn::insertDefault() {
+ if (data.empty()) {
+ ++num_of_defaults_in_prefix;
+ } else {
+ data.back()->insert_default();
+ }
+}
+
+void ColumnObject::Subcolumn::insertManyDefaults(size_t length) {
+ if (data.empty()) {
+ num_of_defaults_in_prefix += length;
+ } else {
+ data.back()->insert_many_defaults(length);
+ }
+}
+
+void ColumnObject::Subcolumn::pop_back(size_t n) {
+ assert(n <= size());
+ size_t num_removed = 0;
+ for (auto it = data.rbegin(); it != data.rend(); ++it) {
+ if (n == 0) break;
+ auto& column = *it;
+ if (n < column->size()) {
+ column->pop_back(n);
+ n = 0;
+ } else {
+ ++num_removed;
+ n -= column->size();
+ }
+ }
+ data.resize(data.size() - num_removed);
+ num_of_defaults_in_prefix -= n;
+}
+
+Field ColumnObject::Subcolumn::getLastField() const {
+ if (data.empty()) return Field();
+ const auto& last_part = data.back();
+ assert(!last_part->empty());
+ return (*last_part)[last_part->size() - 1];
+}
+
+ColumnObject::Subcolumn ColumnObject::Subcolumn::recreate_with_default_values(
+ const FieldInfo& field_info) const {
+ auto scalar_type = field_info.scalar_type;
+ if (is_nullable) {
+ scalar_type = make_nullable(scalar_type);
+ }
+ Subcolumn new_subcolumn;
+ new_subcolumn.least_common_type =
+ LeastCommonType {create_array_of_type(scalar_type, field_info.num_dimensions)};
+ new_subcolumn.is_nullable = is_nullable;
+ new_subcolumn.num_of_defaults_in_prefix = num_of_defaults_in_prefix;
+ new_subcolumn.data.reserve(data.size());
+ for (const auto& part : data) {
+ new_subcolumn.data.push_back(
+ recreate_column_with_default_value(part, scalar_type, field_info.num_dimensions));
+ }
+ return new_subcolumn;
+}
+
+IColumn& ColumnObject::Subcolumn::get_finalized_column() {
+ assert(is_finalized());
+ return *data[0];
+}
+
+const IColumn& ColumnObject::Subcolumn::get_finalized_column() const {
+ assert(is_finalized());
+ return *data[0];
+}
+
+const ColumnPtr& ColumnObject::Subcolumn::get_finalized_column_ptr() const {
+ assert(is_finalized());
+ return data[0];
+}
+
+ColumnObject::Subcolumn::LeastCommonType::LeastCommonType(DataTypePtr type_)
+ : type(std::move(type_)),
+ base_type(getBaseTypeOfArray(type)),
+ num_dimensions(getNumberOfDimensions(*type)) {}
+
+ColumnObject::ColumnObject(bool is_nullable_) : is_nullable(is_nullable_), num_rows(0) {}
+
+ColumnObject::ColumnObject(Subcolumns&& subcolumns_, bool is_nullable_)
+ : is_nullable(is_nullable_),
+ subcolumns(std::move(subcolumns_)),
+ num_rows(subcolumns.empty() ? 0 : (*subcolumns.begin())->data.size()) {
+ check_consistency();
+}
+
+void ColumnObject::check_consistency() const {
+ if (subcolumns.empty()) return;
+ for (const auto& leaf : subcolumns) {
+ if (num_rows != leaf->data.size()) {
+ assert(false);
+ }
+ }
+}
+
+size_t ColumnObject::size() const {
+#ifndef NDEBUG
+ check_consistency();
+#endif
+ return num_rows;
+}
+
+MutableColumnPtr ColumnObject::clone_resized(size_t new_size) const {
+ /// cloneResized with new_size == 0 is used for cloneEmpty().
+ if (new_size != 0) {
+ LOG(FATAL) << "ColumnObject doesn't support resize to non-zero length";
+ }
+ return ColumnObject::create(is_nullable);
+}
+
+size_t ColumnObject::byte_size() const {
+ size_t res = 0;
+ for (const auto& entry : subcolumns) {
+ res += entry->data.byteSize();
+ }
+ return res;
+}
+
+size_t ColumnObject::allocated_bytes() const {
+ size_t res = 0;
+ for (const auto& entry : subcolumns) {
+ res += entry->data.allocatedBytes();
+ }
+ return res;
+}
+
+void ColumnObject::for_each_subcolumn(ColumnCallback callback) {
+ if (!is_finalized()) assert(false);
+ for (auto& entry : subcolumns) {
+ callback(entry->data.data.back());
+ }
+}
+
+Status ColumnObject::try_insert_from(const IColumn& src, size_t n) {
+ return try_insert(src[n]);
+}
+
+Status ColumnObject::try_insert(const Field& field) {
+ const auto& object = field.get<const Object&>();
+ phmap::flat_hash_set<StringRef, StringRefHash> inserted;
+ size_t old_size = size();
+ for (const auto& [key_str, value] : object) {
+ PathInData key(key_str);
+ inserted.insert(key_str);
+ if (!has_subcolumn(key)) {
+ bool succ = add_sub_column(key, old_size);
+ if (!succ) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to add sub column {}", key.get_path()));
+ }
+ }
+ auto* subcolumn = get_subcolumn(key);
+ if (!subcolumn) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to find sub column {}", key.get_path()));
+ }
+ RETURN_IF_ERROR(subcolumn->insert(value));
+ }
+ for (auto& entry : subcolumns) {
+ if (!inserted.contains(entry->path.get_path())) {
+ entry->data.insertDefault();
+ }
+ }
+ ++num_rows;
+ return Status::OK();
+}
+
+void ColumnObject::insert_default() {
+ for (auto& entry : subcolumns) {
+ entry->data.insertDefault();
+ }
+ ++num_rows;
+}
+
+Field ColumnObject::operator[](size_t n) const {
+ if (!is_finalized()) assert(false);
+ Object object;
+ for (const auto& entry : subcolumns) {
+ object[entry->path.get_path()] = (*entry->data.data.back())[n];
+ }
+ return object;
+}
+
+void ColumnObject::get(size_t n, Field& res) const {
+ if (!is_finalized()) assert(false);
+ auto& object = res.get<Object&>();
+ for (const auto& entry : subcolumns) {
+ auto it = object.try_emplace(entry->path.get_path()).first;
+ entry->data.data.back()->get(n, it->second);
+ }
+}
+
+Status ColumnObject::try_insert_indices_from(const IColumn& src, const int* indices_begin,
+ const int* indices_end) {
+ for (auto x = indices_begin; x != indices_end; ++x) {
+ if (*x == -1) {
+ ColumnObject::insert_default();
+ } else {
+ RETURN_IF_ERROR(ColumnObject::try_insert_from(src, *x));
+ }
+ }
+ finalize();
+ return Status::OK();
+}
+
+Status ColumnObject::try_insert_range_from(const IColumn& src, size_t start, size_t length) {
+ const auto& src_object = assert_cast<const ColumnObject&>(src);
+ if (UNLIKELY(src_object.empty())) {
+ return Status::OK();
+ }
+ for (auto& entry : subcolumns) {
+ if (src_object.has_subcolumn(entry->path)) {
+ auto* subcolumn = src_object.get_subcolumn(entry->path);
+ if (!subcolumn) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to find sub column {}", entry->path.get_path()));
+ }
+ RETURN_IF_ERROR(entry->data.insertRangeFrom(*subcolumn, start, length));
+ } else {
+ entry->data.insertManyDefaults(length);
+ }
+ }
+ for (const auto& entry : src_object.subcolumns) {
+ if (!has_subcolumn(entry->path)) {
+ bool succ = false;
+ if (entry->path.has_nested_part()) {
+ const auto& base_type = entry->data.get_least_common_typeBase();
+ FieldInfo field_info {
+ .scalar_type = base_type,
+ .have_nulls = base_type->is_nullable(),
+ .need_convert = false,
+ .num_dimensions = entry->data.get_dimensions(),
+ };
+ succ = add_nested_subcolumn(entry->path, field_info, num_rows);
+ } else {
+ succ = add_sub_column(entry->path, num_rows);
+ }
+ if (!succ) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to add column {}", entry->path.get_path()));
+ }
+ auto* subcolumn = get_subcolumn(entry->path);
+ if (!subcolumn) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to find sub column {}", entry->path.get_path()));
+ }
+ RETURN_IF_ERROR(subcolumn->insertRangeFrom(entry->data, start, length));
+ }
+ }
+ num_rows += length;
+ finalize();
+ return Status::OK();
+}
+
+void ColumnObject::pop_back(size_t length) {
+ for (auto& entry : subcolumns) entry->data.pop_back(length);
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
for (auto& entry : subcolumns) { entry->data.pop_back(length);
}
```
##########
be/src/vec/columns/column_object.cpp:
##########
@@ -0,0 +1,860 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.cpp
+// and modified by Doris
+
+#include <parallel_hashmap/phmap.h>
+#include <vec/columns/column_array.h>
+#include <vec/columns/column_nullable.h>
+#include <vec/columns/column_object.h>
+#include <vec/columns/columns_number.h>
+#include <vec/common/field_visitors.h>
+#include <vec/common/hash_table/hash_set.h>
+#include <vec/common/object_util.h>
+#include <vec/common/pod_array_fwd.h>
+#include <vec/core/field.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_nothing.h>
+#include <vec/data_types/get_least_supertype.h>
+
+#include <vec/data_types/data_type_factory.hpp>
+
+namespace doris::vectorized {
+namespace {
+
+DataTypePtr create_array_of_type(DataTypePtr type, size_t num_dimensions) {
+ for (size_t i = 0; i < num_dimensions; ++i)
+ type = std::make_shared<DataTypeArray>(std::move(type));
+ return type;
+}
+
+DataTypePtr getBaseTypeOfArray(const DataTypePtr& type) {
+ /// Get raw pointers to avoid extra copying of type pointers.
+ const DataTypeArray* last_array = nullptr;
+ const auto* current_type = type.get();
+ while (const auto* type_array = typeid_cast<const DataTypeArray*>(current_type)) {
+ current_type = type_array->get_nested_type().get();
+ last_array = type_array;
+ }
+ return last_array ? last_array->get_nested_type() : type;
+}
+
+size_t getNumberOfDimensions(const IDataType& type) {
+ if (const auto* type_array = typeid_cast<const DataTypeArray*>(&type)) {
+ return type_array->get_number_of_dimensions();
+ }
+ return 0;
+}
+
+DataTypePtr get_data_type_by_column(const IColumn& column) {
+ auto idx = column.get_data_type();
+ if (WhichDataType(idx).is_simple()) {
+ return DataTypeFactory::instance().get(String(getTypeName(idx)));
+ }
+ if (WhichDataType(idx).is_nothing()) {
+ return std::make_shared<DataTypeNothing>();
+ }
+ if (const auto* column_array = check_and_get_column<ColumnArray>(&column)) {
+ return std::make_shared<DataTypeArray>(get_data_type_by_column(column_array->get_data()));
+ }
+ if (const auto* column_nullable = check_and_get_column<ColumnNullable>(&column)) {
+ return make_nullable(get_data_type_by_column(column_nullable->get_nested_column()));
+ }
+ // TODO add more types
+ assert(false);
+ return nullptr;
+}
+
+/// Recreates column with default scalar values and keeps sizes of arrays.
+ColumnPtr recreate_column_with_default_value(const ColumnPtr& column,
+ const DataTypePtr& scalar_type,
+ size_t num_dimensions) {
+ const auto* column_array = check_and_get_column<ColumnArray>(column.get());
+ if (column_array && num_dimensions) {
+ return ColumnArray::create(
+ recreate_column_with_default_value(column_array->get_data_ptr(), scalar_type,
+ num_dimensions - 1),
+ IColumn::mutate(column_array->get_offsets_ptr()));
+ }
+ return create_array_of_type(scalar_type, num_dimensions)
+ ->create_column()
+ ->clone_resized(column->size());
+}
+
+Array create_empty_array_field(size_t num_dimensions) {
+ assert(num_dimensions != 0);
+ Array array;
+ Array* current_array = &array;
+ for (size_t i = 1; i < num_dimensions; ++i) {
+ current_array->push_back(Array());
+ current_array = ¤t_array->back().get<Array&>();
+ }
+ return array;
+}
+
+/// Replaces NULL fields to given field or empty array.
+class FieldVisitorReplaceNull : public StaticVisitor<Field> {
+public:
+ explicit FieldVisitorReplaceNull(const Field& replacement_, size_t num_dimensions_)
+ : replacement(replacement_), num_dimensions(num_dimensions_) {}
+ Field operator()(const Null&) const {
+ return num_dimensions ? create_empty_array_field(num_dimensions) : replacement;
+ }
+ Field operator()(const Array& x) const {
+ assert(num_dimensions > 0);
+ const size_t size = x.size();
+ Array res(size);
+ for (size_t i = 0; i < size; ++i) {
+ res[i] = apply_visitor(FieldVisitorReplaceNull(replacement, num_dimensions - 1), x[i]);
+ }
+ return res;
+ }
+ template <typename T>
+ Field operator()(const T& x) const {
+ return x;
+ }
+
+private:
+ const Field& replacement;
+ size_t num_dimensions;
+};
+
+/// Calculates number of dimensions in array field.
+/// Returns 0 for scalar fields.
+class FieldVisitorToNumberOfDimensions : public StaticVisitor<size_t> {
+public:
+ explicit FieldVisitorToNumberOfDimensions(Status* st) : _st(st) {}
+ size_t operator()(const Array& x) const {
+ const size_t size = x.size();
+ std::optional<size_t> dimensions;
+ for (size_t i = 0; i < size; ++i) {
+ /// Do not count Nulls, because they will be replaced by default
+ /// values with proper number of dimensions.
+ if (x[i].is_null()) continue;
+ size_t current_dimensions = apply_visitor(*this, x[i]);
+ if (!dimensions) {
+ dimensions = current_dimensions;
+ } else if (current_dimensions != *dimensions) {
+ *_st = Status::InvalidArgument(
+ "Number of dimensions mismatched among array elements");
+ return 0;
+ }
+ }
+ return 1 + dimensions.value_or(0);
+ }
+ template <typename T>
+ size_t operator()(const T&) const {
+ return 0;
+ }
+
+private:
+ mutable Status* _st;
+};
+
+/// Visitor that allows to get type of scalar field
+/// or least common type of scalars in array.
+/// More optimized version of FieldToDataType.
+class FieldVisitorToScalarType : public StaticVisitor<size_t> {
+public:
+ using FieldType = Field::Types::Which;
+ size_t operator()(const Array& x) {
+ size_t size = x.size();
+ for (size_t i = 0; i < size; ++i) apply_visitor(*this, x[i]);
+ return 0;
+ }
+ // TODO doris not support unsigned integers for now
+ // treat as signed integers
+ size_t operator()(const UInt64& x) {
+ field_types.insert(FieldType::UInt64);
+ if (x <= std::numeric_limits<Int8>::max()) {
+ type_indexes.insert(TypeIndex::Int8);
+ } else if (x <= std::numeric_limits<Int16>::max()) {
+ type_indexes.insert(TypeIndex::Int16);
+ } else if (x <= std::numeric_limits<Int32>::max()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Int64& x) {
+ // Only support Int32 and Int64
+ field_types.insert(FieldType::Int64);
+ if (x <= std::numeric_limits<Int32>::max() && x >= std::numeric_limits<Int32>::min()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Null&) {
+ have_nulls = true;
+ return 0;
+ }
+ size_t operator()(const Int128I& x) { LOG(FATAL) << "not implemented"; }
+ template <typename T>
+ size_t operator()(const T&) {
+ Field::EnumToType<Field::Types::Array>::Type a;
+ field_types.insert(Field::TypeToEnum<NearestFieldType<T>>::value);
+ type_indexes.insert(TypeId<NearestFieldType<T>>::value);
+ return 0;
+ }
+ Status get_scalar_type(DataTypePtr* type) const {
+ return get_least_supertype(type_indexes, type, true /*compatible with string type*/);
+ }
+ bool contain_nulls() const { return have_nulls; }
+ bool need_convert_field() const { return field_types.size() > 1; }
+
+private:
+ phmap::flat_hash_set<TypeIndex> type_indexes;
+ phmap::flat_hash_set<FieldType> field_types;
+ bool have_nulls = false;
+};
+
+} // namespace
+Status get_field_info(const Field& field, FieldInfo* info) {
+ FieldVisitorToScalarType to_scalar_type_visitor;
+ apply_visitor(to_scalar_type_visitor, field);
+ DataTypePtr type = nullptr;
+ RETURN_IF_ERROR(to_scalar_type_visitor.get_scalar_type(&type));
+ // array item's dimension may missmatch, eg. [1, 2, [1, 2, 3]]
+ Status num_to_dimensions_status;
+ *info = {
+ type,
+ to_scalar_type_visitor.contain_nulls(),
+ to_scalar_type_visitor.need_convert_field(),
+ apply_visitor(FieldVisitorToNumberOfDimensions(&num_to_dimensions_status), field),
+ };
+ RETURN_IF_ERROR(num_to_dimensions_status);
+ return Status::OK();
+}
+
+ColumnObject::Subcolumn::Subcolumn(MutableColumnPtr&& data_, bool is_nullable_)
+ : least_common_type(get_data_type_by_column(*data_)), is_nullable(is_nullable_) {
+ data.push_back(std::move(data_));
+}
+
+ColumnObject::Subcolumn::Subcolumn(size_t size_, bool is_nullable_)
+ : least_common_type(std::make_shared<DataTypeNothing>()),
+ is_nullable(is_nullable_),
+ num_of_defaults_in_prefix(size_) {}
+
+size_t ColumnObject::Subcolumn::Subcolumn::size() const {
+ size_t res = num_of_defaults_in_prefix;
+ for (const auto& part : data) {
+ res += part->size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::byteSize() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->byte_size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::allocatedBytes() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->allocated_bytes();
+ }
+ return res;
+}
+
+Status ColumnObject::Subcolumn::insert(Field field) {
+ FieldInfo info;
+ RETURN_IF_ERROR(get_field_info(field, &info));
+ return insert(std::move(field), std::move(info));
+}
+
+void ColumnObject::Subcolumn::add_new_column_part(DataTypePtr type) {
+ data.push_back(type->create_column());
+ least_common_type = LeastCommonType {std::move(type)};
+}
+
+Status ColumnObject::Subcolumn::insert(Field field, FieldInfo info) {
+ auto base_type = std::move(info.scalar_type);
+ if (is_nothing(base_type)) {
+ insertDefault();
+ return Status::OK();
+ }
+ auto column_dim = least_common_type.get_dimensions();
+ auto value_dim = info.num_dimensions;
+ if (is_nothing(least_common_type.getBase())) {
+ column_dim = value_dim;
+ }
+ if (is_nothing(base_type)) {
+ value_dim = column_dim;
+ }
+ if (value_dim != column_dim) {
+ return Status::InvalidArgument(
+ "Dimension of types mismatched between inserted value and column.");
+ }
+ if (is_nullable && !is_nothing(base_type)) {
+ base_type = make_nullable(base_type);
+ }
+ // alawys nullable at present
+ if (!is_nullable && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+ // need replace muli dimensions array which contains null. eg. [[1, 2, 3], null] -> [[1, 2, 3], []]
+ // since column array doesnt known null's dimension
+ if (info.num_dimensions >= 2 && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+
+ bool type_changed = false;
+ const auto& least_common_base_type = least_common_type.getBase();
+ if (data.empty()) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ } else if (!least_common_base_type->equals(*base_type) && !is_nothing(base_type)) {
+ if (!object_util::is_conversion_required_between_integers(*base_type,
+ *least_common_base_type)) {
+ RETURN_IF_ERROR(
+ get_least_supertype(DataTypes {std::move(base_type), least_common_base_type},
+ &base_type, true /*compatible with string type*/));
+ type_changed = true;
+ if (!least_common_base_type->equals(*base_type)) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ }
+ }
+ }
+
+ if (type_changed || info.need_convert) {
+ Field new_field;
+ RETURN_IF_ERROR(convert_field_to_type(field, *least_common_type.get(), &new_field));
+ field = new_field;
+ }
+
+ data.back()->insert(field);
+ return Status::OK();
+}
+
+Status ColumnObject::Subcolumn::insertRangeFrom(const Subcolumn& src, size_t start, size_t length) {
+ assert(src.is_finalized());
+ const auto& src_column = src.data.back();
+ const auto& src_type = src.least_common_type.get();
+ if (data.empty()) {
+ add_new_column_part(src.least_common_type.get());
+ data.back()->insert_range_from(*src_column, start, length);
+ } else if (least_common_type.get()->equals(*src_type)) {
+ data.back()->insert_range_from(*src_column, start, length);
+ } else {
+ DataTypePtr new_least_common_type = nullptr;
+ RETURN_IF_ERROR(get_least_supertype(DataTypes {least_common_type.get(), src_type},
+ &new_least_common_type,
+ true /*compatible with string type*/));
+ ColumnPtr casted_column;
+ RETURN_IF_ERROR(object_util::cast_column({src_column, src_type, ""}, new_least_common_type,
+ &casted_column));
+ if (!least_common_type.get()->equals(*new_least_common_type))
+ add_new_column_part(std::move(new_least_common_type));
+ data.back()->insert_range_from(*casted_column, start, length);
+ }
+ return Status::OK();
+}
+
+bool ColumnObject::Subcolumn::is_finalized() const {
+ return data.empty() || (data.size() == 1 && num_of_defaults_in_prefix == 0);
+}
+
+template <typename Func>
+ColumnPtr ColumnObject::apply_for_subcolumns(Func&& func, std::string_view func_name) const {
+ if (!is_finalized()) {
+ LOG(FATAL) << "Cannot " << func_name << " non-finalized ColumnObject";
+ }
+ auto res = ColumnObject::create(is_nullable);
+ for (const auto& subcolumn : subcolumns) {
+ auto new_subcolumn = func(subcolumn->data.get_finalized_column());
+ res->add_sub_column(subcolumn->path, new_subcolumn->assume_mutable());
+ }
+ return res;
+}
+ColumnPtr ColumnObject::index(const IColumn& indexes, size_t limit) const {
+ return apply_for_subcolumns(
+ [&](const auto& subcolumn) { return subcolumn.index(indexes, limit); }, "index");
+}
+
+void ColumnObject::Subcolumn::finalize() {
+ if (is_finalized()) {
+ return;
+ }
+ if (data.size() == 1 && num_of_defaults_in_prefix == 0) {
+ data[0] = data[0]->convert_to_full_column_if_const();
+ return;
+ }
+ const auto& to_type = least_common_type.get();
+ auto result_column = to_type->create_column();
+ if (num_of_defaults_in_prefix) {
+ result_column->insert_many_defaults(num_of_defaults_in_prefix);
+ }
+ for (auto& part : data) {
+ part = part->convert_to_full_column_if_const();
+ auto from_type = get_data_type_by_column(*part);
+ size_t part_size = part->size();
+ if (!from_type->equals(*to_type)) {
+ auto offsets = ColumnUInt64::create();
+ auto& offsets_data = offsets->get_data();
+ /// We need to convert only non-default values and then recreate column
+ /// with default value of new type, because default values (which represents misses in data)
+ /// may be inconsistent between types (e.g "0" in UInt64 and empty string in String).
+ part->get_indices_of_non_default_rows(offsets_data, 0, part_size);
+ if (offsets->size() == part_size) {
+ ColumnPtr ptr;
+ object_util::cast_column({part, from_type, ""}, to_type, &ptr);
+ part = ptr;
+ } else {
+ auto values = part->index(*offsets, offsets->size());
+ object_util::cast_column({values, from_type, ""}, to_type, &values);
+ part = values->create_with_offsets(offsets_data, to_type->get_default(), part_size,
+ /*shift=*/0);
+ }
+ }
+ result_column->insert_range_from(*part, 0, part_size);
+ }
+ data = {std::move(result_column)};
+ num_of_defaults_in_prefix = 0;
+}
+
+void ColumnObject::Subcolumn::insertDefault() {
+ if (data.empty()) {
+ ++num_of_defaults_in_prefix;
+ } else {
+ data.back()->insert_default();
+ }
+}
+
+void ColumnObject::Subcolumn::insertManyDefaults(size_t length) {
+ if (data.empty()) {
+ num_of_defaults_in_prefix += length;
+ } else {
+ data.back()->insert_many_defaults(length);
+ }
+}
+
+void ColumnObject::Subcolumn::pop_back(size_t n) {
+ assert(n <= size());
+ size_t num_removed = 0;
+ for (auto it = data.rbegin(); it != data.rend(); ++it) {
+ if (n == 0) break;
+ auto& column = *it;
+ if (n < column->size()) {
+ column->pop_back(n);
+ n = 0;
+ } else {
+ ++num_removed;
+ n -= column->size();
+ }
+ }
+ data.resize(data.size() - num_removed);
+ num_of_defaults_in_prefix -= n;
+}
+
+Field ColumnObject::Subcolumn::getLastField() const {
+ if (data.empty()) return Field();
+ const auto& last_part = data.back();
+ assert(!last_part->empty());
+ return (*last_part)[last_part->size() - 1];
+}
+
+ColumnObject::Subcolumn ColumnObject::Subcolumn::recreate_with_default_values(
+ const FieldInfo& field_info) const {
+ auto scalar_type = field_info.scalar_type;
+ if (is_nullable) {
+ scalar_type = make_nullable(scalar_type);
+ }
+ Subcolumn new_subcolumn;
+ new_subcolumn.least_common_type =
+ LeastCommonType {create_array_of_type(scalar_type, field_info.num_dimensions)};
+ new_subcolumn.is_nullable = is_nullable;
+ new_subcolumn.num_of_defaults_in_prefix = num_of_defaults_in_prefix;
+ new_subcolumn.data.reserve(data.size());
+ for (const auto& part : data) {
+ new_subcolumn.data.push_back(
+ recreate_column_with_default_value(part, scalar_type, field_info.num_dimensions));
+ }
+ return new_subcolumn;
+}
+
+IColumn& ColumnObject::Subcolumn::get_finalized_column() {
+ assert(is_finalized());
+ return *data[0];
+}
+
+const IColumn& ColumnObject::Subcolumn::get_finalized_column() const {
+ assert(is_finalized());
+ return *data[0];
+}
+
+const ColumnPtr& ColumnObject::Subcolumn::get_finalized_column_ptr() const {
+ assert(is_finalized());
+ return data[0];
+}
+
+ColumnObject::Subcolumn::LeastCommonType::LeastCommonType(DataTypePtr type_)
+ : type(std::move(type_)),
+ base_type(getBaseTypeOfArray(type)),
+ num_dimensions(getNumberOfDimensions(*type)) {}
+
+ColumnObject::ColumnObject(bool is_nullable_) : is_nullable(is_nullable_), num_rows(0) {}
+
+ColumnObject::ColumnObject(Subcolumns&& subcolumns_, bool is_nullable_)
+ : is_nullable(is_nullable_),
+ subcolumns(std::move(subcolumns_)),
+ num_rows(subcolumns.empty() ? 0 : (*subcolumns.begin())->data.size()) {
+ check_consistency();
+}
+
+void ColumnObject::check_consistency() const {
+ if (subcolumns.empty()) return;
+ for (const auto& leaf : subcolumns) {
+ if (num_rows != leaf->data.size()) {
+ assert(false);
+ }
+ }
+}
+
+size_t ColumnObject::size() const {
+#ifndef NDEBUG
+ check_consistency();
+#endif
+ return num_rows;
+}
+
+MutableColumnPtr ColumnObject::clone_resized(size_t new_size) const {
+ /// cloneResized with new_size == 0 is used for cloneEmpty().
+ if (new_size != 0) {
+ LOG(FATAL) << "ColumnObject doesn't support resize to non-zero length";
+ }
+ return ColumnObject::create(is_nullable);
+}
+
+size_t ColumnObject::byte_size() const {
+ size_t res = 0;
+ for (const auto& entry : subcolumns) {
+ res += entry->data.byteSize();
+ }
+ return res;
+}
+
+size_t ColumnObject::allocated_bytes() const {
+ size_t res = 0;
+ for (const auto& entry : subcolumns) {
+ res += entry->data.allocatedBytes();
+ }
+ return res;
+}
+
+void ColumnObject::for_each_subcolumn(ColumnCallback callback) {
+ if (!is_finalized()) assert(false);
+ for (auto& entry : subcolumns) {
+ callback(entry->data.data.back());
+ }
+}
+
+Status ColumnObject::try_insert_from(const IColumn& src, size_t n) {
+ return try_insert(src[n]);
+}
+
+Status ColumnObject::try_insert(const Field& field) {
+ const auto& object = field.get<const Object&>();
+ phmap::flat_hash_set<StringRef, StringRefHash> inserted;
+ size_t old_size = size();
+ for (const auto& [key_str, value] : object) {
+ PathInData key(key_str);
+ inserted.insert(key_str);
+ if (!has_subcolumn(key)) {
+ bool succ = add_sub_column(key, old_size);
+ if (!succ) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to add sub column {}", key.get_path()));
+ }
+ }
+ auto* subcolumn = get_subcolumn(key);
+ if (!subcolumn) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to find sub column {}", key.get_path()));
+ }
+ RETURN_IF_ERROR(subcolumn->insert(value));
+ }
+ for (auto& entry : subcolumns) {
+ if (!inserted.contains(entry->path.get_path())) {
+ entry->data.insertDefault();
+ }
+ }
+ ++num_rows;
+ return Status::OK();
+}
+
+void ColumnObject::insert_default() {
+ for (auto& entry : subcolumns) {
+ entry->data.insertDefault();
+ }
+ ++num_rows;
+}
+
+Field ColumnObject::operator[](size_t n) const {
+ if (!is_finalized()) assert(false);
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!is_finalized()) { assert(false);
}
```
##########
be/src/vec/columns/column_object.cpp:
##########
@@ -0,0 +1,860 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.cpp
+// and modified by Doris
+
+#include <parallel_hashmap/phmap.h>
+#include <vec/columns/column_array.h>
+#include <vec/columns/column_nullable.h>
+#include <vec/columns/column_object.h>
+#include <vec/columns/columns_number.h>
+#include <vec/common/field_visitors.h>
+#include <vec/common/hash_table/hash_set.h>
+#include <vec/common/object_util.h>
+#include <vec/common/pod_array_fwd.h>
+#include <vec/core/field.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_nothing.h>
+#include <vec/data_types/get_least_supertype.h>
+
+#include <vec/data_types/data_type_factory.hpp>
+
+namespace doris::vectorized {
+namespace {
+
+DataTypePtr create_array_of_type(DataTypePtr type, size_t num_dimensions) {
+ for (size_t i = 0; i < num_dimensions; ++i)
+ type = std::make_shared<DataTypeArray>(std::move(type));
+ return type;
+}
+
+DataTypePtr getBaseTypeOfArray(const DataTypePtr& type) {
+ /// Get raw pointers to avoid extra copying of type pointers.
+ const DataTypeArray* last_array = nullptr;
+ const auto* current_type = type.get();
+ while (const auto* type_array = typeid_cast<const DataTypeArray*>(current_type)) {
+ current_type = type_array->get_nested_type().get();
+ last_array = type_array;
+ }
+ return last_array ? last_array->get_nested_type() : type;
+}
+
+size_t getNumberOfDimensions(const IDataType& type) {
+ if (const auto* type_array = typeid_cast<const DataTypeArray*>(&type)) {
+ return type_array->get_number_of_dimensions();
+ }
+ return 0;
+}
+
+DataTypePtr get_data_type_by_column(const IColumn& column) {
+ auto idx = column.get_data_type();
+ if (WhichDataType(idx).is_simple()) {
+ return DataTypeFactory::instance().get(String(getTypeName(idx)));
+ }
+ if (WhichDataType(idx).is_nothing()) {
+ return std::make_shared<DataTypeNothing>();
+ }
+ if (const auto* column_array = check_and_get_column<ColumnArray>(&column)) {
+ return std::make_shared<DataTypeArray>(get_data_type_by_column(column_array->get_data()));
+ }
+ if (const auto* column_nullable = check_and_get_column<ColumnNullable>(&column)) {
+ return make_nullable(get_data_type_by_column(column_nullable->get_nested_column()));
+ }
+ // TODO add more types
+ assert(false);
+ return nullptr;
+}
+
+/// Recreates column with default scalar values and keeps sizes of arrays.
+ColumnPtr recreate_column_with_default_value(const ColumnPtr& column,
+ const DataTypePtr& scalar_type,
+ size_t num_dimensions) {
+ const auto* column_array = check_and_get_column<ColumnArray>(column.get());
+ if (column_array && num_dimensions) {
+ return ColumnArray::create(
+ recreate_column_with_default_value(column_array->get_data_ptr(), scalar_type,
+ num_dimensions - 1),
+ IColumn::mutate(column_array->get_offsets_ptr()));
+ }
+ return create_array_of_type(scalar_type, num_dimensions)
+ ->create_column()
+ ->clone_resized(column->size());
+}
+
+Array create_empty_array_field(size_t num_dimensions) {
+ assert(num_dimensions != 0);
+ Array array;
+ Array* current_array = &array;
+ for (size_t i = 1; i < num_dimensions; ++i) {
+ current_array->push_back(Array());
+ current_array = ¤t_array->back().get<Array&>();
+ }
+ return array;
+}
+
+/// Replaces NULL fields to given field or empty array.
+class FieldVisitorReplaceNull : public StaticVisitor<Field> {
+public:
+ explicit FieldVisitorReplaceNull(const Field& replacement_, size_t num_dimensions_)
+ : replacement(replacement_), num_dimensions(num_dimensions_) {}
+ Field operator()(const Null&) const {
+ return num_dimensions ? create_empty_array_field(num_dimensions) : replacement;
+ }
+ Field operator()(const Array& x) const {
+ assert(num_dimensions > 0);
+ const size_t size = x.size();
+ Array res(size);
+ for (size_t i = 0; i < size; ++i) {
+ res[i] = apply_visitor(FieldVisitorReplaceNull(replacement, num_dimensions - 1), x[i]);
+ }
+ return res;
+ }
+ template <typename T>
+ Field operator()(const T& x) const {
+ return x;
+ }
+
+private:
+ const Field& replacement;
+ size_t num_dimensions;
+};
+
+/// Calculates number of dimensions in array field.
+/// Returns 0 for scalar fields.
+class FieldVisitorToNumberOfDimensions : public StaticVisitor<size_t> {
+public:
+ explicit FieldVisitorToNumberOfDimensions(Status* st) : _st(st) {}
+ size_t operator()(const Array& x) const {
+ const size_t size = x.size();
+ std::optional<size_t> dimensions;
+ for (size_t i = 0; i < size; ++i) {
+ /// Do not count Nulls, because they will be replaced by default
+ /// values with proper number of dimensions.
+ if (x[i].is_null()) continue;
+ size_t current_dimensions = apply_visitor(*this, x[i]);
+ if (!dimensions) {
+ dimensions = current_dimensions;
+ } else if (current_dimensions != *dimensions) {
+ *_st = Status::InvalidArgument(
+ "Number of dimensions mismatched among array elements");
+ return 0;
+ }
+ }
+ return 1 + dimensions.value_or(0);
+ }
+ template <typename T>
+ size_t operator()(const T&) const {
+ return 0;
+ }
+
+private:
+ mutable Status* _st;
+};
+
+/// Visitor that allows to get type of scalar field
+/// or least common type of scalars in array.
+/// More optimized version of FieldToDataType.
+class FieldVisitorToScalarType : public StaticVisitor<size_t> {
+public:
+ using FieldType = Field::Types::Which;
+ size_t operator()(const Array& x) {
+ size_t size = x.size();
+ for (size_t i = 0; i < size; ++i) apply_visitor(*this, x[i]);
+ return 0;
+ }
+ // TODO doris not support unsigned integers for now
+ // treat as signed integers
+ size_t operator()(const UInt64& x) {
+ field_types.insert(FieldType::UInt64);
+ if (x <= std::numeric_limits<Int8>::max()) {
+ type_indexes.insert(TypeIndex::Int8);
+ } else if (x <= std::numeric_limits<Int16>::max()) {
+ type_indexes.insert(TypeIndex::Int16);
+ } else if (x <= std::numeric_limits<Int32>::max()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Int64& x) {
+ // Only support Int32 and Int64
+ field_types.insert(FieldType::Int64);
+ if (x <= std::numeric_limits<Int32>::max() && x >= std::numeric_limits<Int32>::min()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Null&) {
+ have_nulls = true;
+ return 0;
+ }
+ size_t operator()(const Int128I& x) { LOG(FATAL) << "not implemented"; }
+ template <typename T>
+ size_t operator()(const T&) {
+ Field::EnumToType<Field::Types::Array>::Type a;
+ field_types.insert(Field::TypeToEnum<NearestFieldType<T>>::value);
+ type_indexes.insert(TypeId<NearestFieldType<T>>::value);
+ return 0;
+ }
+ Status get_scalar_type(DataTypePtr* type) const {
+ return get_least_supertype(type_indexes, type, true /*compatible with string type*/);
+ }
+ bool contain_nulls() const { return have_nulls; }
+ bool need_convert_field() const { return field_types.size() > 1; }
+
+private:
+ phmap::flat_hash_set<TypeIndex> type_indexes;
+ phmap::flat_hash_set<FieldType> field_types;
+ bool have_nulls = false;
+};
+
+} // namespace
+Status get_field_info(const Field& field, FieldInfo* info) {
+ FieldVisitorToScalarType to_scalar_type_visitor;
+ apply_visitor(to_scalar_type_visitor, field);
+ DataTypePtr type = nullptr;
+ RETURN_IF_ERROR(to_scalar_type_visitor.get_scalar_type(&type));
+ // array item's dimension may missmatch, eg. [1, 2, [1, 2, 3]]
+ Status num_to_dimensions_status;
+ *info = {
+ type,
+ to_scalar_type_visitor.contain_nulls(),
+ to_scalar_type_visitor.need_convert_field(),
+ apply_visitor(FieldVisitorToNumberOfDimensions(&num_to_dimensions_status), field),
+ };
+ RETURN_IF_ERROR(num_to_dimensions_status);
+ return Status::OK();
+}
+
+ColumnObject::Subcolumn::Subcolumn(MutableColumnPtr&& data_, bool is_nullable_)
+ : least_common_type(get_data_type_by_column(*data_)), is_nullable(is_nullable_) {
+ data.push_back(std::move(data_));
+}
+
+ColumnObject::Subcolumn::Subcolumn(size_t size_, bool is_nullable_)
+ : least_common_type(std::make_shared<DataTypeNothing>()),
+ is_nullable(is_nullable_),
+ num_of_defaults_in_prefix(size_) {}
+
+size_t ColumnObject::Subcolumn::Subcolumn::size() const {
+ size_t res = num_of_defaults_in_prefix;
+ for (const auto& part : data) {
+ res += part->size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::byteSize() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->byte_size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::allocatedBytes() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->allocated_bytes();
+ }
+ return res;
+}
+
+Status ColumnObject::Subcolumn::insert(Field field) {
+ FieldInfo info;
+ RETURN_IF_ERROR(get_field_info(field, &info));
+ return insert(std::move(field), std::move(info));
+}
+
+void ColumnObject::Subcolumn::add_new_column_part(DataTypePtr type) {
+ data.push_back(type->create_column());
+ least_common_type = LeastCommonType {std::move(type)};
+}
+
+Status ColumnObject::Subcolumn::insert(Field field, FieldInfo info) {
+ auto base_type = std::move(info.scalar_type);
+ if (is_nothing(base_type)) {
+ insertDefault();
+ return Status::OK();
+ }
+ auto column_dim = least_common_type.get_dimensions();
+ auto value_dim = info.num_dimensions;
+ if (is_nothing(least_common_type.getBase())) {
+ column_dim = value_dim;
+ }
+ if (is_nothing(base_type)) {
+ value_dim = column_dim;
+ }
+ if (value_dim != column_dim) {
+ return Status::InvalidArgument(
+ "Dimension of types mismatched between inserted value and column.");
+ }
+ if (is_nullable && !is_nothing(base_type)) {
+ base_type = make_nullable(base_type);
+ }
+ // alawys nullable at present
+ if (!is_nullable && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+ // need replace muli dimensions array which contains null. eg. [[1, 2, 3], null] -> [[1, 2, 3], []]
+ // since column array doesnt known null's dimension
+ if (info.num_dimensions >= 2 && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+
+ bool type_changed = false;
+ const auto& least_common_base_type = least_common_type.getBase();
+ if (data.empty()) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ } else if (!least_common_base_type->equals(*base_type) && !is_nothing(base_type)) {
+ if (!object_util::is_conversion_required_between_integers(*base_type,
+ *least_common_base_type)) {
+ RETURN_IF_ERROR(
+ get_least_supertype(DataTypes {std::move(base_type), least_common_base_type},
+ &base_type, true /*compatible with string type*/));
+ type_changed = true;
+ if (!least_common_base_type->equals(*base_type)) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ }
+ }
+ }
+
+ if (type_changed || info.need_convert) {
+ Field new_field;
+ RETURN_IF_ERROR(convert_field_to_type(field, *least_common_type.get(), &new_field));
+ field = new_field;
+ }
+
+ data.back()->insert(field);
+ return Status::OK();
+}
+
+Status ColumnObject::Subcolumn::insertRangeFrom(const Subcolumn& src, size_t start, size_t length) {
+ assert(src.is_finalized());
+ const auto& src_column = src.data.back();
+ const auto& src_type = src.least_common_type.get();
+ if (data.empty()) {
+ add_new_column_part(src.least_common_type.get());
+ data.back()->insert_range_from(*src_column, start, length);
+ } else if (least_common_type.get()->equals(*src_type)) {
+ data.back()->insert_range_from(*src_column, start, length);
+ } else {
+ DataTypePtr new_least_common_type = nullptr;
+ RETURN_IF_ERROR(get_least_supertype(DataTypes {least_common_type.get(), src_type},
+ &new_least_common_type,
+ true /*compatible with string type*/));
+ ColumnPtr casted_column;
+ RETURN_IF_ERROR(object_util::cast_column({src_column, src_type, ""}, new_least_common_type,
+ &casted_column));
+ if (!least_common_type.get()->equals(*new_least_common_type))
+ add_new_column_part(std::move(new_least_common_type));
+ data.back()->insert_range_from(*casted_column, start, length);
+ }
+ return Status::OK();
+}
+
+bool ColumnObject::Subcolumn::is_finalized() const {
+ return data.empty() || (data.size() == 1 && num_of_defaults_in_prefix == 0);
+}
+
+template <typename Func>
+ColumnPtr ColumnObject::apply_for_subcolumns(Func&& func, std::string_view func_name) const {
+ if (!is_finalized()) {
+ LOG(FATAL) << "Cannot " << func_name << " non-finalized ColumnObject";
+ }
+ auto res = ColumnObject::create(is_nullable);
+ for (const auto& subcolumn : subcolumns) {
+ auto new_subcolumn = func(subcolumn->data.get_finalized_column());
+ res->add_sub_column(subcolumn->path, new_subcolumn->assume_mutable());
+ }
+ return res;
+}
+ColumnPtr ColumnObject::index(const IColumn& indexes, size_t limit) const {
+ return apply_for_subcolumns(
+ [&](const auto& subcolumn) { return subcolumn.index(indexes, limit); }, "index");
+}
+
+void ColumnObject::Subcolumn::finalize() {
+ if (is_finalized()) {
+ return;
+ }
+ if (data.size() == 1 && num_of_defaults_in_prefix == 0) {
+ data[0] = data[0]->convert_to_full_column_if_const();
+ return;
+ }
+ const auto& to_type = least_common_type.get();
+ auto result_column = to_type->create_column();
+ if (num_of_defaults_in_prefix) {
+ result_column->insert_many_defaults(num_of_defaults_in_prefix);
+ }
+ for (auto& part : data) {
+ part = part->convert_to_full_column_if_const();
+ auto from_type = get_data_type_by_column(*part);
+ size_t part_size = part->size();
+ if (!from_type->equals(*to_type)) {
+ auto offsets = ColumnUInt64::create();
+ auto& offsets_data = offsets->get_data();
+ /// We need to convert only non-default values and then recreate column
+ /// with default value of new type, because default values (which represents misses in data)
+ /// may be inconsistent between types (e.g "0" in UInt64 and empty string in String).
+ part->get_indices_of_non_default_rows(offsets_data, 0, part_size);
+ if (offsets->size() == part_size) {
+ ColumnPtr ptr;
+ object_util::cast_column({part, from_type, ""}, to_type, &ptr);
+ part = ptr;
+ } else {
+ auto values = part->index(*offsets, offsets->size());
+ object_util::cast_column({values, from_type, ""}, to_type, &values);
+ part = values->create_with_offsets(offsets_data, to_type->get_default(), part_size,
+ /*shift=*/0);
+ }
+ }
+ result_column->insert_range_from(*part, 0, part_size);
+ }
+ data = {std::move(result_column)};
+ num_of_defaults_in_prefix = 0;
+}
+
+void ColumnObject::Subcolumn::insertDefault() {
+ if (data.empty()) {
+ ++num_of_defaults_in_prefix;
+ } else {
+ data.back()->insert_default();
+ }
+}
+
+void ColumnObject::Subcolumn::insertManyDefaults(size_t length) {
+ if (data.empty()) {
+ num_of_defaults_in_prefix += length;
+ } else {
+ data.back()->insert_many_defaults(length);
+ }
+}
+
+void ColumnObject::Subcolumn::pop_back(size_t n) {
+ assert(n <= size());
+ size_t num_removed = 0;
+ for (auto it = data.rbegin(); it != data.rend(); ++it) {
+ if (n == 0) break;
+ auto& column = *it;
+ if (n < column->size()) {
+ column->pop_back(n);
+ n = 0;
+ } else {
+ ++num_removed;
+ n -= column->size();
+ }
+ }
+ data.resize(data.size() - num_removed);
+ num_of_defaults_in_prefix -= n;
+}
+
+Field ColumnObject::Subcolumn::getLastField() const {
+ if (data.empty()) return Field();
+ const auto& last_part = data.back();
+ assert(!last_part->empty());
+ return (*last_part)[last_part->size() - 1];
+}
+
+ColumnObject::Subcolumn ColumnObject::Subcolumn::recreate_with_default_values(
+ const FieldInfo& field_info) const {
+ auto scalar_type = field_info.scalar_type;
+ if (is_nullable) {
+ scalar_type = make_nullable(scalar_type);
+ }
+ Subcolumn new_subcolumn;
+ new_subcolumn.least_common_type =
+ LeastCommonType {create_array_of_type(scalar_type, field_info.num_dimensions)};
+ new_subcolumn.is_nullable = is_nullable;
+ new_subcolumn.num_of_defaults_in_prefix = num_of_defaults_in_prefix;
+ new_subcolumn.data.reserve(data.size());
+ for (const auto& part : data) {
+ new_subcolumn.data.push_back(
+ recreate_column_with_default_value(part, scalar_type, field_info.num_dimensions));
+ }
+ return new_subcolumn;
+}
+
+IColumn& ColumnObject::Subcolumn::get_finalized_column() {
+ assert(is_finalized());
+ return *data[0];
+}
+
+const IColumn& ColumnObject::Subcolumn::get_finalized_column() const {
+ assert(is_finalized());
+ return *data[0];
+}
+
+const ColumnPtr& ColumnObject::Subcolumn::get_finalized_column_ptr() const {
+ assert(is_finalized());
+ return data[0];
+}
+
+ColumnObject::Subcolumn::LeastCommonType::LeastCommonType(DataTypePtr type_)
+ : type(std::move(type_)),
+ base_type(getBaseTypeOfArray(type)),
+ num_dimensions(getNumberOfDimensions(*type)) {}
+
+ColumnObject::ColumnObject(bool is_nullable_) : is_nullable(is_nullable_), num_rows(0) {}
+
+ColumnObject::ColumnObject(Subcolumns&& subcolumns_, bool is_nullable_)
+ : is_nullable(is_nullable_),
+ subcolumns(std::move(subcolumns_)),
+ num_rows(subcolumns.empty() ? 0 : (*subcolumns.begin())->data.size()) {
+ check_consistency();
+}
+
+void ColumnObject::check_consistency() const {
+ if (subcolumns.empty()) return;
+ for (const auto& leaf : subcolumns) {
+ if (num_rows != leaf->data.size()) {
+ assert(false);
+ }
+ }
+}
+
+size_t ColumnObject::size() const {
+#ifndef NDEBUG
+ check_consistency();
+#endif
+ return num_rows;
+}
+
+MutableColumnPtr ColumnObject::clone_resized(size_t new_size) const {
+ /// cloneResized with new_size == 0 is used for cloneEmpty().
+ if (new_size != 0) {
+ LOG(FATAL) << "ColumnObject doesn't support resize to non-zero length";
+ }
+ return ColumnObject::create(is_nullable);
+}
+
+size_t ColumnObject::byte_size() const {
+ size_t res = 0;
+ for (const auto& entry : subcolumns) {
+ res += entry->data.byteSize();
+ }
+ return res;
+}
+
+size_t ColumnObject::allocated_bytes() const {
+ size_t res = 0;
+ for (const auto& entry : subcolumns) {
+ res += entry->data.allocatedBytes();
+ }
+ return res;
+}
+
+void ColumnObject::for_each_subcolumn(ColumnCallback callback) {
+ if (!is_finalized()) assert(false);
+ for (auto& entry : subcolumns) {
+ callback(entry->data.data.back());
+ }
+}
+
+Status ColumnObject::try_insert_from(const IColumn& src, size_t n) {
+ return try_insert(src[n]);
+}
+
+Status ColumnObject::try_insert(const Field& field) {
+ const auto& object = field.get<const Object&>();
+ phmap::flat_hash_set<StringRef, StringRefHash> inserted;
+ size_t old_size = size();
+ for (const auto& [key_str, value] : object) {
+ PathInData key(key_str);
+ inserted.insert(key_str);
+ if (!has_subcolumn(key)) {
+ bool succ = add_sub_column(key, old_size);
+ if (!succ) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to add sub column {}", key.get_path()));
+ }
+ }
+ auto* subcolumn = get_subcolumn(key);
+ if (!subcolumn) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to find sub column {}", key.get_path()));
+ }
+ RETURN_IF_ERROR(subcolumn->insert(value));
+ }
+ for (auto& entry : subcolumns) {
+ if (!inserted.contains(entry->path.get_path())) {
+ entry->data.insertDefault();
+ }
+ }
+ ++num_rows;
+ return Status::OK();
+}
+
+void ColumnObject::insert_default() {
+ for (auto& entry : subcolumns) {
+ entry->data.insertDefault();
+ }
+ ++num_rows;
+}
+
+Field ColumnObject::operator[](size_t n) const {
+ if (!is_finalized()) assert(false);
+ Object object;
+ for (const auto& entry : subcolumns) {
+ object[entry->path.get_path()] = (*entry->data.data.back())[n];
+ }
+ return object;
+}
+
+void ColumnObject::get(size_t n, Field& res) const {
+ if (!is_finalized()) assert(false);
+ auto& object = res.get<Object&>();
+ for (const auto& entry : subcolumns) {
+ auto it = object.try_emplace(entry->path.get_path()).first;
+ entry->data.data.back()->get(n, it->second);
+ }
+}
+
+Status ColumnObject::try_insert_indices_from(const IColumn& src, const int* indices_begin,
+ const int* indices_end) {
+ for (auto x = indices_begin; x != indices_end; ++x) {
+ if (*x == -1) {
+ ColumnObject::insert_default();
+ } else {
+ RETURN_IF_ERROR(ColumnObject::try_insert_from(src, *x));
+ }
+ }
+ finalize();
+ return Status::OK();
+}
+
+Status ColumnObject::try_insert_range_from(const IColumn& src, size_t start, size_t length) {
+ const auto& src_object = assert_cast<const ColumnObject&>(src);
+ if (UNLIKELY(src_object.empty())) {
+ return Status::OK();
+ }
+ for (auto& entry : subcolumns) {
+ if (src_object.has_subcolumn(entry->path)) {
+ auto* subcolumn = src_object.get_subcolumn(entry->path);
+ if (!subcolumn) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to find sub column {}", entry->path.get_path()));
+ }
+ RETURN_IF_ERROR(entry->data.insertRangeFrom(*subcolumn, start, length));
+ } else {
+ entry->data.insertManyDefaults(length);
+ }
+ }
+ for (const auto& entry : src_object.subcolumns) {
+ if (!has_subcolumn(entry->path)) {
+ bool succ = false;
+ if (entry->path.has_nested_part()) {
+ const auto& base_type = entry->data.get_least_common_typeBase();
+ FieldInfo field_info {
+ .scalar_type = base_type,
+ .have_nulls = base_type->is_nullable(),
+ .need_convert = false,
+ .num_dimensions = entry->data.get_dimensions(),
+ };
+ succ = add_nested_subcolumn(entry->path, field_info, num_rows);
+ } else {
+ succ = add_sub_column(entry->path, num_rows);
+ }
+ if (!succ) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to add column {}", entry->path.get_path()));
+ }
+ auto* subcolumn = get_subcolumn(entry->path);
+ if (!subcolumn) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to find sub column {}", entry->path.get_path()));
+ }
+ RETURN_IF_ERROR(subcolumn->insertRangeFrom(entry->data, start, length));
+ }
+ }
+ num_rows += length;
+ finalize();
+ return Status::OK();
+}
+
+void ColumnObject::pop_back(size_t length) {
+ for (auto& entry : subcolumns) entry->data.pop_back(length);
+ num_rows -= length;
+}
+
+const ColumnObject::Subcolumn* ColumnObject::get_subcolumn(const PathInData& key) const {
+ const auto* node = subcolumns.find_leaf(key);
+ if (node == nullptr) {
+ VLOG_DEBUG << "There is no subcolumn " << key.get_path();
+ return nullptr;
+ }
+ return &node->data;
+}
+
+ColumnObject::Subcolumn* ColumnObject::get_subcolumn(const PathInData& key) {
+ const auto* node = subcolumns.find_leaf(key);
+ if (node == nullptr) {
+ VLOG_DEBUG << "There is no subcolumn " << key.get_path();
+ return nullptr;
+ }
+ return &const_cast<Subcolumns::Node*>(node)->data;
+}
+
+bool ColumnObject::has_subcolumn(const PathInData& key) const {
+ return subcolumns.find_leaf(key) != nullptr;
+}
+
+bool ColumnObject::add_sub_column(const PathInData& key, MutableColumnPtr&& subcolumn) {
+ size_t new_size = subcolumn->size();
+ bool inserted = subcolumns.add(key, Subcolumn(std::move(subcolumn), is_nullable));
+ if (!inserted) {
+ VLOG_DEBUG << "Duplicated sub column " << key.get_path();
+ return false;
+ }
+ if (num_rows == 0) {
+ num_rows = new_size;
+ } else if (new_size != num_rows) {
+ VLOG_DEBUG << "Size of subcolumn is in consistent with column";
+ return false;
+ }
+ return true;
+}
+
+bool ColumnObject::add_sub_column(const PathInData& key, size_t new_size) {
+ bool inserted = subcolumns.add(key, Subcolumn(new_size, is_nullable));
+ if (!inserted) {
+ VLOG_DEBUG << "Duplicated sub column " << key.get_path();
+ return false;
+ }
+ if (num_rows == 0) {
+ num_rows = new_size;
+ } else if (new_size != num_rows) {
+ VLOG_DEBUG << "Size of subcolumn is in consistent with column";
+ return false;
+ }
+ return true;
+}
+
+bool ColumnObject::add_nested_subcolumn(const PathInData& key, const FieldInfo& field_info,
+ size_t new_size) {
+ assert(key.has_nested_part());
+ bool inserted = false;
+ /// We find node that represents the same Nested type as @key.
+ const auto* nested_node = subcolumns.find_best_match(key);
+ if (nested_node) {
+ /// Find any leaf of Nested subcolumn.
+ const auto* leaf = subcolumns.find_leaf(nested_node, [&](const auto&) { return true; });
+ assert(leaf);
+ /// Recreate subcolumn with default values and the same sizes of arrays.
+ auto new_subcolumn = leaf->data.recreate_with_default_values(field_info);
+ /// It's possible that we have already inserted value from current row
+ /// to this subcolumn. So, adjust size to expected.
+ if (new_subcolumn.size() > new_size)
+ new_subcolumn.pop_back(new_subcolumn.size() - new_size);
+ assert(new_subcolumn.size() == new_size);
+ inserted = subcolumns.add(key, new_subcolumn);
+ } else {
+ /// If node was not found just add subcolumn with empty arrays.
+ inserted = subcolumns.add(key, Subcolumn(new_size, is_nullable));
+ }
+ if (!inserted) {
+ VLOG_DEBUG << "Subcolumn already exists";
+ return false;
+ }
+ if (num_rows == 0) num_rows = new_size;
+ return true;
+}
+
+PathsInData ColumnObject::getKeys() const {
+ PathsInData keys;
+ keys.reserve(subcolumns.size());
+ for (const auto& entry : subcolumns) {
+ keys.emplace_back(entry->path);
+ }
+ return keys;
+}
+
+void ColumnObject::remove_subcolumns(const std::unordered_set<std::string>& keys) {
+ Subcolumns new_subcolumns;
+ for (auto& entry : subcolumns) {
+ if (keys.count(entry->path.get_path()) == 0) {
+ new_subcolumns.add(entry->path, entry->data);
+ }
+ }
+ std::swap(subcolumns, new_subcolumns);
+}
+
+bool ColumnObject::is_finalized() const {
+ return std::all_of(subcolumns.begin(), subcolumns.end(),
+ [](const auto& entry) { return entry->data.is_finalized(); });
+}
+
+void ColumnObject::finalize() {
+ size_t old_size = size();
+ Subcolumns new_subcolumns;
+ for (auto&& entry : subcolumns) {
+ const auto& least_common_type = entry->data.get_least_common_type();
+ /// Do not add subcolumns, which consists only from NULLs.
+ if (is_nothing(getBaseTypeOfArray(least_common_type))) continue;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (is_nothing(getBaseTypeOfArray(least_common_type))) { continue;
}
```
##########
be/src/vec/columns/column_object.cpp:
##########
@@ -0,0 +1,860 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.cpp
+// and modified by Doris
+
+#include <parallel_hashmap/phmap.h>
+#include <vec/columns/column_array.h>
+#include <vec/columns/column_nullable.h>
+#include <vec/columns/column_object.h>
+#include <vec/columns/columns_number.h>
+#include <vec/common/field_visitors.h>
+#include <vec/common/hash_table/hash_set.h>
+#include <vec/common/object_util.h>
+#include <vec/common/pod_array_fwd.h>
+#include <vec/core/field.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_nothing.h>
+#include <vec/data_types/get_least_supertype.h>
+
+#include <vec/data_types/data_type_factory.hpp>
+
+namespace doris::vectorized {
+namespace {
+
+DataTypePtr create_array_of_type(DataTypePtr type, size_t num_dimensions) {
+ for (size_t i = 0; i < num_dimensions; ++i)
+ type = std::make_shared<DataTypeArray>(std::move(type));
+ return type;
+}
+
+DataTypePtr getBaseTypeOfArray(const DataTypePtr& type) {
+ /// Get raw pointers to avoid extra copying of type pointers.
+ const DataTypeArray* last_array = nullptr;
+ const auto* current_type = type.get();
+ while (const auto* type_array = typeid_cast<const DataTypeArray*>(current_type)) {
+ current_type = type_array->get_nested_type().get();
+ last_array = type_array;
+ }
+ return last_array ? last_array->get_nested_type() : type;
+}
+
+size_t getNumberOfDimensions(const IDataType& type) {
+ if (const auto* type_array = typeid_cast<const DataTypeArray*>(&type)) {
+ return type_array->get_number_of_dimensions();
+ }
+ return 0;
+}
+
+DataTypePtr get_data_type_by_column(const IColumn& column) {
+ auto idx = column.get_data_type();
+ if (WhichDataType(idx).is_simple()) {
+ return DataTypeFactory::instance().get(String(getTypeName(idx)));
+ }
+ if (WhichDataType(idx).is_nothing()) {
+ return std::make_shared<DataTypeNothing>();
+ }
+ if (const auto* column_array = check_and_get_column<ColumnArray>(&column)) {
+ return std::make_shared<DataTypeArray>(get_data_type_by_column(column_array->get_data()));
+ }
+ if (const auto* column_nullable = check_and_get_column<ColumnNullable>(&column)) {
+ return make_nullable(get_data_type_by_column(column_nullable->get_nested_column()));
+ }
+ // TODO add more types
+ assert(false);
+ return nullptr;
+}
+
+/// Recreates column with default scalar values and keeps sizes of arrays.
+ColumnPtr recreate_column_with_default_value(const ColumnPtr& column,
+ const DataTypePtr& scalar_type,
+ size_t num_dimensions) {
+ const auto* column_array = check_and_get_column<ColumnArray>(column.get());
+ if (column_array && num_dimensions) {
+ return ColumnArray::create(
+ recreate_column_with_default_value(column_array->get_data_ptr(), scalar_type,
+ num_dimensions - 1),
+ IColumn::mutate(column_array->get_offsets_ptr()));
+ }
+ return create_array_of_type(scalar_type, num_dimensions)
+ ->create_column()
+ ->clone_resized(column->size());
+}
+
+Array create_empty_array_field(size_t num_dimensions) {
+ assert(num_dimensions != 0);
+ Array array;
+ Array* current_array = &array;
+ for (size_t i = 1; i < num_dimensions; ++i) {
+ current_array->push_back(Array());
+ current_array = ¤t_array->back().get<Array&>();
+ }
+ return array;
+}
+
+/// Replaces NULL fields to given field or empty array.
+class FieldVisitorReplaceNull : public StaticVisitor<Field> {
+public:
+ explicit FieldVisitorReplaceNull(const Field& replacement_, size_t num_dimensions_)
+ : replacement(replacement_), num_dimensions(num_dimensions_) {}
+ Field operator()(const Null&) const {
+ return num_dimensions ? create_empty_array_field(num_dimensions) : replacement;
+ }
+ Field operator()(const Array& x) const {
+ assert(num_dimensions > 0);
+ const size_t size = x.size();
+ Array res(size);
+ for (size_t i = 0; i < size; ++i) {
+ res[i] = apply_visitor(FieldVisitorReplaceNull(replacement, num_dimensions - 1), x[i]);
+ }
+ return res;
+ }
+ template <typename T>
+ Field operator()(const T& x) const {
+ return x;
+ }
+
+private:
+ const Field& replacement;
+ size_t num_dimensions;
+};
+
+/// Calculates number of dimensions in array field.
+/// Returns 0 for scalar fields.
+class FieldVisitorToNumberOfDimensions : public StaticVisitor<size_t> {
+public:
+ explicit FieldVisitorToNumberOfDimensions(Status* st) : _st(st) {}
+ size_t operator()(const Array& x) const {
+ const size_t size = x.size();
+ std::optional<size_t> dimensions;
+ for (size_t i = 0; i < size; ++i) {
+ /// Do not count Nulls, because they will be replaced by default
+ /// values with proper number of dimensions.
+ if (x[i].is_null()) continue;
+ size_t current_dimensions = apply_visitor(*this, x[i]);
+ if (!dimensions) {
+ dimensions = current_dimensions;
+ } else if (current_dimensions != *dimensions) {
+ *_st = Status::InvalidArgument(
+ "Number of dimensions mismatched among array elements");
+ return 0;
+ }
+ }
+ return 1 + dimensions.value_or(0);
+ }
+ template <typename T>
+ size_t operator()(const T&) const {
+ return 0;
+ }
+
+private:
+ mutable Status* _st;
+};
+
+/// Visitor that allows to get type of scalar field
+/// or least common type of scalars in array.
+/// More optimized version of FieldToDataType.
+class FieldVisitorToScalarType : public StaticVisitor<size_t> {
+public:
+ using FieldType = Field::Types::Which;
+ size_t operator()(const Array& x) {
+ size_t size = x.size();
+ for (size_t i = 0; i < size; ++i) apply_visitor(*this, x[i]);
+ return 0;
+ }
+ // TODO doris not support unsigned integers for now
+ // treat as signed integers
+ size_t operator()(const UInt64& x) {
+ field_types.insert(FieldType::UInt64);
+ if (x <= std::numeric_limits<Int8>::max()) {
+ type_indexes.insert(TypeIndex::Int8);
+ } else if (x <= std::numeric_limits<Int16>::max()) {
+ type_indexes.insert(TypeIndex::Int16);
+ } else if (x <= std::numeric_limits<Int32>::max()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Int64& x) {
+ // Only support Int32 and Int64
+ field_types.insert(FieldType::Int64);
+ if (x <= std::numeric_limits<Int32>::max() && x >= std::numeric_limits<Int32>::min()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Null&) {
+ have_nulls = true;
+ return 0;
+ }
+ size_t operator()(const Int128I& x) { LOG(FATAL) << "not implemented"; }
+ template <typename T>
+ size_t operator()(const T&) {
+ Field::EnumToType<Field::Types::Array>::Type a;
+ field_types.insert(Field::TypeToEnum<NearestFieldType<T>>::value);
+ type_indexes.insert(TypeId<NearestFieldType<T>>::value);
+ return 0;
+ }
+ Status get_scalar_type(DataTypePtr* type) const {
+ return get_least_supertype(type_indexes, type, true /*compatible with string type*/);
+ }
+ bool contain_nulls() const { return have_nulls; }
+ bool need_convert_field() const { return field_types.size() > 1; }
+
+private:
+ phmap::flat_hash_set<TypeIndex> type_indexes;
+ phmap::flat_hash_set<FieldType> field_types;
+ bool have_nulls = false;
+};
+
+} // namespace
+Status get_field_info(const Field& field, FieldInfo* info) {
+ FieldVisitorToScalarType to_scalar_type_visitor;
+ apply_visitor(to_scalar_type_visitor, field);
+ DataTypePtr type = nullptr;
+ RETURN_IF_ERROR(to_scalar_type_visitor.get_scalar_type(&type));
+ // array item's dimension may missmatch, eg. [1, 2, [1, 2, 3]]
+ Status num_to_dimensions_status;
+ *info = {
+ type,
+ to_scalar_type_visitor.contain_nulls(),
+ to_scalar_type_visitor.need_convert_field(),
+ apply_visitor(FieldVisitorToNumberOfDimensions(&num_to_dimensions_status), field),
+ };
+ RETURN_IF_ERROR(num_to_dimensions_status);
+ return Status::OK();
+}
+
+ColumnObject::Subcolumn::Subcolumn(MutableColumnPtr&& data_, bool is_nullable_)
+ : least_common_type(get_data_type_by_column(*data_)), is_nullable(is_nullable_) {
+ data.push_back(std::move(data_));
+}
+
+ColumnObject::Subcolumn::Subcolumn(size_t size_, bool is_nullable_)
+ : least_common_type(std::make_shared<DataTypeNothing>()),
+ is_nullable(is_nullable_),
+ num_of_defaults_in_prefix(size_) {}
+
+size_t ColumnObject::Subcolumn::Subcolumn::size() const {
+ size_t res = num_of_defaults_in_prefix;
+ for (const auto& part : data) {
+ res += part->size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::byteSize() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->byte_size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::allocatedBytes() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->allocated_bytes();
+ }
+ return res;
+}
+
+Status ColumnObject::Subcolumn::insert(Field field) {
+ FieldInfo info;
+ RETURN_IF_ERROR(get_field_info(field, &info));
+ return insert(std::move(field), std::move(info));
+}
+
+void ColumnObject::Subcolumn::add_new_column_part(DataTypePtr type) {
+ data.push_back(type->create_column());
+ least_common_type = LeastCommonType {std::move(type)};
+}
+
+Status ColumnObject::Subcolumn::insert(Field field, FieldInfo info) {
+ auto base_type = std::move(info.scalar_type);
+ if (is_nothing(base_type)) {
+ insertDefault();
+ return Status::OK();
+ }
+ auto column_dim = least_common_type.get_dimensions();
+ auto value_dim = info.num_dimensions;
+ if (is_nothing(least_common_type.getBase())) {
+ column_dim = value_dim;
+ }
+ if (is_nothing(base_type)) {
+ value_dim = column_dim;
+ }
+ if (value_dim != column_dim) {
+ return Status::InvalidArgument(
+ "Dimension of types mismatched between inserted value and column.");
+ }
+ if (is_nullable && !is_nothing(base_type)) {
+ base_type = make_nullable(base_type);
+ }
+ // alawys nullable at present
+ if (!is_nullable && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+ // need replace muli dimensions array which contains null. eg. [[1, 2, 3], null] -> [[1, 2, 3], []]
+ // since column array doesnt known null's dimension
+ if (info.num_dimensions >= 2 && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+
+ bool type_changed = false;
+ const auto& least_common_base_type = least_common_type.getBase();
+ if (data.empty()) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ } else if (!least_common_base_type->equals(*base_type) && !is_nothing(base_type)) {
+ if (!object_util::is_conversion_required_between_integers(*base_type,
+ *least_common_base_type)) {
+ RETURN_IF_ERROR(
+ get_least_supertype(DataTypes {std::move(base_type), least_common_base_type},
+ &base_type, true /*compatible with string type*/));
+ type_changed = true;
+ if (!least_common_base_type->equals(*base_type)) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ }
+ }
+ }
+
+ if (type_changed || info.need_convert) {
+ Field new_field;
+ RETURN_IF_ERROR(convert_field_to_type(field, *least_common_type.get(), &new_field));
+ field = new_field;
+ }
+
+ data.back()->insert(field);
+ return Status::OK();
+}
+
+Status ColumnObject::Subcolumn::insertRangeFrom(const Subcolumn& src, size_t start, size_t length) {
+ assert(src.is_finalized());
+ const auto& src_column = src.data.back();
+ const auto& src_type = src.least_common_type.get();
+ if (data.empty()) {
+ add_new_column_part(src.least_common_type.get());
+ data.back()->insert_range_from(*src_column, start, length);
+ } else if (least_common_type.get()->equals(*src_type)) {
+ data.back()->insert_range_from(*src_column, start, length);
+ } else {
+ DataTypePtr new_least_common_type = nullptr;
+ RETURN_IF_ERROR(get_least_supertype(DataTypes {least_common_type.get(), src_type},
+ &new_least_common_type,
+ true /*compatible with string type*/));
+ ColumnPtr casted_column;
+ RETURN_IF_ERROR(object_util::cast_column({src_column, src_type, ""}, new_least_common_type,
+ &casted_column));
+ if (!least_common_type.get()->equals(*new_least_common_type))
+ add_new_column_part(std::move(new_least_common_type));
+ data.back()->insert_range_from(*casted_column, start, length);
+ }
+ return Status::OK();
+}
+
+bool ColumnObject::Subcolumn::is_finalized() const {
+ return data.empty() || (data.size() == 1 && num_of_defaults_in_prefix == 0);
+}
+
+template <typename Func>
+ColumnPtr ColumnObject::apply_for_subcolumns(Func&& func, std::string_view func_name) const {
+ if (!is_finalized()) {
+ LOG(FATAL) << "Cannot " << func_name << " non-finalized ColumnObject";
+ }
+ auto res = ColumnObject::create(is_nullable);
+ for (const auto& subcolumn : subcolumns) {
+ auto new_subcolumn = func(subcolumn->data.get_finalized_column());
+ res->add_sub_column(subcolumn->path, new_subcolumn->assume_mutable());
+ }
+ return res;
+}
+ColumnPtr ColumnObject::index(const IColumn& indexes, size_t limit) const {
+ return apply_for_subcolumns(
+ [&](const auto& subcolumn) { return subcolumn.index(indexes, limit); }, "index");
+}
+
+void ColumnObject::Subcolumn::finalize() {
+ if (is_finalized()) {
+ return;
+ }
+ if (data.size() == 1 && num_of_defaults_in_prefix == 0) {
+ data[0] = data[0]->convert_to_full_column_if_const();
+ return;
+ }
+ const auto& to_type = least_common_type.get();
+ auto result_column = to_type->create_column();
+ if (num_of_defaults_in_prefix) {
+ result_column->insert_many_defaults(num_of_defaults_in_prefix);
+ }
+ for (auto& part : data) {
+ part = part->convert_to_full_column_if_const();
+ auto from_type = get_data_type_by_column(*part);
+ size_t part_size = part->size();
+ if (!from_type->equals(*to_type)) {
+ auto offsets = ColumnUInt64::create();
+ auto& offsets_data = offsets->get_data();
+ /// We need to convert only non-default values and then recreate column
+ /// with default value of new type, because default values (which represents misses in data)
+ /// may be inconsistent between types (e.g "0" in UInt64 and empty string in String).
+ part->get_indices_of_non_default_rows(offsets_data, 0, part_size);
+ if (offsets->size() == part_size) {
+ ColumnPtr ptr;
+ object_util::cast_column({part, from_type, ""}, to_type, &ptr);
+ part = ptr;
+ } else {
+ auto values = part->index(*offsets, offsets->size());
+ object_util::cast_column({values, from_type, ""}, to_type, &values);
+ part = values->create_with_offsets(offsets_data, to_type->get_default(), part_size,
+ /*shift=*/0);
+ }
+ }
+ result_column->insert_range_from(*part, 0, part_size);
+ }
+ data = {std::move(result_column)};
+ num_of_defaults_in_prefix = 0;
+}
+
+void ColumnObject::Subcolumn::insertDefault() {
+ if (data.empty()) {
+ ++num_of_defaults_in_prefix;
+ } else {
+ data.back()->insert_default();
+ }
+}
+
+void ColumnObject::Subcolumn::insertManyDefaults(size_t length) {
+ if (data.empty()) {
+ num_of_defaults_in_prefix += length;
+ } else {
+ data.back()->insert_many_defaults(length);
+ }
+}
+
+void ColumnObject::Subcolumn::pop_back(size_t n) {
+ assert(n <= size());
+ size_t num_removed = 0;
+ for (auto it = data.rbegin(); it != data.rend(); ++it) {
+ if (n == 0) break;
+ auto& column = *it;
+ if (n < column->size()) {
+ column->pop_back(n);
+ n = 0;
+ } else {
+ ++num_removed;
+ n -= column->size();
+ }
+ }
+ data.resize(data.size() - num_removed);
+ num_of_defaults_in_prefix -= n;
+}
+
+Field ColumnObject::Subcolumn::getLastField() const {
+ if (data.empty()) return Field();
+ const auto& last_part = data.back();
+ assert(!last_part->empty());
+ return (*last_part)[last_part->size() - 1];
+}
+
+ColumnObject::Subcolumn ColumnObject::Subcolumn::recreate_with_default_values(
+ const FieldInfo& field_info) const {
+ auto scalar_type = field_info.scalar_type;
+ if (is_nullable) {
+ scalar_type = make_nullable(scalar_type);
+ }
+ Subcolumn new_subcolumn;
+ new_subcolumn.least_common_type =
+ LeastCommonType {create_array_of_type(scalar_type, field_info.num_dimensions)};
+ new_subcolumn.is_nullable = is_nullable;
+ new_subcolumn.num_of_defaults_in_prefix = num_of_defaults_in_prefix;
+ new_subcolumn.data.reserve(data.size());
+ for (const auto& part : data) {
+ new_subcolumn.data.push_back(
+ recreate_column_with_default_value(part, scalar_type, field_info.num_dimensions));
+ }
+ return new_subcolumn;
+}
+
+IColumn& ColumnObject::Subcolumn::get_finalized_column() {
+ assert(is_finalized());
+ return *data[0];
+}
+
+const IColumn& ColumnObject::Subcolumn::get_finalized_column() const {
+ assert(is_finalized());
+ return *data[0];
+}
+
+const ColumnPtr& ColumnObject::Subcolumn::get_finalized_column_ptr() const {
+ assert(is_finalized());
+ return data[0];
+}
+
+ColumnObject::Subcolumn::LeastCommonType::LeastCommonType(DataTypePtr type_)
+ : type(std::move(type_)),
+ base_type(getBaseTypeOfArray(type)),
+ num_dimensions(getNumberOfDimensions(*type)) {}
+
+ColumnObject::ColumnObject(bool is_nullable_) : is_nullable(is_nullable_), num_rows(0) {}
+
+ColumnObject::ColumnObject(Subcolumns&& subcolumns_, bool is_nullable_)
+ : is_nullable(is_nullable_),
+ subcolumns(std::move(subcolumns_)),
+ num_rows(subcolumns.empty() ? 0 : (*subcolumns.begin())->data.size()) {
+ check_consistency();
+}
+
+void ColumnObject::check_consistency() const {
+ if (subcolumns.empty()) return;
+ for (const auto& leaf : subcolumns) {
+ if (num_rows != leaf->data.size()) {
+ assert(false);
+ }
+ }
+}
+
+size_t ColumnObject::size() const {
+#ifndef NDEBUG
+ check_consistency();
+#endif
+ return num_rows;
+}
+
+MutableColumnPtr ColumnObject::clone_resized(size_t new_size) const {
+ /// cloneResized with new_size == 0 is used for cloneEmpty().
+ if (new_size != 0) {
+ LOG(FATAL) << "ColumnObject doesn't support resize to non-zero length";
+ }
+ return ColumnObject::create(is_nullable);
+}
+
+size_t ColumnObject::byte_size() const {
+ size_t res = 0;
+ for (const auto& entry : subcolumns) {
+ res += entry->data.byteSize();
+ }
+ return res;
+}
+
+size_t ColumnObject::allocated_bytes() const {
+ size_t res = 0;
+ for (const auto& entry : subcolumns) {
+ res += entry->data.allocatedBytes();
+ }
+ return res;
+}
+
+void ColumnObject::for_each_subcolumn(ColumnCallback callback) {
+ if (!is_finalized()) assert(false);
+ for (auto& entry : subcolumns) {
+ callback(entry->data.data.back());
+ }
+}
+
+Status ColumnObject::try_insert_from(const IColumn& src, size_t n) {
+ return try_insert(src[n]);
+}
+
+Status ColumnObject::try_insert(const Field& field) {
+ const auto& object = field.get<const Object&>();
+ phmap::flat_hash_set<StringRef, StringRefHash> inserted;
+ size_t old_size = size();
+ for (const auto& [key_str, value] : object) {
+ PathInData key(key_str);
+ inserted.insert(key_str);
+ if (!has_subcolumn(key)) {
+ bool succ = add_sub_column(key, old_size);
+ if (!succ) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to add sub column {}", key.get_path()));
+ }
+ }
+ auto* subcolumn = get_subcolumn(key);
+ if (!subcolumn) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to find sub column {}", key.get_path()));
+ }
+ RETURN_IF_ERROR(subcolumn->insert(value));
+ }
+ for (auto& entry : subcolumns) {
+ if (!inserted.contains(entry->path.get_path())) {
+ entry->data.insertDefault();
+ }
+ }
+ ++num_rows;
+ return Status::OK();
+}
+
+void ColumnObject::insert_default() {
+ for (auto& entry : subcolumns) {
+ entry->data.insertDefault();
+ }
+ ++num_rows;
+}
+
+Field ColumnObject::operator[](size_t n) const {
+ if (!is_finalized()) assert(false);
+ Object object;
+ for (const auto& entry : subcolumns) {
+ object[entry->path.get_path()] = (*entry->data.data.back())[n];
+ }
+ return object;
+}
+
+void ColumnObject::get(size_t n, Field& res) const {
+ if (!is_finalized()) assert(false);
+ auto& object = res.get<Object&>();
+ for (const auto& entry : subcolumns) {
+ auto it = object.try_emplace(entry->path.get_path()).first;
+ entry->data.data.back()->get(n, it->second);
+ }
+}
+
+Status ColumnObject::try_insert_indices_from(const IColumn& src, const int* indices_begin,
+ const int* indices_end) {
+ for (auto x = indices_begin; x != indices_end; ++x) {
+ if (*x == -1) {
+ ColumnObject::insert_default();
+ } else {
+ RETURN_IF_ERROR(ColumnObject::try_insert_from(src, *x));
+ }
+ }
+ finalize();
+ return Status::OK();
+}
+
+Status ColumnObject::try_insert_range_from(const IColumn& src, size_t start, size_t length) {
+ const auto& src_object = assert_cast<const ColumnObject&>(src);
+ if (UNLIKELY(src_object.empty())) {
+ return Status::OK();
+ }
+ for (auto& entry : subcolumns) {
+ if (src_object.has_subcolumn(entry->path)) {
+ auto* subcolumn = src_object.get_subcolumn(entry->path);
+ if (!subcolumn) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to find sub column {}", entry->path.get_path()));
+ }
+ RETURN_IF_ERROR(entry->data.insertRangeFrom(*subcolumn, start, length));
+ } else {
+ entry->data.insertManyDefaults(length);
+ }
+ }
+ for (const auto& entry : src_object.subcolumns) {
+ if (!has_subcolumn(entry->path)) {
+ bool succ = false;
+ if (entry->path.has_nested_part()) {
+ const auto& base_type = entry->data.get_least_common_typeBase();
+ FieldInfo field_info {
+ .scalar_type = base_type,
+ .have_nulls = base_type->is_nullable(),
+ .need_convert = false,
+ .num_dimensions = entry->data.get_dimensions(),
+ };
+ succ = add_nested_subcolumn(entry->path, field_info, num_rows);
+ } else {
+ succ = add_sub_column(entry->path, num_rows);
+ }
+ if (!succ) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to add column {}", entry->path.get_path()));
+ }
+ auto* subcolumn = get_subcolumn(entry->path);
+ if (!subcolumn) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to find sub column {}", entry->path.get_path()));
+ }
+ RETURN_IF_ERROR(subcolumn->insertRangeFrom(entry->data, start, length));
+ }
+ }
+ num_rows += length;
+ finalize();
+ return Status::OK();
+}
+
+void ColumnObject::pop_back(size_t length) {
+ for (auto& entry : subcolumns) entry->data.pop_back(length);
+ num_rows -= length;
+}
+
+const ColumnObject::Subcolumn* ColumnObject::get_subcolumn(const PathInData& key) const {
+ const auto* node = subcolumns.find_leaf(key);
+ if (node == nullptr) {
+ VLOG_DEBUG << "There is no subcolumn " << key.get_path();
+ return nullptr;
+ }
+ return &node->data;
+}
+
+ColumnObject::Subcolumn* ColumnObject::get_subcolumn(const PathInData& key) {
+ const auto* node = subcolumns.find_leaf(key);
+ if (node == nullptr) {
+ VLOG_DEBUG << "There is no subcolumn " << key.get_path();
+ return nullptr;
+ }
+ return &const_cast<Subcolumns::Node*>(node)->data;
+}
+
+bool ColumnObject::has_subcolumn(const PathInData& key) const {
+ return subcolumns.find_leaf(key) != nullptr;
+}
+
+bool ColumnObject::add_sub_column(const PathInData& key, MutableColumnPtr&& subcolumn) {
+ size_t new_size = subcolumn->size();
+ bool inserted = subcolumns.add(key, Subcolumn(std::move(subcolumn), is_nullable));
+ if (!inserted) {
+ VLOG_DEBUG << "Duplicated sub column " << key.get_path();
+ return false;
+ }
+ if (num_rows == 0) {
+ num_rows = new_size;
+ } else if (new_size != num_rows) {
+ VLOG_DEBUG << "Size of subcolumn is in consistent with column";
+ return false;
+ }
+ return true;
+}
+
+bool ColumnObject::add_sub_column(const PathInData& key, size_t new_size) {
+ bool inserted = subcolumns.add(key, Subcolumn(new_size, is_nullable));
+ if (!inserted) {
+ VLOG_DEBUG << "Duplicated sub column " << key.get_path();
+ return false;
+ }
+ if (num_rows == 0) {
+ num_rows = new_size;
+ } else if (new_size != num_rows) {
+ VLOG_DEBUG << "Size of subcolumn is in consistent with column";
+ return false;
+ }
+ return true;
+}
+
+bool ColumnObject::add_nested_subcolumn(const PathInData& key, const FieldInfo& field_info,
+ size_t new_size) {
+ assert(key.has_nested_part());
+ bool inserted = false;
+ /// We find node that represents the same Nested type as @key.
+ const auto* nested_node = subcolumns.find_best_match(key);
+ if (nested_node) {
+ /// Find any leaf of Nested subcolumn.
+ const auto* leaf = subcolumns.find_leaf(nested_node, [&](const auto&) { return true; });
Review Comment:
warning: static member accessed through instance [readability-static-accessed-through-instance]
```suggestion
const auto* leaf = doris::vectorized::ColumnObject::Subcolumns::find_leaf(nested_node, [&](const auto&) { return true; });
```
##########
be/src/vec/columns/columns_common.cpp:
##########
@@ -279,4 +279,19 @@ INSTANTIATE(Float64, ColumnArray::Offset64)
#undef INSTANTIATE
+namespace detail {
+template <typename T>
+const PaddedPODArray<T>* get_indexes_data(const IColumn& indexes) {
+ auto* column = typeid_cast<const ColumnVector<T>*>(&indexes);
+ if (column) return &column->get_data();
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (column) { return &column->get_data();
}
```
##########
be/src/vec/columns/columns_common.h:
##########
@@ -51,4 +51,43 @@ void filter_arrays_impl_only_data(const PaddedPODArray<T>& src_elems,
PaddedPODArray<T>& res_elems, const IColumn::Filter& filt,
ssize_t result_size_hint);
+namespace detail {
+template <typename T>
+const PaddedPODArray<T>* get_indexes_data(const IColumn& indexes);
+}
+
+/// Check limit <= indexes->size() and call column.index_impl(const PaddedPodArray<Type> & indexes, UInt64 limit).
+template <typename Column>
+ColumnPtr select_index_impl(const Column& column, const IColumn& indexes, size_t limit) {
+ if (limit == 0) limit = indexes.size();
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (limit == 0) { limit = indexes.size();
}
```
##########
be/src/vec/columns/column_object.h:
##########
@@ -0,0 +1,364 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.h
+// and modified by Doris
+
+#pragma once
+#include <vec/columns/column.h>
+#include <vec/columns/subcolumn_tree.h>
+#include <vec/common/hash_table/hash_map.h>
+#include <vec/common/pod_array.h>
+#include <vec/core/field.h>
+#include <vec/core/names.h>
+#include <vec/data_types/data_type.h>
+#include <vec/json/json_parser.h>
+
+#include "common/status.h"
+namespace doris::vectorized {
+
+/// Info that represents a scalar or array field in a decomposed view.
+/// It allows to recreate field with different number
+/// of dimensions or nullability.
+struct FieldInfo {
+ /// The common type of of all scalars in field.
+ DataTypePtr scalar_type;
+ /// Do we have NULL scalar in field.
+ bool have_nulls;
+ /// If true then we have scalars with different types in array and
+ /// we need to convert scalars to the common type.
+ bool need_convert;
+ /// Number of dimension in array. 0 if field is scalar.
+ size_t num_dimensions;
+};
+Status get_field_info(const Field& field, FieldInfo* info);
+/** A column that represents object with dynamic set of subcolumns.
+ * Subcolumns are identified by paths in document and are stored in
+ * a trie-like structure. ColumnObject is not suitable for writing into tables
+ * and it should be converted to Tuple with fixed set of subcolumns before that.
+ */
+class ColumnObject final : public COWHelper<IColumn, ColumnObject> {
+public:
+ /** Class that represents one subcolumn.
+ * It stores values in several parts of column
+ * and keeps current common type of all parts.
+ * We add a new column part with a new type, when we insert a field,
+ * which can't be converted to the current common type.
+ * After insertion of all values subcolumn should be finalized
+ * for writing and other operations.
+ */
+ class Subcolumn {
+ public:
+ Subcolumn() = default;
+
+ Subcolumn(size_t size_, bool is_nullable_);
+
+ Subcolumn(MutableColumnPtr&& data_, bool is_nullable_);
+
+ size_t size() const;
+
+ size_t byteSize() const;
+
+ size_t allocatedBytes() const;
+
+ bool is_finalized() const;
+
+ const DataTypePtr& get_least_common_type() const { return least_common_type.get(); }
+
+ const DataTypePtr& get_least_common_typeBase() const { return least_common_type.getBase(); }
+
+ size_t get_dimensions() const { return least_common_type.get_dimensions(); }
+
+ /// Checks the consistency of column's parts stored in @data.
+ void checkTypes() const;
+
+ /// Inserts a field, which scalars can be arbitrary, but number of
+ /// dimensions should be consistent with current common type.
+ /// return Status::InvalidArgument when meet conflict types
+ Status insert(Field field);
+
+ Status insert(Field field, FieldInfo info);
+
+ void insertDefault();
+
+ void insertManyDefaults(size_t length);
+
+ Status insertRangeFrom(const Subcolumn& src, size_t start, size_t length);
+
+ void pop_back(size_t n);
+
+ /// Converts all column's parts to the common type and
+ /// creates a single column that stores all values.
+ void finalize();
+
+ /// Returns last inserted field.
+ Field getLastField() const;
+
+ /// Recreates subcolumn with default scalar values and keeps sizes of arrays.
+ /// Used to create columns of type Nested with consistent array sizes.
+ Subcolumn recreate_with_default_values(const FieldInfo& field_info) const;
+
+ /// Returns single column if subcolumn in finalizes.
+ /// Otherwise -- undefined behaviour.
+ IColumn& get_finalized_column();
+
+ const IColumn& get_finalized_column() const;
+
+ const ColumnPtr& get_finalized_column_ptr() const;
+
+ friend class ColumnObject;
+
+ private:
+ class LeastCommonType {
+ public:
+ LeastCommonType() = default;
+
+ explicit LeastCommonType(DataTypePtr type_);
+
+ const DataTypePtr& get() const { return type; }
+
+ const DataTypePtr& getBase() const { return base_type; }
+
+ size_t get_dimensions() const { return num_dimensions; }
+
+ private:
+ DataTypePtr type;
+ DataTypePtr base_type;
+ size_t num_dimensions = 0;
+ };
+ void add_new_column_part(DataTypePtr type);
+
+ /// Current least common type of all values inserted to this subcolumn.
+ LeastCommonType least_common_type;
+ /// If true then common type type of subcolumn is Nullable
+ /// and default values are NULLs.
+ bool is_nullable = false;
+ /// Parts of column. Parts should be in increasing order in terms of subtypes/supertypes.
+ /// That means that the least common type for i-th prefix is the type of i-th part
+ /// and it's the supertype for all type of column from 0 to i-1.
+ std::vector<WrappedPtr> data;
+ /// Until we insert any non-default field we don't know further
+ /// least common type and we count number of defaults in prefix,
+ /// which will be converted to the default type of final common type.
+ size_t num_of_defaults_in_prefix = 0;
+ };
+ using Subcolumns = SubcolumnsTree<Subcolumn>;
+
+private:
+ /// If true then all subcolumns are nullable.
+ const bool is_nullable;
+ Subcolumns subcolumns;
+ size_t num_rows;
+
+public:
+ static constexpr auto COLUMN_NAME_DUMMY = "_dummy";
+
+ explicit ColumnObject(bool is_nullable_);
+
+ ColumnObject(Subcolumns&& subcolumns_, bool is_nullable_);
+
+ ~ColumnObject() = default;
+
+ bool can_be_inside_nullable() const override { return true; }
+
+ /// Checks that all subcolumns have consistent sizes.
+ void check_consistency() const;
+
+ bool has_subcolumn(const PathInData& key) const;
+
+ // return null if not found
+ const Subcolumn* get_subcolumn(const PathInData& key) const;
+
+ // return null if not found
+ Subcolumn* get_subcolumn(const PathInData& key);
+
+ void incr_num_rows() { ++num_rows; }
+
+ /// Adds a subcolumn from existing IColumn.
+ bool add_sub_column(const PathInData& key, MutableColumnPtr&& subcolumn);
+
+ /// Adds a subcolumn of specific size with default values.
+ bool add_sub_column(const PathInData& key, size_t new_size);
+
+ /// Adds a subcolumn of type Nested of specific size with default values.
+ /// It cares about consistency of sizes of Nested arrays.
+ bool add_nested_subcolumn(const PathInData& key, const FieldInfo& field_info, size_t new_size);
+
+ const Subcolumns& get_subcolumns() const { return subcolumns; }
+
+ Subcolumns& get_subcolumns() { return subcolumns; }
+
+ PathsInData getKeys() const;
+
+ std::string get_keys_str() const {
+ std::stringstream ss;
+ bool first = true;
+ for (auto& k : getKeys()) {
+ if (first) {
+ first = false;
+ } else {
+ ss << ", ";
+ }
+ ss << k.get_path();
+ }
+
+ return ss.str();
+ }
+
+ void remove_subcolumns(const std::unordered_set<std::string>& keys);
+
+ /// Finalizes all subcolumns.
+ void finalize();
+
+ bool is_finalized() const;
+
+ /// Part of interface
+ const char* get_family_name() const override { return "Variant"; }
+
+ TypeIndex get_data_type() const override { return TypeIndex::VARIANT; }
+
+ size_t size() const override;
+
+ MutableColumnPtr clone_resized(size_t new_size) const override;
+
+ size_t byte_size() const override;
+
+ size_t allocated_bytes() const override;
+
+ void for_each_subcolumn(ColumnCallback callback) override;
+
+ // Do nothing, call try_insert instead
+ void insert(const Field& field) override {
+ Status st = try_insert(field);
+ if (!st.ok()) LOG(FATAL) << "insert return ERROR status: " << st;
+ }
+
+ // Do nothing, call try_insert_range_from instead
+ void insert_range_from(const IColumn& src, size_t start, size_t length) override {
+ Status st = try_insert_range_from(src, start, length);
+ if (!st.ok()) LOG(FATAL) << "insert_range_from return ERROR status: " << st;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!st.ok()) { LOG(FATAL) << "insert_range_from return ERROR status: " << st;
}
```
##########
be/src/vec/columns/column_object.cpp:
##########
@@ -0,0 +1,860 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.cpp
+// and modified by Doris
+
+#include <parallel_hashmap/phmap.h>
+#include <vec/columns/column_array.h>
+#include <vec/columns/column_nullable.h>
+#include <vec/columns/column_object.h>
+#include <vec/columns/columns_number.h>
+#include <vec/common/field_visitors.h>
+#include <vec/common/hash_table/hash_set.h>
+#include <vec/common/object_util.h>
+#include <vec/common/pod_array_fwd.h>
+#include <vec/core/field.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_nothing.h>
+#include <vec/data_types/get_least_supertype.h>
+
+#include <vec/data_types/data_type_factory.hpp>
+
+namespace doris::vectorized {
+namespace {
+
+DataTypePtr create_array_of_type(DataTypePtr type, size_t num_dimensions) {
+ for (size_t i = 0; i < num_dimensions; ++i)
+ type = std::make_shared<DataTypeArray>(std::move(type));
+ return type;
+}
+
+DataTypePtr getBaseTypeOfArray(const DataTypePtr& type) {
+ /// Get raw pointers to avoid extra copying of type pointers.
+ const DataTypeArray* last_array = nullptr;
+ const auto* current_type = type.get();
+ while (const auto* type_array = typeid_cast<const DataTypeArray*>(current_type)) {
+ current_type = type_array->get_nested_type().get();
+ last_array = type_array;
+ }
+ return last_array ? last_array->get_nested_type() : type;
+}
+
+size_t getNumberOfDimensions(const IDataType& type) {
+ if (const auto* type_array = typeid_cast<const DataTypeArray*>(&type)) {
+ return type_array->get_number_of_dimensions();
+ }
+ return 0;
+}
+
+DataTypePtr get_data_type_by_column(const IColumn& column) {
+ auto idx = column.get_data_type();
+ if (WhichDataType(idx).is_simple()) {
+ return DataTypeFactory::instance().get(String(getTypeName(idx)));
+ }
+ if (WhichDataType(idx).is_nothing()) {
+ return std::make_shared<DataTypeNothing>();
+ }
+ if (const auto* column_array = check_and_get_column<ColumnArray>(&column)) {
+ return std::make_shared<DataTypeArray>(get_data_type_by_column(column_array->get_data()));
+ }
+ if (const auto* column_nullable = check_and_get_column<ColumnNullable>(&column)) {
+ return make_nullable(get_data_type_by_column(column_nullable->get_nested_column()));
+ }
+ // TODO add more types
+ assert(false);
+ return nullptr;
+}
+
+/// Recreates column with default scalar values and keeps sizes of arrays.
+ColumnPtr recreate_column_with_default_value(const ColumnPtr& column,
+ const DataTypePtr& scalar_type,
+ size_t num_dimensions) {
+ const auto* column_array = check_and_get_column<ColumnArray>(column.get());
+ if (column_array && num_dimensions) {
+ return ColumnArray::create(
+ recreate_column_with_default_value(column_array->get_data_ptr(), scalar_type,
+ num_dimensions - 1),
+ IColumn::mutate(column_array->get_offsets_ptr()));
+ }
+ return create_array_of_type(scalar_type, num_dimensions)
+ ->create_column()
+ ->clone_resized(column->size());
+}
+
+Array create_empty_array_field(size_t num_dimensions) {
+ assert(num_dimensions != 0);
+ Array array;
+ Array* current_array = &array;
+ for (size_t i = 1; i < num_dimensions; ++i) {
+ current_array->push_back(Array());
+ current_array = ¤t_array->back().get<Array&>();
+ }
+ return array;
+}
+
+/// Replaces NULL fields to given field or empty array.
+class FieldVisitorReplaceNull : public StaticVisitor<Field> {
+public:
+ explicit FieldVisitorReplaceNull(const Field& replacement_, size_t num_dimensions_)
+ : replacement(replacement_), num_dimensions(num_dimensions_) {}
+ Field operator()(const Null&) const {
+ return num_dimensions ? create_empty_array_field(num_dimensions) : replacement;
+ }
+ Field operator()(const Array& x) const {
+ assert(num_dimensions > 0);
+ const size_t size = x.size();
+ Array res(size);
+ for (size_t i = 0; i < size; ++i) {
+ res[i] = apply_visitor(FieldVisitorReplaceNull(replacement, num_dimensions - 1), x[i]);
+ }
+ return res;
+ }
+ template <typename T>
+ Field operator()(const T& x) const {
+ return x;
+ }
+
+private:
+ const Field& replacement;
+ size_t num_dimensions;
+};
+
+/// Calculates number of dimensions in array field.
+/// Returns 0 for scalar fields.
+class FieldVisitorToNumberOfDimensions : public StaticVisitor<size_t> {
+public:
+ explicit FieldVisitorToNumberOfDimensions(Status* st) : _st(st) {}
+ size_t operator()(const Array& x) const {
+ const size_t size = x.size();
+ std::optional<size_t> dimensions;
+ for (size_t i = 0; i < size; ++i) {
+ /// Do not count Nulls, because they will be replaced by default
+ /// values with proper number of dimensions.
+ if (x[i].is_null()) continue;
+ size_t current_dimensions = apply_visitor(*this, x[i]);
+ if (!dimensions) {
+ dimensions = current_dimensions;
+ } else if (current_dimensions != *dimensions) {
+ *_st = Status::InvalidArgument(
+ "Number of dimensions mismatched among array elements");
+ return 0;
+ }
+ }
+ return 1 + dimensions.value_or(0);
+ }
+ template <typename T>
+ size_t operator()(const T&) const {
+ return 0;
+ }
+
+private:
+ mutable Status* _st;
+};
+
+/// Visitor that allows to get type of scalar field
+/// or least common type of scalars in array.
+/// More optimized version of FieldToDataType.
+class FieldVisitorToScalarType : public StaticVisitor<size_t> {
+public:
+ using FieldType = Field::Types::Which;
+ size_t operator()(const Array& x) {
+ size_t size = x.size();
+ for (size_t i = 0; i < size; ++i) apply_visitor(*this, x[i]);
+ return 0;
+ }
+ // TODO doris not support unsigned integers for now
+ // treat as signed integers
+ size_t operator()(const UInt64& x) {
+ field_types.insert(FieldType::UInt64);
+ if (x <= std::numeric_limits<Int8>::max()) {
+ type_indexes.insert(TypeIndex::Int8);
+ } else if (x <= std::numeric_limits<Int16>::max()) {
+ type_indexes.insert(TypeIndex::Int16);
+ } else if (x <= std::numeric_limits<Int32>::max()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Int64& x) {
+ // Only support Int32 and Int64
+ field_types.insert(FieldType::Int64);
+ if (x <= std::numeric_limits<Int32>::max() && x >= std::numeric_limits<Int32>::min()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Null&) {
+ have_nulls = true;
+ return 0;
+ }
+ size_t operator()(const Int128I& x) { LOG(FATAL) << "not implemented"; }
+ template <typename T>
+ size_t operator()(const T&) {
+ Field::EnumToType<Field::Types::Array>::Type a;
+ field_types.insert(Field::TypeToEnum<NearestFieldType<T>>::value);
+ type_indexes.insert(TypeId<NearestFieldType<T>>::value);
+ return 0;
+ }
+ Status get_scalar_type(DataTypePtr* type) const {
+ return get_least_supertype(type_indexes, type, true /*compatible with string type*/);
+ }
+ bool contain_nulls() const { return have_nulls; }
+ bool need_convert_field() const { return field_types.size() > 1; }
+
+private:
+ phmap::flat_hash_set<TypeIndex> type_indexes;
+ phmap::flat_hash_set<FieldType> field_types;
+ bool have_nulls = false;
+};
+
+} // namespace
+Status get_field_info(const Field& field, FieldInfo* info) {
+ FieldVisitorToScalarType to_scalar_type_visitor;
+ apply_visitor(to_scalar_type_visitor, field);
+ DataTypePtr type = nullptr;
+ RETURN_IF_ERROR(to_scalar_type_visitor.get_scalar_type(&type));
+ // array item's dimension may missmatch, eg. [1, 2, [1, 2, 3]]
+ Status num_to_dimensions_status;
+ *info = {
+ type,
+ to_scalar_type_visitor.contain_nulls(),
+ to_scalar_type_visitor.need_convert_field(),
+ apply_visitor(FieldVisitorToNumberOfDimensions(&num_to_dimensions_status), field),
+ };
+ RETURN_IF_ERROR(num_to_dimensions_status);
+ return Status::OK();
+}
+
+ColumnObject::Subcolumn::Subcolumn(MutableColumnPtr&& data_, bool is_nullable_)
+ : least_common_type(get_data_type_by_column(*data_)), is_nullable(is_nullable_) {
+ data.push_back(std::move(data_));
+}
+
+ColumnObject::Subcolumn::Subcolumn(size_t size_, bool is_nullable_)
+ : least_common_type(std::make_shared<DataTypeNothing>()),
+ is_nullable(is_nullable_),
+ num_of_defaults_in_prefix(size_) {}
+
+size_t ColumnObject::Subcolumn::Subcolumn::size() const {
+ size_t res = num_of_defaults_in_prefix;
+ for (const auto& part : data) {
+ res += part->size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::byteSize() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->byte_size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::allocatedBytes() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->allocated_bytes();
+ }
+ return res;
+}
+
+Status ColumnObject::Subcolumn::insert(Field field) {
+ FieldInfo info;
+ RETURN_IF_ERROR(get_field_info(field, &info));
+ return insert(std::move(field), std::move(info));
+}
+
+void ColumnObject::Subcolumn::add_new_column_part(DataTypePtr type) {
+ data.push_back(type->create_column());
+ least_common_type = LeastCommonType {std::move(type)};
+}
+
+Status ColumnObject::Subcolumn::insert(Field field, FieldInfo info) {
+ auto base_type = std::move(info.scalar_type);
+ if (is_nothing(base_type)) {
+ insertDefault();
+ return Status::OK();
+ }
+ auto column_dim = least_common_type.get_dimensions();
+ auto value_dim = info.num_dimensions;
+ if (is_nothing(least_common_type.getBase())) {
+ column_dim = value_dim;
+ }
+ if (is_nothing(base_type)) {
+ value_dim = column_dim;
+ }
+ if (value_dim != column_dim) {
+ return Status::InvalidArgument(
+ "Dimension of types mismatched between inserted value and column.");
+ }
+ if (is_nullable && !is_nothing(base_type)) {
+ base_type = make_nullable(base_type);
+ }
+ // alawys nullable at present
+ if (!is_nullable && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+ // need replace muli dimensions array which contains null. eg. [[1, 2, 3], null] -> [[1, 2, 3], []]
+ // since column array doesnt known null's dimension
+ if (info.num_dimensions >= 2 && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+
+ bool type_changed = false;
+ const auto& least_common_base_type = least_common_type.getBase();
+ if (data.empty()) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ } else if (!least_common_base_type->equals(*base_type) && !is_nothing(base_type)) {
+ if (!object_util::is_conversion_required_between_integers(*base_type,
+ *least_common_base_type)) {
+ RETURN_IF_ERROR(
+ get_least_supertype(DataTypes {std::move(base_type), least_common_base_type},
+ &base_type, true /*compatible with string type*/));
+ type_changed = true;
+ if (!least_common_base_type->equals(*base_type)) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ }
+ }
+ }
+
+ if (type_changed || info.need_convert) {
+ Field new_field;
+ RETURN_IF_ERROR(convert_field_to_type(field, *least_common_type.get(), &new_field));
+ field = new_field;
+ }
+
+ data.back()->insert(field);
+ return Status::OK();
+}
+
+Status ColumnObject::Subcolumn::insertRangeFrom(const Subcolumn& src, size_t start, size_t length) {
+ assert(src.is_finalized());
+ const auto& src_column = src.data.back();
+ const auto& src_type = src.least_common_type.get();
+ if (data.empty()) {
+ add_new_column_part(src.least_common_type.get());
+ data.back()->insert_range_from(*src_column, start, length);
+ } else if (least_common_type.get()->equals(*src_type)) {
+ data.back()->insert_range_from(*src_column, start, length);
+ } else {
+ DataTypePtr new_least_common_type = nullptr;
+ RETURN_IF_ERROR(get_least_supertype(DataTypes {least_common_type.get(), src_type},
+ &new_least_common_type,
+ true /*compatible with string type*/));
+ ColumnPtr casted_column;
+ RETURN_IF_ERROR(object_util::cast_column({src_column, src_type, ""}, new_least_common_type,
+ &casted_column));
+ if (!least_common_type.get()->equals(*new_least_common_type))
+ add_new_column_part(std::move(new_least_common_type));
+ data.back()->insert_range_from(*casted_column, start, length);
+ }
+ return Status::OK();
+}
+
+bool ColumnObject::Subcolumn::is_finalized() const {
+ return data.empty() || (data.size() == 1 && num_of_defaults_in_prefix == 0);
+}
+
+template <typename Func>
+ColumnPtr ColumnObject::apply_for_subcolumns(Func&& func, std::string_view func_name) const {
+ if (!is_finalized()) {
+ LOG(FATAL) << "Cannot " << func_name << " non-finalized ColumnObject";
+ }
+ auto res = ColumnObject::create(is_nullable);
+ for (const auto& subcolumn : subcolumns) {
+ auto new_subcolumn = func(subcolumn->data.get_finalized_column());
+ res->add_sub_column(subcolumn->path, new_subcolumn->assume_mutable());
+ }
+ return res;
+}
+ColumnPtr ColumnObject::index(const IColumn& indexes, size_t limit) const {
+ return apply_for_subcolumns(
+ [&](const auto& subcolumn) { return subcolumn.index(indexes, limit); }, "index");
+}
+
+void ColumnObject::Subcolumn::finalize() {
+ if (is_finalized()) {
+ return;
+ }
+ if (data.size() == 1 && num_of_defaults_in_prefix == 0) {
+ data[0] = data[0]->convert_to_full_column_if_const();
+ return;
+ }
+ const auto& to_type = least_common_type.get();
+ auto result_column = to_type->create_column();
+ if (num_of_defaults_in_prefix) {
+ result_column->insert_many_defaults(num_of_defaults_in_prefix);
+ }
+ for (auto& part : data) {
+ part = part->convert_to_full_column_if_const();
+ auto from_type = get_data_type_by_column(*part);
+ size_t part_size = part->size();
+ if (!from_type->equals(*to_type)) {
+ auto offsets = ColumnUInt64::create();
+ auto& offsets_data = offsets->get_data();
+ /// We need to convert only non-default values and then recreate column
+ /// with default value of new type, because default values (which represents misses in data)
+ /// may be inconsistent between types (e.g "0" in UInt64 and empty string in String).
+ part->get_indices_of_non_default_rows(offsets_data, 0, part_size);
+ if (offsets->size() == part_size) {
+ ColumnPtr ptr;
+ object_util::cast_column({part, from_type, ""}, to_type, &ptr);
+ part = ptr;
+ } else {
+ auto values = part->index(*offsets, offsets->size());
+ object_util::cast_column({values, from_type, ""}, to_type, &values);
+ part = values->create_with_offsets(offsets_data, to_type->get_default(), part_size,
+ /*shift=*/0);
+ }
+ }
+ result_column->insert_range_from(*part, 0, part_size);
+ }
+ data = {std::move(result_column)};
+ num_of_defaults_in_prefix = 0;
+}
+
+void ColumnObject::Subcolumn::insertDefault() {
+ if (data.empty()) {
+ ++num_of_defaults_in_prefix;
+ } else {
+ data.back()->insert_default();
+ }
+}
+
+void ColumnObject::Subcolumn::insertManyDefaults(size_t length) {
+ if (data.empty()) {
+ num_of_defaults_in_prefix += length;
+ } else {
+ data.back()->insert_many_defaults(length);
+ }
+}
+
+void ColumnObject::Subcolumn::pop_back(size_t n) {
+ assert(n <= size());
+ size_t num_removed = 0;
+ for (auto it = data.rbegin(); it != data.rend(); ++it) {
+ if (n == 0) break;
+ auto& column = *it;
+ if (n < column->size()) {
+ column->pop_back(n);
+ n = 0;
+ } else {
+ ++num_removed;
+ n -= column->size();
+ }
+ }
+ data.resize(data.size() - num_removed);
+ num_of_defaults_in_prefix -= n;
+}
+
+Field ColumnObject::Subcolumn::getLastField() const {
+ if (data.empty()) return Field();
+ const auto& last_part = data.back();
+ assert(!last_part->empty());
+ return (*last_part)[last_part->size() - 1];
+}
+
+ColumnObject::Subcolumn ColumnObject::Subcolumn::recreate_with_default_values(
+ const FieldInfo& field_info) const {
+ auto scalar_type = field_info.scalar_type;
+ if (is_nullable) {
+ scalar_type = make_nullable(scalar_type);
+ }
+ Subcolumn new_subcolumn;
+ new_subcolumn.least_common_type =
+ LeastCommonType {create_array_of_type(scalar_type, field_info.num_dimensions)};
+ new_subcolumn.is_nullable = is_nullable;
+ new_subcolumn.num_of_defaults_in_prefix = num_of_defaults_in_prefix;
+ new_subcolumn.data.reserve(data.size());
+ for (const auto& part : data) {
+ new_subcolumn.data.push_back(
+ recreate_column_with_default_value(part, scalar_type, field_info.num_dimensions));
+ }
+ return new_subcolumn;
+}
+
+IColumn& ColumnObject::Subcolumn::get_finalized_column() {
+ assert(is_finalized());
+ return *data[0];
+}
+
+const IColumn& ColumnObject::Subcolumn::get_finalized_column() const {
+ assert(is_finalized());
+ return *data[0];
+}
+
+const ColumnPtr& ColumnObject::Subcolumn::get_finalized_column_ptr() const {
+ assert(is_finalized());
+ return data[0];
+}
+
+ColumnObject::Subcolumn::LeastCommonType::LeastCommonType(DataTypePtr type_)
+ : type(std::move(type_)),
+ base_type(getBaseTypeOfArray(type)),
+ num_dimensions(getNumberOfDimensions(*type)) {}
+
+ColumnObject::ColumnObject(bool is_nullable_) : is_nullable(is_nullable_), num_rows(0) {}
+
+ColumnObject::ColumnObject(Subcolumns&& subcolumns_, bool is_nullable_)
+ : is_nullable(is_nullable_),
+ subcolumns(std::move(subcolumns_)),
+ num_rows(subcolumns.empty() ? 0 : (*subcolumns.begin())->data.size()) {
+ check_consistency();
+}
+
+void ColumnObject::check_consistency() const {
+ if (subcolumns.empty()) return;
+ for (const auto& leaf : subcolumns) {
+ if (num_rows != leaf->data.size()) {
+ assert(false);
+ }
+ }
+}
+
+size_t ColumnObject::size() const {
+#ifndef NDEBUG
+ check_consistency();
+#endif
+ return num_rows;
+}
+
+MutableColumnPtr ColumnObject::clone_resized(size_t new_size) const {
+ /// cloneResized with new_size == 0 is used for cloneEmpty().
+ if (new_size != 0) {
+ LOG(FATAL) << "ColumnObject doesn't support resize to non-zero length";
+ }
+ return ColumnObject::create(is_nullable);
+}
+
+size_t ColumnObject::byte_size() const {
+ size_t res = 0;
+ for (const auto& entry : subcolumns) {
+ res += entry->data.byteSize();
+ }
+ return res;
+}
+
+size_t ColumnObject::allocated_bytes() const {
+ size_t res = 0;
+ for (const auto& entry : subcolumns) {
+ res += entry->data.allocatedBytes();
+ }
+ return res;
+}
+
+void ColumnObject::for_each_subcolumn(ColumnCallback callback) {
+ if (!is_finalized()) assert(false);
+ for (auto& entry : subcolumns) {
+ callback(entry->data.data.back());
+ }
+}
+
+Status ColumnObject::try_insert_from(const IColumn& src, size_t n) {
+ return try_insert(src[n]);
+}
+
+Status ColumnObject::try_insert(const Field& field) {
+ const auto& object = field.get<const Object&>();
+ phmap::flat_hash_set<StringRef, StringRefHash> inserted;
+ size_t old_size = size();
+ for (const auto& [key_str, value] : object) {
+ PathInData key(key_str);
+ inserted.insert(key_str);
+ if (!has_subcolumn(key)) {
+ bool succ = add_sub_column(key, old_size);
+ if (!succ) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to add sub column {}", key.get_path()));
+ }
+ }
+ auto* subcolumn = get_subcolumn(key);
+ if (!subcolumn) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to find sub column {}", key.get_path()));
+ }
+ RETURN_IF_ERROR(subcolumn->insert(value));
+ }
+ for (auto& entry : subcolumns) {
+ if (!inserted.contains(entry->path.get_path())) {
+ entry->data.insertDefault();
+ }
+ }
+ ++num_rows;
+ return Status::OK();
+}
+
+void ColumnObject::insert_default() {
+ for (auto& entry : subcolumns) {
+ entry->data.insertDefault();
+ }
+ ++num_rows;
+}
+
+Field ColumnObject::operator[](size_t n) const {
+ if (!is_finalized()) assert(false);
+ Object object;
+ for (const auto& entry : subcolumns) {
+ object[entry->path.get_path()] = (*entry->data.data.back())[n];
+ }
+ return object;
+}
+
+void ColumnObject::get(size_t n, Field& res) const {
+ if (!is_finalized()) assert(false);
+ auto& object = res.get<Object&>();
+ for (const auto& entry : subcolumns) {
+ auto it = object.try_emplace(entry->path.get_path()).first;
+ entry->data.data.back()->get(n, it->second);
+ }
+}
+
+Status ColumnObject::try_insert_indices_from(const IColumn& src, const int* indices_begin,
+ const int* indices_end) {
+ for (auto x = indices_begin; x != indices_end; ++x) {
+ if (*x == -1) {
+ ColumnObject::insert_default();
+ } else {
+ RETURN_IF_ERROR(ColumnObject::try_insert_from(src, *x));
+ }
+ }
+ finalize();
+ return Status::OK();
+}
+
+Status ColumnObject::try_insert_range_from(const IColumn& src, size_t start, size_t length) {
+ const auto& src_object = assert_cast<const ColumnObject&>(src);
+ if (UNLIKELY(src_object.empty())) {
+ return Status::OK();
+ }
+ for (auto& entry : subcolumns) {
+ if (src_object.has_subcolumn(entry->path)) {
+ auto* subcolumn = src_object.get_subcolumn(entry->path);
+ if (!subcolumn) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to find sub column {}", entry->path.get_path()));
+ }
+ RETURN_IF_ERROR(entry->data.insertRangeFrom(*subcolumn, start, length));
+ } else {
+ entry->data.insertManyDefaults(length);
+ }
+ }
+ for (const auto& entry : src_object.subcolumns) {
+ if (!has_subcolumn(entry->path)) {
+ bool succ = false;
+ if (entry->path.has_nested_part()) {
+ const auto& base_type = entry->data.get_least_common_typeBase();
+ FieldInfo field_info {
+ .scalar_type = base_type,
+ .have_nulls = base_type->is_nullable(),
+ .need_convert = false,
+ .num_dimensions = entry->data.get_dimensions(),
+ };
+ succ = add_nested_subcolumn(entry->path, field_info, num_rows);
+ } else {
+ succ = add_sub_column(entry->path, num_rows);
+ }
+ if (!succ) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to add column {}", entry->path.get_path()));
+ }
+ auto* subcolumn = get_subcolumn(entry->path);
+ if (!subcolumn) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to find sub column {}", entry->path.get_path()));
+ }
+ RETURN_IF_ERROR(subcolumn->insertRangeFrom(entry->data, start, length));
+ }
+ }
+ num_rows += length;
+ finalize();
+ return Status::OK();
+}
+
+void ColumnObject::pop_back(size_t length) {
+ for (auto& entry : subcolumns) entry->data.pop_back(length);
+ num_rows -= length;
+}
+
+const ColumnObject::Subcolumn* ColumnObject::get_subcolumn(const PathInData& key) const {
+ const auto* node = subcolumns.find_leaf(key);
+ if (node == nullptr) {
+ VLOG_DEBUG << "There is no subcolumn " << key.get_path();
+ return nullptr;
+ }
+ return &node->data;
+}
+
+ColumnObject::Subcolumn* ColumnObject::get_subcolumn(const PathInData& key) {
+ const auto* node = subcolumns.find_leaf(key);
+ if (node == nullptr) {
+ VLOG_DEBUG << "There is no subcolumn " << key.get_path();
+ return nullptr;
+ }
+ return &const_cast<Subcolumns::Node*>(node)->data;
+}
+
+bool ColumnObject::has_subcolumn(const PathInData& key) const {
+ return subcolumns.find_leaf(key) != nullptr;
+}
+
+bool ColumnObject::add_sub_column(const PathInData& key, MutableColumnPtr&& subcolumn) {
+ size_t new_size = subcolumn->size();
+ bool inserted = subcolumns.add(key, Subcolumn(std::move(subcolumn), is_nullable));
+ if (!inserted) {
+ VLOG_DEBUG << "Duplicated sub column " << key.get_path();
+ return false;
+ }
+ if (num_rows == 0) {
+ num_rows = new_size;
+ } else if (new_size != num_rows) {
+ VLOG_DEBUG << "Size of subcolumn is in consistent with column";
+ return false;
+ }
+ return true;
+}
+
+bool ColumnObject::add_sub_column(const PathInData& key, size_t new_size) {
+ bool inserted = subcolumns.add(key, Subcolumn(new_size, is_nullable));
+ if (!inserted) {
+ VLOG_DEBUG << "Duplicated sub column " << key.get_path();
+ return false;
+ }
+ if (num_rows == 0) {
+ num_rows = new_size;
+ } else if (new_size != num_rows) {
+ VLOG_DEBUG << "Size of subcolumn is in consistent with column";
+ return false;
+ }
+ return true;
+}
+
+bool ColumnObject::add_nested_subcolumn(const PathInData& key, const FieldInfo& field_info,
+ size_t new_size) {
+ assert(key.has_nested_part());
+ bool inserted = false;
+ /// We find node that represents the same Nested type as @key.
+ const auto* nested_node = subcolumns.find_best_match(key);
+ if (nested_node) {
+ /// Find any leaf of Nested subcolumn.
+ const auto* leaf = subcolumns.find_leaf(nested_node, [&](const auto&) { return true; });
+ assert(leaf);
+ /// Recreate subcolumn with default values and the same sizes of arrays.
+ auto new_subcolumn = leaf->data.recreate_with_default_values(field_info);
+ /// It's possible that we have already inserted value from current row
+ /// to this subcolumn. So, adjust size to expected.
+ if (new_subcolumn.size() > new_size)
+ new_subcolumn.pop_back(new_subcolumn.size() - new_size);
+ assert(new_subcolumn.size() == new_size);
+ inserted = subcolumns.add(key, new_subcolumn);
+ } else {
+ /// If node was not found just add subcolumn with empty arrays.
+ inserted = subcolumns.add(key, Subcolumn(new_size, is_nullable));
+ }
+ if (!inserted) {
+ VLOG_DEBUG << "Subcolumn already exists";
+ return false;
+ }
+ if (num_rows == 0) num_rows = new_size;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (num_rows == 0) { num_rows = new_size;
}
```
##########
be/src/vec/columns/column_object.cpp:
##########
@@ -0,0 +1,860 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnObject.cpp
+// and modified by Doris
+
+#include <parallel_hashmap/phmap.h>
+#include <vec/columns/column_array.h>
+#include <vec/columns/column_nullable.h>
+#include <vec/columns/column_object.h>
+#include <vec/columns/columns_number.h>
+#include <vec/common/field_visitors.h>
+#include <vec/common/hash_table/hash_set.h>
+#include <vec/common/object_util.h>
+#include <vec/common/pod_array_fwd.h>
+#include <vec/core/field.h>
+#include <vec/data_types/convert_field_to_type.h>
+#include <vec/data_types/data_type_array.h>
+#include <vec/data_types/data_type_nothing.h>
+#include <vec/data_types/get_least_supertype.h>
+
+#include <vec/data_types/data_type_factory.hpp>
+
+namespace doris::vectorized {
+namespace {
+
+DataTypePtr create_array_of_type(DataTypePtr type, size_t num_dimensions) {
+ for (size_t i = 0; i < num_dimensions; ++i)
+ type = std::make_shared<DataTypeArray>(std::move(type));
+ return type;
+}
+
+DataTypePtr getBaseTypeOfArray(const DataTypePtr& type) {
+ /// Get raw pointers to avoid extra copying of type pointers.
+ const DataTypeArray* last_array = nullptr;
+ const auto* current_type = type.get();
+ while (const auto* type_array = typeid_cast<const DataTypeArray*>(current_type)) {
+ current_type = type_array->get_nested_type().get();
+ last_array = type_array;
+ }
+ return last_array ? last_array->get_nested_type() : type;
+}
+
+size_t getNumberOfDimensions(const IDataType& type) {
+ if (const auto* type_array = typeid_cast<const DataTypeArray*>(&type)) {
+ return type_array->get_number_of_dimensions();
+ }
+ return 0;
+}
+
+DataTypePtr get_data_type_by_column(const IColumn& column) {
+ auto idx = column.get_data_type();
+ if (WhichDataType(idx).is_simple()) {
+ return DataTypeFactory::instance().get(String(getTypeName(idx)));
+ }
+ if (WhichDataType(idx).is_nothing()) {
+ return std::make_shared<DataTypeNothing>();
+ }
+ if (const auto* column_array = check_and_get_column<ColumnArray>(&column)) {
+ return std::make_shared<DataTypeArray>(get_data_type_by_column(column_array->get_data()));
+ }
+ if (const auto* column_nullable = check_and_get_column<ColumnNullable>(&column)) {
+ return make_nullable(get_data_type_by_column(column_nullable->get_nested_column()));
+ }
+ // TODO add more types
+ assert(false);
+ return nullptr;
+}
+
+/// Recreates column with default scalar values and keeps sizes of arrays.
+ColumnPtr recreate_column_with_default_value(const ColumnPtr& column,
+ const DataTypePtr& scalar_type,
+ size_t num_dimensions) {
+ const auto* column_array = check_and_get_column<ColumnArray>(column.get());
+ if (column_array && num_dimensions) {
+ return ColumnArray::create(
+ recreate_column_with_default_value(column_array->get_data_ptr(), scalar_type,
+ num_dimensions - 1),
+ IColumn::mutate(column_array->get_offsets_ptr()));
+ }
+ return create_array_of_type(scalar_type, num_dimensions)
+ ->create_column()
+ ->clone_resized(column->size());
+}
+
+Array create_empty_array_field(size_t num_dimensions) {
+ assert(num_dimensions != 0);
+ Array array;
+ Array* current_array = &array;
+ for (size_t i = 1; i < num_dimensions; ++i) {
+ current_array->push_back(Array());
+ current_array = ¤t_array->back().get<Array&>();
+ }
+ return array;
+}
+
+/// Replaces NULL fields to given field or empty array.
+class FieldVisitorReplaceNull : public StaticVisitor<Field> {
+public:
+ explicit FieldVisitorReplaceNull(const Field& replacement_, size_t num_dimensions_)
+ : replacement(replacement_), num_dimensions(num_dimensions_) {}
+ Field operator()(const Null&) const {
+ return num_dimensions ? create_empty_array_field(num_dimensions) : replacement;
+ }
+ Field operator()(const Array& x) const {
+ assert(num_dimensions > 0);
+ const size_t size = x.size();
+ Array res(size);
+ for (size_t i = 0; i < size; ++i) {
+ res[i] = apply_visitor(FieldVisitorReplaceNull(replacement, num_dimensions - 1), x[i]);
+ }
+ return res;
+ }
+ template <typename T>
+ Field operator()(const T& x) const {
+ return x;
+ }
+
+private:
+ const Field& replacement;
+ size_t num_dimensions;
+};
+
+/// Calculates number of dimensions in array field.
+/// Returns 0 for scalar fields.
+class FieldVisitorToNumberOfDimensions : public StaticVisitor<size_t> {
+public:
+ explicit FieldVisitorToNumberOfDimensions(Status* st) : _st(st) {}
+ size_t operator()(const Array& x) const {
+ const size_t size = x.size();
+ std::optional<size_t> dimensions;
+ for (size_t i = 0; i < size; ++i) {
+ /// Do not count Nulls, because they will be replaced by default
+ /// values with proper number of dimensions.
+ if (x[i].is_null()) continue;
+ size_t current_dimensions = apply_visitor(*this, x[i]);
+ if (!dimensions) {
+ dimensions = current_dimensions;
+ } else if (current_dimensions != *dimensions) {
+ *_st = Status::InvalidArgument(
+ "Number of dimensions mismatched among array elements");
+ return 0;
+ }
+ }
+ return 1 + dimensions.value_or(0);
+ }
+ template <typename T>
+ size_t operator()(const T&) const {
+ return 0;
+ }
+
+private:
+ mutable Status* _st;
+};
+
+/// Visitor that allows to get type of scalar field
+/// or least common type of scalars in array.
+/// More optimized version of FieldToDataType.
+class FieldVisitorToScalarType : public StaticVisitor<size_t> {
+public:
+ using FieldType = Field::Types::Which;
+ size_t operator()(const Array& x) {
+ size_t size = x.size();
+ for (size_t i = 0; i < size; ++i) apply_visitor(*this, x[i]);
+ return 0;
+ }
+ // TODO doris not support unsigned integers for now
+ // treat as signed integers
+ size_t operator()(const UInt64& x) {
+ field_types.insert(FieldType::UInt64);
+ if (x <= std::numeric_limits<Int8>::max()) {
+ type_indexes.insert(TypeIndex::Int8);
+ } else if (x <= std::numeric_limits<Int16>::max()) {
+ type_indexes.insert(TypeIndex::Int16);
+ } else if (x <= std::numeric_limits<Int32>::max()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Int64& x) {
+ // Only support Int32 and Int64
+ field_types.insert(FieldType::Int64);
+ if (x <= std::numeric_limits<Int32>::max() && x >= std::numeric_limits<Int32>::min()) {
+ type_indexes.insert(TypeIndex::Int32);
+ } else {
+ type_indexes.insert(TypeIndex::Int64);
+ }
+ return 0;
+ }
+ size_t operator()(const Null&) {
+ have_nulls = true;
+ return 0;
+ }
+ size_t operator()(const Int128I& x) { LOG(FATAL) << "not implemented"; }
+ template <typename T>
+ size_t operator()(const T&) {
+ Field::EnumToType<Field::Types::Array>::Type a;
+ field_types.insert(Field::TypeToEnum<NearestFieldType<T>>::value);
+ type_indexes.insert(TypeId<NearestFieldType<T>>::value);
+ return 0;
+ }
+ Status get_scalar_type(DataTypePtr* type) const {
+ return get_least_supertype(type_indexes, type, true /*compatible with string type*/);
+ }
+ bool contain_nulls() const { return have_nulls; }
+ bool need_convert_field() const { return field_types.size() > 1; }
+
+private:
+ phmap::flat_hash_set<TypeIndex> type_indexes;
+ phmap::flat_hash_set<FieldType> field_types;
+ bool have_nulls = false;
+};
+
+} // namespace
+Status get_field_info(const Field& field, FieldInfo* info) {
+ FieldVisitorToScalarType to_scalar_type_visitor;
+ apply_visitor(to_scalar_type_visitor, field);
+ DataTypePtr type = nullptr;
+ RETURN_IF_ERROR(to_scalar_type_visitor.get_scalar_type(&type));
+ // array item's dimension may missmatch, eg. [1, 2, [1, 2, 3]]
+ Status num_to_dimensions_status;
+ *info = {
+ type,
+ to_scalar_type_visitor.contain_nulls(),
+ to_scalar_type_visitor.need_convert_field(),
+ apply_visitor(FieldVisitorToNumberOfDimensions(&num_to_dimensions_status), field),
+ };
+ RETURN_IF_ERROR(num_to_dimensions_status);
+ return Status::OK();
+}
+
+ColumnObject::Subcolumn::Subcolumn(MutableColumnPtr&& data_, bool is_nullable_)
+ : least_common_type(get_data_type_by_column(*data_)), is_nullable(is_nullable_) {
+ data.push_back(std::move(data_));
+}
+
+ColumnObject::Subcolumn::Subcolumn(size_t size_, bool is_nullable_)
+ : least_common_type(std::make_shared<DataTypeNothing>()),
+ is_nullable(is_nullable_),
+ num_of_defaults_in_prefix(size_) {}
+
+size_t ColumnObject::Subcolumn::Subcolumn::size() const {
+ size_t res = num_of_defaults_in_prefix;
+ for (const auto& part : data) {
+ res += part->size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::byteSize() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->byte_size();
+ }
+ return res;
+}
+
+size_t ColumnObject::Subcolumn::Subcolumn::allocatedBytes() const {
+ size_t res = 0;
+ for (const auto& part : data) {
+ res += part->allocated_bytes();
+ }
+ return res;
+}
+
+Status ColumnObject::Subcolumn::insert(Field field) {
+ FieldInfo info;
+ RETURN_IF_ERROR(get_field_info(field, &info));
+ return insert(std::move(field), std::move(info));
+}
+
+void ColumnObject::Subcolumn::add_new_column_part(DataTypePtr type) {
+ data.push_back(type->create_column());
+ least_common_type = LeastCommonType {std::move(type)};
+}
+
+Status ColumnObject::Subcolumn::insert(Field field, FieldInfo info) {
+ auto base_type = std::move(info.scalar_type);
+ if (is_nothing(base_type)) {
+ insertDefault();
+ return Status::OK();
+ }
+ auto column_dim = least_common_type.get_dimensions();
+ auto value_dim = info.num_dimensions;
+ if (is_nothing(least_common_type.getBase())) {
+ column_dim = value_dim;
+ }
+ if (is_nothing(base_type)) {
+ value_dim = column_dim;
+ }
+ if (value_dim != column_dim) {
+ return Status::InvalidArgument(
+ "Dimension of types mismatched between inserted value and column.");
+ }
+ if (is_nullable && !is_nothing(base_type)) {
+ base_type = make_nullable(base_type);
+ }
+ // alawys nullable at present
+ if (!is_nullable && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+ // need replace muli dimensions array which contains null. eg. [[1, 2, 3], null] -> [[1, 2, 3], []]
+ // since column array doesnt known null's dimension
+ if (info.num_dimensions >= 2 && info.have_nulls) {
+ field = apply_visitor(FieldVisitorReplaceNull(base_type->get_default(), value_dim),
+ std::move(field));
+ }
+
+ bool type_changed = false;
+ const auto& least_common_base_type = least_common_type.getBase();
+ if (data.empty()) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ } else if (!least_common_base_type->equals(*base_type) && !is_nothing(base_type)) {
+ if (!object_util::is_conversion_required_between_integers(*base_type,
+ *least_common_base_type)) {
+ RETURN_IF_ERROR(
+ get_least_supertype(DataTypes {std::move(base_type), least_common_base_type},
+ &base_type, true /*compatible with string type*/));
+ type_changed = true;
+ if (!least_common_base_type->equals(*base_type)) {
+ add_new_column_part(create_array_of_type(std::move(base_type), value_dim));
+ }
+ }
+ }
+
+ if (type_changed || info.need_convert) {
+ Field new_field;
+ RETURN_IF_ERROR(convert_field_to_type(field, *least_common_type.get(), &new_field));
+ field = new_field;
+ }
+
+ data.back()->insert(field);
+ return Status::OK();
+}
+
+Status ColumnObject::Subcolumn::insertRangeFrom(const Subcolumn& src, size_t start, size_t length) {
+ assert(src.is_finalized());
+ const auto& src_column = src.data.back();
+ const auto& src_type = src.least_common_type.get();
+ if (data.empty()) {
+ add_new_column_part(src.least_common_type.get());
+ data.back()->insert_range_from(*src_column, start, length);
+ } else if (least_common_type.get()->equals(*src_type)) {
+ data.back()->insert_range_from(*src_column, start, length);
+ } else {
+ DataTypePtr new_least_common_type = nullptr;
+ RETURN_IF_ERROR(get_least_supertype(DataTypes {least_common_type.get(), src_type},
+ &new_least_common_type,
+ true /*compatible with string type*/));
+ ColumnPtr casted_column;
+ RETURN_IF_ERROR(object_util::cast_column({src_column, src_type, ""}, new_least_common_type,
+ &casted_column));
+ if (!least_common_type.get()->equals(*new_least_common_type))
+ add_new_column_part(std::move(new_least_common_type));
+ data.back()->insert_range_from(*casted_column, start, length);
+ }
+ return Status::OK();
+}
+
+bool ColumnObject::Subcolumn::is_finalized() const {
+ return data.empty() || (data.size() == 1 && num_of_defaults_in_prefix == 0);
+}
+
+template <typename Func>
+ColumnPtr ColumnObject::apply_for_subcolumns(Func&& func, std::string_view func_name) const {
+ if (!is_finalized()) {
+ LOG(FATAL) << "Cannot " << func_name << " non-finalized ColumnObject";
+ }
+ auto res = ColumnObject::create(is_nullable);
+ for (const auto& subcolumn : subcolumns) {
+ auto new_subcolumn = func(subcolumn->data.get_finalized_column());
+ res->add_sub_column(subcolumn->path, new_subcolumn->assume_mutable());
+ }
+ return res;
+}
+ColumnPtr ColumnObject::index(const IColumn& indexes, size_t limit) const {
+ return apply_for_subcolumns(
+ [&](const auto& subcolumn) { return subcolumn.index(indexes, limit); }, "index");
+}
+
+void ColumnObject::Subcolumn::finalize() {
+ if (is_finalized()) {
+ return;
+ }
+ if (data.size() == 1 && num_of_defaults_in_prefix == 0) {
+ data[0] = data[0]->convert_to_full_column_if_const();
+ return;
+ }
+ const auto& to_type = least_common_type.get();
+ auto result_column = to_type->create_column();
+ if (num_of_defaults_in_prefix) {
+ result_column->insert_many_defaults(num_of_defaults_in_prefix);
+ }
+ for (auto& part : data) {
+ part = part->convert_to_full_column_if_const();
+ auto from_type = get_data_type_by_column(*part);
+ size_t part_size = part->size();
+ if (!from_type->equals(*to_type)) {
+ auto offsets = ColumnUInt64::create();
+ auto& offsets_data = offsets->get_data();
+ /// We need to convert only non-default values and then recreate column
+ /// with default value of new type, because default values (which represents misses in data)
+ /// may be inconsistent between types (e.g "0" in UInt64 and empty string in String).
+ part->get_indices_of_non_default_rows(offsets_data, 0, part_size);
+ if (offsets->size() == part_size) {
+ ColumnPtr ptr;
+ object_util::cast_column({part, from_type, ""}, to_type, &ptr);
+ part = ptr;
+ } else {
+ auto values = part->index(*offsets, offsets->size());
+ object_util::cast_column({values, from_type, ""}, to_type, &values);
+ part = values->create_with_offsets(offsets_data, to_type->get_default(), part_size,
+ /*shift=*/0);
+ }
+ }
+ result_column->insert_range_from(*part, 0, part_size);
+ }
+ data = {std::move(result_column)};
+ num_of_defaults_in_prefix = 0;
+}
+
+void ColumnObject::Subcolumn::insertDefault() {
+ if (data.empty()) {
+ ++num_of_defaults_in_prefix;
+ } else {
+ data.back()->insert_default();
+ }
+}
+
+void ColumnObject::Subcolumn::insertManyDefaults(size_t length) {
+ if (data.empty()) {
+ num_of_defaults_in_prefix += length;
+ } else {
+ data.back()->insert_many_defaults(length);
+ }
+}
+
+void ColumnObject::Subcolumn::pop_back(size_t n) {
+ assert(n <= size());
+ size_t num_removed = 0;
+ for (auto it = data.rbegin(); it != data.rend(); ++it) {
+ if (n == 0) break;
+ auto& column = *it;
+ if (n < column->size()) {
+ column->pop_back(n);
+ n = 0;
+ } else {
+ ++num_removed;
+ n -= column->size();
+ }
+ }
+ data.resize(data.size() - num_removed);
+ num_of_defaults_in_prefix -= n;
+}
+
+Field ColumnObject::Subcolumn::getLastField() const {
+ if (data.empty()) return Field();
+ const auto& last_part = data.back();
+ assert(!last_part->empty());
+ return (*last_part)[last_part->size() - 1];
+}
+
+ColumnObject::Subcolumn ColumnObject::Subcolumn::recreate_with_default_values(
+ const FieldInfo& field_info) const {
+ auto scalar_type = field_info.scalar_type;
+ if (is_nullable) {
+ scalar_type = make_nullable(scalar_type);
+ }
+ Subcolumn new_subcolumn;
+ new_subcolumn.least_common_type =
+ LeastCommonType {create_array_of_type(scalar_type, field_info.num_dimensions)};
+ new_subcolumn.is_nullable = is_nullable;
+ new_subcolumn.num_of_defaults_in_prefix = num_of_defaults_in_prefix;
+ new_subcolumn.data.reserve(data.size());
+ for (const auto& part : data) {
+ new_subcolumn.data.push_back(
+ recreate_column_with_default_value(part, scalar_type, field_info.num_dimensions));
+ }
+ return new_subcolumn;
+}
+
+IColumn& ColumnObject::Subcolumn::get_finalized_column() {
+ assert(is_finalized());
+ return *data[0];
+}
+
+const IColumn& ColumnObject::Subcolumn::get_finalized_column() const {
+ assert(is_finalized());
+ return *data[0];
+}
+
+const ColumnPtr& ColumnObject::Subcolumn::get_finalized_column_ptr() const {
+ assert(is_finalized());
+ return data[0];
+}
+
+ColumnObject::Subcolumn::LeastCommonType::LeastCommonType(DataTypePtr type_)
+ : type(std::move(type_)),
+ base_type(getBaseTypeOfArray(type)),
+ num_dimensions(getNumberOfDimensions(*type)) {}
+
+ColumnObject::ColumnObject(bool is_nullable_) : is_nullable(is_nullable_), num_rows(0) {}
+
+ColumnObject::ColumnObject(Subcolumns&& subcolumns_, bool is_nullable_)
+ : is_nullable(is_nullable_),
+ subcolumns(std::move(subcolumns_)),
+ num_rows(subcolumns.empty() ? 0 : (*subcolumns.begin())->data.size()) {
+ check_consistency();
+}
+
+void ColumnObject::check_consistency() const {
+ if (subcolumns.empty()) return;
+ for (const auto& leaf : subcolumns) {
+ if (num_rows != leaf->data.size()) {
+ assert(false);
+ }
+ }
+}
+
+size_t ColumnObject::size() const {
+#ifndef NDEBUG
+ check_consistency();
+#endif
+ return num_rows;
+}
+
+MutableColumnPtr ColumnObject::clone_resized(size_t new_size) const {
+ /// cloneResized with new_size == 0 is used for cloneEmpty().
+ if (new_size != 0) {
+ LOG(FATAL) << "ColumnObject doesn't support resize to non-zero length";
+ }
+ return ColumnObject::create(is_nullable);
+}
+
+size_t ColumnObject::byte_size() const {
+ size_t res = 0;
+ for (const auto& entry : subcolumns) {
+ res += entry->data.byteSize();
+ }
+ return res;
+}
+
+size_t ColumnObject::allocated_bytes() const {
+ size_t res = 0;
+ for (const auto& entry : subcolumns) {
+ res += entry->data.allocatedBytes();
+ }
+ return res;
+}
+
+void ColumnObject::for_each_subcolumn(ColumnCallback callback) {
+ if (!is_finalized()) assert(false);
+ for (auto& entry : subcolumns) {
+ callback(entry->data.data.back());
+ }
+}
+
+Status ColumnObject::try_insert_from(const IColumn& src, size_t n) {
+ return try_insert(src[n]);
+}
+
+Status ColumnObject::try_insert(const Field& field) {
+ const auto& object = field.get<const Object&>();
+ phmap::flat_hash_set<StringRef, StringRefHash> inserted;
+ size_t old_size = size();
+ for (const auto& [key_str, value] : object) {
+ PathInData key(key_str);
+ inserted.insert(key_str);
+ if (!has_subcolumn(key)) {
+ bool succ = add_sub_column(key, old_size);
+ if (!succ) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to add sub column {}", key.get_path()));
+ }
+ }
+ auto* subcolumn = get_subcolumn(key);
+ if (!subcolumn) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to find sub column {}", key.get_path()));
+ }
+ RETURN_IF_ERROR(subcolumn->insert(value));
+ }
+ for (auto& entry : subcolumns) {
+ if (!inserted.contains(entry->path.get_path())) {
+ entry->data.insertDefault();
+ }
+ }
+ ++num_rows;
+ return Status::OK();
+}
+
+void ColumnObject::insert_default() {
+ for (auto& entry : subcolumns) {
+ entry->data.insertDefault();
+ }
+ ++num_rows;
+}
+
+Field ColumnObject::operator[](size_t n) const {
+ if (!is_finalized()) assert(false);
+ Object object;
+ for (const auto& entry : subcolumns) {
+ object[entry->path.get_path()] = (*entry->data.data.back())[n];
+ }
+ return object;
+}
+
+void ColumnObject::get(size_t n, Field& res) const {
+ if (!is_finalized()) assert(false);
+ auto& object = res.get<Object&>();
+ for (const auto& entry : subcolumns) {
+ auto it = object.try_emplace(entry->path.get_path()).first;
+ entry->data.data.back()->get(n, it->second);
+ }
+}
+
+Status ColumnObject::try_insert_indices_from(const IColumn& src, const int* indices_begin,
+ const int* indices_end) {
+ for (auto x = indices_begin; x != indices_end; ++x) {
+ if (*x == -1) {
+ ColumnObject::insert_default();
+ } else {
+ RETURN_IF_ERROR(ColumnObject::try_insert_from(src, *x));
+ }
+ }
+ finalize();
+ return Status::OK();
+}
+
+Status ColumnObject::try_insert_range_from(const IColumn& src, size_t start, size_t length) {
+ const auto& src_object = assert_cast<const ColumnObject&>(src);
+ if (UNLIKELY(src_object.empty())) {
+ return Status::OK();
+ }
+ for (auto& entry : subcolumns) {
+ if (src_object.has_subcolumn(entry->path)) {
+ auto* subcolumn = src_object.get_subcolumn(entry->path);
+ if (!subcolumn) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to find sub column {}", entry->path.get_path()));
+ }
+ RETURN_IF_ERROR(entry->data.insertRangeFrom(*subcolumn, start, length));
+ } else {
+ entry->data.insertManyDefaults(length);
+ }
+ }
+ for (const auto& entry : src_object.subcolumns) {
+ if (!has_subcolumn(entry->path)) {
+ bool succ = false;
+ if (entry->path.has_nested_part()) {
+ const auto& base_type = entry->data.get_least_common_typeBase();
+ FieldInfo field_info {
+ .scalar_type = base_type,
+ .have_nulls = base_type->is_nullable(),
+ .need_convert = false,
+ .num_dimensions = entry->data.get_dimensions(),
+ };
+ succ = add_nested_subcolumn(entry->path, field_info, num_rows);
+ } else {
+ succ = add_sub_column(entry->path, num_rows);
+ }
+ if (!succ) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to add column {}", entry->path.get_path()));
+ }
+ auto* subcolumn = get_subcolumn(entry->path);
+ if (!subcolumn) {
+ return Status::InvalidArgument(
+ fmt::format("Failed to find sub column {}", entry->path.get_path()));
+ }
+ RETURN_IF_ERROR(subcolumn->insertRangeFrom(entry->data, start, length));
+ }
+ }
+ num_rows += length;
+ finalize();
+ return Status::OK();
+}
+
+void ColumnObject::pop_back(size_t length) {
+ for (auto& entry : subcolumns) entry->data.pop_back(length);
+ num_rows -= length;
+}
+
+const ColumnObject::Subcolumn* ColumnObject::get_subcolumn(const PathInData& key) const {
+ const auto* node = subcolumns.find_leaf(key);
+ if (node == nullptr) {
+ VLOG_DEBUG << "There is no subcolumn " << key.get_path();
+ return nullptr;
+ }
+ return &node->data;
+}
+
+ColumnObject::Subcolumn* ColumnObject::get_subcolumn(const PathInData& key) {
+ const auto* node = subcolumns.find_leaf(key);
+ if (node == nullptr) {
+ VLOG_DEBUG << "There is no subcolumn " << key.get_path();
+ return nullptr;
+ }
+ return &const_cast<Subcolumns::Node*>(node)->data;
+}
+
+bool ColumnObject::has_subcolumn(const PathInData& key) const {
+ return subcolumns.find_leaf(key) != nullptr;
+}
+
+bool ColumnObject::add_sub_column(const PathInData& key, MutableColumnPtr&& subcolumn) {
+ size_t new_size = subcolumn->size();
+ bool inserted = subcolumns.add(key, Subcolumn(std::move(subcolumn), is_nullable));
+ if (!inserted) {
+ VLOG_DEBUG << "Duplicated sub column " << key.get_path();
+ return false;
+ }
+ if (num_rows == 0) {
+ num_rows = new_size;
+ } else if (new_size != num_rows) {
+ VLOG_DEBUG << "Size of subcolumn is in consistent with column";
+ return false;
+ }
+ return true;
+}
+
+bool ColumnObject::add_sub_column(const PathInData& key, size_t new_size) {
+ bool inserted = subcolumns.add(key, Subcolumn(new_size, is_nullable));
+ if (!inserted) {
+ VLOG_DEBUG << "Duplicated sub column " << key.get_path();
+ return false;
+ }
+ if (num_rows == 0) {
+ num_rows = new_size;
+ } else if (new_size != num_rows) {
+ VLOG_DEBUG << "Size of subcolumn is in consistent with column";
+ return false;
+ }
+ return true;
+}
+
+bool ColumnObject::add_nested_subcolumn(const PathInData& key, const FieldInfo& field_info,
+ size_t new_size) {
+ assert(key.has_nested_part());
+ bool inserted = false;
+ /// We find node that represents the same Nested type as @key.
+ const auto* nested_node = subcolumns.find_best_match(key);
+ if (nested_node) {
+ /// Find any leaf of Nested subcolumn.
+ const auto* leaf = subcolumns.find_leaf(nested_node, [&](const auto&) { return true; });
+ assert(leaf);
+ /// Recreate subcolumn with default values and the same sizes of arrays.
+ auto new_subcolumn = leaf->data.recreate_with_default_values(field_info);
+ /// It's possible that we have already inserted value from current row
+ /// to this subcolumn. So, adjust size to expected.
+ if (new_subcolumn.size() > new_size)
+ new_subcolumn.pop_back(new_subcolumn.size() - new_size);
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (new_subcolumn.size() > new_size) {
new_subcolumn.pop_back(new_subcolumn.size() - new_size);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16319: [DRAFT] Dynamic table
Posted by github-actions.
github-actions[bot] commented on code in PR #16319:
URL: https://github.com/apache/doris/pull/16319#discussion_r1092848363
##########
be/src/exec/base_scanner.cpp:
##########
@@ -251,6 +266,28 @@
std::move(column_ptr), slot_desc->get_data_type_ptr(), slot_desc->col_name()));
}
+ // handle dynamic generated columns
+ if (!_full_base_schema_view->empty()) {
+ assert(_is_dynamic_schema);
+ for (size_t x = dest_block->columns(); x < _src_block.columns(); ++x) {
+ auto& column_type_name = _src_block.get_by_position(x);
+ const TColumn& tcolumn =
+ _full_base_schema_view->column_name_to_column[column_type_name.name];
+ auto original_type = vectorized::DataTypeFactory::instance().create_data_type(tcolumn);
+ // type conflict free path, always cast to original type
+ if (!column_type_name.type->equals(*original_type)) {
+ vectorized::ColumnPtr column_ptr;
+ RETURN_IF_ERROR(vectorized::object_util::cast_column(column_type_name,
+ original_type, &column_ptr));
+ column_type_name.column = column_ptr;
+ column_type_name.type = original_type;
+ }
+ dest_block->insert(vectorized::ColumnWithTypeAndName(std::move(column_type_name.column),
+ std::move(column_type_name.type),
Review Comment:
warning: passing result of std::move() as a const reference argument; no move will actually happen [performance-move-const-arg]
```suggestion
column_type_name.type,
```
##########
be/src/exec/base_scanner.cpp:
##########
@@ -251,6 +266,28 @@ Status BaseScanner::_materialize_dest_block(vectorized::Block* dest_block) {
std::move(column_ptr), slot_desc->get_data_type_ptr(), slot_desc->col_name()));
}
+ // handle dynamic generated columns
+ if (!_full_base_schema_view->empty()) {
+ assert(_is_dynamic_schema);
+ for (size_t x = dest_block->columns(); x < _src_block.columns(); ++x) {
+ auto& column_type_name = _src_block.get_by_position(x);
+ const TColumn& tcolumn =
+ _full_base_schema_view->column_name_to_column[column_type_name.name];
+ auto original_type = vectorized::DataTypeFactory::instance().create_data_type(tcolumn);
+ // type conflict free path, always cast to original type
+ if (!column_type_name.type->equals(*original_type)) {
+ vectorized::ColumnPtr column_ptr;
+ RETURN_IF_ERROR(vectorized::object_util::cast_column(column_type_name,
+ original_type, &column_ptr));
+ column_type_name.column = column_ptr;
+ column_type_name.type = original_type;
+ }
+ dest_block->insert(vectorized::ColumnWithTypeAndName(std::move(column_type_name.column),
Review Comment:
warning: passing result of std::move() as a const reference argument; no move will actually happen [performance-move-const-arg]
```suggestion
dest_block->insert(vectorized::ColumnWithTypeAndName(column_type_name.column,
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ return;
+ }
+
+ const char* chars = reinterpret_cast<const char*>(value);
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN:
+ stream->write(chars, sizeof(bool));
+ return;
+
+ case TYPE_TINYINT:
+ stream->write(chars, sizeof(int8_t));
+ break;
+
+ case TYPE_SMALLINT:
+ stream->write(chars, sizeof(int16_t));
+ break;
+
+ case TYPE_INT:
+ stream->write(chars, sizeof(int32_t));
+ break;
+
+ case TYPE_BIGINT:
+ stream->write(chars, sizeof(int64_t));
+ break;
+
+ case TYPE_FLOAT:
+ stream->write(chars, sizeof(float));
+ break;
+
+ case TYPE_DOUBLE:
+ stream->write(chars, sizeof(double));
+ break;
+
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING:
+ string_val = reinterpret_cast<const StringValue*>(value);
Review Comment:
warning: unknown type name 'StringValue'; did you mean 'StringVal'? [clang-diagnostic-error]
```suggestion
string_val = reinterpret_cast<const StringVal*>(value);
```
**be/src/udf/udf.h:393:** 'StringVal' declared here
```cpp
struct StringVal : public AnyVal {
^
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
Review Comment:
warning: out-of-line definition of 'print_value_as_bytes' does not match any declaration in 'doris::RawValue' [clang-diagnostic-error]
```cpp
void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
^
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ return;
+ }
+
+ const char* chars = reinterpret_cast<const char*>(value);
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN:
+ stream->write(chars, sizeof(bool));
+ return;
+
+ case TYPE_TINYINT:
+ stream->write(chars, sizeof(int8_t));
+ break;
+
+ case TYPE_SMALLINT:
+ stream->write(chars, sizeof(int16_t));
+ break;
+
+ case TYPE_INT:
+ stream->write(chars, sizeof(int32_t));
+ break;
+
+ case TYPE_BIGINT:
+ stream->write(chars, sizeof(int64_t));
+ break;
+
+ case TYPE_FLOAT:
+ stream->write(chars, sizeof(float));
+ break;
+
+ case TYPE_DOUBLE:
+ stream->write(chars, sizeof(double));
+ break;
+
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ stream->write(static_cast<char*>(string_val->ptr), string_val->len);
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ stream->write(chars, sizeof(DateTimeValue));
+ break;
+
+ case TYPE_DATEV2:
+ stream->write(chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>));
+ break;
+
+ case TYPE_DATETIMEV2:
+ stream->write(
+ chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>));
+ break;
+
+ case TYPE_DECIMALV2:
+ stream->write(chars, sizeof(DecimalV2Value));
+ break;
+
+ case TYPE_DECIMAL32:
+ stream->write(chars, 4);
+ break;
+
+ case TYPE_DECIMAL64:
+ stream->write(chars, 8);
+ break;
+
+ case TYPE_DECIMAL128:
+ stream->write(chars, 16);
+ break;
+
+ case TYPE_LARGEINT:
+ stream->write(chars, sizeof(__int128));
+ break;
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ *stream << "NULL";
+ return;
+ }
+
+ int old_precision = stream->precision();
+ std::ios_base::fmtflags old_flags = stream->flags();
+
+ if (scale > -1) {
+ stream->precision(scale);
+ // Setting 'fixed' causes precision to set the number of digits printed after the
+ // decimal (by default it sets the maximum number of digits total).
+ *stream << std::fixed;
+ }
+
+ std::string tmp;
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_BOOLEAN: {
+ bool val = *reinterpret_cast<const bool*>(value);
+ *stream << (val ? "true" : "false");
+ return;
+ }
+
+ case TYPE_TINYINT:
+ // Extra casting for chars since they should not be interpreted as ASCII.
+ *stream << static_cast<int>(*reinterpret_cast<const int8_t*>(value));
+ break;
+
+ case TYPE_SMALLINT:
+ *stream << *reinterpret_cast<const int16_t*>(value);
+ break;
+
+ case TYPE_INT:
+ *stream << *reinterpret_cast<const int32_t*>(value);
+ break;
+
+ case TYPE_BIGINT:
+ *stream << *reinterpret_cast<const int64_t*>(value);
+ break;
+
+ case TYPE_FLOAT:
+ *stream << *reinterpret_cast<const float*>(value);
+ break;
+
+ case TYPE_DOUBLE:
+ *stream << *reinterpret_cast<const double*>(value);
+ break;
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_STRING:
+ case TYPE_VARIANT:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ tmp.assign(static_cast<char*>(string_val->ptr), string_val->len);
Review Comment:
warning: static_cast from 'uint8_t *' (aka 'unsigned char *') to 'char *' is not allowed [clang-diagnostic-error]
```cpp
tmp.assign(static_cast<char*>(string_val->ptr), string_val->len);
^
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ return;
+ }
+
+ const char* chars = reinterpret_cast<const char*>(value);
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN:
+ stream->write(chars, sizeof(bool));
+ return;
+
+ case TYPE_TINYINT:
+ stream->write(chars, sizeof(int8_t));
+ break;
+
+ case TYPE_SMALLINT:
+ stream->write(chars, sizeof(int16_t));
+ break;
+
+ case TYPE_INT:
+ stream->write(chars, sizeof(int32_t));
+ break;
+
+ case TYPE_BIGINT:
+ stream->write(chars, sizeof(int64_t));
+ break;
+
+ case TYPE_FLOAT:
+ stream->write(chars, sizeof(float));
+ break;
+
+ case TYPE_DOUBLE:
+ stream->write(chars, sizeof(double));
+ break;
+
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ stream->write(static_cast<char*>(string_val->ptr), string_val->len);
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ stream->write(chars, sizeof(DateTimeValue));
+ break;
+
+ case TYPE_DATEV2:
+ stream->write(chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>));
+ break;
+
+ case TYPE_DATETIMEV2:
+ stream->write(
+ chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>));
+ break;
+
+ case TYPE_DECIMALV2:
+ stream->write(chars, sizeof(DecimalV2Value));
+ break;
+
+ case TYPE_DECIMAL32:
+ stream->write(chars, 4);
+ break;
+
+ case TYPE_DECIMAL64:
+ stream->write(chars, 8);
+ break;
+
+ case TYPE_DECIMAL128:
+ stream->write(chars, 16);
+ break;
+
+ case TYPE_LARGEINT:
+ stream->write(chars, sizeof(__int128));
+ break;
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ *stream << "NULL";
+ return;
+ }
+
+ int old_precision = stream->precision();
+ std::ios_base::fmtflags old_flags = stream->flags();
+
+ if (scale > -1) {
+ stream->precision(scale);
+ // Setting 'fixed' causes precision to set the number of digits printed after the
+ // decimal (by default it sets the maximum number of digits total).
+ *stream << std::fixed;
+ }
+
+ std::string tmp;
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_BOOLEAN: {
+ bool val = *reinterpret_cast<const bool*>(value);
+ *stream << (val ? "true" : "false");
+ return;
+ }
+
+ case TYPE_TINYINT:
+ // Extra casting for chars since they should not be interpreted as ASCII.
+ *stream << static_cast<int>(*reinterpret_cast<const int8_t*>(value));
+ break;
+
+ case TYPE_SMALLINT:
+ *stream << *reinterpret_cast<const int16_t*>(value);
+ break;
+
+ case TYPE_INT:
+ *stream << *reinterpret_cast<const int32_t*>(value);
+ break;
+
+ case TYPE_BIGINT:
+ *stream << *reinterpret_cast<const int64_t*>(value);
+ break;
+
+ case TYPE_FLOAT:
+ *stream << *reinterpret_cast<const float*>(value);
+ break;
+
+ case TYPE_DOUBLE:
+ *stream << *reinterpret_cast<const double*>(value);
+ break;
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_STRING:
+ case TYPE_VARIANT:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ tmp.assign(static_cast<char*>(string_val->ptr), string_val->len);
+ *stream << tmp;
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ *stream << *reinterpret_cast<const DateTimeValue*>(value);
+ break;
+
+ case TYPE_DATEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>*>(value);
+ break;
+
+ case TYPE_DATETIMEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>*>(
+ value);
+ break;
+
+ case TYPE_DECIMALV2:
+ *stream << DecimalV2Value(reinterpret_cast<const PackedInt128*>(value)->value).to_string();
+ break;
+
+ case TYPE_DECIMAL32: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal32*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL64: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal64*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL128: {
Review Comment:
warning: use of undeclared identifier 'TYPE_DECIMAL128'; did you mean 'TYPE_DECIMAL128I'? [clang-diagnostic-error]
```suggestion
case TYPE_DECIMAL128I: {
```
**be/src/runtime/define_primitive_type.h:54:** 'TYPE_DECIMAL128I' declared here
```cpp
TYPE_DECIMAL128I, /* 30 */
^
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ return;
+ }
+
+ const char* chars = reinterpret_cast<const char*>(value);
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN:
+ stream->write(chars, sizeof(bool));
+ return;
+
+ case TYPE_TINYINT:
+ stream->write(chars, sizeof(int8_t));
+ break;
+
+ case TYPE_SMALLINT:
+ stream->write(chars, sizeof(int16_t));
+ break;
+
+ case TYPE_INT:
+ stream->write(chars, sizeof(int32_t));
+ break;
+
+ case TYPE_BIGINT:
+ stream->write(chars, sizeof(int64_t));
+ break;
+
+ case TYPE_FLOAT:
+ stream->write(chars, sizeof(float));
+ break;
+
+ case TYPE_DOUBLE:
+ stream->write(chars, sizeof(double));
+ break;
+
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ stream->write(static_cast<char*>(string_val->ptr), string_val->len);
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ stream->write(chars, sizeof(DateTimeValue));
+ break;
+
+ case TYPE_DATEV2:
+ stream->write(chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>));
+ break;
+
+ case TYPE_DATETIMEV2:
+ stream->write(
+ chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>));
+ break;
+
+ case TYPE_DECIMALV2:
+ stream->write(chars, sizeof(DecimalV2Value));
+ break;
+
+ case TYPE_DECIMAL32:
+ stream->write(chars, 4);
+ break;
+
+ case TYPE_DECIMAL64:
+ stream->write(chars, 8);
+ break;
+
+ case TYPE_DECIMAL128:
+ stream->write(chars, 16);
+ break;
+
+ case TYPE_LARGEINT:
+ stream->write(chars, sizeof(__int128));
+ break;
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ *stream << "NULL";
+ return;
+ }
+
+ int old_precision = stream->precision();
+ std::ios_base::fmtflags old_flags = stream->flags();
+
+ if (scale > -1) {
+ stream->precision(scale);
+ // Setting 'fixed' causes precision to set the number of digits printed after the
+ // decimal (by default it sets the maximum number of digits total).
+ *stream << std::fixed;
+ }
+
+ std::string tmp;
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_BOOLEAN: {
+ bool val = *reinterpret_cast<const bool*>(value);
+ *stream << (val ? "true" : "false");
+ return;
+ }
+
+ case TYPE_TINYINT:
+ // Extra casting for chars since they should not be interpreted as ASCII.
+ *stream << static_cast<int>(*reinterpret_cast<const int8_t*>(value));
+ break;
+
+ case TYPE_SMALLINT:
+ *stream << *reinterpret_cast<const int16_t*>(value);
+ break;
+
+ case TYPE_INT:
+ *stream << *reinterpret_cast<const int32_t*>(value);
+ break;
+
+ case TYPE_BIGINT:
+ *stream << *reinterpret_cast<const int64_t*>(value);
+ break;
+
+ case TYPE_FLOAT:
+ *stream << *reinterpret_cast<const float*>(value);
+ break;
+
+ case TYPE_DOUBLE:
+ *stream << *reinterpret_cast<const double*>(value);
+ break;
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_STRING:
+ case TYPE_VARIANT:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ tmp.assign(static_cast<char*>(string_val->ptr), string_val->len);
+ *stream << tmp;
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ *stream << *reinterpret_cast<const DateTimeValue*>(value);
+ break;
+
+ case TYPE_DATEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>*>(value);
+ break;
+
+ case TYPE_DATETIMEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>*>(
+ value);
+ break;
+
+ case TYPE_DECIMALV2:
+ *stream << DecimalV2Value(reinterpret_cast<const PackedInt128*>(value)->value).to_string();
+ break;
+
+ case TYPE_DECIMAL32: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal32*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL64: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal64*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL128: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal128*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_LARGEINT:
+ *stream << reinterpret_cast<const PackedInt128*>(value)->value;
+ break;
+
+ case TYPE_ARRAY: {
+ const CollectionValue* src = reinterpret_cast<const CollectionValue*>(value);
+ auto children_type = type.children.at(0);
+ auto iter = src->iterator(children_type.type);
+ *stream << "[";
+ print_value(iter.get(), children_type, scale, stream);
+ iter.next();
+ for (; iter.has_next(); iter.next()) {
+ *stream << ", ";
+ print_value(iter.get(), children_type, scale, stream);
+ }
+ *stream << "]";
+ break;
+ }
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+
+ stream->precision(old_precision);
+ // Undo setting stream to fixed
+ stream->flags(old_flags);
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::string* str) {
+ if (value == nullptr) {
+ *str = "NULL";
+ return;
+ }
+
+ std::stringstream out;
+ out.precision(ASCII_PRECISION);
+ const StringValue* string_val = nullptr;
+ std::string tmp;
+ bool val = false;
+
+ // Special case types that we can print more efficiently without using a std::stringstream
+ switch (type.type) {
+ case TYPE_BOOLEAN:
+ val = *reinterpret_cast<const bool*>(value);
+ *str = (val ? "true" : "false");
+ return;
+
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_OBJECT:
+ case TYPE_HLL:
+ case TYPE_QUANTILE_STATE:
+ case TYPE_STRING: {
+ string_val = reinterpret_cast<const StringValue*>(value);
+ std::stringstream ss;
+ ss << "ptr:" << (void*)string_val->ptr << " len:" << string_val->len;
+ tmp = ss.str();
+ if (string_val->len <= 1000) {
+ tmp.assign(static_cast<char*>(string_val->ptr), string_val->len);
Review Comment:
warning: static_cast from 'uint8_t *' (aka 'unsigned char *') to 'char *' is not allowed [clang-diagnostic-error]
```cpp
tmp.assign(static_cast<char*>(string_val->ptr), string_val->len);
^
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ return;
+ }
+
+ const char* chars = reinterpret_cast<const char*>(value);
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN:
+ stream->write(chars, sizeof(bool));
+ return;
+
+ case TYPE_TINYINT:
+ stream->write(chars, sizeof(int8_t));
+ break;
+
+ case TYPE_SMALLINT:
+ stream->write(chars, sizeof(int16_t));
+ break;
+
+ case TYPE_INT:
+ stream->write(chars, sizeof(int32_t));
+ break;
+
+ case TYPE_BIGINT:
+ stream->write(chars, sizeof(int64_t));
+ break;
+
+ case TYPE_FLOAT:
+ stream->write(chars, sizeof(float));
+ break;
+
+ case TYPE_DOUBLE:
+ stream->write(chars, sizeof(double));
+ break;
+
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ stream->write(static_cast<char*>(string_val->ptr), string_val->len);
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ stream->write(chars, sizeof(DateTimeValue));
+ break;
+
+ case TYPE_DATEV2:
+ stream->write(chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>));
+ break;
+
+ case TYPE_DATETIMEV2:
+ stream->write(
+ chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>));
+ break;
+
+ case TYPE_DECIMALV2:
+ stream->write(chars, sizeof(DecimalV2Value));
+ break;
+
+ case TYPE_DECIMAL32:
+ stream->write(chars, 4);
+ break;
+
+ case TYPE_DECIMAL64:
+ stream->write(chars, 8);
+ break;
+
+ case TYPE_DECIMAL128:
+ stream->write(chars, 16);
+ break;
+
+ case TYPE_LARGEINT:
+ stream->write(chars, sizeof(__int128));
+ break;
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ *stream << "NULL";
+ return;
+ }
+
+ int old_precision = stream->precision();
+ std::ios_base::fmtflags old_flags = stream->flags();
+
+ if (scale > -1) {
+ stream->precision(scale);
+ // Setting 'fixed' causes precision to set the number of digits printed after the
+ // decimal (by default it sets the maximum number of digits total).
+ *stream << std::fixed;
+ }
+
+ std::string tmp;
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_BOOLEAN: {
+ bool val = *reinterpret_cast<const bool*>(value);
+ *stream << (val ? "true" : "false");
+ return;
+ }
+
+ case TYPE_TINYINT:
+ // Extra casting for chars since they should not be interpreted as ASCII.
+ *stream << static_cast<int>(*reinterpret_cast<const int8_t*>(value));
+ break;
+
+ case TYPE_SMALLINT:
+ *stream << *reinterpret_cast<const int16_t*>(value);
+ break;
+
+ case TYPE_INT:
+ *stream << *reinterpret_cast<const int32_t*>(value);
+ break;
+
+ case TYPE_BIGINT:
+ *stream << *reinterpret_cast<const int64_t*>(value);
+ break;
+
+ case TYPE_FLOAT:
+ *stream << *reinterpret_cast<const float*>(value);
+ break;
+
+ case TYPE_DOUBLE:
+ *stream << *reinterpret_cast<const double*>(value);
+ break;
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_STRING:
+ case TYPE_VARIANT:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ tmp.assign(static_cast<char*>(string_val->ptr), string_val->len);
+ *stream << tmp;
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ *stream << *reinterpret_cast<const DateTimeValue*>(value);
+ break;
+
+ case TYPE_DATEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>*>(value);
+ break;
+
+ case TYPE_DATETIMEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>*>(
+ value);
+ break;
+
+ case TYPE_DECIMALV2:
+ *stream << DecimalV2Value(reinterpret_cast<const PackedInt128*>(value)->value).to_string();
+ break;
+
+ case TYPE_DECIMAL32: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal32*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL64: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal64*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL128: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal128*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_LARGEINT:
+ *stream << reinterpret_cast<const PackedInt128*>(value)->value;
+ break;
+
+ case TYPE_ARRAY: {
+ const CollectionValue* src = reinterpret_cast<const CollectionValue*>(value);
+ auto children_type = type.children.at(0);
+ auto iter = src->iterator(children_type.type);
+ *stream << "[";
+ print_value(iter.get(), children_type, scale, stream);
+ iter.next();
+ for (; iter.has_next(); iter.next()) {
+ *stream << ", ";
+ print_value(iter.get(), children_type, scale, stream);
+ }
+ *stream << "]";
+ break;
+ }
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+
+ stream->precision(old_precision);
+ // Undo setting stream to fixed
+ stream->flags(old_flags);
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::string* str) {
+ if (value == nullptr) {
+ *str = "NULL";
+ return;
+ }
+
+ std::stringstream out;
+ out.precision(ASCII_PRECISION);
+ const StringValue* string_val = nullptr;
+ std::string tmp;
+ bool val = false;
+
+ // Special case types that we can print more efficiently without using a std::stringstream
+ switch (type.type) {
+ case TYPE_BOOLEAN:
+ val = *reinterpret_cast<const bool*>(value);
+ *str = (val ? "true" : "false");
+ return;
+
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_OBJECT:
+ case TYPE_HLL:
+ case TYPE_QUANTILE_STATE:
+ case TYPE_STRING: {
+ string_val = reinterpret_cast<const StringValue*>(value);
+ std::stringstream ss;
+ ss << "ptr:" << (void*)string_val->ptr << " len:" << string_val->len;
+ tmp = ss.str();
+ if (string_val->len <= 1000) {
+ tmp.assign(static_cast<char*>(string_val->ptr), string_val->len);
+ }
+ str->swap(tmp);
+ return;
+ }
+ case TYPE_NULL: {
+ *str = "NULL";
+ return;
+ }
+ default:
+ print_value(value, type, scale, &out);
+ }
+
+ *str = out.str();
+}
+
+void RawValue::write(const void* value, void* dst, const TypeDescriptor& type, MemPool* pool) {
+ DCHECK(value != nullptr);
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN: {
+ *reinterpret_cast<bool*>(dst) = *reinterpret_cast<const bool*>(value);
+ break;
+ }
+
+ case TYPE_TINYINT: {
+ *reinterpret_cast<int8_t*>(dst) = *reinterpret_cast<const int8_t*>(value);
+ break;
+ }
+
+ case TYPE_SMALLINT: {
+ *reinterpret_cast<int16_t*>(dst) = *reinterpret_cast<const int16_t*>(value);
+ break;
+ }
+
+ case TYPE_INT: {
+ *reinterpret_cast<int32_t*>(dst) = *reinterpret_cast<const int32_t*>(value);
+ break;
+ }
+
+ case TYPE_BIGINT: {
+ *reinterpret_cast<int64_t*>(dst) = *reinterpret_cast<const int64_t*>(value);
+ break;
+ }
+
+ case TYPE_LARGEINT: {
+ *reinterpret_cast<PackedInt128*>(dst) = *reinterpret_cast<const PackedInt128*>(value);
+ break;
+ }
+
+ case TYPE_FLOAT: {
+ *reinterpret_cast<float*>(dst) = *reinterpret_cast<const float*>(value);
+ break;
+ }
+
+ case TYPE_TIME:
+ case TYPE_DOUBLE: {
+ *reinterpret_cast<double*>(dst) = *reinterpret_cast<const double*>(value);
+ break;
+ }
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ *reinterpret_cast<DateTimeValue*>(dst) = *reinterpret_cast<const DateTimeValue*>(value);
+ break;
+
+ case TYPE_DATEV2:
+ *reinterpret_cast<doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>*>(
+ dst) =
+ *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>*>(
+ value);
+ break;
+
+ case TYPE_DATETIMEV2:
+ *reinterpret_cast<doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>*>(
+ dst) =
+ *reinterpret_cast<const doris::vectorized::DateV2Value<
+ doris::vectorized::DateTimeV2ValueType>*>(value);
+ break;
+
+ case TYPE_DECIMALV2:
+ *reinterpret_cast<PackedInt128*>(dst) = *reinterpret_cast<const PackedInt128*>(value);
+ break;
+
+ case TYPE_DECIMAL32:
+ *reinterpret_cast<doris::vectorized::Decimal32*>(dst) =
+ *reinterpret_cast<const doris::vectorized::Decimal32*>(value);
+ break;
+ case TYPE_DECIMAL64:
+ *reinterpret_cast<doris::vectorized::Decimal64*>(dst) =
+ *reinterpret_cast<const doris::vectorized::Decimal64*>(value);
+ break;
+ case TYPE_DECIMAL128:
+ *reinterpret_cast<doris::vectorized::Decimal128*>(dst) =
+ *reinterpret_cast<const doris::vectorized::Decimal128*>(value);
+ break;
+
+ case TYPE_OBJECT:
+ case TYPE_HLL:
+ case TYPE_QUANTILE_STATE:
+ case TYPE_VARCHAR:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING: {
+ const StringValue* src = reinterpret_cast<const StringValue*>(value);
Review Comment:
warning: unknown type name 'StringValue'; did you mean 'StringVal'? [clang-diagnostic-error]
```suggestion
const StringVal* src = reinterpret_cast<const StringValue*>(value);
```
**be/src/udf/udf.h:393:** 'StringVal' declared here
```cpp
struct StringVal : public AnyVal {
^
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ return;
+ }
+
+ const char* chars = reinterpret_cast<const char*>(value);
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN:
+ stream->write(chars, sizeof(bool));
+ return;
+
+ case TYPE_TINYINT:
+ stream->write(chars, sizeof(int8_t));
+ break;
+
+ case TYPE_SMALLINT:
+ stream->write(chars, sizeof(int16_t));
+ break;
+
+ case TYPE_INT:
+ stream->write(chars, sizeof(int32_t));
+ break;
+
+ case TYPE_BIGINT:
+ stream->write(chars, sizeof(int64_t));
+ break;
+
+ case TYPE_FLOAT:
+ stream->write(chars, sizeof(float));
+ break;
+
+ case TYPE_DOUBLE:
+ stream->write(chars, sizeof(double));
+ break;
+
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ stream->write(static_cast<char*>(string_val->ptr), string_val->len);
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ stream->write(chars, sizeof(DateTimeValue));
+ break;
+
+ case TYPE_DATEV2:
+ stream->write(chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>));
+ break;
+
+ case TYPE_DATETIMEV2:
+ stream->write(
+ chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>));
+ break;
+
+ case TYPE_DECIMALV2:
+ stream->write(chars, sizeof(DecimalV2Value));
+ break;
+
+ case TYPE_DECIMAL32:
+ stream->write(chars, 4);
+ break;
+
+ case TYPE_DECIMAL64:
+ stream->write(chars, 8);
+ break;
+
+ case TYPE_DECIMAL128:
+ stream->write(chars, 16);
+ break;
+
+ case TYPE_LARGEINT:
+ stream->write(chars, sizeof(__int128));
+ break;
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ *stream << "NULL";
+ return;
+ }
+
+ int old_precision = stream->precision();
+ std::ios_base::fmtflags old_flags = stream->flags();
+
+ if (scale > -1) {
+ stream->precision(scale);
+ // Setting 'fixed' causes precision to set the number of digits printed after the
+ // decimal (by default it sets the maximum number of digits total).
+ *stream << std::fixed;
+ }
+
+ std::string tmp;
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_BOOLEAN: {
+ bool val = *reinterpret_cast<const bool*>(value);
+ *stream << (val ? "true" : "false");
+ return;
+ }
+
+ case TYPE_TINYINT:
+ // Extra casting for chars since they should not be interpreted as ASCII.
+ *stream << static_cast<int>(*reinterpret_cast<const int8_t*>(value));
+ break;
+
+ case TYPE_SMALLINT:
+ *stream << *reinterpret_cast<const int16_t*>(value);
+ break;
+
+ case TYPE_INT:
+ *stream << *reinterpret_cast<const int32_t*>(value);
+ break;
+
+ case TYPE_BIGINT:
+ *stream << *reinterpret_cast<const int64_t*>(value);
+ break;
+
+ case TYPE_FLOAT:
+ *stream << *reinterpret_cast<const float*>(value);
+ break;
+
+ case TYPE_DOUBLE:
+ *stream << *reinterpret_cast<const double*>(value);
+ break;
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_STRING:
+ case TYPE_VARIANT:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ tmp.assign(static_cast<char*>(string_val->ptr), string_val->len);
+ *stream << tmp;
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ *stream << *reinterpret_cast<const DateTimeValue*>(value);
+ break;
+
+ case TYPE_DATEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>*>(value);
+ break;
+
+ case TYPE_DATETIMEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>*>(
+ value);
+ break;
+
+ case TYPE_DECIMALV2:
+ *stream << DecimalV2Value(reinterpret_cast<const PackedInt128*>(value)->value).to_string();
+ break;
+
+ case TYPE_DECIMAL32: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal32*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL64: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal64*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL128: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal128*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_LARGEINT:
+ *stream << reinterpret_cast<const PackedInt128*>(value)->value;
+ break;
+
+ case TYPE_ARRAY: {
+ const CollectionValue* src = reinterpret_cast<const CollectionValue*>(value);
+ auto children_type = type.children.at(0);
+ auto iter = src->iterator(children_type.type);
+ *stream << "[";
+ print_value(iter.get(), children_type, scale, stream);
+ iter.next();
+ for (; iter.has_next(); iter.next()) {
+ *stream << ", ";
+ print_value(iter.get(), children_type, scale, stream);
+ }
+ *stream << "]";
+ break;
+ }
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+
+ stream->precision(old_precision);
+ // Undo setting stream to fixed
+ stream->flags(old_flags);
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::string* str) {
+ if (value == nullptr) {
+ *str = "NULL";
+ return;
+ }
+
+ std::stringstream out;
+ out.precision(ASCII_PRECISION);
+ const StringValue* string_val = nullptr;
+ std::string tmp;
+ bool val = false;
+
+ // Special case types that we can print more efficiently without using a std::stringstream
+ switch (type.type) {
+ case TYPE_BOOLEAN:
+ val = *reinterpret_cast<const bool*>(value);
+ *str = (val ? "true" : "false");
+ return;
+
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_OBJECT:
+ case TYPE_HLL:
+ case TYPE_QUANTILE_STATE:
+ case TYPE_STRING: {
+ string_val = reinterpret_cast<const StringValue*>(value);
+ std::stringstream ss;
+ ss << "ptr:" << (void*)string_val->ptr << " len:" << string_val->len;
+ tmp = ss.str();
+ if (string_val->len <= 1000) {
+ tmp.assign(static_cast<char*>(string_val->ptr), string_val->len);
+ }
+ str->swap(tmp);
+ return;
+ }
+ case TYPE_NULL: {
+ *str = "NULL";
+ return;
+ }
+ default:
+ print_value(value, type, scale, &out);
+ }
+
+ *str = out.str();
+}
+
+void RawValue::write(const void* value, void* dst, const TypeDescriptor& type, MemPool* pool) {
+ DCHECK(value != nullptr);
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN: {
+ *reinterpret_cast<bool*>(dst) = *reinterpret_cast<const bool*>(value);
+ break;
+ }
+
+ case TYPE_TINYINT: {
+ *reinterpret_cast<int8_t*>(dst) = *reinterpret_cast<const int8_t*>(value);
+ break;
+ }
+
+ case TYPE_SMALLINT: {
+ *reinterpret_cast<int16_t*>(dst) = *reinterpret_cast<const int16_t*>(value);
+ break;
+ }
+
+ case TYPE_INT: {
+ *reinterpret_cast<int32_t*>(dst) = *reinterpret_cast<const int32_t*>(value);
+ break;
+ }
+
+ case TYPE_BIGINT: {
+ *reinterpret_cast<int64_t*>(dst) = *reinterpret_cast<const int64_t*>(value);
+ break;
+ }
+
+ case TYPE_LARGEINT: {
+ *reinterpret_cast<PackedInt128*>(dst) = *reinterpret_cast<const PackedInt128*>(value);
+ break;
+ }
+
+ case TYPE_FLOAT: {
+ *reinterpret_cast<float*>(dst) = *reinterpret_cast<const float*>(value);
+ break;
+ }
+
+ case TYPE_TIME:
+ case TYPE_DOUBLE: {
+ *reinterpret_cast<double*>(dst) = *reinterpret_cast<const double*>(value);
+ break;
+ }
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ *reinterpret_cast<DateTimeValue*>(dst) = *reinterpret_cast<const DateTimeValue*>(value);
+ break;
+
+ case TYPE_DATEV2:
+ *reinterpret_cast<doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>*>(
+ dst) =
+ *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>*>(
+ value);
+ break;
+
+ case TYPE_DATETIMEV2:
+ *reinterpret_cast<doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>*>(
+ dst) =
+ *reinterpret_cast<const doris::vectorized::DateV2Value<
+ doris::vectorized::DateTimeV2ValueType>*>(value);
+ break;
+
+ case TYPE_DECIMALV2:
+ *reinterpret_cast<PackedInt128*>(dst) = *reinterpret_cast<const PackedInt128*>(value);
+ break;
+
+ case TYPE_DECIMAL32:
+ *reinterpret_cast<doris::vectorized::Decimal32*>(dst) =
+ *reinterpret_cast<const doris::vectorized::Decimal32*>(value);
+ break;
+ case TYPE_DECIMAL64:
+ *reinterpret_cast<doris::vectorized::Decimal64*>(dst) =
+ *reinterpret_cast<const doris::vectorized::Decimal64*>(value);
+ break;
+ case TYPE_DECIMAL128:
Review Comment:
warning: use of undeclared identifier 'TYPE_DECIMAL128'; did you mean 'TYPE_DECIMAL128I'? [clang-diagnostic-error]
```suggestion
case TYPE_DECIMAL128I:
```
**be/src/runtime/define_primitive_type.h:54:** 'TYPE_DECIMAL128I' declared here
```cpp
TYPE_DECIMAL128I, /* 30 */
^
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ return;
+ }
+
+ const char* chars = reinterpret_cast<const char*>(value);
+ const StringValue* string_val = nullptr;
Review Comment:
warning: unknown type name 'StringValue'; did you mean 'StringVal'? [clang-diagnostic-error]
```suggestion
const StringVal* string_val = nullptr;
```
**be/src/udf/udf.h:393:** 'StringVal' declared here
```cpp
struct StringVal : public AnyVal {
^
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ return;
+ }
+
+ const char* chars = reinterpret_cast<const char*>(value);
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN:
+ stream->write(chars, sizeof(bool));
+ return;
+
+ case TYPE_TINYINT:
+ stream->write(chars, sizeof(int8_t));
+ break;
+
+ case TYPE_SMALLINT:
+ stream->write(chars, sizeof(int16_t));
+ break;
+
+ case TYPE_INT:
+ stream->write(chars, sizeof(int32_t));
+ break;
+
+ case TYPE_BIGINT:
+ stream->write(chars, sizeof(int64_t));
+ break;
+
+ case TYPE_FLOAT:
+ stream->write(chars, sizeof(float));
+ break;
+
+ case TYPE_DOUBLE:
+ stream->write(chars, sizeof(double));
+ break;
+
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ stream->write(static_cast<char*>(string_val->ptr), string_val->len);
Review Comment:
warning: static_cast from 'uint8_t *' (aka 'unsigned char *') to 'char *' is not allowed [clang-diagnostic-error]
```cpp
stream->write(static_cast<char*>(string_val->ptr), string_val->len);
^
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ return;
+ }
+
+ const char* chars = reinterpret_cast<const char*>(value);
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN:
+ stream->write(chars, sizeof(bool));
+ return;
+
+ case TYPE_TINYINT:
+ stream->write(chars, sizeof(int8_t));
+ break;
+
+ case TYPE_SMALLINT:
+ stream->write(chars, sizeof(int16_t));
+ break;
+
+ case TYPE_INT:
+ stream->write(chars, sizeof(int32_t));
+ break;
+
+ case TYPE_BIGINT:
+ stream->write(chars, sizeof(int64_t));
+ break;
+
+ case TYPE_FLOAT:
+ stream->write(chars, sizeof(float));
+ break;
+
+ case TYPE_DOUBLE:
+ stream->write(chars, sizeof(double));
+ break;
+
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ stream->write(static_cast<char*>(string_val->ptr), string_val->len);
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ stream->write(chars, sizeof(DateTimeValue));
+ break;
+
+ case TYPE_DATEV2:
+ stream->write(chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>));
+ break;
+
+ case TYPE_DATETIMEV2:
+ stream->write(
+ chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>));
+ break;
+
+ case TYPE_DECIMALV2:
+ stream->write(chars, sizeof(DecimalV2Value));
+ break;
+
+ case TYPE_DECIMAL32:
+ stream->write(chars, 4);
+ break;
+
+ case TYPE_DECIMAL64:
+ stream->write(chars, 8);
+ break;
+
+ case TYPE_DECIMAL128:
Review Comment:
warning: use of undeclared identifier 'TYPE_DECIMAL128'; did you mean 'TYPE_DECIMAL128I'? [clang-diagnostic-error]
```suggestion
case TYPE_DECIMAL128I:
```
**be/src/runtime/define_primitive_type.h:54:** 'TYPE_DECIMAL128I' declared here
```cpp
TYPE_DECIMAL128I, /* 30 */
^
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ return;
+ }
+
+ const char* chars = reinterpret_cast<const char*>(value);
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN:
+ stream->write(chars, sizeof(bool));
+ return;
+
+ case TYPE_TINYINT:
+ stream->write(chars, sizeof(int8_t));
+ break;
+
+ case TYPE_SMALLINT:
+ stream->write(chars, sizeof(int16_t));
+ break;
+
+ case TYPE_INT:
+ stream->write(chars, sizeof(int32_t));
+ break;
+
+ case TYPE_BIGINT:
+ stream->write(chars, sizeof(int64_t));
+ break;
+
+ case TYPE_FLOAT:
+ stream->write(chars, sizeof(float));
+ break;
+
+ case TYPE_DOUBLE:
+ stream->write(chars, sizeof(double));
+ break;
+
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ stream->write(static_cast<char*>(string_val->ptr), string_val->len);
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ stream->write(chars, sizeof(DateTimeValue));
+ break;
+
+ case TYPE_DATEV2:
+ stream->write(chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>));
+ break;
+
+ case TYPE_DATETIMEV2:
+ stream->write(
+ chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>));
+ break;
+
+ case TYPE_DECIMALV2:
+ stream->write(chars, sizeof(DecimalV2Value));
+ break;
+
+ case TYPE_DECIMAL32:
+ stream->write(chars, 4);
+ break;
+
+ case TYPE_DECIMAL64:
+ stream->write(chars, 8);
+ break;
+
+ case TYPE_DECIMAL128:
+ stream->write(chars, 16);
+ break;
+
+ case TYPE_LARGEINT:
+ stream->write(chars, sizeof(__int128));
+ break;
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ *stream << "NULL";
+ return;
+ }
+
+ int old_precision = stream->precision();
+ std::ios_base::fmtflags old_flags = stream->flags();
+
+ if (scale > -1) {
+ stream->precision(scale);
+ // Setting 'fixed' causes precision to set the number of digits printed after the
+ // decimal (by default it sets the maximum number of digits total).
+ *stream << std::fixed;
+ }
+
+ std::string tmp;
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_BOOLEAN: {
+ bool val = *reinterpret_cast<const bool*>(value);
+ *stream << (val ? "true" : "false");
+ return;
+ }
+
+ case TYPE_TINYINT:
+ // Extra casting for chars since they should not be interpreted as ASCII.
+ *stream << static_cast<int>(*reinterpret_cast<const int8_t*>(value));
+ break;
+
+ case TYPE_SMALLINT:
+ *stream << *reinterpret_cast<const int16_t*>(value);
+ break;
+
+ case TYPE_INT:
+ *stream << *reinterpret_cast<const int32_t*>(value);
+ break;
+
+ case TYPE_BIGINT:
+ *stream << *reinterpret_cast<const int64_t*>(value);
+ break;
+
+ case TYPE_FLOAT:
+ *stream << *reinterpret_cast<const float*>(value);
+ break;
+
+ case TYPE_DOUBLE:
+ *stream << *reinterpret_cast<const double*>(value);
+ break;
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_STRING:
+ case TYPE_VARIANT:
+ string_val = reinterpret_cast<const StringValue*>(value);
Review Comment:
warning: unknown type name 'StringValue'; did you mean 'StringVal'? [clang-diagnostic-error]
```suggestion
string_val = reinterpret_cast<const StringVal*>(value);
```
**be/src/udf/udf.h:393:** 'StringVal' declared here
```cpp
struct StringVal : public AnyVal {
^
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ return;
+ }
+
+ const char* chars = reinterpret_cast<const char*>(value);
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN:
+ stream->write(chars, sizeof(bool));
+ return;
+
+ case TYPE_TINYINT:
+ stream->write(chars, sizeof(int8_t));
+ break;
+
+ case TYPE_SMALLINT:
+ stream->write(chars, sizeof(int16_t));
+ break;
+
+ case TYPE_INT:
+ stream->write(chars, sizeof(int32_t));
+ break;
+
+ case TYPE_BIGINT:
+ stream->write(chars, sizeof(int64_t));
+ break;
+
+ case TYPE_FLOAT:
+ stream->write(chars, sizeof(float));
+ break;
+
+ case TYPE_DOUBLE:
+ stream->write(chars, sizeof(double));
+ break;
+
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ stream->write(static_cast<char*>(string_val->ptr), string_val->len);
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ stream->write(chars, sizeof(DateTimeValue));
+ break;
+
+ case TYPE_DATEV2:
+ stream->write(chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>));
+ break;
+
+ case TYPE_DATETIMEV2:
+ stream->write(
+ chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>));
+ break;
+
+ case TYPE_DECIMALV2:
+ stream->write(chars, sizeof(DecimalV2Value));
+ break;
+
+ case TYPE_DECIMAL32:
+ stream->write(chars, 4);
+ break;
+
+ case TYPE_DECIMAL64:
+ stream->write(chars, 8);
+ break;
+
+ case TYPE_DECIMAL128:
+ stream->write(chars, 16);
+ break;
+
+ case TYPE_LARGEINT:
+ stream->write(chars, sizeof(__int128));
+ break;
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
Review Comment:
warning: out-of-line definition of 'print_value' does not match any declaration in 'doris::RawValue' [clang-diagnostic-error]
```cpp
void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
^
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ return;
+ }
+
+ const char* chars = reinterpret_cast<const char*>(value);
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN:
+ stream->write(chars, sizeof(bool));
+ return;
+
+ case TYPE_TINYINT:
+ stream->write(chars, sizeof(int8_t));
+ break;
+
+ case TYPE_SMALLINT:
+ stream->write(chars, sizeof(int16_t));
+ break;
+
+ case TYPE_INT:
+ stream->write(chars, sizeof(int32_t));
+ break;
+
+ case TYPE_BIGINT:
+ stream->write(chars, sizeof(int64_t));
+ break;
+
+ case TYPE_FLOAT:
+ stream->write(chars, sizeof(float));
+ break;
+
+ case TYPE_DOUBLE:
+ stream->write(chars, sizeof(double));
+ break;
+
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ stream->write(static_cast<char*>(string_val->ptr), string_val->len);
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ stream->write(chars, sizeof(DateTimeValue));
+ break;
+
+ case TYPE_DATEV2:
+ stream->write(chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>));
+ break;
+
+ case TYPE_DATETIMEV2:
+ stream->write(
+ chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>));
+ break;
+
+ case TYPE_DECIMALV2:
+ stream->write(chars, sizeof(DecimalV2Value));
+ break;
+
+ case TYPE_DECIMAL32:
+ stream->write(chars, 4);
+ break;
+
+ case TYPE_DECIMAL64:
+ stream->write(chars, 8);
+ break;
+
+ case TYPE_DECIMAL128:
+ stream->write(chars, 16);
+ break;
+
+ case TYPE_LARGEINT:
+ stream->write(chars, sizeof(__int128));
+ break;
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ *stream << "NULL";
+ return;
+ }
+
+ int old_precision = stream->precision();
+ std::ios_base::fmtflags old_flags = stream->flags();
+
+ if (scale > -1) {
+ stream->precision(scale);
+ // Setting 'fixed' causes precision to set the number of digits printed after the
+ // decimal (by default it sets the maximum number of digits total).
+ *stream << std::fixed;
+ }
+
+ std::string tmp;
+ const StringValue* string_val = nullptr;
Review Comment:
warning: unknown type name 'StringValue'; did you mean 'StringVal'? [clang-diagnostic-error]
```suggestion
const StringVal* string_val = nullptr;
```
**be/src/udf/udf.h:393:** 'StringVal' declared here
```cpp
struct StringVal : public AnyVal {
^
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ return;
+ }
+
+ const char* chars = reinterpret_cast<const char*>(value);
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN:
+ stream->write(chars, sizeof(bool));
+ return;
+
+ case TYPE_TINYINT:
+ stream->write(chars, sizeof(int8_t));
+ break;
+
+ case TYPE_SMALLINT:
+ stream->write(chars, sizeof(int16_t));
+ break;
+
+ case TYPE_INT:
+ stream->write(chars, sizeof(int32_t));
+ break;
+
+ case TYPE_BIGINT:
+ stream->write(chars, sizeof(int64_t));
+ break;
+
+ case TYPE_FLOAT:
+ stream->write(chars, sizeof(float));
+ break;
+
+ case TYPE_DOUBLE:
+ stream->write(chars, sizeof(double));
+ break;
+
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ stream->write(static_cast<char*>(string_val->ptr), string_val->len);
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ stream->write(chars, sizeof(DateTimeValue));
+ break;
+
+ case TYPE_DATEV2:
+ stream->write(chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>));
+ break;
+
+ case TYPE_DATETIMEV2:
+ stream->write(
+ chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>));
+ break;
+
+ case TYPE_DECIMALV2:
+ stream->write(chars, sizeof(DecimalV2Value));
+ break;
+
+ case TYPE_DECIMAL32:
+ stream->write(chars, 4);
+ break;
+
+ case TYPE_DECIMAL64:
+ stream->write(chars, 8);
+ break;
+
+ case TYPE_DECIMAL128:
+ stream->write(chars, 16);
+ break;
+
+ case TYPE_LARGEINT:
+ stream->write(chars, sizeof(__int128));
+ break;
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ *stream << "NULL";
+ return;
+ }
+
+ int old_precision = stream->precision();
+ std::ios_base::fmtflags old_flags = stream->flags();
+
+ if (scale > -1) {
+ stream->precision(scale);
+ // Setting 'fixed' causes precision to set the number of digits printed after the
+ // decimal (by default it sets the maximum number of digits total).
+ *stream << std::fixed;
+ }
+
+ std::string tmp;
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_BOOLEAN: {
+ bool val = *reinterpret_cast<const bool*>(value);
+ *stream << (val ? "true" : "false");
+ return;
+ }
+
+ case TYPE_TINYINT:
+ // Extra casting for chars since they should not be interpreted as ASCII.
+ *stream << static_cast<int>(*reinterpret_cast<const int8_t*>(value));
+ break;
+
+ case TYPE_SMALLINT:
+ *stream << *reinterpret_cast<const int16_t*>(value);
+ break;
+
+ case TYPE_INT:
+ *stream << *reinterpret_cast<const int32_t*>(value);
+ break;
+
+ case TYPE_BIGINT:
+ *stream << *reinterpret_cast<const int64_t*>(value);
+ break;
+
+ case TYPE_FLOAT:
+ *stream << *reinterpret_cast<const float*>(value);
+ break;
+
+ case TYPE_DOUBLE:
+ *stream << *reinterpret_cast<const double*>(value);
+ break;
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_STRING:
+ case TYPE_VARIANT:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ tmp.assign(static_cast<char*>(string_val->ptr), string_val->len);
+ *stream << tmp;
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ *stream << *reinterpret_cast<const DateTimeValue*>(value);
+ break;
+
+ case TYPE_DATEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>*>(value);
+ break;
+
+ case TYPE_DATETIMEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>*>(
+ value);
+ break;
+
+ case TYPE_DECIMALV2:
+ *stream << DecimalV2Value(reinterpret_cast<const PackedInt128*>(value)->value).to_string();
+ break;
+
+ case TYPE_DECIMAL32: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal32*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL64: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal64*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL128: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal128*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_LARGEINT:
+ *stream << reinterpret_cast<const PackedInt128*>(value)->value;
+ break;
+
+ case TYPE_ARRAY: {
+ const CollectionValue* src = reinterpret_cast<const CollectionValue*>(value);
+ auto children_type = type.children.at(0);
+ auto iter = src->iterator(children_type.type);
+ *stream << "[";
+ print_value(iter.get(), children_type, scale, stream);
+ iter.next();
+ for (; iter.has_next(); iter.next()) {
+ *stream << ", ";
+ print_value(iter.get(), children_type, scale, stream);
+ }
+ *stream << "]";
+ break;
+ }
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+
+ stream->precision(old_precision);
+ // Undo setting stream to fixed
+ stream->flags(old_flags);
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::string* str) {
+ if (value == nullptr) {
+ *str = "NULL";
+ return;
+ }
+
+ std::stringstream out;
+ out.precision(ASCII_PRECISION);
+ const StringValue* string_val = nullptr;
Review Comment:
warning: unknown type name 'StringValue'; did you mean 'StringVal'? [clang-diagnostic-error]
```suggestion
const StringVal* string_val = nullptr;
```
**be/src/udf/udf.h:393:** 'StringVal' declared here
```cpp
struct StringVal : public AnyVal {
^
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ return;
+ }
+
+ const char* chars = reinterpret_cast<const char*>(value);
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN:
+ stream->write(chars, sizeof(bool));
+ return;
+
+ case TYPE_TINYINT:
+ stream->write(chars, sizeof(int8_t));
+ break;
+
+ case TYPE_SMALLINT:
+ stream->write(chars, sizeof(int16_t));
+ break;
+
+ case TYPE_INT:
+ stream->write(chars, sizeof(int32_t));
+ break;
+
+ case TYPE_BIGINT:
+ stream->write(chars, sizeof(int64_t));
+ break;
+
+ case TYPE_FLOAT:
+ stream->write(chars, sizeof(float));
+ break;
+
+ case TYPE_DOUBLE:
+ stream->write(chars, sizeof(double));
+ break;
+
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ stream->write(static_cast<char*>(string_val->ptr), string_val->len);
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ stream->write(chars, sizeof(DateTimeValue));
+ break;
+
+ case TYPE_DATEV2:
+ stream->write(chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>));
+ break;
+
+ case TYPE_DATETIMEV2:
+ stream->write(
+ chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>));
+ break;
+
+ case TYPE_DECIMALV2:
+ stream->write(chars, sizeof(DecimalV2Value));
+ break;
+
+ case TYPE_DECIMAL32:
+ stream->write(chars, 4);
+ break;
+
+ case TYPE_DECIMAL64:
+ stream->write(chars, 8);
+ break;
+
+ case TYPE_DECIMAL128:
+ stream->write(chars, 16);
+ break;
+
+ case TYPE_LARGEINT:
+ stream->write(chars, sizeof(__int128));
+ break;
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ *stream << "NULL";
+ return;
+ }
+
+ int old_precision = stream->precision();
+ std::ios_base::fmtflags old_flags = stream->flags();
+
+ if (scale > -1) {
+ stream->precision(scale);
+ // Setting 'fixed' causes precision to set the number of digits printed after the
+ // decimal (by default it sets the maximum number of digits total).
+ *stream << std::fixed;
+ }
+
+ std::string tmp;
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_BOOLEAN: {
+ bool val = *reinterpret_cast<const bool*>(value);
+ *stream << (val ? "true" : "false");
+ return;
+ }
+
+ case TYPE_TINYINT:
+ // Extra casting for chars since they should not be interpreted as ASCII.
+ *stream << static_cast<int>(*reinterpret_cast<const int8_t*>(value));
+ break;
+
+ case TYPE_SMALLINT:
+ *stream << *reinterpret_cast<const int16_t*>(value);
+ break;
+
+ case TYPE_INT:
+ *stream << *reinterpret_cast<const int32_t*>(value);
+ break;
+
+ case TYPE_BIGINT:
+ *stream << *reinterpret_cast<const int64_t*>(value);
+ break;
+
+ case TYPE_FLOAT:
+ *stream << *reinterpret_cast<const float*>(value);
+ break;
+
+ case TYPE_DOUBLE:
+ *stream << *reinterpret_cast<const double*>(value);
+ break;
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_STRING:
+ case TYPE_VARIANT:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ tmp.assign(static_cast<char*>(string_val->ptr), string_val->len);
+ *stream << tmp;
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ *stream << *reinterpret_cast<const DateTimeValue*>(value);
+ break;
+
+ case TYPE_DATEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>*>(value);
+ break;
+
+ case TYPE_DATETIMEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>*>(
+ value);
+ break;
+
+ case TYPE_DECIMALV2:
+ *stream << DecimalV2Value(reinterpret_cast<const PackedInt128*>(value)->value).to_string();
+ break;
+
+ case TYPE_DECIMAL32: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal32*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL64: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal64*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL128: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal128*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_LARGEINT:
+ *stream << reinterpret_cast<const PackedInt128*>(value)->value;
+ break;
+
+ case TYPE_ARRAY: {
+ const CollectionValue* src = reinterpret_cast<const CollectionValue*>(value);
+ auto children_type = type.children.at(0);
+ auto iter = src->iterator(children_type.type);
+ *stream << "[";
+ print_value(iter.get(), children_type, scale, stream);
+ iter.next();
+ for (; iter.has_next(); iter.next()) {
+ *stream << ", ";
+ print_value(iter.get(), children_type, scale, stream);
+ }
+ *stream << "]";
+ break;
+ }
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+
+ stream->precision(old_precision);
+ // Undo setting stream to fixed
+ stream->flags(old_flags);
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
Review Comment:
warning: out-of-line definition of 'print_value' does not match any declaration in 'doris::RawValue' [clang-diagnostic-error]
```cpp
void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
^
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ return;
+ }
+
+ const char* chars = reinterpret_cast<const char*>(value);
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN:
+ stream->write(chars, sizeof(bool));
+ return;
+
+ case TYPE_TINYINT:
+ stream->write(chars, sizeof(int8_t));
+ break;
+
+ case TYPE_SMALLINT:
+ stream->write(chars, sizeof(int16_t));
+ break;
+
+ case TYPE_INT:
+ stream->write(chars, sizeof(int32_t));
+ break;
+
+ case TYPE_BIGINT:
+ stream->write(chars, sizeof(int64_t));
+ break;
+
+ case TYPE_FLOAT:
+ stream->write(chars, sizeof(float));
+ break;
+
+ case TYPE_DOUBLE:
+ stream->write(chars, sizeof(double));
+ break;
+
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ stream->write(static_cast<char*>(string_val->ptr), string_val->len);
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ stream->write(chars, sizeof(DateTimeValue));
+ break;
+
+ case TYPE_DATEV2:
+ stream->write(chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>));
+ break;
+
+ case TYPE_DATETIMEV2:
+ stream->write(
+ chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>));
+ break;
+
+ case TYPE_DECIMALV2:
+ stream->write(chars, sizeof(DecimalV2Value));
+ break;
+
+ case TYPE_DECIMAL32:
+ stream->write(chars, 4);
+ break;
+
+ case TYPE_DECIMAL64:
+ stream->write(chars, 8);
+ break;
+
+ case TYPE_DECIMAL128:
+ stream->write(chars, 16);
+ break;
+
+ case TYPE_LARGEINT:
+ stream->write(chars, sizeof(__int128));
+ break;
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ *stream << "NULL";
+ return;
+ }
+
+ int old_precision = stream->precision();
+ std::ios_base::fmtflags old_flags = stream->flags();
+
+ if (scale > -1) {
+ stream->precision(scale);
+ // Setting 'fixed' causes precision to set the number of digits printed after the
+ // decimal (by default it sets the maximum number of digits total).
+ *stream << std::fixed;
+ }
+
+ std::string tmp;
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_BOOLEAN: {
+ bool val = *reinterpret_cast<const bool*>(value);
+ *stream << (val ? "true" : "false");
+ return;
+ }
+
+ case TYPE_TINYINT:
+ // Extra casting for chars since they should not be interpreted as ASCII.
+ *stream << static_cast<int>(*reinterpret_cast<const int8_t*>(value));
+ break;
+
+ case TYPE_SMALLINT:
+ *stream << *reinterpret_cast<const int16_t*>(value);
+ break;
+
+ case TYPE_INT:
+ *stream << *reinterpret_cast<const int32_t*>(value);
+ break;
+
+ case TYPE_BIGINT:
+ *stream << *reinterpret_cast<const int64_t*>(value);
+ break;
+
+ case TYPE_FLOAT:
+ *stream << *reinterpret_cast<const float*>(value);
+ break;
+
+ case TYPE_DOUBLE:
+ *stream << *reinterpret_cast<const double*>(value);
+ break;
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_STRING:
+ case TYPE_VARIANT:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ tmp.assign(static_cast<char*>(string_val->ptr), string_val->len);
+ *stream << tmp;
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ *stream << *reinterpret_cast<const DateTimeValue*>(value);
+ break;
+
+ case TYPE_DATEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>*>(value);
+ break;
+
+ case TYPE_DATETIMEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>*>(
+ value);
+ break;
+
+ case TYPE_DECIMALV2:
+ *stream << DecimalV2Value(reinterpret_cast<const PackedInt128*>(value)->value).to_string();
+ break;
+
+ case TYPE_DECIMAL32: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal32*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL64: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal64*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL128: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal128*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_LARGEINT:
+ *stream << reinterpret_cast<const PackedInt128*>(value)->value;
+ break;
+
+ case TYPE_ARRAY: {
+ const CollectionValue* src = reinterpret_cast<const CollectionValue*>(value);
+ auto children_type = type.children.at(0);
+ auto iter = src->iterator(children_type.type);
+ *stream << "[";
+ print_value(iter.get(), children_type, scale, stream);
+ iter.next();
+ for (; iter.has_next(); iter.next()) {
+ *stream << ", ";
+ print_value(iter.get(), children_type, scale, stream);
+ }
+ *stream << "]";
+ break;
+ }
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+
+ stream->precision(old_precision);
+ // Undo setting stream to fixed
+ stream->flags(old_flags);
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::string* str) {
+ if (value == nullptr) {
+ *str = "NULL";
+ return;
+ }
+
+ std::stringstream out;
+ out.precision(ASCII_PRECISION);
+ const StringValue* string_val = nullptr;
+ std::string tmp;
+ bool val = false;
+
+ // Special case types that we can print more efficiently without using a std::stringstream
+ switch (type.type) {
+ case TYPE_BOOLEAN:
+ val = *reinterpret_cast<const bool*>(value);
+ *str = (val ? "true" : "false");
+ return;
+
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_OBJECT:
+ case TYPE_HLL:
+ case TYPE_QUANTILE_STATE:
+ case TYPE_STRING: {
+ string_val = reinterpret_cast<const StringValue*>(value);
Review Comment:
warning: unknown type name 'StringValue'; did you mean 'StringVal'? [clang-diagnostic-error]
```suggestion
string_val = reinterpret_cast<const StringVal*>(value);
```
**be/src/udf/udf.h:393:** 'StringVal' declared here
```cpp
struct StringVal : public AnyVal {
^
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ return;
+ }
+
+ const char* chars = reinterpret_cast<const char*>(value);
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN:
+ stream->write(chars, sizeof(bool));
+ return;
+
+ case TYPE_TINYINT:
+ stream->write(chars, sizeof(int8_t));
+ break;
+
+ case TYPE_SMALLINT:
+ stream->write(chars, sizeof(int16_t));
+ break;
+
+ case TYPE_INT:
+ stream->write(chars, sizeof(int32_t));
+ break;
+
+ case TYPE_BIGINT:
+ stream->write(chars, sizeof(int64_t));
+ break;
+
+ case TYPE_FLOAT:
+ stream->write(chars, sizeof(float));
+ break;
+
+ case TYPE_DOUBLE:
+ stream->write(chars, sizeof(double));
+ break;
+
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ stream->write(static_cast<char*>(string_val->ptr), string_val->len);
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ stream->write(chars, sizeof(DateTimeValue));
+ break;
+
+ case TYPE_DATEV2:
+ stream->write(chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>));
+ break;
+
+ case TYPE_DATETIMEV2:
+ stream->write(
+ chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>));
+ break;
+
+ case TYPE_DECIMALV2:
+ stream->write(chars, sizeof(DecimalV2Value));
+ break;
+
+ case TYPE_DECIMAL32:
+ stream->write(chars, 4);
+ break;
+
+ case TYPE_DECIMAL64:
+ stream->write(chars, 8);
+ break;
+
+ case TYPE_DECIMAL128:
+ stream->write(chars, 16);
+ break;
+
+ case TYPE_LARGEINT:
+ stream->write(chars, sizeof(__int128));
+ break;
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ *stream << "NULL";
+ return;
+ }
+
+ int old_precision = stream->precision();
+ std::ios_base::fmtflags old_flags = stream->flags();
+
+ if (scale > -1) {
+ stream->precision(scale);
+ // Setting 'fixed' causes precision to set the number of digits printed after the
+ // decimal (by default it sets the maximum number of digits total).
+ *stream << std::fixed;
+ }
+
+ std::string tmp;
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_BOOLEAN: {
+ bool val = *reinterpret_cast<const bool*>(value);
+ *stream << (val ? "true" : "false");
+ return;
+ }
+
+ case TYPE_TINYINT:
+ // Extra casting for chars since they should not be interpreted as ASCII.
+ *stream << static_cast<int>(*reinterpret_cast<const int8_t*>(value));
+ break;
+
+ case TYPE_SMALLINT:
+ *stream << *reinterpret_cast<const int16_t*>(value);
+ break;
+
+ case TYPE_INT:
+ *stream << *reinterpret_cast<const int32_t*>(value);
+ break;
+
+ case TYPE_BIGINT:
+ *stream << *reinterpret_cast<const int64_t*>(value);
+ break;
+
+ case TYPE_FLOAT:
+ *stream << *reinterpret_cast<const float*>(value);
+ break;
+
+ case TYPE_DOUBLE:
+ *stream << *reinterpret_cast<const double*>(value);
+ break;
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_STRING:
+ case TYPE_VARIANT:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ tmp.assign(static_cast<char*>(string_val->ptr), string_val->len);
+ *stream << tmp;
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ *stream << *reinterpret_cast<const DateTimeValue*>(value);
+ break;
+
+ case TYPE_DATEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>*>(value);
+ break;
+
+ case TYPE_DATETIMEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>*>(
+ value);
+ break;
+
+ case TYPE_DECIMALV2:
+ *stream << DecimalV2Value(reinterpret_cast<const PackedInt128*>(value)->value).to_string();
+ break;
+
+ case TYPE_DECIMAL32: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal32*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL64: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal64*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL128: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal128*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_LARGEINT:
+ *stream << reinterpret_cast<const PackedInt128*>(value)->value;
+ break;
+
+ case TYPE_ARRAY: {
+ const CollectionValue* src = reinterpret_cast<const CollectionValue*>(value);
+ auto children_type = type.children.at(0);
+ auto iter = src->iterator(children_type.type);
Review Comment:
warning: no member named 'iterator' in 'doris::CollectionValue' [clang-diagnostic-error]
```cpp
auto iter = src->iterator(children_type.type);
^
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ return;
+ }
+
+ const char* chars = reinterpret_cast<const char*>(value);
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN:
+ stream->write(chars, sizeof(bool));
+ return;
+
+ case TYPE_TINYINT:
+ stream->write(chars, sizeof(int8_t));
+ break;
+
+ case TYPE_SMALLINT:
+ stream->write(chars, sizeof(int16_t));
+ break;
+
+ case TYPE_INT:
+ stream->write(chars, sizeof(int32_t));
+ break;
+
+ case TYPE_BIGINT:
+ stream->write(chars, sizeof(int64_t));
+ break;
+
+ case TYPE_FLOAT:
+ stream->write(chars, sizeof(float));
+ break;
+
+ case TYPE_DOUBLE:
+ stream->write(chars, sizeof(double));
+ break;
+
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ stream->write(static_cast<char*>(string_val->ptr), string_val->len);
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ stream->write(chars, sizeof(DateTimeValue));
+ break;
+
+ case TYPE_DATEV2:
+ stream->write(chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>));
+ break;
+
+ case TYPE_DATETIMEV2:
+ stream->write(
+ chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>));
+ break;
+
+ case TYPE_DECIMALV2:
+ stream->write(chars, sizeof(DecimalV2Value));
+ break;
+
+ case TYPE_DECIMAL32:
+ stream->write(chars, 4);
+ break;
+
+ case TYPE_DECIMAL64:
+ stream->write(chars, 8);
+ break;
+
+ case TYPE_DECIMAL128:
+ stream->write(chars, 16);
+ break;
+
+ case TYPE_LARGEINT:
+ stream->write(chars, sizeof(__int128));
+ break;
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ *stream << "NULL";
+ return;
+ }
+
+ int old_precision = stream->precision();
+ std::ios_base::fmtflags old_flags = stream->flags();
+
+ if (scale > -1) {
+ stream->precision(scale);
+ // Setting 'fixed' causes precision to set the number of digits printed after the
+ // decimal (by default it sets the maximum number of digits total).
+ *stream << std::fixed;
+ }
+
+ std::string tmp;
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_BOOLEAN: {
+ bool val = *reinterpret_cast<const bool*>(value);
+ *stream << (val ? "true" : "false");
+ return;
+ }
+
+ case TYPE_TINYINT:
+ // Extra casting for chars since they should not be interpreted as ASCII.
+ *stream << static_cast<int>(*reinterpret_cast<const int8_t*>(value));
+ break;
+
+ case TYPE_SMALLINT:
+ *stream << *reinterpret_cast<const int16_t*>(value);
+ break;
+
+ case TYPE_INT:
+ *stream << *reinterpret_cast<const int32_t*>(value);
+ break;
+
+ case TYPE_BIGINT:
+ *stream << *reinterpret_cast<const int64_t*>(value);
+ break;
+
+ case TYPE_FLOAT:
+ *stream << *reinterpret_cast<const float*>(value);
+ break;
+
+ case TYPE_DOUBLE:
+ *stream << *reinterpret_cast<const double*>(value);
+ break;
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_STRING:
+ case TYPE_VARIANT:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ tmp.assign(static_cast<char*>(string_val->ptr), string_val->len);
+ *stream << tmp;
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ *stream << *reinterpret_cast<const DateTimeValue*>(value);
+ break;
+
+ case TYPE_DATEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>*>(value);
+ break;
+
+ case TYPE_DATETIMEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>*>(
+ value);
+ break;
+
+ case TYPE_DECIMALV2:
+ *stream << DecimalV2Value(reinterpret_cast<const PackedInt128*>(value)->value).to_string();
+ break;
+
+ case TYPE_DECIMAL32: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal32*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL64: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal64*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL128: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal128*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_LARGEINT:
+ *stream << reinterpret_cast<const PackedInt128*>(value)->value;
+ break;
+
+ case TYPE_ARRAY: {
+ const CollectionValue* src = reinterpret_cast<const CollectionValue*>(value);
+ auto children_type = type.children.at(0);
+ auto iter = src->iterator(children_type.type);
+ *stream << "[";
+ print_value(iter.get(), children_type, scale, stream);
+ iter.next();
+ for (; iter.has_next(); iter.next()) {
+ *stream << ", ";
+ print_value(iter.get(), children_type, scale, stream);
+ }
+ *stream << "]";
+ break;
+ }
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+
+ stream->precision(old_precision);
+ // Undo setting stream to fixed
+ stream->flags(old_flags);
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::string* str) {
+ if (value == nullptr) {
+ *str = "NULL";
+ return;
+ }
+
+ std::stringstream out;
+ out.precision(ASCII_PRECISION);
+ const StringValue* string_val = nullptr;
+ std::string tmp;
+ bool val = false;
+
+ // Special case types that we can print more efficiently without using a std::stringstream
+ switch (type.type) {
+ case TYPE_BOOLEAN:
+ val = *reinterpret_cast<const bool*>(value);
+ *str = (val ? "true" : "false");
+ return;
+
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_OBJECT:
+ case TYPE_HLL:
+ case TYPE_QUANTILE_STATE:
+ case TYPE_STRING: {
+ string_val = reinterpret_cast<const StringValue*>(value);
+ std::stringstream ss;
+ ss << "ptr:" << (void*)string_val->ptr << " len:" << string_val->len;
+ tmp = ss.str();
+ if (string_val->len <= 1000) {
+ tmp.assign(static_cast<char*>(string_val->ptr), string_val->len);
+ }
+ str->swap(tmp);
+ return;
+ }
+ case TYPE_NULL: {
+ *str = "NULL";
+ return;
+ }
+ default:
+ print_value(value, type, scale, &out);
+ }
+
+ *str = out.str();
+}
+
+void RawValue::write(const void* value, void* dst, const TypeDescriptor& type, MemPool* pool) {
Review Comment:
warning: out-of-line definition of 'write' does not match any declaration in 'doris::RawValue' [clang-diagnostic-error]
```cpp
void RawValue::write(const void* value, void* dst, const TypeDescriptor& type, MemPool* pool) {
^
```
##########
be/src/vec/columns/column.h:
##########
@@ -448,6 +449,26 @@ class IColumn : public COW<IColumn> {
LOG(FATAL) << "not support";
};
+ /// Appends one field multiple times. Can be optimized in inherited classes.
+ virtual void insert_many(const Field& field, size_t length) {
+ for (size_t i = 0; i < length; ++i) insert(field);
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
for (size_t i = 0; i < length; ++i) { insert(field);
}
```
##########
be/src/runtime/raw_value.cpp:
##########
@@ -0,0 +1,651 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/be/src/runtime/raw-value.cpp
+// and modified by Doris
+
+#include "runtime/raw_value.h"
+
+#include <sstream>
+
+#include "common/consts.h"
+#include "runtime/collection_value.h"
+#include "runtime/large_int_value.h"
+#include "runtime/tuple.h"
+#include "util/types.h"
+#include "vec/io/io_helper.h"
+
+namespace doris {
+
+const int RawValue::ASCII_PRECISION = 16; // print 16 digits for double/float
+
+void RawValue::print_value_as_bytes(const void* value, const TypeDescriptor& type,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ return;
+ }
+
+ const char* chars = reinterpret_cast<const char*>(value);
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN:
+ stream->write(chars, sizeof(bool));
+ return;
+
+ case TYPE_TINYINT:
+ stream->write(chars, sizeof(int8_t));
+ break;
+
+ case TYPE_SMALLINT:
+ stream->write(chars, sizeof(int16_t));
+ break;
+
+ case TYPE_INT:
+ stream->write(chars, sizeof(int32_t));
+ break;
+
+ case TYPE_BIGINT:
+ stream->write(chars, sizeof(int64_t));
+ break;
+
+ case TYPE_FLOAT:
+ stream->write(chars, sizeof(float));
+ break;
+
+ case TYPE_DOUBLE:
+ stream->write(chars, sizeof(double));
+ break;
+
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ stream->write(static_cast<char*>(string_val->ptr), string_val->len);
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ stream->write(chars, sizeof(DateTimeValue));
+ break;
+
+ case TYPE_DATEV2:
+ stream->write(chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>));
+ break;
+
+ case TYPE_DATETIMEV2:
+ stream->write(
+ chars,
+ sizeof(doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>));
+ break;
+
+ case TYPE_DECIMALV2:
+ stream->write(chars, sizeof(DecimalV2Value));
+ break;
+
+ case TYPE_DECIMAL32:
+ stream->write(chars, 4);
+ break;
+
+ case TYPE_DECIMAL64:
+ stream->write(chars, 8);
+ break;
+
+ case TYPE_DECIMAL128:
+ stream->write(chars, 16);
+ break;
+
+ case TYPE_LARGEINT:
+ stream->write(chars, sizeof(__int128));
+ break;
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::stringstream* stream) {
+ if (value == nullptr) {
+ *stream << "NULL";
+ return;
+ }
+
+ int old_precision = stream->precision();
+ std::ios_base::fmtflags old_flags = stream->flags();
+
+ if (scale > -1) {
+ stream->precision(scale);
+ // Setting 'fixed' causes precision to set the number of digits printed after the
+ // decimal (by default it sets the maximum number of digits total).
+ *stream << std::fixed;
+ }
+
+ std::string tmp;
+ const StringValue* string_val = nullptr;
+
+ switch (type.type) {
+ case TYPE_BOOLEAN: {
+ bool val = *reinterpret_cast<const bool*>(value);
+ *stream << (val ? "true" : "false");
+ return;
+ }
+
+ case TYPE_TINYINT:
+ // Extra casting for chars since they should not be interpreted as ASCII.
+ *stream << static_cast<int>(*reinterpret_cast<const int8_t*>(value));
+ break;
+
+ case TYPE_SMALLINT:
+ *stream << *reinterpret_cast<const int16_t*>(value);
+ break;
+
+ case TYPE_INT:
+ *stream << *reinterpret_cast<const int32_t*>(value);
+ break;
+
+ case TYPE_BIGINT:
+ *stream << *reinterpret_cast<const int64_t*>(value);
+ break;
+
+ case TYPE_FLOAT:
+ *stream << *reinterpret_cast<const float*>(value);
+ break;
+
+ case TYPE_DOUBLE:
+ *stream << *reinterpret_cast<const double*>(value);
+ break;
+ case TYPE_HLL:
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_STRING:
+ case TYPE_VARIANT:
+ string_val = reinterpret_cast<const StringValue*>(value);
+ tmp.assign(static_cast<char*>(string_val->ptr), string_val->len);
+ *stream << tmp;
+ return;
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ *stream << *reinterpret_cast<const DateTimeValue*>(value);
+ break;
+
+ case TYPE_DATEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>*>(value);
+ break;
+
+ case TYPE_DATETIMEV2:
+ *stream << *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>*>(
+ value);
+ break;
+
+ case TYPE_DECIMALV2:
+ *stream << DecimalV2Value(reinterpret_cast<const PackedInt128*>(value)->value).to_string();
+ break;
+
+ case TYPE_DECIMAL32: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal32*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL64: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal64*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_DECIMAL128: {
+ auto decimal_val = reinterpret_cast<const doris::vectorized::Decimal128*>(value);
+ write_text(*decimal_val, type.scale, *stream);
+ break;
+ }
+
+ case TYPE_LARGEINT:
+ *stream << reinterpret_cast<const PackedInt128*>(value)->value;
+ break;
+
+ case TYPE_ARRAY: {
+ const CollectionValue* src = reinterpret_cast<const CollectionValue*>(value);
+ auto children_type = type.children.at(0);
+ auto iter = src->iterator(children_type.type);
+ *stream << "[";
+ print_value(iter.get(), children_type, scale, stream);
+ iter.next();
+ for (; iter.has_next(); iter.next()) {
+ *stream << ", ";
+ print_value(iter.get(), children_type, scale, stream);
+ }
+ *stream << "]";
+ break;
+ }
+
+ default:
+ DCHECK(false) << "bad RawValue::print_value() type: " << type;
+ }
+
+ stream->precision(old_precision);
+ // Undo setting stream to fixed
+ stream->flags(old_flags);
+}
+
+void RawValue::print_value(const void* value, const TypeDescriptor& type, int scale,
+ std::string* str) {
+ if (value == nullptr) {
+ *str = "NULL";
+ return;
+ }
+
+ std::stringstream out;
+ out.precision(ASCII_PRECISION);
+ const StringValue* string_val = nullptr;
+ std::string tmp;
+ bool val = false;
+
+ // Special case types that we can print more efficiently without using a std::stringstream
+ switch (type.type) {
+ case TYPE_BOOLEAN:
+ val = *reinterpret_cast<const bool*>(value);
+ *str = (val ? "true" : "false");
+ return;
+
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_OBJECT:
+ case TYPE_HLL:
+ case TYPE_QUANTILE_STATE:
+ case TYPE_STRING: {
+ string_val = reinterpret_cast<const StringValue*>(value);
+ std::stringstream ss;
+ ss << "ptr:" << (void*)string_val->ptr << " len:" << string_val->len;
+ tmp = ss.str();
+ if (string_val->len <= 1000) {
+ tmp.assign(static_cast<char*>(string_val->ptr), string_val->len);
+ }
+ str->swap(tmp);
+ return;
+ }
+ case TYPE_NULL: {
+ *str = "NULL";
+ return;
+ }
+ default:
+ print_value(value, type, scale, &out);
+ }
+
+ *str = out.str();
+}
+
+void RawValue::write(const void* value, void* dst, const TypeDescriptor& type, MemPool* pool) {
+ DCHECK(value != nullptr);
+
+ switch (type.type) {
+ case TYPE_NULL:
+ break;
+ case TYPE_BOOLEAN: {
+ *reinterpret_cast<bool*>(dst) = *reinterpret_cast<const bool*>(value);
+ break;
+ }
+
+ case TYPE_TINYINT: {
+ *reinterpret_cast<int8_t*>(dst) = *reinterpret_cast<const int8_t*>(value);
+ break;
+ }
+
+ case TYPE_SMALLINT: {
+ *reinterpret_cast<int16_t*>(dst) = *reinterpret_cast<const int16_t*>(value);
+ break;
+ }
+
+ case TYPE_INT: {
+ *reinterpret_cast<int32_t*>(dst) = *reinterpret_cast<const int32_t*>(value);
+ break;
+ }
+
+ case TYPE_BIGINT: {
+ *reinterpret_cast<int64_t*>(dst) = *reinterpret_cast<const int64_t*>(value);
+ break;
+ }
+
+ case TYPE_LARGEINT: {
+ *reinterpret_cast<PackedInt128*>(dst) = *reinterpret_cast<const PackedInt128*>(value);
+ break;
+ }
+
+ case TYPE_FLOAT: {
+ *reinterpret_cast<float*>(dst) = *reinterpret_cast<const float*>(value);
+ break;
+ }
+
+ case TYPE_TIME:
+ case TYPE_DOUBLE: {
+ *reinterpret_cast<double*>(dst) = *reinterpret_cast<const double*>(value);
+ break;
+ }
+
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ *reinterpret_cast<DateTimeValue*>(dst) = *reinterpret_cast<const DateTimeValue*>(value);
+ break;
+
+ case TYPE_DATEV2:
+ *reinterpret_cast<doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>*>(
+ dst) =
+ *reinterpret_cast<
+ const doris::vectorized::DateV2Value<doris::vectorized::DateV2ValueType>*>(
+ value);
+ break;
+
+ case TYPE_DATETIMEV2:
+ *reinterpret_cast<doris::vectorized::DateV2Value<doris::vectorized::DateTimeV2ValueType>*>(
+ dst) =
+ *reinterpret_cast<const doris::vectorized::DateV2Value<
+ doris::vectorized::DateTimeV2ValueType>*>(value);
+ break;
+
+ case TYPE_DECIMALV2:
+ *reinterpret_cast<PackedInt128*>(dst) = *reinterpret_cast<const PackedInt128*>(value);
+ break;
+
+ case TYPE_DECIMAL32:
+ *reinterpret_cast<doris::vectorized::Decimal32*>(dst) =
+ *reinterpret_cast<const doris::vectorized::Decimal32*>(value);
+ break;
+ case TYPE_DECIMAL64:
+ *reinterpret_cast<doris::vectorized::Decimal64*>(dst) =
+ *reinterpret_cast<const doris::vectorized::Decimal64*>(value);
+ break;
+ case TYPE_DECIMAL128:
+ *reinterpret_cast<doris::vectorized::Decimal128*>(dst) =
+ *reinterpret_cast<const doris::vectorized::Decimal128*>(value);
+ break;
+
+ case TYPE_OBJECT:
+ case TYPE_HLL:
+ case TYPE_QUANTILE_STATE:
+ case TYPE_VARCHAR:
+ case TYPE_CHAR:
+ case TYPE_VARIANT:
+ case TYPE_STRING: {
+ const StringValue* src = reinterpret_cast<const StringValue*>(value);
Review Comment:
warning: unknown type name 'StringValue'; did you mean 'StringVal'? [clang-diagnostic-error]
```suggestion
const StringValue* src = reinterpret_cast<const StringVal*>(value);
```
**be/src/udf/udf.h:393:** 'StringVal' declared here
```cpp
struct StringVal : public AnyVal {
^
```
##########
be/src/vec/columns/column_array.cpp:
##########
@@ -46,6 +46,34 @@
*/
static constexpr size_t max_array_size_as_field = 1000000;
+template <typename T>
+ColumnPtr ColumnArray::index_impl(const PaddedPODArray<T>& indexes, size_t limit) const {
+ assert(limit <= indexes.size());
+ if (limit == 0) return ColumnArray::create(data->clone_empty());
+ /// Convert indexes to UInt64 in case of overflow.
+ auto nested_indexes_column = ColumnUInt64::create();
+ PaddedPODArray<UInt64>& nested_indexes = nested_indexes_column->get_data();
+ nested_indexes.reserve(get_offsets().back());
+ auto res = ColumnArray::create(data->clone_empty());
+ Offsets64& res_offsets = res->get_offsets();
+ res_offsets.resize(limit);
+ size_t current_offset = 0;
+ for (size_t i = 0; i < limit; ++i) {
+ for (size_t j = 0; j < size_at(indexes[i]); ++j)
+ nested_indexes.push_back(offset_at(indexes[i]) + j);
+ current_offset += size_at(indexes[i]);
+ res_offsets[i] = current_offset;
+ }
+ if (current_offset != 0) res->data = data->index(*nested_indexes_column, current_offset);
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (current_offset != 0) { res->data = data->index(*nested_indexes_column, current_offset);
}
```
##########
be/src/vec/columns/column_array.cpp:
##########
@@ -46,6 +46,34 @@
*/
static constexpr size_t max_array_size_as_field = 1000000;
+template <typename T>
+ColumnPtr ColumnArray::index_impl(const PaddedPODArray<T>& indexes, size_t limit) const {
+ assert(limit <= indexes.size());
+ if (limit == 0) return ColumnArray::create(data->clone_empty());
+ /// Convert indexes to UInt64 in case of overflow.
+ auto nested_indexes_column = ColumnUInt64::create();
+ PaddedPODArray<UInt64>& nested_indexes = nested_indexes_column->get_data();
+ nested_indexes.reserve(get_offsets().back());
+ auto res = ColumnArray::create(data->clone_empty());
+ Offsets64& res_offsets = res->get_offsets();
+ res_offsets.resize(limit);
+ size_t current_offset = 0;
+ for (size_t i = 0; i < limit; ++i) {
+ for (size_t j = 0; j < size_at(indexes[i]); ++j)
+ nested_indexes.push_back(offset_at(indexes[i]) + j);
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
for (size_t j = 0; j < size_at(indexes[i]); ++j) {
nested_indexes.push_back(offset_at(indexes[i]) + j);
}
```
##########
be/src/vec/columns/column_array.cpp:
##########
@@ -46,6 +46,34 @@ extern const int TOO_LARGE_ARRAY_SIZE;
*/
static constexpr size_t max_array_size_as_field = 1000000;
+template <typename T>
+ColumnPtr ColumnArray::index_impl(const PaddedPODArray<T>& indexes, size_t limit) const {
+ assert(limit <= indexes.size());
+ if (limit == 0) return ColumnArray::create(data->clone_empty());
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (limit == 0) { return ColumnArray::create(data->clone_empty());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org