You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "eldenmoon (via GitHub)" <gi...@apache.org> on 2023/12/27 13:58:49 UTC
[PR] [feature](Variant) add variant config in frontend meta [doris]
eldenmoon opened a new pull request, #29185:
URL: https://github.com/apache/doris/pull/29185
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Variant) add variant config in frontend meta [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #29185:
URL: https://github.com/apache/doris/pull/29185#discussion_r1437063851
##########
be/src/olap/tablet.h:
##########
@@ -562,6 +563,9 @@ class Tablet final : public BaseTablet {
std::unique_ptr<segment_v2::ColumnIterator>* column_iterator,
OlapReaderStatistics* stats);
+ void set_variant_config(VariantConfig binlog_config);
+ const VariantConfig& variant_config() const;
+
Review Comment:
warning: parameter 'rowset_id' is const-qualified in the function declaration; const-qualification of parameters only has an effect in function definitions [readability-avoid-const-params-in-decls]
```suggestion
bool _contains_rowset(RowsetId rowset_id);
```
##########
be/src/olap/tablet_meta.h:
##########
@@ -251,6 +254,11 @@ class TabletMeta {
return _time_series_compaction_time_threshold_seconds;
}
+ const VariantConfig& variant_config() const { return _variant_config; }
+ void set_variant_config(VariantConfig variant_config) {
+ _variant_config = std::move(variant_config);
+ }
+
private:
Status _save_meta(DataDir* data_dir);
Review Comment:
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
<details>
<summary>Additional context</summary>
**be/src/olap/tablet_meta.h:256:** previously declared here
```cpp
private:
^
```
</details>
##########
be/src/vec/columns/column_object.cpp:
##########
@@ -583,6 +587,11 @@ ColumnObject::ColumnObject(bool is_nullable_, bool create_root_)
}
}
+ColumnObject::ColumnObject(VariantConfig variant_config)
+ : is_nullable(true), num_rows(0), config(std::move(variant_config)) {
+ subcolumns.create_root(Subcolumn(0, is_nullable, true /*root*/));
+}
Review Comment:
warning: method 'check_consistency' can be made static [readability-convert-member-functions-to-static]
be/src/vec/columns/column_object.h:234:
```diff
- void check_consistency() const;
+ static void check_consistency() ;
```
```suggestion
void ColumnObject::check_consistency() {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Variant) add variant config in frontend meta [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29185:
URL: https://github.com/apache/doris/pull/29185#issuecomment-1870947908
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.2 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17183801652 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Variant) add variant config in frontend meta [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29185:
URL: https://github.com/apache/doris/pull/29185#issuecomment-1870806129
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Variant) add variant config in frontend meta [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #29185:
URL: https://github.com/apache/doris/pull/29185#issuecomment-1882930833
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Variant) add variant config in frontend meta [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29185:
URL: https://github.com/apache/doris/pull/29185#issuecomment-1883065836
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.72 seconds
stream load tsv: 562 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 21.8 seconds inserted 10000000 Rows, about 458K ops/s
storage size: 17183641059 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Variant) add variant config in frontend meta [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29185:
URL: https://github.com/apache/doris/pull/29185#issuecomment-1870960285
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 51fe44dde380f14aa9d60239f3b6d36d9dbe933e, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4964 4653 4646 4646
q2 370 153 139 139
q3 1480 1319 1248 1248
q4 1134 981 958 958
q5 3190 3153 3169 3153
q6 257 127 129 127
q7 1042 510 493 493
q8 2258 2259 2241 2241
q9 6676 6695 6683 6683
q10 3219 3302 3267 3267
q11 332 219 210 210
q12 347 208 211 208
q13 4144 3439 3448 3439
q14 251 213 214 213
q15 573 523 528 523
q16 446 382 386 382
q17 1047 782 604 604
q18 7133 6803 6754 6754
q19 1639 1644 1636 1636
q20 558 325 297 297
q21 3174 2729 2696 2696
q22 365 299 298 298
Total cold run time: 44599 ms
Total hot run time: 40215 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4589 4592 4579 4579
q2 293 166 172 166
q3 3374 3373 3369 3369
q4 2229 2205 2203 2203
q5 5727 5724 5718 5718
q6 242 117 121 117
q7 2388 1866 1894 1866
q8 3616 3634 3622 3622
q9 9068 9027 9001 9001
q10 3826 3900 3906 3900
q11 492 366 384 366
q12 768 602 592 592
q13 3922 3224 3199 3199
q14 288 259 252 252
q15 570 511 528 511
q16 506 464 480 464
q17 1979 1970 1966 1966
q18 8650 8279 8184 8184
q19 1764 1761 1763 1761
q20 2227 1938 1922 1922
q21 6145 5779 5802 5779
q22 536 480 474 474
Total cold run time: 63199 ms
Total hot run time: 60011 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Variant) add variant config in frontend meta [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #29185:
URL: https://github.com/apache/doris/pull/29185#issuecomment-1870875784
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Variant) add variant config in frontend meta [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon closed pull request #29185: [feature](Variant) add variant config in frontend meta
URL: https://github.com/apache/doris/pull/29185
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Variant) add variant config in frontend meta [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #29185:
URL: https://github.com/apache/doris/pull/29185#discussion_r1437064322
##########
be/src/olap/variant_config.cpp:
##########
@@ -0,0 +1,57 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/variant_config.h"
+
+#include <fmt/format.h>
Review Comment:
warning: 'fmt/format.h' file not found [clang-diagnostic-error]
```cpp
#include <fmt/format.h>
^
```
##########
be/src/vec/columns/column_object.cpp:
##########
@@ -441,8 +443,10 @@ ColumnPtr ColumnObject::index(const IColumn& indexes, size_t limit) const {
[&](const auto& subcolumn) { return subcolumn.index(indexes, limit); });
}
-bool ColumnObject::Subcolumn::check_if_sparse_column(size_t num_rows) {
- if (num_rows < config::variant_threshold_rows_to_estimate_sparse_column) {
+bool ColumnObject::Subcolumn::check_if_sparse_column(size_t num_rows,
Review Comment:
warning: method 'check_if_sparse_column' can be made static [readability-convert-member-functions-to-static]
be/src/vec/columns/column_object.h:144:
```diff
- bool check_if_sparse_column(size_t num_rows, int rows_threshold_to_estimate_spasr,
+ static bool check_if_sparse_column(size_t num_rows, int rows_threshold_to_estimate_spasr,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Variant) add variant config in frontend meta [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #29185:
URL: https://github.com/apache/doris/pull/29185#discussion_r1437372357
##########
be/src/vec/runtime/ipv4_value.h:
##########
@@ -34,87 +33,41 @@ class IPv4Value {
explicit IPv4Value(vectorized::IPv4 ipv4) { _value = ipv4; }
- explicit IPv4Value(std::string ipv4) {}
-
- [[nodiscard]] const vectorized::IPv4& value() const { return _value; }
+ const vectorized::IPv4& value() const { return _value; }
vectorized::IPv4& value() { return _value; }
void set_value(vectorized::IPv4 ipv4) { _value = ipv4; }
- bool from_string(std::string ipv4) { return from_string(_value, ipv4); }
-
- [[nodiscard]] std::string to_string() const { return to_string(_value); }
+ bool from_string(const std::string& ipv4_str) { return from_string(_value, ipv4_str); }
- static bool from_string(vectorized::IPv4& value, std::string ipv4) {
- remove_ipv4_space(ipv4);
+ std::string to_string() const { return to_string(_value); }
- // shortest ipv4 string is `0.0.0.0` whose length is 7
- if (ipv4.size() < 7 || !is_valid_string(ipv4)) {
+ static bool from_string(vectorized::IPv4& value, const std::string& ipv4_str) {
+ if (ipv4_str.empty()) {
return false;
}
-
- vectorized::IPv4 octets[4] = {0};
- std::istringstream iss(ipv4);
- std::string octet;
- uint8_t octet_index = 0;
-
- while (getline(iss, octet, '.')) {
- if (octet_index >= 4) {
- return false;
- }
-
- StringParser::ParseResult result;
- vectorized::IPv4 val = StringParser::string_to_unsigned_int<vectorized::IPv4>(
- octet.c_str(), octet.length(), &result);
- if (result != StringParser::PARSE_SUCCESS || val > 255) {
- return false;
- }
-
- octets[octet_index++] = val;
- }
-
- if (octet_index != 4) {
+ int64_t parse_value;
+ const char* src = ipv4_str.c_str();
+ const char* end = ipv4_str.c_str() + ipv4_str.size() - 1;
+ while (std::isspace(*src)) ++src;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
while (std::isspace(*src)) { ++src;
}
```
##########
be/src/vec/runtime/ipv4_value.h:
##########
@@ -34,87 +33,41 @@
explicit IPv4Value(vectorized::IPv4 ipv4) { _value = ipv4; }
- explicit IPv4Value(std::string ipv4) {}
-
- [[nodiscard]] const vectorized::IPv4& value() const { return _value; }
+ const vectorized::IPv4& value() const { return _value; }
vectorized::IPv4& value() { return _value; }
void set_value(vectorized::IPv4 ipv4) { _value = ipv4; }
- bool from_string(std::string ipv4) { return from_string(_value, ipv4); }
-
- [[nodiscard]] std::string to_string() const { return to_string(_value); }
+ bool from_string(const std::string& ipv4_str) { return from_string(_value, ipv4_str); }
- static bool from_string(vectorized::IPv4& value, std::string ipv4) {
- remove_ipv4_space(ipv4);
+ std::string to_string() const { return to_string(_value); }
- // shortest ipv4 string is `0.0.0.0` whose length is 7
- if (ipv4.size() < 7 || !is_valid_string(ipv4)) {
+ static bool from_string(vectorized::IPv4& value, const std::string& ipv4_str) {
+ if (ipv4_str.empty()) {
return false;
}
-
- vectorized::IPv4 octets[4] = {0};
- std::istringstream iss(ipv4);
- std::string octet;
- uint8_t octet_index = 0;
-
- while (getline(iss, octet, '.')) {
- if (octet_index >= 4) {
- return false;
- }
-
- StringParser::ParseResult result;
- vectorized::IPv4 val = StringParser::string_to_unsigned_int<vectorized::IPv4>(
- octet.c_str(), octet.length(), &result);
- if (result != StringParser::PARSE_SUCCESS || val > 255) {
- return false;
- }
-
- octets[octet_index++] = val;
- }
-
- if (octet_index != 4) {
+ int64_t parse_value;
+ const char* src = ipv4_str.c_str();
+ const char* end = ipv4_str.c_str() + ipv4_str.size() - 1;
+ while (std::isspace(*src)) ++src;
+ while (std::isspace(*end)) --end;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
while (std::isspace(*end)) { --end;
}
```
##########
be/src/vec/exec/scan/vscanner.cpp:
##########
@@ -58,9 +60,30 @@ Status VScanner::prepare(RuntimeState* state, const VExprContextSPtrs& conjuncts
}
}
+ const auto& projections = _parent ? _parent->_projections : _local_state->_projections;
+ if (!projections.empty()) {
+ _projections.resize(projections.size());
+ for (size_t i = 0; i != projections.size(); ++i) {
+ RETURN_IF_ERROR(projections[i]->clone(state, _projections[i]));
+ }
+ }
+
return Status::OK();
}
+Status VScanner::get_block_after_projects(RuntimeState* state, vectorized::Block* block,
+ bool* eos) {
+ auto& row_descriptor =
+ _parent ? _parent->_row_descriptor : _local_state->_parent->row_descriptor();
+ if (_output_row_descriptor) {
+ _origin_block.clear_column_data(row_descriptor.num_materialized_slots());
+ auto status = get_block(state, &_origin_block, eos);
+ if (UNLIKELY(!status.ok())) return status;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (UNLIKELY(!status.ok())) { return status;
}
```
##########
be/src/vec/runtime/ipv6_value.h:
##########
@@ -35,224 +34,35 @@ class IPv6Value {
explicit IPv6Value(vectorized::IPv6 ipv6) { _value = ipv6; }
- [[nodiscard]] const vectorized::IPv6& value() const { return _value; }
+ const vectorized::IPv6& value() const { return _value; }
vectorized::IPv6& value() { return _value; }
void set_value(vectorized::IPv6 ipv6) { _value = ipv6; }
- bool from_string(std::string ipv6) { return from_string(_value, ipv6); }
-
- bool from_binary_string(std::string ipv6_binary) {
- return from_binary_string(_value, ipv6_binary);
- }
-
- static bool from_string(vectorized::IPv6& x, std::string ipv6) {
- remove_ipv6_space(ipv6);
-
- if (ipv6.empty() || !is_valid_string(ipv6)) {
- return false;
- }
-
- std::transform(ipv6.begin(), ipv6.end(), ipv6.begin(),
- [](unsigned char ch) { return std::tolower(ch); });
- std::istringstream iss(ipv6);
- std::string field;
- uint16_t fields[8] = {0};
- uint8_t zero_index = 0;
- uint8_t num_field = 0;
- uint8_t right_field_num = 0;
-
- while (num_field < 8) {
- if (!getline(iss, field, ':')) {
- break;
- }
-
- if (field.empty()) {
- zero_index = num_field;
- fields[num_field++] = 0;
- } else {
- try {
- if (field.size() > 4 || field > "ffff") {
- return false;
- }
-
- fields[num_field++] = std::stoi(field, nullptr, 16);
- } catch (const std::exception& /*e*/) {
- return false;
- }
- }
- }
-
- if (zero_index != 0) {
- right_field_num = num_field - zero_index - 1;
-
- for (uint8_t i = 7; i > 7 - right_field_num; --i) {
- fields[i] = fields[zero_index + right_field_num + i - 7];
- fields[zero_index + right_field_num + i - 7] = 0;
- }
- }
-
- uint64_t high = (static_cast<uint64_t>(fields[0]) << 48) |
- (static_cast<uint64_t>(fields[1]) << 32) |
- (static_cast<uint64_t>(fields[2]) << 16) | static_cast<uint64_t>(fields[3]);
- uint64_t low = (static_cast<uint64_t>(fields[4]) << 48) |
- (static_cast<uint64_t>(fields[5]) << 32) |
- (static_cast<uint64_t>(fields[6]) << 16) | static_cast<uint64_t>(fields[7]);
-
- x = static_cast<vectorized::IPv6>(high) << 64 | low;
- return true;
- }
+ bool from_string(const std::string& ipv6_str) { return from_string(_value, ipv6_str); }
- static bool from_binary_string(vectorized::IPv6& x, std::string ipv6_binary_str) {
- // Accepts a FixedString(16) value containing the IPv6 address in binary format
- if (ipv6_binary_str.size() != 16) {
+ static bool from_string(vectorized::IPv6& value, const std::string& ipv6_str) {
+ if (ipv6_str.empty()) {
return false;
}
-
- uint64_t high = 0;
- uint64_t low = 0;
-
- const uint8_t* ipv6_binary = reinterpret_cast<const uint8_t*>(ipv6_binary_str.c_str());
-
- for (int i = 0; i < 8; ++i) {
- high |= (static_cast<uint64_t>(ipv6_binary[i]) << (56 - i * 8));
- }
-
- for (int i = 8; i < 16; ++i) {
- low |= (static_cast<uint64_t>(ipv6_binary[i]) << (56 - (i - 8) * 8));
- }
-
- x = static_cast<vectorized::IPv6>(high) << 64 | low;
- return true;
- }
-
- [[nodiscard]] std::string to_string() const { return to_string(_value); }
-
- static std::string to_string(vectorized::IPv6 x) {
- // "0000:0000:0000:0000:0000:0000:0000:0000"
- if (x == 0) {
- return "::";
- }
-
- uint64_t low = static_cast<uint64_t>(x);
- uint64_t high = static_cast<uint64_t>(x >> 64);
-
- uint16_t fields[8] = {static_cast<uint16_t>((high >> 48) & 0xFFFF),
- static_cast<uint16_t>((high >> 32) & 0xFFFF),
- static_cast<uint16_t>((high >> 16) & 0xFFFF),
- static_cast<uint16_t>(high & 0xFFFF),
- static_cast<uint16_t>((low >> 48) & 0xFFFF),
- static_cast<uint16_t>((low >> 32) & 0xFFFF),
- static_cast<uint16_t>((low >> 16) & 0xFFFF),
- static_cast<uint16_t>(low & 0xFFFF)};
-
- uint8_t zero_start = 0, zero_end = 0;
-
- while (zero_start < 8 && zero_end < 8) {
- if (fields[zero_start] != 0) {
- zero_start++;
- zero_end = zero_start;
- continue;
- }
-
- while (zero_end < 7 && fields[zero_end + 1] == 0) {
- zero_end++;
- }
-
- if (zero_end > zero_start) {
- break;
- }
-
- zero_start++;
- zero_end = zero_start;
- }
-
- std::stringstream ss;
-
- if (zero_start == zero_end) {
- for (uint8_t i = 0; i < 7; ++i) {
- ss << std::hex << fields[i] << ":";
- }
- ss << std::hex << fields[7];
- } else {
- for (uint8_t i = 0; i < zero_start; ++i) {
- ss << std::hex << fields[i] << ":";
- }
-
- if (zero_end == 7) {
- ss << ":";
- } else {
- for (uint8_t j = zero_end + 1; j < 8; ++j) {
- ss << std::hex << ":" << fields[j];
- }
- }
- }
-
- return ss.str();
- }
-
- [[nodiscard]] std::string to_binary_string() const { return to_binary_string(_value); }
-
- static std::string to_binary_string(vectorized::IPv6 x) {
- uint64_t low = static_cast<uint64_t>(x);
- uint64_t high = static_cast<uint64_t>(x >> 64);
-
- uint8_t fields[16] = {static_cast<uint8_t>((high >> 56) & 0xFF),
- static_cast<uint8_t>((high >> 48) & 0xFF),
- static_cast<uint8_t>((high >> 40) & 0xFF),
- static_cast<uint8_t>((high >> 32) & 0xFF),
- static_cast<uint8_t>((high >> 24) & 0xFF),
- static_cast<uint8_t>((high >> 16) & 0xFF),
- static_cast<uint8_t>((high >> 8) & 0xFF),
- static_cast<uint8_t>(high & 0xFF),
- static_cast<uint8_t>((low >> 56) & 0xFF),
- static_cast<uint8_t>((low >> 48) & 0xFF),
- static_cast<uint8_t>((low >> 40) & 0xFF),
- static_cast<uint8_t>((low >> 32) & 0xFF),
- static_cast<uint8_t>((low >> 24) & 0xFF),
- static_cast<uint8_t>((low >> 16) & 0xFF),
- static_cast<uint8_t>((low >> 8) & 0xFF),
- static_cast<uint8_t>(low & 0xFF)};
-
- std::stringstream ss;
-
- for (int i = 0; i < 16; ++i) {
- ss << (char)fields[i];
- }
-
- return ss.str();
+ const char* src = ipv6_str.c_str();
+ const char* end = ipv6_str.c_str() + ipv6_str.size() - 1;
+ while (std::isspace(*src)) ++src;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
while (std::isspace(*src)) { ++src;
}
```
##########
be/src/vec/runtime/ipv6_value.h:
##########
@@ -35,224 +34,35 @@
explicit IPv6Value(vectorized::IPv6 ipv6) { _value = ipv6; }
- [[nodiscard]] const vectorized::IPv6& value() const { return _value; }
+ const vectorized::IPv6& value() const { return _value; }
vectorized::IPv6& value() { return _value; }
void set_value(vectorized::IPv6 ipv6) { _value = ipv6; }
- bool from_string(std::string ipv6) { return from_string(_value, ipv6); }
-
- bool from_binary_string(std::string ipv6_binary) {
- return from_binary_string(_value, ipv6_binary);
- }
-
- static bool from_string(vectorized::IPv6& x, std::string ipv6) {
- remove_ipv6_space(ipv6);
-
- if (ipv6.empty() || !is_valid_string(ipv6)) {
- return false;
- }
-
- std::transform(ipv6.begin(), ipv6.end(), ipv6.begin(),
- [](unsigned char ch) { return std::tolower(ch); });
- std::istringstream iss(ipv6);
- std::string field;
- uint16_t fields[8] = {0};
- uint8_t zero_index = 0;
- uint8_t num_field = 0;
- uint8_t right_field_num = 0;
-
- while (num_field < 8) {
- if (!getline(iss, field, ':')) {
- break;
- }
-
- if (field.empty()) {
- zero_index = num_field;
- fields[num_field++] = 0;
- } else {
- try {
- if (field.size() > 4 || field > "ffff") {
- return false;
- }
-
- fields[num_field++] = std::stoi(field, nullptr, 16);
- } catch (const std::exception& /*e*/) {
- return false;
- }
- }
- }
-
- if (zero_index != 0) {
- right_field_num = num_field - zero_index - 1;
-
- for (uint8_t i = 7; i > 7 - right_field_num; --i) {
- fields[i] = fields[zero_index + right_field_num + i - 7];
- fields[zero_index + right_field_num + i - 7] = 0;
- }
- }
-
- uint64_t high = (static_cast<uint64_t>(fields[0]) << 48) |
- (static_cast<uint64_t>(fields[1]) << 32) |
- (static_cast<uint64_t>(fields[2]) << 16) | static_cast<uint64_t>(fields[3]);
- uint64_t low = (static_cast<uint64_t>(fields[4]) << 48) |
- (static_cast<uint64_t>(fields[5]) << 32) |
- (static_cast<uint64_t>(fields[6]) << 16) | static_cast<uint64_t>(fields[7]);
-
- x = static_cast<vectorized::IPv6>(high) << 64 | low;
- return true;
- }
+ bool from_string(const std::string& ipv6_str) { return from_string(_value, ipv6_str); }
- static bool from_binary_string(vectorized::IPv6& x, std::string ipv6_binary_str) {
- // Accepts a FixedString(16) value containing the IPv6 address in binary format
- if (ipv6_binary_str.size() != 16) {
+ static bool from_string(vectorized::IPv6& value, const std::string& ipv6_str) {
+ if (ipv6_str.empty()) {
return false;
}
-
- uint64_t high = 0;
- uint64_t low = 0;
-
- const uint8_t* ipv6_binary = reinterpret_cast<const uint8_t*>(ipv6_binary_str.c_str());
-
- for (int i = 0; i < 8; ++i) {
- high |= (static_cast<uint64_t>(ipv6_binary[i]) << (56 - i * 8));
- }
-
- for (int i = 8; i < 16; ++i) {
- low |= (static_cast<uint64_t>(ipv6_binary[i]) << (56 - (i - 8) * 8));
- }
-
- x = static_cast<vectorized::IPv6>(high) << 64 | low;
- return true;
- }
-
- [[nodiscard]] std::string to_string() const { return to_string(_value); }
-
- static std::string to_string(vectorized::IPv6 x) {
- // "0000:0000:0000:0000:0000:0000:0000:0000"
- if (x == 0) {
- return "::";
- }
-
- uint64_t low = static_cast<uint64_t>(x);
- uint64_t high = static_cast<uint64_t>(x >> 64);
-
- uint16_t fields[8] = {static_cast<uint16_t>((high >> 48) & 0xFFFF),
- static_cast<uint16_t>((high >> 32) & 0xFFFF),
- static_cast<uint16_t>((high >> 16) & 0xFFFF),
- static_cast<uint16_t>(high & 0xFFFF),
- static_cast<uint16_t>((low >> 48) & 0xFFFF),
- static_cast<uint16_t>((low >> 32) & 0xFFFF),
- static_cast<uint16_t>((low >> 16) & 0xFFFF),
- static_cast<uint16_t>(low & 0xFFFF)};
-
- uint8_t zero_start = 0, zero_end = 0;
-
- while (zero_start < 8 && zero_end < 8) {
- if (fields[zero_start] != 0) {
- zero_start++;
- zero_end = zero_start;
- continue;
- }
-
- while (zero_end < 7 && fields[zero_end + 1] == 0) {
- zero_end++;
- }
-
- if (zero_end > zero_start) {
- break;
- }
-
- zero_start++;
- zero_end = zero_start;
- }
-
- std::stringstream ss;
-
- if (zero_start == zero_end) {
- for (uint8_t i = 0; i < 7; ++i) {
- ss << std::hex << fields[i] << ":";
- }
- ss << std::hex << fields[7];
- } else {
- for (uint8_t i = 0; i < zero_start; ++i) {
- ss << std::hex << fields[i] << ":";
- }
-
- if (zero_end == 7) {
- ss << ":";
- } else {
- for (uint8_t j = zero_end + 1; j < 8; ++j) {
- ss << std::hex << ":" << fields[j];
- }
- }
- }
-
- return ss.str();
- }
-
- [[nodiscard]] std::string to_binary_string() const { return to_binary_string(_value); }
-
- static std::string to_binary_string(vectorized::IPv6 x) {
- uint64_t low = static_cast<uint64_t>(x);
- uint64_t high = static_cast<uint64_t>(x >> 64);
-
- uint8_t fields[16] = {static_cast<uint8_t>((high >> 56) & 0xFF),
- static_cast<uint8_t>((high >> 48) & 0xFF),
- static_cast<uint8_t>((high >> 40) & 0xFF),
- static_cast<uint8_t>((high >> 32) & 0xFF),
- static_cast<uint8_t>((high >> 24) & 0xFF),
- static_cast<uint8_t>((high >> 16) & 0xFF),
- static_cast<uint8_t>((high >> 8) & 0xFF),
- static_cast<uint8_t>(high & 0xFF),
- static_cast<uint8_t>((low >> 56) & 0xFF),
- static_cast<uint8_t>((low >> 48) & 0xFF),
- static_cast<uint8_t>((low >> 40) & 0xFF),
- static_cast<uint8_t>((low >> 32) & 0xFF),
- static_cast<uint8_t>((low >> 24) & 0xFF),
- static_cast<uint8_t>((low >> 16) & 0xFF),
- static_cast<uint8_t>((low >> 8) & 0xFF),
- static_cast<uint8_t>(low & 0xFF)};
-
- std::stringstream ss;
-
- for (int i = 0; i < 16; ++i) {
- ss << (char)fields[i];
- }
-
- return ss.str();
+ const char* src = ipv6_str.c_str();
+ const char* end = ipv6_str.c_str() + ipv6_str.size() - 1;
+ while (std::isspace(*src)) ++src;
+ while (std::isspace(*end)) --end;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
while (std::isspace(*end)) { --end;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Variant) add variant config in frontend meta [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #29185:
URL: https://github.com/apache/doris/pull/29185#issuecomment-1882815564
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org