You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "wyxxxcat (via GitHub)" <gi...@apache.org> on 2024/02/20 10:13:02 UTC
[PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
wyxxxcat opened a new pull request, #31163:
URL: https://github.com/apache/doris/pull/31163
## Proposed changes
Issue Number: #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1956153170
<details>
<summary>TPC-H: <b>Total hot run time: 41495 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 840cd2ff453dc888342a2d7017bf8c554d73f671, data reload: false
------ Round 1 ----------------------------------
q1 17684 5003 4885 4885
q2 2035 147 134 134
q3 10571 1020 1013 1013
q4 4652 964 984 964
q5 7698 3204 3290 3204
q6 193 143 138 138
q7 1284 775 764 764
q8 9245 2054 2076 2054
q9 7589 6677 6658 6658
q10 8312 2650 2656 2650
q11 430 228 215 215
q12 764 332 337 332
q13 17991 3645 3657 3645
q14 290 263 254 254
q15 594 510 508 508
q16 476 408 418 408
q17 932 885 862 862
q18 7453 6746 6656 6656
q19 1558 1505 1501 1501
q20 632 348 330 330
q21 6640 3966 3988 3966
q22 884 358 354 354
Total cold run time: 107907 ms
Total hot run time: 41495 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4859 4850 4834 4834
q2 289 186 192 186
q3 3610 3614 3590 3590
q4 2580 2554 2560 2554
q5 5807 5772 5764 5764
q6 207 132 131 131
q7 2268 1670 1645 1645
q8 3039 3098 3092 3092
q9 8729 8684 8671 8671
q10 6770 4231 4234 4231
q11 545 411 408 408
q12 781 563 534 534
q13 4557 3441 3421 3421
q14 259 241 230 230
q15 635 514 498 498
q16 486 464 449 449
q17 1699 1617 1612 1612
q18 8476 7662 7744 7662
q19 1641 1629 1650 1629
q20 2132 1848 1817 1817
q21 6768 6355 6358 6355
q22 581 523 527 523
Total cold run time: 66718 ms
Total hot run time: 59836 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1954317122
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1956194904
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1954910098
TeamCity be ut coverage result:
Function Coverage: 35.66% (8528/23917)
Line Coverage: 27.61% (69119/250374)
Region Coverage: 26.70% (35819/134130)
Branch Coverage: 23.53% (18317/77854)
Coverage Report: http://coverage.selectdb-in.cc/coverage/45cb021250160648ef60000ebbcdf80ffbae9170_45cb021250160648ef60000ebbcdf80ffbae9170/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "wyxxxcat (via GitHub)" <gi...@apache.org>.
wyxxxcat commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1955775602
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on code in PR #31163:
URL: https://github.com/apache/doris/pull/31163#discussion_r1496180572
##########
be/test/vec/columns/column_hash_func_test.cpp:
##########
@@ -113,157 +84,4 @@ TEST(HashFuncTest, ArraySimpleBenchmarkTest) {
std::cout << time_t << "ns" << std::endl;
}
-TEST(HashFuncTest, ArrayNestedArrayTest) {
- DataTypes dataTypes = create_scala_data_types();
-
- DataTypePtr d = std::make_shared<DataTypeInt64>();
- MutableColumnPtr scala_mutable_col = d->create_column();
- DataTypePtr nested_array_ptr = std::make_shared<DataTypeArray>(d);
- DataTypePtr array_ptr = std::make_shared<DataTypeArray>(nested_array_ptr);
- MutableColumnPtr array_mutable_col = array_ptr->create_column();
-
- Array a, a1, a2, a3, nested, nested1;
- nested.push_back(Int64(1));
- nested1.push_back(Int64(2));
-
- // a: [[1], [2]]
- a.push_back(nested);
- a.push_back(nested1);
- // a1: [[2], [1]]
- a1.push_back(nested1);
- a1.push_back(nested);
-
- // a2: [[], [1]]
- a2.push_back(Array());
- a2.push_back(nested);
- // a3: [[1], []]
- a3.push_back(nested);
- a3.push_back(Array());
-
- array_mutable_col->insert(a);
- array_mutable_col->insert(a1);
- array_mutable_col->insert(a2);
- array_mutable_col->insert(a3);
-
- auto nested_col =
- reinterpret_cast<vectorized::ColumnArray*>(array_mutable_col.get())->get_data_ptr();
- EXPECT_EQ(nested_col->size(), 8);
-
- std::vector<uint64_t> xx_hash_vals(4);
- std::vector<uint32_t> crc_hash_vals(4);
- auto* __restrict xx_hashes = xx_hash_vals.data();
- auto* __restrict crc_hashes = crc_hash_vals.data();
-
- // xxHash
- EXPECT_NO_FATAL_FAILURE(array_mutable_col->update_hashes_with_value(xx_hashes));
- EXPECT_TRUE(xx_hashes[0] != xx_hashes[1]);
- EXPECT_TRUE(xx_hashes[2] != xx_hashes[3]);
- // crcHash
- EXPECT_NO_FATAL_FAILURE(
- array_mutable_col->update_crcs_with_value(crc_hashes, PrimitiveType::TYPE_ARRAY, 4));
- EXPECT_TRUE(crc_hashes[0] != crc_hashes[1]);
- EXPECT_TRUE(crc_hashes[2] != crc_hashes[3]);
-}
-
-TEST(HashFuncTest, ArrayCornerCaseTest) {
- DataTypes dataTypes = create_scala_data_types();
-
- DataTypePtr d = std::make_shared<DataTypeInt64>();
- DataTypePtr a = std::make_shared<DataTypeArray>(d);
- MutableColumnPtr array_mutable_col = a->create_column();
- Array a1, a2;
- a1.push_back(Int64(1));
- a1.push_back(Int64(2));
- a1.push_back(Int64(3));
- array_mutable_col->insert(a1);
- array_mutable_col->insert(a1);
- a2.push_back(Int64(11));
- a2.push_back(Int64(12));
- a2.push_back(Int64(13));
- array_mutable_col->insert(a2);
-
- EXPECT_EQ(array_mutable_col->size(), 3);
-
- std::vector<uint64_t> sip_hash_vals(3);
- std::vector<uint64_t> xx_hash_vals(3);
- std::vector<uint32_t> crc_hash_vals(3);
- auto* __restrict sip_hashes = sip_hash_vals.data();
- auto* __restrict xx_hashes = xx_hash_vals.data();
- auto* __restrict crc_hashes = crc_hash_vals.data();
-
- // sipHash
- std::vector<SipHash> siphashs(3);
- array_mutable_col->update_hashes_with_value(siphashs);
- EXPECT_NO_FATAL_FAILURE(array_mutable_col->update_hashes_with_value(siphashs));
Review Comment:
don't delete the whole testcase. only delete sipHash related
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1956034850
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1956090475
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "wyxxxcat (via GitHub)" <gi...@apache.org>.
wyxxxcat commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1954778164
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "wyxxxcat (via GitHub)" <gi...@apache.org>.
wyxxxcat commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1954784225
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1954315790
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1953918679
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1954307209
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1955834394
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1954779010
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/7977919163") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In build.sh line 285:
local submodule_path=$1
^---------------------^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 286:
local submodule_name=$2
^---------------------^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 287:
local archive_url=$3
^------------------^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 289:
set +e
^----^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 290:
cd "${DORIS_HOME}"
^----------------^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 291:
echo "Update ${submodule_name} submodule ..."
^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 292:
git submodule update --init --recursive "${submodule_path}"
^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 293:
exit_code=$?
^----------^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 294:
if [[ "${exit_code}" -eq 0 ]]; then
^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
^------------------------^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 295:
cd "${submodule_path}"
^--------------------^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 296:
submodule_commit_id=$(git rev-parse HEAD)
^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
^----------------^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 297:
cd -
^--^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 298:
expect_submodule_commit_id=$(git ls-tree HEAD "${submodule_path}" | awk '{print $3}')
^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 299:
echo "Current commit ID of ${submodule_name} submodule: ${submodule_commit_id}, expected is ${expect_submodule_commit_id}"
^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 301:
set -e
^----^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 302:
if [[ "${exit_code}" -ne 0 ]]; then
^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
^------------------------^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 303:
set +e
^----^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 305:
submodule_commit=$(git ls-tree HEAD "${submodule_path}" | awk '{print $3}')
^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 306:
exit_code=$?
^----------^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 307:
if [[ "${exit_code}" = "0" ]]; then
^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
^------------------------^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 308:
commit_specific_url=$(echo "${archive_url}" | sed "s/refs\/heads/${submodule_commit}/")
^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 310:
commit_specific_url="${archive_url}"
^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 312:
set -e
^----^ SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 313:
echo "Update ${submodule_name} submodule failed, start to download and extract ${commit_specific_url}"
^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 315:
mkdir -p "${DORIS_HOME}/${submodule_path}"
^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
In build.sh line 316:
curl -L "${commit_specific_url}" | tar -xz -C "${DORIS_HOME}/${submodule_path}" --strip-components=1
^-- SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
For more information:
https://www.shellcheck.net/wiki/SC2317 -- Command appears to be unreachable...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' found no issues.
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "wyxxxcat (via GitHub)" <gi...@apache.org>.
wyxxxcat commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1955821904
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1956181466
<details>
<summary>ClickBench: <b>Total hot run time: 30.76 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 840cd2ff453dc888342a2d7017bf8c554d73f671, data reload: false
query1 0.03 0.02 0.02
query2 0.06 0.03 0.02
query3 0.23 0.08 0.07
query4 1.65 0.09 0.08
query5 0.49 0.48 0.49
query6 1.37 0.61 0.61
query7 0.02 0.01 0.01
query8 0.04 0.02 0.02
query9 0.53 0.45 0.46
query10 0.48 0.50 0.51
query11 0.13 0.10 0.09
query12 0.12 0.10 0.10
query13 0.59 0.60 0.59
query14 0.77 0.78 0.81
query15 0.82 0.80 0.81
query16 0.33 0.33 0.33
query17 0.89 0.90 0.90
query18 0.18 0.15 0.17
query19 1.78 1.71 1.70
query20 0.01 0.02 0.01
query21 15.39 0.60 0.59
query22 3.07 3.89 2.23
query23 17.24 1.07 1.00
query24 2.01 0.56 0.20
query25 0.61 0.07 0.06
query26 0.17 0.15 0.14
query27 0.06 0.06 0.05
query28 12.20 0.84 0.82
query29 12.56 3.40 3.31
query30 0.51 0.50 0.46
query31 2.77 0.36 0.37
query32 3.34 0.48 0.48
query33 3.09 3.15 3.14
query34 15.55 4.51 4.47
query35 4.48 4.53 4.52
query36 1.07 0.95 0.96
query37 0.07 0.05 0.04
query38 0.04 0.03 0.03
query39 0.02 0.01 0.01
query40 0.20 0.15 0.15
query41 0.07 0.02 0.01
query42 0.03 0.01 0.02
query43 0.02 0.02 0.02
Total cold run time: 105.09 s
Total hot run time: 30.76 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1954920954
<details>
<summary>TPC-DS: <b>Total hot run time: 178319 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 45cb021250160648ef60000ebbcdf80ffbae9170, data reload: false
query1 914 362 346 346
query2 6504 1821 1800 1800
query3 6708 209 220 209
query4 23215 21134 21249 21134
query5 4296 386 464 386
query6 259 167 167 167
query7 4611 297 301 297
query8 251 192 214 192
query9 8445 2846 2867 2846
query10 415 228 224 224
query11 15044 14544 14523 14523
query12 138 83 82 82
query13 1686 416 414 414
query14 9080 7885 7867 7867
query15 214 189 193 189
query16 7429 264 251 251
query17 1409 552 542 542
query18 1953 273 269 269
query19 194 151 146 146
query20 87 84 87 84
query21 191 114 116 114
query22 4837 4724 4665 4665
query23 32697 31689 31521 31521
query24 12744 3474 3410 3410
query25 646 372 360 360
query26 1872 155 162 155
query27 3041 312 317 312
query28 6645 1830 1823 1823
query29 1129 653 613 613
query30 287 143 146 143
query31 938 764 757 757
query32 102 63 58 58
query33 711 236 236 236
query34 1056 490 503 490
query35 931 850 835 835
query36 966 903 881 881
query37 183 64 61 61
query38 3346 3211 3195 3195
query39 1377 1353 1328 1328
query40 286 103 103 103
query41 36 35 34 34
query42 106 102 95 95
query43 490 441 458 441
query44 1074 697 705 697
query45 195 190 181 181
query46 1065 786 765 765
query47 1664 1494 1534 1494
query48 410 349 345 345
query49 1220 309 306 306
query50 795 365 376 365
query51 5356 5280 5226 5226
query52 104 92 96 92
query53 390 315 307 307
query54 286 227 233 227
query55 85 80 82 80
query56 229 214 194 194
query57 1092 958 969 958
query58 227 205 199 199
query59 2240 2181 2238 2181
query60 251 212 222 212
query61 87 84 84 84
query62 593 383 381 381
query63 330 292 285 285
query64 6403 3097 3140 3097
query65 3283 3255 3236 3236
query66 1346 329 348 329
query67 14651 14348 14424 14348
query68 5166 570 541 541
query69 523 369 354 354
query70 1188 1244 1222 1222
query71 380 251 254 251
query72 6359 2776 2635 2635
query73 697 308 318 308
query74 6912 6451 6434 6434
query75 3212 2568 2534 2534
query76 3236 1160 1215 1160
query77 367 240 236 236
query78 9461 8795 8823 8795
query79 972 505 505 505
query80 517 344 354 344
query81 441 210 207 207
query82 166 84 87 84
query83 141 126 120 120
query84 234 79 76 76
query85 1021 354 346 346
query86 299 315 305 305
query87 3435 3284 3292 3284
query88 2701 2281 2291 2281
query89 455 358 366 358
query90 1940 165 167 165
query91 162 130 128 128
query92 55 47 53 47
query93 958 477 457 457
query94 1108 181 180 180
query95 472 373 8741 373
query96 589 265 266 265
query97 4447 4248 4309 4248
query98 229 209 197 197
query99 1042 729 692 692
Total cold run time: 269730 ms
Total hot run time: 178319 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "wyxxxcat (via GitHub)" <gi...@apache.org>.
wyxxxcat commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1954281933
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on code in PR #31163:
URL: https://github.com/apache/doris/pull/31163#discussion_r1496178190
##########
be/src/vec/columns/column.h:
##########
@@ -366,19 +353,11 @@ class IColumn : public COW<IColumn> {
/// null_data to mark whether need to do hash compute, null_data == nullptr
/// means all element need to do hash function, else only *null_data != 0 need to do hash func
/// do xxHash here, faster than other hash method
- virtual void update_hashes_with_value(std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data = nullptr) const {
- LOG(FATAL) << get_name() << " update_hashes_with_value siphash not supported";
- }
/// Update state of hash function with value of n elements to avoid the virtual function call
/// null_data to mark whether need to do hash compute, null_data == nullptr
/// means all element need to do hash function, else only *null_data != 0 need to do hash func
/// do xxHash here, faster than other sip hash
- virtual void update_hashes_with_value(uint64_t* __restrict hashes,
Review Comment:
Don't delete this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1955785041
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1954904807
<details>
<summary>TPC-H: <b>Total hot run time: 41659 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 45cb021250160648ef60000ebbcdf80ffbae9170, data reload: false
------ Round 1 ----------------------------------
q1 17589 5606 4894 4894
q2 2049 144 139 139
q3 10788 1057 1044 1044
q4 4991 1006 1009 1006
q5 7709 3275 3326 3275
q6 198 138 140 138
q7 1271 795 773 773
q8 9259 2103 2081 2081
q9 7735 6778 6706 6706
q10 8315 2671 2642 2642
q11 412 215 209 209
q12 761 334 355 334
q13 18017 3697 3684 3684
q14 289 254 255 254
q15 621 520 497 497
q16 487 410 416 410
q17 942 882 842 842
q18 7620 6722 6651 6651
q19 1563 1511 1517 1511
q20 635 357 323 323
q21 5939 3903 3964 3903
q22 884 351 343 343
Total cold run time: 108074 ms
Total hot run time: 41659 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4879 4888 4897 4888
q2 296 184 196 184
q3 3605 3602 3579 3579
q4 2561 2552 2536 2536
q5 5796 5791 5815 5791
q6 219 129 126 126
q7 2259 1701 1714 1701
q8 3038 3129 3141 3129
q9 8803 8775 8751 8751
q10 6932 4246 4256 4246
q11 532 387 371 371
q12 781 546 552 546
q13 4223 3455 3402 3402
q14 268 237 250 237
q15 599 499 534 499
q16 482 465 452 452
q17 1694 1643 1634 1634
q18 8329 7649 7581 7581
q19 1645 1653 1641 1641
q20 2131 1830 1833 1830
q21 6634 6225 6292 6225
q22 586 533 529 529
Total cold run time: 66292 ms
Total hot run time: 59878 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "wyxxxcat (via GitHub)" <gi...@apache.org>.
wyxxxcat commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1956076693
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1954302274
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1954799909
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1953904699
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1955866386
TeamCity be ut coverage result:
Function Coverage: 35.69% (8548/23949)
Line Coverage: 27.57% (69334/251487)
Region Coverage: 26.71% (35968/134651)
Branch Coverage: 23.52% (18391/78178)
Coverage Report: http://coverage.selectdb-in.cc/coverage/486106f6b393a0c5470ac04d5683f741ca5ff1d3_486106f6b393a0c5470ac04d5683f741ca5ff1d3/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "wyxxxcat (via GitHub)" <gi...@apache.org>.
wyxxxcat commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1956070376
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1953917540
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1954932900
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 45cb021250160648ef60000ebbcdf80ffbae9170 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.2 seconds inserted 10000000 Rows, about 757K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1956186462
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 840cd2ff453dc888342a2d7017bf8c554d73f671 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.5 seconds inserted 10000000 Rows, about 740K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "wyxxxcat (via GitHub)" <gi...@apache.org>.
wyxxxcat closed pull request #31163: [chore](columns) remove update_hashes_with_value for SipHash
URL: https://github.com/apache/doris/pull/31163
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1956046982
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1955860060
<details>
<summary>TPC-H: <b>Total hot run time: 41694 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 486106f6b393a0c5470ac04d5683f741ca5ff1d3, data reload: false
------ Round 1 ----------------------------------
q1 5245 4840 4860 4840
q2 1546 142 136 136
q3 5357 1023 1020 1020
q4 4769 972 1000 972
q5 7726 3265 3296 3265
q6 196 138 139 138
q7 1274 786 783 783
q8 9414 2104 2110 2104
q9 7736 6771 6700 6700
q10 8376 2674 2643 2643
q11 411 229 220 220
q12 1406 340 343 340
q13 18055 3668 3722 3668
q14 297 267 259 259
q15 585 520 493 493
q16 487 413 418 413
q17 963 888 860 860
q18 7455 6667 6738 6667
q19 1556 1489 1492 1489
q20 645 367 345 345
q21 6144 3992 3999 3992
q22 869 347 355 347
Total cold run time: 90512 ms
Total hot run time: 41694 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4889 4826 4823 4823
q2 288 183 182 182
q3 3618 3600 3584 3584
q4 2506 2561 2494 2494
q5 5761 5746 5765 5746
q6 212 126 133 126
q7 2262 1671 1679 1671
q8 3008 3071 3112 3071
q9 8764 8743 8705 8705
q10 6839 4259 4259 4259
q11 533 396 412 396
q12 792 559 566 559
q13 4237 3410 3400 3400
q14 273 241 246 241
q15 584 505 509 505
q16 482 433 424 424
q17 1690 1604 1602 1602
q18 8353 7620 7644 7620
q19 1659 1654 1642 1642
q20 2133 1847 1851 1847
q21 6609 6261 6136 6136
q22 582 511 513 511
Total cold run time: 66074 ms
Total hot run time: 59544 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1956172381
<details>
<summary>TPC-DS: <b>Total hot run time: 176982 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 840cd2ff453dc888342a2d7017bf8c554d73f671, data reload: false
query1 926 361 344 344
query2 6521 1732 1691 1691
query3 6696 202 203 202
query4 23347 21323 21231 21231
query5 4241 377 369 369
query6 259 170 161 161
query7 4613 295 285 285
query8 247 194 202 194
query9 8445 2870 2844 2844
query10 421 238 230 230
query11 15006 14549 14485 14485
query12 145 89 81 81
query13 1702 420 415 415
query14 9086 7541 7676 7541
query15 209 196 188 188
query16 7388 252 252 252
query17 1427 576 536 536
query18 1945 270 262 262
query19 192 141 146 141
query20 87 83 82 82
query21 185 122 123 122
query22 4738 4652 4773 4652
query23 32537 31588 31525 31525
query24 12766 3364 3421 3364
query25 636 369 358 358
query26 1877 155 169 155
query27 3040 307 309 307
query28 6632 1882 1858 1858
query29 1150 620 610 610
query30 278 134 148 134
query31 936 748 767 748
query32 96 61 58 58
query33 728 231 235 231
query34 1082 486 491 486
query35 935 831 803 803
query36 962 887 951 887
query37 177 62 64 62
query38 3307 3196 3157 3157
query39 1367 1334 1341 1334
query40 282 103 103 103
query41 36 35 34 34
query42 106 95 94 94
query43 481 440 443 440
query44 1057 693 706 693
query45 200 179 176 176
query46 1038 785 750 750
query47 1636 1590 1482 1482
query48 421 349 358 349
query49 1231 299 302 299
query50 774 371 372 371
query51 5277 5135 5200 5135
query52 110 95 90 90
query53 390 299 294 294
query54 292 216 232 216
query55 85 78 76 76
query56 217 214 197 197
query57 1014 948 959 948
query58 217 204 202 202
query59 2254 2110 2062 2062
query60 247 222 214 214
query61 82 89 82 82
query62 593 375 371 371
query63 335 288 293 288
query64 6427 3039 3139 3039
query65 3292 3278 3247 3247
query66 1349 338 321 321
query67 14560 14258 14140 14140
query68 5151 570 556 556
query69 515 354 362 354
query70 1262 1194 1208 1194
query71 436 250 275 250
query72 6288 2760 2590 2590
query73 704 309 314 309
query74 6898 6529 6443 6443
query75 3255 2588 2559 2559
query76 3319 1124 1180 1124
query77 360 238 227 227
query78 9437 8802 8674 8674
query79 993 502 510 502
query80 515 354 345 345
query81 437 208 206 206
query82 163 84 85 84
query83 137 118 124 118
query84 224 80 75 75
query85 1029 342 338 338
query86 286 303 298 298
query87 3517 3260 3232 3232
query88 2763 2294 2314 2294
query89 437 363 360 360
query90 1977 162 168 162
query91 154 126 125 125
query92 56 50 47 47
query93 994 518 498 498
query94 1128 183 182 182
query95 491 383 371 371
query96 574 273 263 263
query97 4463 4224 4264 4224
query98 221 206 201 201
query99 1077 689 679 679
Total cold run time: 269596 ms
Total hot run time: 176982 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on code in PR #31163:
URL: https://github.com/apache/doris/pull/31163#discussion_r1495570526
##########
be/src/vec/columns/column_array.cpp:
##########
@@ -291,11 +291,6 @@ void ColumnArray::update_hash_with_value(size_t n, SipHash& hash) const {
for (size_t i = 0; i < array_size; ++i) get_data().update_hash_with_value(offset + i, hash);
}
-void ColumnArray::update_hashes_with_value(std::vector<SipHash>& hashes,
- const uint8_t* __restrict null_data) const {
- SIP_HASHES_FUNCTION_COLUMN_IMPL();
Review Comment:
check if it's safe to delete this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1953899584
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1953921939
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "wyxxxcat (via GitHub)" <gi...@apache.org>.
wyxxxcat commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1954287103
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1954311322
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1953883379
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1956135219
TeamCity be ut coverage result:
Function Coverage: 35.69% (8546/23948)
Line Coverage: 27.57% (69340/251478)
Region Coverage: 26.71% (35972/134651)
Branch Coverage: 23.53% (18397/78176)
Coverage Report: http://coverage.selectdb-in.cc/coverage/840cd2ff453dc888342a2d7017bf8c554d73f671_840cd2ff453dc888342a2d7017bf8c554d73f671/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1955852951
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1954928627
<details>
<summary>ClickBench: <b>Total hot run time: 31.44 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 45cb021250160648ef60000ebbcdf80ffbae9170, data reload: false
query1 0.03 0.03 0.02
query2 0.06 0.02 0.02
query3 0.24 0.07 0.07
query4 1.68 0.09 0.08
query5 0.49 0.49 0.48
query6 1.39 0.61 0.61
query7 0.02 0.01 0.01
query8 0.04 0.02 0.03
query9 0.51 0.46 0.44
query10 0.48 0.50 0.49
query11 0.12 0.10 0.09
query12 0.12 0.09 0.10
query13 0.59 0.59 0.58
query14 0.76 0.79 0.78
query15 0.81 0.80 0.79
query16 0.33 0.34 0.33
query17 0.90 0.90 0.91
query18 0.19 0.16 0.16
query19 1.77 1.64 1.69
query20 0.02 0.01 0.01
query21 15.41 0.62 0.61
query22 2.92 3.76 2.89
query23 17.50 1.00 0.99
query24 2.01 0.54 0.38
query25 0.60 0.06 0.06
query26 0.17 0.13 0.14
query27 0.05 0.05 0.05
query28 12.05 0.86 0.83
query29 12.51 3.36 3.26
query30 0.54 0.49 0.49
query31 2.76 0.35 0.36
query32 3.37 0.48 0.47
query33 3.09 3.17 3.11
query34 15.35 4.51 4.49
query35 4.51 4.49 4.50
query36 1.08 0.95 0.94
query37 0.06 0.05 0.05
query38 0.04 0.02 0.03
query39 0.02 0.01 0.02
query40 0.17 0.16 0.15
query41 0.07 0.02 0.02
query42 0.02 0.02 0.02
query43 0.03 0.02 0.02
Total cold run time: 104.88 s
Total hot run time: 31.44 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on code in PR #31163:
URL: https://github.com/apache/doris/pull/31163#discussion_r1496805012
##########
be/src/vec/columns/column.h:
##########
@@ -395,6 +387,10 @@ class IColumn : public COW<IColumn> {
LOG(FATAL) << get_name() << "update_crcs_with_value not supported";
}
+ virtual void update_hashes_with_value(uint64_t* __restrict hashes,
Review Comment:
dont move them
##########
be/test/vec/columns/column_hash_func_test.cpp:
##########
@@ -113,157 +84,4 @@ TEST(HashFuncTest, ArraySimpleBenchmarkTest) {
std::cout << time_t << "ns" << std::endl;
}
-TEST(HashFuncTest, ArrayNestedArrayTest) {
- DataTypes dataTypes = create_scala_data_types();
-
- DataTypePtr d = std::make_shared<DataTypeInt64>();
- MutableColumnPtr scala_mutable_col = d->create_column();
- DataTypePtr nested_array_ptr = std::make_shared<DataTypeArray>(d);
- DataTypePtr array_ptr = std::make_shared<DataTypeArray>(nested_array_ptr);
- MutableColumnPtr array_mutable_col = array_ptr->create_column();
-
- Array a, a1, a2, a3, nested, nested1;
- nested.push_back(Int64(1));
- nested1.push_back(Int64(2));
-
- // a: [[1], [2]]
- a.push_back(nested);
- a.push_back(nested1);
- // a1: [[2], [1]]
- a1.push_back(nested1);
- a1.push_back(nested);
-
- // a2: [[], [1]]
- a2.push_back(Array());
- a2.push_back(nested);
- // a3: [[1], []]
- a3.push_back(nested);
- a3.push_back(Array());
-
- array_mutable_col->insert(a);
- array_mutable_col->insert(a1);
- array_mutable_col->insert(a2);
- array_mutable_col->insert(a3);
-
- auto nested_col =
- reinterpret_cast<vectorized::ColumnArray*>(array_mutable_col.get())->get_data_ptr();
- EXPECT_EQ(nested_col->size(), 8);
-
- std::vector<uint64_t> xx_hash_vals(4);
- std::vector<uint32_t> crc_hash_vals(4);
- auto* __restrict xx_hashes = xx_hash_vals.data();
- auto* __restrict crc_hashes = crc_hash_vals.data();
-
- // xxHash
- EXPECT_NO_FATAL_FAILURE(array_mutable_col->update_hashes_with_value(xx_hashes));
- EXPECT_TRUE(xx_hashes[0] != xx_hashes[1]);
- EXPECT_TRUE(xx_hashes[2] != xx_hashes[3]);
- // crcHash
- EXPECT_NO_FATAL_FAILURE(
- array_mutable_col->update_crcs_with_value(crc_hashes, PrimitiveType::TYPE_ARRAY, 4));
- EXPECT_TRUE(crc_hashes[0] != crc_hashes[1]);
- EXPECT_TRUE(crc_hashes[2] != crc_hashes[3]);
-}
-
-TEST(HashFuncTest, ArrayCornerCaseTest) {
- DataTypes dataTypes = create_scala_data_types();
-
- DataTypePtr d = std::make_shared<DataTypeInt64>();
- DataTypePtr a = std::make_shared<DataTypeArray>(d);
- MutableColumnPtr array_mutable_col = a->create_column();
- Array a1, a2;
- a1.push_back(Int64(1));
- a1.push_back(Int64(2));
- a1.push_back(Int64(3));
- array_mutable_col->insert(a1);
- array_mutable_col->insert(a1);
- a2.push_back(Int64(11));
- a2.push_back(Int64(12));
- a2.push_back(Int64(13));
- array_mutable_col->insert(a2);
-
- EXPECT_EQ(array_mutable_col->size(), 3);
-
- std::vector<uint64_t> sip_hash_vals(3);
- std::vector<uint64_t> xx_hash_vals(3);
- std::vector<uint32_t> crc_hash_vals(3);
- auto* __restrict sip_hashes = sip_hash_vals.data();
- auto* __restrict xx_hashes = xx_hash_vals.data();
- auto* __restrict crc_hashes = crc_hash_vals.data();
-
- // sipHash
- std::vector<SipHash> siphashs(3);
- array_mutable_col->update_hashes_with_value(siphashs);
- EXPECT_NO_FATAL_FAILURE(array_mutable_col->update_hashes_with_value(siphashs));
Review Comment:
this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "wyxxxcat (via GitHub)" <gi...@apache.org>.
wyxxxcat closed pull request #31163: [chore](columns) remove update_hashes_with_value for SipHash
URL: https://github.com/apache/doris/pull/31163
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "wyxxxcat (via GitHub)" <gi...@apache.org>.
wyxxxcat commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1955844175
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1956051326
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "wyxxxcat (via GitHub)" <gi...@apache.org>.
wyxxxcat commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1956057848
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](columns) remove update_hashes_with_value for SipHash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31163:
URL: https://github.com/apache/doris/pull/31163#issuecomment-1956131054
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org