You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "liugddx (via GitHub)" <gi...@apache.org> on 2023/11/27 14:46:40 UTC
[PR] [regression test](broker load) add case for without filepath [doris]
liugddx opened a new pull request, #27658:
URL: https://github.com/apache/doris/pull/27658
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [regression test](broker load) add case for without filepath [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27658:
URL: https://github.com/apache/doris/pull/27658#issuecomment-1840637509
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 41522040609e05856b33ea89605b0788c6f9e862, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4641 4501 4469 4469
q2 362 149 160 149
q3 1464 1261 1240 1240
q4 1113 962 916 916
q5 3150 3163 3201 3163
q6 247 125 126 125
q7 1008 488 488 488
q8 2210 2238 2193 2193
q9 6639 6717 6683 6683
q10 3189 3257 3269 3257
q11 322 211 207 207
q12 353 212 216 212
q13 4553 3802 3823 3802
q14 237 212 221 212
q15 568 528 519 519
q16 433 388 380 380
q17 1022 648 558 558
q18 7436 8084 7828 7828
q19 1527 1419 1385 1385
q20 521 335 324 324
q21 3109 2645 2675 2645
q22 348 288 293 288
Total cold run time: 44452 ms
Total hot run time: 41043 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4393 4408 4371 4371
q2 266 167 164 164
q3 3511 3521 3516 3516
q4 2387 2377 2378 2377
q5 5750 5755 5743 5743
q6 238 120 120 120
q7 2376 1861 1880 1861
q8 3511 3501 3518 3501
q9 9080 9018 9010 9010
q10 3914 3993 3973 3973
q11 500 422 399 399
q12 765 595 589 589
q13 4282 3536 3560 3536
q14 274 240 242 240
q15 569 524 527 524
q16 502 455 429 429
q17 1874 1851 1862 1851
q18 8539 8252 8229 8229
q19 1716 1779 1725 1725
q20 2253 1959 1966 1959
q21 6535 6140 6158 6140
q22 499 439 422 422
Total cold run time: 63734 ms
Total hot run time: 60679 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [regression test](broker load) add case for without filepath [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27658:
URL: https://github.com/apache/doris/pull/27658#issuecomment-1844053236
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [regression test](broker load) add case for without filepath [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27658:
URL: https://github.com/apache/doris/pull/27658#issuecomment-1833383929
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [regression test](broker load) add case for without filepath [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #27658:
URL: https://github.com/apache/doris/pull/27658#discussion_r1413132515
##########
regression-test/suites/load_p0/broker_load/test_broker_load_without_filepath.groovy:
##########
@@ -0,0 +1,95 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_broker_load_without_filepath", "load_p0") {
+ // define a sql table
+ def testTable = "tbl_test_broker_load_without_filepath"
+
+ def create_test_table = {testTablex ->
+ def result1 = sql """
+ CREATE TABLE IF NOT EXISTS ${testTable} (
+ `k1` BIGINT NOT NULL,
+ `k2` DATE NULL,
+ `k3` INT(11) NOT NULL,
+ `k4` INT(11) NOT NULL,
+ `v5` BIGINT SUM NULL DEFAULT "0"
+ ) ENGINE=OLAP
+ AGGREGATE KEY(`k1`, `k2`, `k3`, `k4`)
+ COMMENT 'OLAP'
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 16
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "storage_format" = "V2"
+ );
+ """
+
+ // DDL/DML return 1 row and 3 column, the only value is update row count
+ assertTrue(result1.size() == 1)
+ assertTrue(result1[0].size() == 1)
+ assertTrue(result1[0][0] == 0, "Create table should update 0 rows")
+
+ // insert 1 row to check whether the table is ok
+ def result2 = sql """ INSERT INTO ${testTable} VALUES
+ (1,2023-09-01,1,1,1)
+ """
+ assertTrue(result2.size() == 1)
+ assertTrue(result2[0].size() == 1)
+ assertTrue(result2[0][0] == 1, "Insert should update 1 rows")
+ }
+
+ def load_from_hdfs_norm = {testTablex, label, hdfsFilePath, format, brokerName, hdfsUser, hdfsPasswd ->
+ try {
+ sql """
Review Comment:
2 issues:
1. If this sql does not throw exception, than your `assertTrue` will not be executed.
2. for exception test, you can just use `test` action, eg:
```
test {
sql "xxxx";
exception "your excepted error message"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [regression test](broker load) add case for without filepath [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27658:
URL: https://github.com/apache/doris/pull/27658#issuecomment-1833383867
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [regression test](broker load) add case for without filepath [doris]
Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on PR #27658:
URL: https://github.com/apache/doris/pull/27658#issuecomment-1830031555
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [regression test](broker load) add case for without filepath [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27658:
URL: https://github.com/apache/doris/pull/27658#issuecomment-1832368303
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.6 seconds
stream load tsv: 573 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162618217 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [regression test](broker load) add case for without filepath [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27658:
URL: https://github.com/apache/doris/pull/27658#issuecomment-1833395690
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit ecfefa1d2040a2a7c84ce3afb91a5e4dedd8fe6d, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4903 4650 4658 4650
q2 352 159 161 159
q3 1510 1306 1263 1263
q4 1150 968 965 965
q5 3262 3261 3247 3247
q6 253 129 125 125
q7 1031 518 548 518
q8 2272 2237 2222 2222
q9 6929 6995 6962 6962
q10 3300 3366 3363 3363
q11 337 212 206 206
q12 354 209 224 209
q13 4663 3981 3868 3868
q14 248 229 215 215
q15 590 547 527 527
q16 438 401 375 375
q17 1029 706 587 587
q18 8138 7744 8037 7744
q19 1549 1546 1503 1503
q20 540 325 294 294
q21 3371 2986 2993 2986
q22 371 298 301 298
Total cold run time: 46590 ms
Total hot run time: 42286 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4607 4619 4541 4541
q2 309 201 249 201
q3 3744 3756 3757 3756
q4 2526 2499 2511 2499
q5 6210 6171 6203 6171
q6 245 124 123 123
q7 2609 1989 1971 1971
q8 3719 3691 3782 3691
q9 9455 9392 9455 9392
q10 4048 4129 4158 4129
q11 653 504 507 504
q12 819 613 598 598
q13 4362 3603 3630 3603
q14 291 252 252 252
q15 581 523 532 523
q16 516 473 480 473
q17 2155 2061 2057 2057
q18 9639 8983 9007 8983
q19 1807 1759 1774 1759
q20 2317 1996 1984 1984
q21 7345 6827 6977 6827
q22 619 551 554 551
Total cold run time: 68576 ms
Total hot run time: 64588 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [regression test](broker load) add case for without filepath [doris]
Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on PR #27658:
URL: https://github.com/apache/doris/pull/27658#issuecomment-1827977558
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [regression test](broker load) add case for without filepath [doris]
Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on PR #27658:
URL: https://github.com/apache/doris/pull/27658#issuecomment-1840172610
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [regression test](broker load) add case for without filepath [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #27658:
URL: https://github.com/apache/doris/pull/27658
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [regression test](broker load) add case for without filepath [doris]
Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on PR #27658:
URL: https://github.com/apache/doris/pull/27658#issuecomment-1832047965
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [regression test](broker load) add case for without filepath [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27658:
URL: https://github.com/apache/doris/pull/27658#issuecomment-1840549498
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.16 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.4 seconds inserted 10000000 Rows, about 352K ops/s
storage size: 17163788579 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org