You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "HowardQin (via GitHub)" <gi...@apache.org> on 2024/01/30 08:48:08 UTC
[PR] [improvement](load) fix lzo decompess issue and support 1.04 lzop format [doris]
HowardQin opened a new pull request, #30573:
URL: https://github.com/apache/doris/pull/30573
Issue Number: close #29406
1. Enable 1.04 lzop file format support
2. Fix issue in lzo decompress code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1925661126
<details>
<summary>TPC-H: <b>Total hot run time: 37015 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d1023484d641fbf6e681f02e8bffbb0adcb72761, data reload: false
------ Round 1 ----------------------------------
q1 17646 4584 4444 4444
q2 2055 154 136 136
q3 10582 921 910 910
q4 4675 745 678 678
q5 7695 2771 2815 2771
q6 181 119 122 119
q7 1156 739 722 722
q8 9305 2015 2008 2008
q9 7302 6369 6363 6363
q10 8138 2442 2467 2442
q11 416 212 206 206
q12 803 279 288 279
q13 18024 3300 3319 3300
q14 272 253 253 253
q15 542 503 503 503
q16 496 400 411 400
q17 948 566 566 566
q18 6820 5905 6122 5905
q19 1531 1345 1329 1329
q20 621 357 336 336
q21 6624 3153 3061 3061
q22 818 291 284 284
Total cold run time: 106650 ms
Total hot run time: 37015 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4616 4506 4453 4453
q2 320 231 224 224
q3 2989 2854 2860 2854
q4 1841 1620 1674 1620
q5 5250 5267 5273 5267
q6 193 116 115 115
q7 2172 1781 1734 1734
q8 3116 3287 3216 3216
q9 8480 8358 8324 8324
q10 5803 3587 3595 3587
q11 542 447 464 447
q12 757 584 585 584
q13 11735 3102 3106 3102
q14 282 257 264 257
q15 541 511 504 504
q16 512 469 473 469
q17 1878 1686 1691 1686
q18 8274 7740 7574 7574
q19 11148 1556 1538 1538
q20 2223 1912 1914 1912
q21 4842 4650 4694 4650
q22 546 486 481 481
Total cold run time: 78060 ms
Total hot run time: 54598 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #30573:
URL: https://github.com/apache/doris/pull/30573#discussion_r1473752925
##########
be/src/olap/utils.cpp:
##########
@@ -19,6 +19,7 @@
// IWYU pragma: no_include <bthread/errno.h>
#include <errno.h> // IWYU pragma: keep
+#include <stdarg.h>
Review Comment:
warning: inclusion of deprecated C++ header 'stdarg.h'; consider using 'cstdarg' instead [modernize-deprecated-headers]
```suggestion
#include <cstdarg>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1925670371
<details>
<summary>ClickBench: <b>Total hot run time: 30.93 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d1023484d641fbf6e681f02e8bffbb0adcb72761, data reload: false
query1 0.04 0.03 0.02
query2 0.05 0.02 0.02
query3 0.22 0.06 0.06
query4 1.67 0.09 0.09
query5 0.53 0.52 0.52
query6 1.19 0.64 0.66
query7 0.02 0.01 0.02
query8 0.04 0.02 0.02
query9 0.56 0.53 0.52
query10 0.55 0.55 0.56
query11 0.12 0.08 0.08
query12 0.11 0.10 0.09
query13 0.60 0.61 0.61
query14 0.79 0.81 0.78
query15 0.79 0.76 0.78
query16 0.38 0.39 0.40
query17 0.98 1.02 1.04
query18 0.22 0.26 0.24
query19 1.80 1.79 1.75
query20 0.02 0.01 0.01
query21 15.40 0.59 0.59
query22 2.75 2.54 1.67
query23 17.35 0.83 0.75
query24 2.65 1.59 0.98
query25 0.31 0.17 0.22
query26 0.63 0.14 0.13
query27 0.05 0.05 0.05
query28 11.07 0.84 0.86
query29 12.52 3.23 3.17
query30 0.58 0.56 0.52
query31 2.79 0.34 0.35
query32 3.37 0.49 0.47
query33 3.23 3.22 3.24
query34 15.76 4.29 4.26
query35 4.31 4.27 4.31
query36 1.11 1.06 1.05
query37 0.06 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.02 0.01
query40 0.16 0.14 0.13
query41 0.07 0.01 0.02
query42 0.02 0.02 0.02
query43 0.02 0.02 0.02
Total cold run time: 104.95 s
Total hot run time: 30.93 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) fix lzo decompess issue and support 1.04 lzop format [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1916359624
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1925668878
<details>
<summary>TPC-DS: <b>Total hot run time: 174177 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d1023484d641fbf6e681f02e8bffbb0adcb72761, data reload: false
query1 946 331 334 331
query2 6566 1924 1894 1894
query3 6702 207 199 199
query4 31851 21981 22040 21981
query5 4249 365 359 359
query6 272 169 171 169
query7 4614 280 275 275
query8 247 170 177 170
query9 9017 2312 2318 2312
query10 400 225 213 213
query11 19040 15332 15251 15251
query12 126 75 75 75
query13 1623 415 409 409
query14 9212 7172 7070 7070
query15 239 184 185 184
query16 8158 251 239 239
query17 1884 524 497 497
query18 2095 272 265 265
query19 371 142 145 142
query20 82 74 81 74
query21 192 132 128 128
query22 4812 4674 4652 4652
query23 31068 30063 29920 29920
query24 6884 2718 2751 2718
query25 545 342 338 338
query26 712 144 143 143
query27 2019 292 300 292
query28 4151 1873 1840 1840
query29 843 620 631 620
query30 279 133 142 133
query31 905 722 732 722
query32 94 51 55 51
query33 451 221 218 218
query34 829 460 466 460
query35 842 756 734 734
query36 1022 938 881 881
query37 98 58 57 57
query38 3245 3085 3073 3073
query39 1303 1240 1238 1238
query40 199 96 90 90
query41 37 35 34 34
query42 98 91 89 89
query43 520 482 454 454
query44 1083 694 700 694
query45 189 176 175 175
query46 1061 661 673 661
query47 1577 1546 1477 1477
query48 453 349 358 349
query49 1039 286 288 286
query50 771 378 370 370
query51 5238 5105 5122 5105
query52 112 91 87 87
query53 335 274 282 274
query54 251 214 220 214
query55 79 77 74 74
query56 216 199 190 190
query57 914 902 895 895
query58 201 173 178 173
query59 2316 2281 2333 2281
query60 240 214 202 202
query61 87 84 83 83
query62 599 341 353 341
query63 282 276 271 271
query64 4456 3494 3656 3494
query65 3260 3251 3239 3239
query66 803 310 308 308
query67 14339 13996 14173 13996
query68 4996 525 522 522
query69 468 338 315 315
query70 1272 1208 1233 1208
query71 336 245 246 245
query72 6422 2867 2692 2692
query73 719 322 319 319
query74 6636 6247 6216 6216
query75 3047 2333 2377 2333
query76 2988 985 955 955
query77 372 227 226 226
query78 9240 8687 8536 8536
query79 4368 500 494 494
query80 1850 350 349 349
query81 539 199 195 195
query82 1373 86 83 83
query83 244 129 127 127
query84 285 78 87 78
query85 1868 329 351 329
query86 479 302 313 302
query87 3399 3228 3255 3228
query88 4457 2310 2309 2309
query89 488 370 372 370
query90 1954 167 160 160
query91 155 123 140 123
query92 51 41 41 41
query93 5835 478 461 461
query94 1199 182 178 178
query95 466 7846 359 359
query96 602 265 276 265
query97 4235 4106 4094 4094
query98 222 190 198 190
query99 1193 739 642 642
Total cold run time: 280501 ms
Total hot run time: 174177 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1925364608
@HowardQin Hi, if the user is using the lzo before doris 2.0, and then upgrade to this version, there already some data , does it could upgrade successfully?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1920728110
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit a0fcb29937b29e0c054e68abe3c07b4054f489d9 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.5 seconds inserted 10000000 Rows, about 740K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1925685270
TeamCity be ut coverage result:
Function Coverage: 36.14% (8627/23871)
Line Coverage: 28.19% (70551/250273)
Region Coverage: 27.21% (36413/133813)
Branch Coverage: 24.00% (18650/77706)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d1023484d641fbf6e681f02e8bffbb0adcb72761_d1023484d641fbf6e681f02e8bffbb0adcb72761/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1925686703
@HowardQin You are right. lzo is only used during read parquet data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1920715714
<details>
<summary>TPC-DS: <b>Total hot run time: 174936 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0fcb29937b29e0c054e68abe3c07b4054f489d9, data reload: false
query1 943 346 335 335
query2 6547 2051 2030 2030
query3 6689 205 200 200
query4 32068 22213 21976 21976
query5 4458 408 402 402
query6 252 154 155 154
query7 4621 267 267 267
query8 228 168 178 168
query9 8928 2332 2292 2292
query10 404 179 191 179
query11 18271 15427 15414 15414
query12 119 67 63 63
query13 1625 398 375 375
query14 10297 7228 7190 7190
query15 233 174 179 174
query16 7474 263 252 252
query17 1873 487 448 448
query18 1958 256 247 247
query19 341 135 130 130
query20 70 71 72 71
query21 198 139 129 129
query22 4992 4801 4544 4544
query23 31275 30355 30303 30303
query24 11471 2739 2722 2722
query25 517 310 293 293
query26 1401 136 136 136
query27 2379 283 278 278
query28 6965 1867 1840 1840
query29 1467 645 611 611
query30 281 130 135 130
query31 939 705 716 705
query32 96 52 54 52
query33 707 206 201 201
query34 1073 451 448 448
query35 858 743 766 743
query36 1328 1192 1248 1192
query37 95 61 61 61
query38 3245 3157 3195 3157
query39 1290 1258 1244 1244
query40 325 89 85 85
query41 40 35 34 34
query42 93 78 83 78
query43 537 526 504 504
query44 1080 695 711 695
query45 188 181 171 171
query46 1047 632 643 632
query47 1489 1529 1447 1447
query48 391 310 313 310
query49 1250 277 272 272
query50 684 315 320 315
query51 5268 5121 5179 5121
query52 95 78 80 78
query53 321 267 266 266
query54 282 179 181 179
query55 80 77 80 77
query56 176 164 160 160
query57 1017 896 907 896
query58 188 154 156 154
query59 2479 2264 2387 2264
query60 206 174 174 174
query61 83 83 82 82
query62 634 362 346 346
query63 290 248 258 248
query64 6400 3668 3249 3249
query65 3231 3228 3231 3228
query66 1384 322 311 311
query67 14211 14128 14219 14128
query68 4269 496 503 496
query69 426 295 307 295
query70 1484 1486 1494 1486
query71 290 211 205 205
query72 5423 3106 2836 2836
query73 684 317 319 317
query74 6698 6227 6239 6227
query75 2948 2297 2291 2291
query76 2511 964 961 961
query77 338 236 229 229
query78 9241 8774 8566 8566
query79 3005 495 514 495
query80 2134 312 314 312
query81 519 194 202 194
query82 846 84 78 78
query83 249 110 114 110
query84 293 76 74 74
query85 2497 338 331 331
query86 516 392 403 392
query87 3469 3306 3258 3258
query88 3827 2183 2171 2171
query89 426 340 342 340
query90 1932 184 184 184
query91 147 117 126 117
query92 60 44 43 43
query93 3155 469 453 453
query94 1299 158 157 157
query95 506 455 451 451
query96 664 315 313 313
query97 4271 4054 4124 4054
query98 203 194 185 185
query99 1168 719 655 655
Total cold run time: 286943 ms
Total hot run time: 174936 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1920353176
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1920723026
<details>
<summary>ClickBench: <b>Total hot run time: 30.55 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a0fcb29937b29e0c054e68abe3c07b4054f489d9, data reload: false
query1 0.04 0.03 0.03
query2 0.05 0.02 0.03
query3 0.23 0.06 0.06
query4 1.68 0.09 0.09
query5 0.52 0.51 0.50
query6 1.23 0.64 0.64
query7 0.02 0.01 0.01
query8 0.04 0.03 0.03
query9 0.54 0.49 0.48
query10 0.55 0.56 0.56
query11 0.12 0.09 0.09
query12 0.11 0.08 0.09
query13 0.59 0.60 0.60
query14 0.78 0.80 0.82
query15 0.81 0.79 0.78
query16 0.38 0.39 0.38
query17 1.03 1.01 1.04
query18 0.25 0.27 0.22
query19 1.89 1.76 1.77
query20 0.01 0.01 0.01
query21 15.40 0.57 0.57
query22 2.86 2.34 2.56
query23 17.11 0.75 0.82
query24 2.63 1.16 0.12
query25 0.27 0.31 0.04
query26 0.29 0.14 0.14
query27 0.05 0.04 0.05
query28 12.16 0.84 0.86
query29 12.49 3.21 3.19
query30 0.62 0.57 0.52
query31 2.79 0.35 0.34
query32 3.36 0.49 0.48
query33 3.21 3.22 3.22
query34 15.77 4.29 4.23
query35 4.28 4.30 4.28
query36 1.10 1.06 1.04
query37 0.06 0.05 0.05
query38 0.03 0.03 0.03
query39 0.02 0.01 0.01
query40 0.16 0.13 0.13
query41 0.07 0.02 0.01
query42 0.03 0.02 0.01
query43 0.03 0.02 0.02
Total cold run time: 105.66 s
Total hot run time: 30.55 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #30573:
URL: https://github.com/apache/doris/pull/30573
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1920694881
<details>
<summary>TPC-H: <b>Total hot run time: 37274 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0fcb29937b29e0c054e68abe3c07b4054f489d9, data reload: false
------ Round 1 ----------------------------------
q1 17706 5050 4491 4491
q2 2044 143 133 133
q3 10707 928 936 928
q4 4701 786 703 703
q5 7667 2813 2883 2813
q6 187 123 123 123
q7 1174 726 706 706
q8 9339 2016 2048 2016
q9 7301 6352 6378 6352
q10 8124 2471 2409 2409
q11 417 212 214 212
q12 788 297 295 295
q13 18006 3317 3270 3270
q14 283 245 239 239
q15 519 501 494 494
q16 477 411 406 406
q17 941 559 491 491
q18 6774 6071 5964 5964
q19 1587 1491 1393 1393
q20 643 332 320 320
q21 6788 3246 3226 3226
q22 801 312 290 290
Total cold run time: 106974 ms
Total hot run time: 37274 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4513 4504 4438 4438
q2 322 219 246 219
q3 3014 2901 2822 2822
q4 1858 1646 1593 1593
q5 5204 5286 5287 5286
q6 199 115 118 115
q7 2167 1756 1776 1756
q8 3138 3236 3258 3236
q9 8373 8344 8362 8344
q10 5905 3574 3590 3574
q11 547 446 451 446
q12 725 602 582 582
q13 6617 3102 3104 3102
q14 300 266 257 257
q15 522 489 496 489
q16 526 482 474 474
q17 1842 1743 1712 1712
q18 8101 7719 7652 7652
q19 8233 1564 1583 1564
q20 2136 1911 1919 1911
q21 4864 4631 4613 4613
q22 546 458 478 458
Total cold run time: 69652 ms
Total hot run time: 54643 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1920789024
TeamCity be ut coverage result:
Function Coverage: 36.15% (8626/23859)
Line Coverage: 28.21% (70505/249962)
Region Coverage: 27.22% (36404/133719)
Branch Coverage: 24.01% (18645/77660)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a0fcb29937b29e0c054e68abe3c07b4054f489d9_a0fcb29937b29e0c054e68abe3c07b4054f489d9/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "HowardQin (via GitHub)" <gi...@apache.org>.
HowardQin commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1925646371
> @HowardQin Hi, if the user is using the lzo before doris 2.0, and then upgrade to this version, there already some data , does it could upgrade successfully?
orc::lzoDecompress() can only decompress files compressed by lzop with level 1 to 6, so it's not 100% compatible with liblzo2.a.
If old data in segment compressed using lzo with level above 6, this can be a problem, but I do not have enough info.
however, currently, doris does not have any code about decompressing lzo data in segment, there's only lzo-decompress logic for loading lzop compressed files. it seems we have already unsupported reading lzo data in segment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1925686946
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "HowardQin (via GitHub)" <gi...@apache.org>.
HowardQin commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1920668312
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "HowardQin (via GitHub)" <gi...@apache.org>.
HowardQin commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1925646576
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1925671251
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit d1023484d641fbf6e681f02e8bffbb0adcb72761 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.7 seconds inserted 10000000 Rows, about 729K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30573:
URL: https://github.com/apache/doris/pull/30573#issuecomment-1925686952
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org