You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xingyingone (via GitHub)" <gi...@apache.org> on 2023/11/29 07:55:07 UTC
[PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
xingyingone opened a new pull request, #27752:
URL: https://github.com/apache/doris/pull/27752
…ed Transfer Encoding
## Proposed changes
Issue Number: close https://github.com/apache/doris/issues/27671
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1831610179
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on code in PR #27752:
URL: https://github.com/apache/doris/pull/27752#discussion_r1408901722
##########
be/src/http/action/stream_load.cpp:
##########
@@ -301,6 +301,14 @@ Status StreamLoadAction::_on_header(HttpRequest* http_req, std::shared_ptr<Strea
}
}
+ if (ctx->body_bytes == 0 && !ctx->is_chunked_transfer) {
Review Comment:
Ctx->body_bytes == 0 must be zero in on_header method, for it hasn't started receive data here yet, and data receiving begins in StreamLoadAction::_handle. I think you can check in _handle method or use http_req->header(HttpHeaders::CONTENT_LENGTH).empty() to check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1851255821
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit cf88064c31aca3e923ac4d59966e7e14f5f51f0c, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4687 4430 4487 4430
q2 363 151 156 151
q3 1450 1247 1166 1166
q4 1113 894 917 894
q5 3115 3123 3154 3123
q6 247 125 129 125
q7 983 476 475 475
q8 2181 2236 2205 2205
q9 6619 6672 6666 6666
q10 3204 3255 3267 3255
q11 324 206 204 204
q12 353 205 204 204
q13 4536 3802 3805 3802
q14 241 211 210 210
q15 561 522 522 522
q16 444 386 394 386
q17 1010 618 552 552
q18 7674 7233 7018 7018
q19 1507 1360 1533 1360
q20 915 775 512 512
q21 3073 2644 2680 2644
q22 342 276 282 276
Total cold run time: 44942 ms
Total hot run time: 40180 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4442 4430 4478 4430
q2 268 162 171 162
q3 3531 3529 3537 3529
q4 2419 2383 2385 2383
q5 5724 5723 5739 5723
q6 241 122 120 120
q7 2387 1889 1884 1884
q8 3512 3511 3519 3511
q9 9076 9020 9001 9001
q10 3869 3988 3987 3987
q11 498 397 392 392
q12 770 591 596 591
q13 4275 3584 3578 3578
q14 288 264 267 264
q15 570 524 511 511
q16 493 455 484 455
q17 1847 1876 1853 1853
q18 8679 8337 8350 8337
q19 1747 1759 1729 1729
q20 2253 1952 1948 1948
q21 6535 6153 6163 6153
q22 502 436 416 416
Total cold run time: 63926 ms
Total hot run time: 60957 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1845421858
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 5d4e4f89fb4256825a01209e9315adccf9dd5bd9, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4688 4432 4472 4432
q2 364 160 159 159
q3 1467 1273 1187 1187
q4 1117 929 905 905
q5 3150 3163 3136 3136
q6 249 125 126 125
q7 996 475 480 475
q8 2193 2227 2218 2218
q9 6683 6706 6689 6689
q10 3235 3519 3289 3289
q11 327 197 205 197
q12 349 210 206 206
q13 4511 3817 3802 3802
q14 241 207 213 207
q15 568 520 528 520
q16 436 384 387 384
q17 1015 592 597 592
q18 7501 6893 7122 6893
q19 1510 1386 1358 1358
q20 1777 297 313 297
q21 3023 2647 2615 2615
q22 348 274 276 274
Total cold run time: 45748 ms
Total hot run time: 39960 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4393 4407 4377 4377
q2 272 166 168 166
q3 3540 3505 3502 3502
q4 2374 2367 2363 2363
q5 5739 5724 5740 5724
q6 245 123 121 121
q7 2400 1876 1867 1867
q8 3532 3516 3519 3516
q9 9086 8977 9004 8977
q10 3892 3987 3996 3987
q11 499 386 391 386
q12 758 595 591 591
q13 4315 3567 3575 3567
q14 284 267 259 259
q15 584 516 528 516
q16 490 453 447 447
q17 1877 1870 1845 1845
q18 8637 8202 8316 8202
q19 1755 1737 1770 1737
q20 2253 1950 1942 1942
q21 6493 6134 6156 6134
q22 519 442 432 432
Total cold run time: 63937 ms
Total hot run time: 60658 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1851226849
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1851248051
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1853859456
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1833593808
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.38 seconds
stream load tsv: 572 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17167412191 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on code in PR #27752:
URL: https://github.com/apache/doris/pull/27752#discussion_r1421905533
##########
regression-test/suites/load_p0/stream_load/test_stream_load.groovy:
##########
@@ -1575,5 +1575,138 @@ suite("test_stream_load", "p0") {
} finally {
sql """ DROP TABLE IF EXISTS ${tableName19} FORCE"""
}
+
+ def sql_result = sql """ select Host, HttpPort from backends() where alive = true limit 1; """
+
+ log.info(sql_result[0][0].toString())
+ log.info(sql_result[0][1].toString())
+ log.info(sql_result[0].size.toString())
+
+ def beHost=sql_result[0][0]
+ def beHttpPort=sql_result[0][1]
+ log.info("${beHost}".toString())
+ log.info("${beHttpPort}".toString());
+
+ //test be : chunked transfer + Content Length
+ try {
+ sql """ DROP TABLE IF EXISTS ${tableName16} """
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tableName16} (
+ `k1` bigint(20) NULL DEFAULT "1",
+ `k2` bigint(20) NULL ,
+ `v1` tinyint(4) NULL,
+ `v2` tinyint(4) NULL,
+ `v3` tinyint(4) NULL,
+ `v4` DATETIME NULL
+ ) ENGINE=OLAP
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 3
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");
+ """
+
+ def command = "curl --location-trusted -u ${context.config.feHttpUser}:${context.config.feHttpPassword} -H column_separator:| -H ${db}:${tableName16} -H Content-Length:0 -H Transfer-Encoding:chunked -H columns:k1,k2,v1,v2,v3 -T ${context.dataPath}/test_chunked_transfer.csv http://${beHost}:${beHttpPort}/api/${db}/${tableName16}/_stream_load"
+ log.info("test chunked transfer command: ${command}")
+ def process = command.execute()
+ code = process.waitFor()
+ out = process.text
+ json2pc = parseJson(out)
Review Comment:
Do not use name "json2pc", json enough
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "xingyingone (via GitHub)" <gi...@apache.org>.
xingyingone commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1833392865
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "xingyingone (via GitHub)" <gi...@apache.org>.
xingyingone commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1833541994
> usually, when a function/branch is added, we need also to provide a UT or regression test. Please take a look at [here](https://doris.apache.org/zh-CN/community/developer-guide/regression-testing)
Okay , I will provide it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1863711499
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1831702019
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on code in PR #27752:
URL: https://github.com/apache/doris/pull/27752#discussion_r1408901722
##########
be/src/http/action/stream_load.cpp:
##########
@@ -301,6 +301,14 @@ Status StreamLoadAction::_on_header(HttpRequest* http_req, std::shared_ptr<Strea
}
}
+ if (ctx->body_bytes == 0 && !ctx->is_chunked_transfer) {
Review Comment:
Ctx->body_bytes == 0 must be zero in on_header method, for it hasn't started receive data here yet, which begins in StreamLoadAction::_handle. I think you can check in _handle or use http_req->header(HttpHeaders::CONTENT_LENGTH).empty() to check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1851384928
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.41 seconds
stream load tsv: 580 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17224878943 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "xingyingone (via GitHub)" <gi...@apache.org>.
xingyingone commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1851221656
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1851244520
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1851263119
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.28 seconds
stream load tsv: 574 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17221208095 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on code in PR #27752:
URL: https://github.com/apache/doris/pull/27752#discussion_r1421905854
##########
regression-test/suites/load_p0/stream_load/test_stream_load.groovy:
##########
@@ -1575,5 +1575,138 @@ suite("test_stream_load", "p0") {
} finally {
sql """ DROP TABLE IF EXISTS ${tableName19} FORCE"""
}
+
+ def sql_result = sql """ select Host, HttpPort from backends() where alive = true limit 1; """
+
+ log.info(sql_result[0][0].toString())
+ log.info(sql_result[0][1].toString())
+ log.info(sql_result[0].size.toString())
+
+ def beHost=sql_result[0][0]
+ def beHttpPort=sql_result[0][1]
+ log.info("${beHost}".toString())
+ log.info("${beHttpPort}".toString());
+
+ //test be : chunked transfer + Content Length
+ try {
+ sql """ DROP TABLE IF EXISTS ${tableName16} """
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tableName16} (
+ `k1` bigint(20) NULL DEFAULT "1",
+ `k2` bigint(20) NULL ,
+ `v1` tinyint(4) NULL,
+ `v2` tinyint(4) NULL,
+ `v3` tinyint(4) NULL,
+ `v4` DATETIME NULL
+ ) ENGINE=OLAP
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 3
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");
+ """
+
+ def command = "curl --location-trusted -u ${context.config.feHttpUser}:${context.config.feHttpPassword} -H column_separator:| -H ${db}:${tableName16} -H Content-Length:0 -H Transfer-Encoding:chunked -H columns:k1,k2,v1,v2,v3 -T ${context.dataPath}/test_chunked_transfer.csv http://${beHost}:${beHttpPort}/api/${db}/${tableName16}/_stream_load"
+ log.info("test chunked transfer command: ${command}")
+ def process = command.execute()
+ code = process.waitFor()
+ out = process.text
+ json2pc = parseJson(out)
+ log.info("test chunked transfer result: ${out}".toString())
+ def json = parseJson(out)
+ assertEquals("fail", json.Status.toLowerCase())
+ } finally {
+ sql """ DROP TABLE IF EXISTS ${tableName16} FORCE"""
+ }
+
+
+ //test be : no chunked transfer + no Content Length
+ try {
+ sql """ DROP TABLE IF EXISTS ${tableName16} """
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tableName16} (
+ `k1` bigint(20) NULL DEFAULT "1",
+ `k2` bigint(20) NULL ,
+ `v1` tinyint(4) NULL,
+ `v2` tinyint(4) NULL,
+ `v3` tinyint(4) NULL,
+ `v4` DATETIME NULL
+ ) ENGINE=OLAP
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 3
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");
+ """
+
+ def command = "curl --location-trusted -u ${context.config.feHttpUser}:${context.config.feHttpPassword} -H column_separator:| -H ${db}:${tableName16} -H Content-Length: -H Transfer-Encoding: -T ${context.dataPath}/test_chunked_transfer.csv http://${beHost}:${beHttpPort}/api/${db}/${tableName16}/_stream_load"
+ log.info("test chunked transfer command: ${command}")
+ def process = command.execute()
+ code = process.waitFor()
+ out = process.text
+ json2pc = parseJson(out)
+ log.info("test chunked transfer result: ${out}".toString())
+ def json = parseJson(out)
+ assertEquals("fail", json.Status.toLowerCase())
+ } finally {
+ sql """ DROP TABLE IF EXISTS ${tableName16} FORCE"""
+ }
+
+
+ //test be : no chunked transfer + Content Length=0
+ try {
+ sql """ DROP TABLE IF EXISTS ${tableName16} """
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tableName16} (
+ `k1` bigint(20) NULL DEFAULT "1",
+ `k2` bigint(20) NULL ,
+ `v1` tinyint(4) NULL,
+ `v2` tinyint(4) NULL,
+ `v3` tinyint(4) NULL,
+ `v4` DATETIME NULL
+ ) ENGINE=OLAP
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 3
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");
+ """
+
+ def command = "curl --location-trusted -u ${context.config.feHttpUser}:${context.config.feHttpPassword} -H column_separator:| -H ${db}:${tableName16} -H Content-Length:0 -H Transfer-Encoding: -H columns:k1,k2,v1,v2,v3 -T ${context.dataPath}/test_chunked_transfer.csv http://${beHost}:${beHttpPort}/api/${db}/${tableName16}/_stream_load"
+ log.info("test chunked transfer command: ${command}")
+ def process = command.execute()
+ code = process.waitFor()
+ out = process.text
+ json2pc = parseJson(out)
+ log.info("test chunked transfer result: ${out}".toString())
+ def json = parseJson(out)
+ assertEquals("success", json.Status.toLowerCase())
+ assertEquals(0, json.NumberTotalRows)
+ } finally {
+ sql """ DROP TABLE IF EXISTS ${tableName16} FORCE"""
+ }
+
+ //test fe : no chunked transfer + Content Length=0
+ try {
+ sql """ DROP TABLE IF EXISTS ${tableName16} """
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tableName16} (
+ `k1` bigint(20) NULL DEFAULT "1",
+ `k2` bigint(20) NULL ,
+ `v1` tinyint(4) NULL,
+ `v2` tinyint(4) NULL,
+ `v3` tinyint(4) NULL,
+ `v4` DATETIME NULL
+ ) ENGINE=OLAP
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 3
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");
+ """
+
+ def command = "curl --location-trusted -u ${context.config.feHttpUser}:${context.config.feHttpPassword} -H column_separator:| -H ${db}:${tableName16} -H Content-Length:0 -H Transfer-Encoding: -H columns:k1,k2,v1,v2,v3 -T ${context.dataPath}/test_chunked_transfer.csv http://${context.config.feHttpAddress}/api/${db}/${tableName16}/_stream_load"
+ log.info("test chunked transfer command: ${command}")
+ def process = command.execute()
+ code = process.waitFor()
+ out = process.text
+ json2pc = parseJson(out)
+ log.info("test chunked transfer result: ${out}".toString())
+ def json = parseJson(out)
+ assertEquals("success", json.Status.toLowerCase())
+ assertEquals(0, json.NumberTotalRows)
+ } finally {
+ sql """ DROP TABLE IF EXISTS ${tableName16} FORCE"""
+ }
+
+
Review Comment:
Can you add test with empty file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1850079410
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "xingyingone (via GitHub)" <gi...@apache.org>.
xingyingone commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1831387898
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on code in PR #27752:
URL: https://github.com/apache/doris/pull/27752#discussion_r1408909411
##########
be/src/http/action/stream_load.cpp:
##########
@@ -301,6 +301,14 @@ Status StreamLoadAction::_on_header(HttpRequest* http_req, std::shared_ptr<Strea
}
}
+ if (ctx->body_bytes == 0 && !ctx->is_chunked_transfer) {
+ LOG(WARNING) << "content_length is empty and transfer-encoding!=chunked, please set "
Review Comment:
It is no need log and return InternalError is enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1831401373
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1853873259
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1853979954
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.15 seconds
stream load tsv: 586 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17217984895 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on code in PR #27752:
URL: https://github.com/apache/doris/pull/27752#discussion_r1409371509
##########
be/src/http/action/stream_load.cpp:
##########
@@ -301,6 +301,14 @@ Status StreamLoadAction::_on_header(HttpRequest* http_req, std::shared_ptr<Strea
}
}
+ if (ctx->body_bytes == 0 && !ctx->is_chunked_transfer) {
Review Comment:
There may be several things to consider here:
1. Forgot to set content length or chunked
2. Incorrect 0 set
3. It is itself 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1833402725
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "xingyingone (via GitHub)" <gi...@apache.org>.
xingyingone commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1850067698
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1850124265
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 1a246077094b1508556a88c3b84435cb6c481fb6, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4675 4465 4463 4463
q2 356 158 164 158
q3 1505 1241 1218 1218
q4 1099 940 909 909
q5 3116 3148 3128 3128
q6 254 125 126 125
q7 995 483 481 481
q8 2194 2234 2191 2191
q9 6673 6619 6668 6619
q10 3202 3236 3264 3236
q11 328 193 194 193
q12 349 209 210 209
q13 4571 3783 3815 3783
q14 243 214 207 207
q15 569 515 519 515
q16 433 390 389 389
q17 1019 589 560 560
q18 7820 7728 7273 7273
q19 1535 1345 1452 1345
q20 538 362 285 285
q21 3108 2675 2705 2675
q22 358 278 289 278
Total cold run time: 44940 ms
Total hot run time: 40240 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4420 4373 4376 4373
q2 266 165 174 165
q3 3549 3531 3528 3528
q4 2386 2392 2376 2376
q5 5725 5746 5734 5734
q6 241 120 120 120
q7 2392 1887 1880 1880
q8 3507 3516 3506 3506
q9 9038 8995 8956 8956
q10 3901 3995 3989 3989
q11 503 398 368 368
q12 764 587 586 586
q13 4320 3590 3548 3548
q14 284 249 244 244
q15 569 519 523 519
q16 499 478 483 478
q17 1889 1851 1854 1851
q18 8774 8331 8363 8331
q19 1744 1730 1775 1730
q20 2248 1949 1946 1946
q21 6543 6203 6193 6193
q22 496 431 422 422
Total cold run time: 64058 ms
Total hot run time: 60843 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #27752:
URL: https://github.com/apache/doris/pull/27752
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "xingyingone (via GitHub)" <gi...@apache.org>.
xingyingone commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1845374983
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1851372564
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 21ecf8f4781a0e4770b6caa3b5077a5404a51dc2, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4680 4490 4941 4490
q2 359 167 159 159
q3 1463 1235 1181 1181
q4 1121 927 906 906
q5 3437 3183 3134 3134
q6 251 129 127 127
q7 990 485 481 481
q8 2236 2233 2185 2185
q9 6698 6668 6655 6655
q10 3222 3271 3253 3253
q11 328 207 211 207
q12 347 207 203 203
q13 4612 3782 5410 3782
q14 233 217 211 211
q15 557 523 512 512
q16 446 386 393 386
q17 1011 602 543 543
q18 8407 7328 7030 7030
q19 1524 1264 1470 1264
q20 628 316 291 291
q21 3096 2665 2702 2665
q22 351 274 282 274
Total cold run time: 45997 ms
Total hot run time: 39939 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4407 4380 4408 4380
q2 268 163 168 163
q3 3551 3562 3543 3543
q4 2404 2380 2393 2380
q5 5763 5747 5756 5747
q6 241 120 121 120
q7 2393 1885 1882 1882
q8 3514 3532 3657 3532
q9 9048 9025 8978 8978
q10 3905 3987 3968 3968
q11 505 373 374 373
q12 766 586 590 586
q13 4329 3603 3544 3544
q14 284 259 253 253
q15 567 514 516 514
q16 512 457 481 457
q17 1894 1866 1838 1838
q18 8728 8251 8378 8251
q19 1746 1757 1718 1718
q20 2260 1954 1943 1943
q21 6546 6179 6200 6179
q22 498 416 430 416
Total cold run time: 64129 ms
Total hot run time: 60765 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on code in PR #27752:
URL: https://github.com/apache/doris/pull/27752#discussion_r1408901722
##########
be/src/http/action/stream_load.cpp:
##########
@@ -301,6 +301,14 @@ Status StreamLoadAction::_on_header(HttpRequest* http_req, std::shared_ptr<Strea
}
}
+ if (ctx->body_bytes == 0 && !ctx->is_chunked_transfer) {
Review Comment:
Ctx->body_bytes == 0 must be zero in on_header method, for it hasn't started receive data here yet, and data receiving begins in StreamLoadAction::_handle. I think you can check in _handle method or use http_req->header(HttpHeaders::CONTENT_LENGTH).empty() to check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "xingyingone (via GitHub)" <gi...@apache.org>.
xingyingone commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1831691465
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "xingyingone (via GitHub)" <gi...@apache.org>.
xingyingone commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1845361399
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1851238738
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit f409d76a9b4a612f75c058afb9803e593b358931, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4650 4458 4454 4454
q2 358 163 153 153
q3 1453 1249 1251 1249
q4 1114 901 861 861
q5 3158 3137 3103 3103
q6 245 127 130 127
q7 949 486 484 484
q8 2207 2198 2177 2177
q9 6640 6665 6633 6633
q10 3188 3239 3241 3239
q11 314 196 201 196
q12 351 211 202 202
q13 4560 3816 3788 3788
q14 259 207 207 207
q15 555 514 521 514
q16 441 381 388 381
q17 986 556 503 503
q18 7681 7865 6969 6969
q19 1518 1316 1488 1316
q20 540 312 309 309
q21 3095 2684 2674 2674
q22 343 275 283 275
Total cold run time: 44605 ms
Total hot run time: 39814 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4417 4389 4363 4363
q2 266 163 171 163
q3 3555 3549 3522 3522
q4 2389 2383 2377 2377
q5 5722 5738 5719 5719
q6 240 121 123 121
q7 2386 1847 1851 1847
q8 3509 3516 3517 3516
q9 9024 9024 9002 9002
q10 3915 4002 3981 3981
q11 508 380 399 380
q12 763 592 600 592
q13 4297 3578 3572 3572
q14 287 246 257 246
q15 570 516 522 516
q16 500 450 483 450
q17 1868 1846 1831 1831
q18 8673 8214 8403 8214
q19 1736 1738 1771 1738
q20 2251 2013 2030 2013
q21 6599 6204 6166 6166
q22 499 428 423 423
Total cold run time: 63974 ms
Total hot run time: 60752 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "xingyingone (via GitHub)" <gi...@apache.org>.
xingyingone commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1853849467
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "xingyingone (via GitHub)" <gi...@apache.org>.
xingyingone commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1857351178
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1831482176
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.68 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17100949496 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1831853125
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.03 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17099903871 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #27752:
URL: https://github.com/apache/doris/pull/27752#discussion_r1424967129
##########
be/src/http/action/stream_load.cpp:
##########
@@ -144,6 +144,13 @@ Status StreamLoadAction::_handle(std::shared_ptr<StreamLoadContext> ctx) {
LOG(WARNING) << "recevie body don't equal with body bytes, body_bytes=" << ctx->body_bytes
<< ", receive_bytes=" << ctx->receive_bytes << ", id=" << ctx->id;
return Status::InternalError("receive body don't equal with body bytes");
+ } else if (ctx->body_bytes == 0 && ctx->receive_bytes && ctx->is_chunked_transfer == false) {
Review Comment:
unlikely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1857391509
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit e5b40a882bfddc5676656dbf1175407f4d780056, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4699 4453 4473 4453
q2 360 156 154 154
q3 1472 1252 1262 1252
q4 1121 909 882 882
q5 3198 3217 3215 3215
q6 243 128 131 128
q7 1017 485 477 477
q8 2206 2223 2199 2199
q9 6689 6687 6661 6661
q10 3240 3294 3300 3294
q11 326 206 206 206
q12 354 206 208 206
q13 4581 3797 3810 3797
q14 239 209 209 209
q15 568 527 532 527
q16 444 382 393 382
q17 1041 668 578 578
q18 7044 6806 6795 6795
q19 1537 1478 1432 1432
q20 548 311 300 300
q21 3084 2655 2661 2655
q22 344 286 286 286
Total cold run time: 44355 ms
Total hot run time: 40088 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4413 4381 4404 4381
q2 266 162 175 162
q3 3505 3521 3508 3508
q4 2405 2401 2391 2391
q5 5728 5760 5721 5721
q6 235 119 119 119
q7 2351 1865 1854 1854
q8 3517 3530 3525 3525
q9 9121 8988 8959 8959
q10 3927 4025 4012 4012
q11 513 383 385 383
q12 774 585 596 585
q13 4306 3545 3607 3545
q14 294 254 254 254
q15 571 518 527 518
q16 503 450 455 450
q17 1901 1875 1871 1871
q18 8612 8237 8227 8227
q19 1748 1769 1769 1769
q20 2250 1937 1933 1933
q21 6502 6175 6158 6158
q22 505 438 437 437
Total cold run time: 63947 ms
Total hot run time: 60762 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1845430011
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.96 seconds
stream load tsv: 583 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17213631497 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1850150439
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.88 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 30.0 seconds inserted 10000000 Rows, about 333K ops/s
storage size: 17219613220 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "xingyingone (via GitHub)" <gi...@apache.org>.
xingyingone commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1851240326
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "xingyingone (via GitHub)" <gi...@apache.org>.
xingyingone commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1851342482
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "xingyingone (via GitHub)" <gi...@apache.org>.
xingyingone commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1831599662
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1833786568
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 7172f5661f222ec3b5e786d18db0b5861ec21300, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4914 4705 4657 4657
q2 353 153 158 153
q3 1516 1331 1266 1266
q4 1163 965 940 940
q5 3264 3230 3236 3230
q6 253 135 132 132
q7 1009 512 539 512
q8 2293 2248 2229 2229
q9 6955 6991 6940 6940
q10 3309 3363 3361 3361
q11 344 214 225 214
q12 349 213 222 213
q13 4683 3859 3886 3859
q14 251 222 212 212
q15 582 552 526 526
q16 439 393 390 390
q17 1038 661 601 601
q18 8030 7833 9241 7833
q19 1585 1573 1519 1519
q20 695 340 298 298
q21 3404 3011 2958 2958
q22 368 293 301 293
Total cold run time: 46797 ms
Total hot run time: 42336 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4592 4588 4609 4588
q2 310 201 242 201
q3 3761 3770 3745 3745
q4 2520 2523 2507 2507
q5 6180 6177 6206 6177
q6 245 123 127 123
q7 2625 1953 1983 1953
q8 3701 3673 3731 3673
q9 9428 9431 9344 9344
q10 4064 4146 4160 4146
q11 598 491 488 488
q12 801 632 622 622
q13 4378 3685 3637 3637
q14 282 240 252 240
q15 591 522 529 522
q16 538 488 481 481
q17 2081 2073 2060 2060
q18 9508 8913 8887 8887
q19 1819 1766 1799 1766
q20 2304 1985 1966 1966
q21 7494 6862 7174 6862
q22 634 565 579 565
Total cold run time: 68454 ms
Total hot run time: 64553 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1845387185
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1845371145
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1851348513
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #27752:
URL: https://github.com/apache/doris/pull/27752#discussion_r1424943855
##########
be/src/http/action/stream_load.cpp:
##########
@@ -302,12 +309,21 @@ Status StreamLoadAction::_on_header(HttpRequest* http_req, std::shared_ptr<Strea
evhttp_request_get_connection(http_req->get_evhttp_request()), csv_max_body_bytes);
#endif
}
-
if (!http_req->header(HttpHeaders::TRANSFER_ENCODING).empty()) {
if (http_req->header(HttpHeaders::TRANSFER_ENCODING).find(CHUNK) != std::string::npos) {
ctx->is_chunked_transfer = true;
}
}
+ if (http_req->header(HttpHeaders::CONTENT_LENGTH).empty() && !ctx->is_chunked_transfer) {
+ LOG(WARNING) << "content_length is empty and transfer-encoding!=chunked, please set "
+ "content_length or transfer-encoding=chunked";
+ return Status::InternalError(
+ "content_length is empty and transfer-encoding!=chunked, please set content_length "
+ "or transfer-encoding=chunked");
+ } else if (!http_req->header(HttpHeaders::CONTENT_LENGTH).empty() && ctx->is_chunked_transfer) {
Review Comment:
ditto, UNLIKELY
##########
regression-test/suites/load_p0/stream_load/test_stream_load.groovy:
##########
@@ -1575,5 +1575,136 @@ suite("test_stream_load", "p0") {
} finally {
sql """ DROP TABLE IF EXISTS ${tableName19} FORCE"""
}
+
+ def sql_result = sql """ select Host, HttpPort from backends() where alive = true limit 1; """
+
+ log.info(sql_result[0][0].toString())
+ log.info(sql_result[0][1].toString())
+ log.info(sql_result[0].size.toString())
+
+ def beHost=sql_result[0][0]
+ def beHttpPort=sql_result[0][1]
+ log.info("${beHost}".toString())
+ log.info("${beHttpPort}".toString());
+
+ //test be : chunked transfer + Content Length
+ try {
+ sql """ DROP TABLE IF EXISTS ${tableName16} """
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tableName16} (
+ `k1` bigint(20) NULL DEFAULT "1",
+ `k2` bigint(20) NULL ,
+ `v1` tinyint(4) NULL,
+ `v2` tinyint(4) NULL,
+ `v3` tinyint(4) NULL,
+ `v4` DATETIME NULL
+ ) ENGINE=OLAP
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 3
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");
+ """
+
+ def command = "curl --location-trusted -u ${context.config.feHttpUser}:${context.config.feHttpPassword} -H column_separator:| -H ${db}:${tableName16} -H Content-Length:0 -H Transfer-Encoding:chunked -H columns:k1,k2,v1,v2,v3 -T ${context.dataPath}/test_chunked_transfer.csv http://${beHost}:${beHttpPort}/api/${db}/${tableName16}/_stream_load"
+ log.info("test chunked transfer command: ${command}")
+ def process = command.execute()
+ code = process.waitFor()
+ out = process.text
+ log.info("test chunked transfer result: ${out}".toString())
+ def json = parseJson(out)
+ assertEquals("fail", json.Status.toLowerCase())
+ assertTrue(json.Message.contains("[INTERNAL_ERROR]please do not set both content_length and transfer-encoding"))
+ } finally {
+ sql """ DROP TABLE IF EXISTS ${tableName16} FORCE"""
+ }
+
+
+ //test be : no chunked transfer + no Content Length
+ try {
+ sql """ DROP TABLE IF EXISTS ${tableName16} """
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tableName16} (
+ `k1` bigint(20) NULL DEFAULT "1",
+ `k2` bigint(20) NULL ,
+ `v1` tinyint(4) NULL,
+ `v2` tinyint(4) NULL,
+ `v3` tinyint(4) NULL,
+ `v4` DATETIME NULL
+ ) ENGINE=OLAP
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 3
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");
+ """
+
+ def command = "curl --location-trusted -u ${context.config.feHttpUser}:${context.config.feHttpPassword} -H column_separator:| -H ${db}:${tableName16} -H Content-Length: -H Transfer-Encoding: -T ${context.dataPath}/test_chunked_transfer.csv http://${beHost}:${beHttpPort}/api/${db}/${tableName16}/_stream_load"
+ log.info("test chunked transfer command: ${command}")
+ def process = command.execute()
+ code = process.waitFor()
+ out = process.text
+ log.info("test chunked transfer result: ${out}".toString())
+ def json = parseJson(out)
+ assertEquals("fail", json.Status.toLowerCase())
+ assertTrue(json.Message.contains("[INTERNAL_ERROR]content_length is empty and transfer-encoding!=chunked, please set content_length or transfer-encoding=chunked"))
+ } finally {
+ sql """ DROP TABLE IF EXISTS ${tableName16} FORCE"""
+ }
+
+
+ //test be : no chunked transfer + Content Length=0
+ try {
+ sql """ DROP TABLE IF EXISTS ${tableName16} """
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tableName16} (
+ `k1` bigint(20) NULL DEFAULT "1",
+ `k2` bigint(20) NULL ,
+ `v1` tinyint(4) NULL,
+ `v2` tinyint(4) NULL,
+ `v3` tinyint(4) NULL,
+ `v4` DATETIME NULL
+ ) ENGINE=OLAP
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 3
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");
+ """
+
+ def command = "curl --location-trusted -u ${context.config.feHttpUser}:${context.config.feHttpPassword} -H column_separator:| -H ${db}:${tableName16} -H Content-Length:0 -H Transfer-Encoding: -H columns:k1,k2,v1,v2,v3 -T ${context.dataPath}/test_chunked_transfer.csv http://${beHost}:${beHttpPort}/api/${db}/${tableName16}/_stream_load"
+ log.info("test chunked transfer command: ${command}")
+ def process = command.execute()
+ code = process.waitFor()
+ out = process.text
+ log.info("test chunked transfer result: ${out}".toString())
+ def json = parseJson(out)
+ assertEquals("success", json.Status.toLowerCase())
+ assertEquals(0, json.NumberTotalRows)
+ } finally {
+ sql """ DROP TABLE IF EXISTS ${tableName16} FORCE"""
+ }
+
+
+ //test fe : no chunked transfer + Content Length=0
+ try {
+ sql """ DROP TABLE IF EXISTS ${tableName16} """
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tableName16} (
+ `k1` bigint(20) NULL DEFAULT "1",
+ `k2` bigint(20) NULL ,
+ `v1` tinyint(4) NULL,
+ `v2` tinyint(4) NULL,
+ `v3` tinyint(4) NULL,
+ `v4` DATETIME NULL
+ ) ENGINE=OLAP
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 3
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");
+ """
+
+ def command = "curl --location-trusted -u ${context.config.feHttpUser}:${context.config.feHttpPassword} -H column_separator:| -H ${db}:${tableName16} -H Content-Length:0 -H Transfer-Encoding: -H columns:k1,k2,v1,v2,v3 -T ${context.dataPath}/test_chunked_transfer.csv http://${context.config.feHttpAddress}/api/${db}/${tableName16}/_stream_load"
+ log.info("test chunked transfer command: ${command}")
+ def process = command.execute()
+ code = process.waitFor()
+ out = process.text
+ log.info("test chunked transfer result: ${out}".toString())
+ def json = parseJson(out)
+ assertEquals("success", json.Status.toLowerCase())
Review Comment:
why success?
##########
be/src/http/action/stream_load.cpp:
##########
@@ -302,12 +309,21 @@ Status StreamLoadAction::_on_header(HttpRequest* http_req, std::shared_ptr<Strea
evhttp_request_get_connection(http_req->get_evhttp_request()), csv_max_body_bytes);
#endif
}
-
if (!http_req->header(HttpHeaders::TRANSFER_ENCODING).empty()) {
if (http_req->header(HttpHeaders::TRANSFER_ENCODING).find(CHUNK) != std::string::npos) {
ctx->is_chunked_transfer = true;
}
}
+ if (http_req->header(HttpHeaders::CONTENT_LENGTH).empty() && !ctx->is_chunked_transfer) {
Review Comment:
please add UNLIKELY to make branch predictor happy
##########
regression-test/suites/load_p0/stream_load/test_stream_load.groovy:
##########
@@ -1575,5 +1575,136 @@ suite("test_stream_load", "p0") {
} finally {
sql """ DROP TABLE IF EXISTS ${tableName19} FORCE"""
}
+
+ def sql_result = sql """ select Host, HttpPort from backends() where alive = true limit 1; """
+
+ log.info(sql_result[0][0].toString())
+ log.info(sql_result[0][1].toString())
+ log.info(sql_result[0].size.toString())
+
+ def beHost=sql_result[0][0]
+ def beHttpPort=sql_result[0][1]
+ log.info("${beHost}".toString())
+ log.info("${beHttpPort}".toString());
+
+ //test be : chunked transfer + Content Length
+ try {
+ sql """ DROP TABLE IF EXISTS ${tableName16} """
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tableName16} (
+ `k1` bigint(20) NULL DEFAULT "1",
+ `k2` bigint(20) NULL ,
+ `v1` tinyint(4) NULL,
+ `v2` tinyint(4) NULL,
+ `v3` tinyint(4) NULL,
+ `v4` DATETIME NULL
+ ) ENGINE=OLAP
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 3
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");
+ """
+
+ def command = "curl --location-trusted -u ${context.config.feHttpUser}:${context.config.feHttpPassword} -H column_separator:| -H ${db}:${tableName16} -H Content-Length:0 -H Transfer-Encoding:chunked -H columns:k1,k2,v1,v2,v3 -T ${context.dataPath}/test_chunked_transfer.csv http://${beHost}:${beHttpPort}/api/${db}/${tableName16}/_stream_load"
+ log.info("test chunked transfer command: ${command}")
+ def process = command.execute()
+ code = process.waitFor()
+ out = process.text
+ log.info("test chunked transfer result: ${out}".toString())
+ def json = parseJson(out)
+ assertEquals("fail", json.Status.toLowerCase())
+ assertTrue(json.Message.contains("[INTERNAL_ERROR]please do not set both content_length and transfer-encoding"))
+ } finally {
+ sql """ DROP TABLE IF EXISTS ${tableName16} FORCE"""
+ }
+
+
+ //test be : no chunked transfer + no Content Length
+ try {
+ sql """ DROP TABLE IF EXISTS ${tableName16} """
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tableName16} (
+ `k1` bigint(20) NULL DEFAULT "1",
+ `k2` bigint(20) NULL ,
+ `v1` tinyint(4) NULL,
+ `v2` tinyint(4) NULL,
+ `v3` tinyint(4) NULL,
+ `v4` DATETIME NULL
+ ) ENGINE=OLAP
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 3
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");
+ """
+
+ def command = "curl --location-trusted -u ${context.config.feHttpUser}:${context.config.feHttpPassword} -H column_separator:| -H ${db}:${tableName16} -H Content-Length: -H Transfer-Encoding: -T ${context.dataPath}/test_chunked_transfer.csv http://${beHost}:${beHttpPort}/api/${db}/${tableName16}/_stream_load"
+ log.info("test chunked transfer command: ${command}")
+ def process = command.execute()
+ code = process.waitFor()
+ out = process.text
+ log.info("test chunked transfer result: ${out}".toString())
+ def json = parseJson(out)
+ assertEquals("fail", json.Status.toLowerCase())
+ assertTrue(json.Message.contains("[INTERNAL_ERROR]content_length is empty and transfer-encoding!=chunked, please set content_length or transfer-encoding=chunked"))
+ } finally {
+ sql """ DROP TABLE IF EXISTS ${tableName16} FORCE"""
+ }
+
+
+ //test be : no chunked transfer + Content Length=0
+ try {
+ sql """ DROP TABLE IF EXISTS ${tableName16} """
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tableName16} (
+ `k1` bigint(20) NULL DEFAULT "1",
+ `k2` bigint(20) NULL ,
+ `v1` tinyint(4) NULL,
+ `v2` tinyint(4) NULL,
+ `v3` tinyint(4) NULL,
+ `v4` DATETIME NULL
+ ) ENGINE=OLAP
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 3
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");
+ """
+
+ def command = "curl --location-trusted -u ${context.config.feHttpUser}:${context.config.feHttpPassword} -H column_separator:| -H ${db}:${tableName16} -H Content-Length:0 -H Transfer-Encoding: -H columns:k1,k2,v1,v2,v3 -T ${context.dataPath}/test_chunked_transfer.csv http://${beHost}:${beHttpPort}/api/${db}/${tableName16}/_stream_load"
+ log.info("test chunked transfer command: ${command}")
+ def process = command.execute()
+ code = process.waitFor()
+ out = process.text
+ log.info("test chunked transfer result: ${out}".toString())
+ def json = parseJson(out)
+ assertEquals("success", json.Status.toLowerCase())
Review Comment:
why success?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "xingyingone (via GitHub)" <gi...@apache.org>.
xingyingone commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1853870810
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1854100905
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit e5b40a882bfddc5676656dbf1175407f4d780056, data reload: false
run tpch-sf100 query with default conf and session variables
q1 50028 31829 31831 31829
q2 368 136 138 136
q3 1517 1189 1197 1189
q4 1116 898 920 898
q5 3166 3190 3196 3190
q6 259 140 143 140
q7 1012 508 481 481
q8 2123 2225 2183 2183
q9 6874 6821 6822 6821
q10 3233 3318 3310 3310
q11 331 203 219 203
q12 353 211 208 208
q13 4536 3825 5465 3825
q14 427 315 326 315
q15 779 1345 1348 1345
q16 434 392 392 392
q17 1011 575 538 538
q18 7698 6892 6899 6892
q19 1553 1518 1341 1341
q20 733 291 308 291
q21 3065 2650 2704 2650
q22 343 284 286 284
Total cold run time: 90959 ms
Total hot run time: 68461 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 30836 31845 31804 31804
q2 278 165 172 165
q3 3599 3606 3601 3601
q4 2402 2383 2392 2383
q5 5752 5773 5777 5773
q6 246 139 134 134
q7 3060 3404 3357 3357
q8 3530 3531 3534 3531
q9 9150 9136 9100 9100
q10 3954 4044 4066 4044
q11 821 898 1353 898
q12 780 602 601 601
q13 4287 3568 3570 3568
q14 520 435 475 435
q15 1104 1324 1353 1324
q16 502 438 463 438
q17 1868 1828 1866 1828
q18 8720 8057 8296 8057
q19 1718 1784 1736 1736
q20 2901 2613 2643 2613
q21 6576 6172 6132 6132
q22 507 435 429 429
Total cold run time: 93111 ms
Total hot run time: 91951 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](stream_load)fix bug when stream without content-length or chunk… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27752:
URL: https://github.com/apache/doris/pull/27752#issuecomment-1857378229
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.51 seconds
stream load tsv: 589 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17221075235 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org