You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "DongLiang-0 (via GitHub)" <gi...@apache.org> on 2023/12/20 06:50:22 UTC
[PR] [fix](tvf)Fixed the avro-scanner projection pushdown failing to query… [doris]
DongLiang-0 opened a new pull request, #28709:
URL: https://github.com/apache/doris/pull/28709
… on multiple BEs
## Proposed changes
Since projection pushdown requires knowing the schema information of the avro file in advance.
In the previous implementation, avro schema information was cached in be.
When there are multiple BEs in the cluster, the other BEs cannot obtain the avro schema, resulting in a NullPointerException.
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tvf)Fixed the avro-scanner projection pushdown failing to query… [doris]
Posted by "DongLiang-0 (via GitHub)" <gi...@apache.org>.
DongLiang-0 commented on PR #28709:
URL: https://github.com/apache/doris/pull/28709#issuecomment-1863941371
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tvf)Fixed the avro-scanner projection pushdown failing to query… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28709:
URL: https://github.com/apache/doris/pull/28709#issuecomment-1864134225
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tvf)Fixed the avro-scanner projection pushdown failing to query… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28709:
URL: https://github.com/apache/doris/pull/28709#issuecomment-1864033308
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 189064e3cdc5c4887777b00a689d665ba74caf39, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4664 4404 4429 4404
q2 361 154 154 154
q3 1481 1242 1221 1221
q4 1117 914 886 886
q5 3169 3175 3165 3165
q6 247 127 126 126
q7 1009 487 484 484
q8 2181 2225 2177 2177
q9 6674 6648 6671 6648
q10 3257 3284 3277 3277
q11 310 196 180 180
q12 350 209 207 207
q13 4557 3834 3805 3805
q14 244 211 212 211
q15 568 517 530 517
q16 441 397 389 389
q17 1004 612 558 558
q18 7207 6871 6900 6871
q19 1516 1370 1434 1370
q20 496 322 320 320
q21 3095 2658 2731 2658
q22 345 276 274 274
Total cold run time: 44293 ms
Total hot run time: 39902 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4322 4353 4342 4342
q2 265 165 179 165
q3 3534 3529 3542 3529
q4 2393 2370 2374 2370
q5 5755 5745 5737 5737
q6 242 120 123 120
q7 2381 1835 1885 1835
q8 3514 3533 3516 3516
q9 9010 9009 8979 8979
q10 3916 4027 4002 4002
q11 487 356 358 356
q12 759 592 623 592
q13 4296 3569 3560 3560
q14 293 258 261 258
q15 580 520 517 517
q16 508 451 469 451
q17 1895 1862 1881 1862
q18 8578 8197 8227 8197
q19 1717 1760 1750 1750
q20 2266 1943 1946 1943
q21 6520 6163 6162 6162
q22 487 442 419 419
Total cold run time: 63718 ms
Total hot run time: 60662 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tvf)Fixed the avro-scanner projection pushdown failing to query… [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28709:
URL: https://github.com/apache/doris/pull/28709#issuecomment-1864035020
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.93 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 30.2 seconds inserted 10000000 Rows, about 331K ops/s
storage size: 17184104018 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tvf)Fixed the avro-scanner projection pushdown failing to query… [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #28709:
URL: https://github.com/apache/doris/pull/28709
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tvf)Fixed the avro-scanner projection pushdown failing to query… [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28709:
URL: https://github.com/apache/doris/pull/28709#issuecomment-1864134178
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org