You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "DongLiang-0 (via GitHub)" <gi...@apache.org> on 2023/12/20 06:50:22 UTC

[PR] [fix](tvf)Fixed the avro-scanner projection pushdown failing to query… [doris]

DongLiang-0 opened a new pull request, #28709:
URL: https://github.com/apache/doris/pull/28709

   … on multiple BEs
   
   ## Proposed changes
   Since projection pushdown requires knowing the schema information of the avro file in advance.
   In the previous implementation, avro schema information was cached in be. 
   When there are multiple BEs in the cluster, the other BEs cannot obtain the avro schema, resulting in a NullPointerException.
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](tvf)Fixed the avro-scanner projection pushdown failing to query… [doris]

Posted by "DongLiang-0 (via GitHub)" <gi...@apache.org>.
DongLiang-0 commented on PR #28709:
URL: https://github.com/apache/doris/pull/28709#issuecomment-1863941371

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](tvf)Fixed the avro-scanner projection pushdown failing to query… [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28709:
URL: https://github.com/apache/doris/pull/28709#issuecomment-1864134225

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](tvf)Fixed the avro-scanner projection pushdown failing to query… [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28709:
URL: https://github.com/apache/doris/pull/28709#issuecomment-1864033308

   
   TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Tpch sf100 test result on commit 189064e3cdc5c4887777b00a689d665ba74caf39, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	4664	4404	4429	4404
   q2	361	154	154	154
   q3	1481	1242	1221	1221
   q4	1117	914	886	886
   q5	3169	3175	3165	3165
   q6	247	127	126	126
   q7	1009	487	484	484
   q8	2181	2225	2177	2177
   q9	6674	6648	6671	6648
   q10	3257	3284	3277	3277
   q11	310	196	180	180
   q12	350	209	207	207
   q13	4557	3834	3805	3805
   q14	244	211	212	211
   q15	568	517	530	517
   q16	441	397	389	389
   q17	1004	612	558	558
   q18	7207	6871	6900	6871
   q19	1516	1370	1434	1370
   q20	496	322	320	320
   q21	3095	2658	2731	2658
   q22	345	276	274	274
   Total cold run time: 44293 ms
   Total hot run time: 39902 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4322	4353	4342	4342
   q2	265	165	179	165
   q3	3534	3529	3542	3529
   q4	2393	2370	2374	2370
   q5	5755	5745	5737	5737
   q6	242	120	123	120
   q7	2381	1835	1885	1835
   q8	3514	3533	3516	3516
   q9	9010	9009	8979	8979
   q10	3916	4027	4002	4002
   q11	487	356	358	356
   q12	759	592	623	592
   q13	4296	3569	3560	3560
   q14	293	258	261	258
   q15	580	520	517	517
   q16	508	451	469	451
   q17	1895	1862	1881	1862
   q18	8578	8197	8227	8197
   q19	1717	1760	1750	1750
   q20	2266	1943	1946	1943
   q21	6520	6163	6162	6162
   q22	487	442	419	419
   Total cold run time: 63718 ms
   Total hot run time: 60662 ms
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](tvf)Fixed the avro-scanner projection pushdown failing to query… [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28709:
URL: https://github.com/apache/doris/pull/28709#issuecomment-1864035020

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 44.93 seconds
    stream load tsv:          578 seconds loaded 74807831229 Bytes, about 123 MB/s
    stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
    stream load orc:          67 seconds loaded 1101869774 Bytes, about 15 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          30.2 seconds inserted 10000000 Rows, about 331K ops/s
    storage size: 17184104018 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](tvf)Fixed the avro-scanner projection pushdown failing to query… [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #28709:
URL: https://github.com/apache/doris/pull/28709


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](tvf)Fixed the avro-scanner projection pushdown failing to query… [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28709:
URL: https://github.com/apache/doris/pull/28709#issuecomment-1864134178

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org