You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Mryange (via GitHub)" <gi...@apache.org> on 2023/10/26 10:28:31 UTC
[PR] Profile 1 [doris]
Mryange opened a new pull request, #25963:
URL: https://github.com/apache/doris/pull/25963
## Proposed changes
pipelineX
```
SimpleProfile:
Instance(1):
RESULT_SINK_OPERATOR (id=0):
- CloseTime: avg 11.897us, max 11.897us, min 11.897us
- InputRows: 0
- OpenTime: avg 134.655us, max 134.655us, min 134.655us
AGGREGATION_OPERATOR (id=789):
- PlanInfo
- output: sum(partial_sum((lo_extendedprice * lo_discount)))[#45]
- group by:
- cardinality=1
- BlocksReturned: 1
- CloseTime: avg 4.668us, max 4.668us, min 4.668us
- OpenTime: avg 31.901us, max 31.901us, min 31.901us
- ProjectionTime: avg 0ns, max 0ns, min 0ns
- RowsReturned: 1
AGGREGATION_SINK_OPERATOR (id=789):
- CloseTime: avg 3.382us, max 3.382us, min 3.382us
- InputRows: 8
- OpenTime: avg 116.611us, max 116.611us, min 116.611us
EXCHANGE_OPERATOR (id=786):
- PlanInfo
- offset: 0
- BlocksReturned: 8
- CloseTime: avg 16.266us, max 16.266us, min 16.266us
- OpenTime: avg 108.464us, max 108.464us, min 108.464us
- ProjectionTime: avg 0ns, max 0ns, min 0ns
- RowsReturned: 8
Instance(8):
DATA_STREAM_SINK_OPERATOR (id=786):
- BlocksSent: 8
- CloseTime: avg 40.989us, max 46.870us, min 34.721us
- InputRows: 8
- OpenTime: avg 172.45us, max 218.876us, min 145.983us
AGGREGATION_OPERATOR (id=783):
- PlanInfo
- output: partial_sum((lo_extendedprice * lo_discount))[#44]
- group by:
- cardinality=1
- BlocksReturned: 8
- CloseTime: avg 6.870us, max 8.781us, min 4.768us
- OpenTime: avg 26.276us, max 28.113us, min 23.632us
- ProjectionTime: avg 0ns, max 0ns, min 0ns
- RowsReturned: 8
AGGREGATION_SINK_OPERATOR (id=783):
- CloseTime: avg 3.619us, max 4.754us, min 2.356us
- InputRows: 11.913972M (11913972)
- OpenTime: avg 102.743us, max 107.788us, min 93.91us
HASH_JOIN_OPERATOR (id=777):
- PlanInfo
- join op: INNER JOIN(BROADCAST)[]
- equal join conjunct: lo_orderdate = d_datekey
- runtime filters: RF000[in_or_bloom] <- d_datekey(-1/0/2097152)
- cardinality=75,006,015
- vec output tuple id: 5
- vIntermediate tuple ids: 4
- hash output slot ids: 36 37
- BlocksReturned: 3.575K (3575)
- CloseTime: avg 13.300us, max 14.626us, min 11.534us
- OpenTime: avg 71.560us, max 76.128us, min 67.947us
- ProbeRows: 11.913972M (11913972)
- ProjectionTime: avg 3.55ms, max 4.545ms, min 1.595ms
- RowsReturned: 11.913972M (11913972)
OLAP_SCAN_OPERATOR (id=756):
- PlanInfo
- TABLE: default_cluster:ssb.lineorder(lineorder), PREAGGREGATION: ON
- PREDICATES: lo_discount <= 3 AND lo_discount >= 1 AND lo_quantity < 25
- runtime filters: RF000[in_or_bloom] -> lo_orderdate
- partitions=7/7, tablets=336/336, tabletList=11074,11076,11078 ...
- cardinality=600037902, avgRowSize=130.72264, numNodes=1
- pushAggOp=NONE
- projections: lo_orderdate, lo_extendedprice, lo_discount
- project output tuple id: 3
- BlocksReturned: 3.575K (3575)
- CloseTime: avg 417.334us, max 644.891us, min 273.816us
- OpenTime: avg 2.119ms, max 2.495ms, min 1.869ms
- ProjectionTime: avg 10.601ms, max 15.483ms, min 5.416ms
- RowsReturned: 11.913972M (11913972)
:
VScanner:
SegmentIterator:
HASH_JOIN_SINK_OPERATOR (id=777):
- CloseTime: avg 0ns, max 0ns, min 0ns
- InputRows: 365
- OpenTime: avg 211.806us, max 229.724us, min 196.571us
EXCHANGE_OPERATOR (id=774):
- PlanInfo
- offset: 0
- BlocksReturned: 1
- CloseTime: avg 14.623us, max 28.378us, min 6.797us
- OpenTime: avg 87.146us, max 101.315us, min 70.978us
- ProjectionTime: avg 0ns, max 0ns, min 0ns
- RowsReturned: 365
Instance(1):
DATA_STREAM_SINK_OPERATOR (id=774):
- BlocksSent: 1
- CloseTime: avg 43.797us, max 43.797us, min 43.797us
- InputRows: 365
- OpenTime: avg 156.41us, max 156.41us, min 156.41us
OLAP_SCAN_OPERATOR (id=765):
- PlanInfo
- TABLE: default_cluster:ssb.dates(dates), PREAGGREGATION: ON
- PREDICATES: d_year = 1993
- partitions=1/1, tablets=1/1, tabletList=11778
- cardinality=2556, avgRowSize=68.56221, numNodes=1
- pushAggOp=NONE
- projections: d_datekey
- project output tuple id: 1
- BlocksReturned: 1
- CloseTime: avg 120.494us, max 120.494us, min 120.494us
- OpenTime: avg 756.687us, max 756.687us, min 756.687us
- ProjectionTime: avg 64.322us, max 64.322us, min 64.322us
- RowsReturned: 365
VScanner:
SegmentIterator:
```
pipeline
```
SimpleProfile:
Instance(1):
VDataBufferSender (dst_fragment_instance_id=-6e2d87aa85b6b5ff--52848c53948863f5):
VAGGREGATION_NODE (id=789):
- PlanInfo
- output: sum(partial_sum((lo_extendedprice * lo_discount)))[#45]
- group by:
- cardinality=1
- BlockCount: 1
- BuildTime: avg 0ns, max 0ns, min 0ns
- ExecTime: avg 254.705us, max 254.705us, min 254.705us
- OutputBytes: 4.03 KB
- RowsReturned: 1
VEXCHANGE_NODE (id=786):
- PlanInfo
- offset: 0
- BlockCount: 8
- ExecTime: avg 153.349us, max 153.349us, min 153.349us
- OutputBytes: 512.00 B
- RowsReturned: 8
Instance(8):
VDataStreamSender (dst_id=786, dst_fragments=[{"91d278557a494a01-ad7b73ac6b779c0b"}]):
- BlocksSent: 8
VAGGREGATION_NODE (id=783):
- PlanInfo
- output: partial_sum((lo_extendedprice * lo_discount))[#44]
- group by:
- cardinality=1
- BlockCount: 8
- BuildTime: avg 82.67ms, max 117.939ms, min 19.573ms
- ExecTime: avg 123.348ms, max 176.956ms, min 29.843ms
- OutputBytes: 512.00 B
- RowsReturned: 8
VHASH_JOIN_NODE (id=777):
- PlanInfo
- join op: INNER JOIN(BROADCAST)[]
- equal join conjunct: lo_orderdate = d_datekey
- runtime filters: RF000[in_or_bloom] <- d_datekey(-1/0/2097152)
- cardinality=75,006,015
- vec output tuple id: 5
- vIntermediate tuple ids: 4
- hash output slot ids: 36 37
- BlockCount: 3.575K (3575)
- ExecTime: avg 683.521ms, max 971.39ms, min 144.256ms
- OutputBytes: 120.94 MB
- RowsReturned: 11.913972M (11913972)
ProbePhase:
- ProbeRows: 11.913972M (11913972)
- ProbeTime: avg 682.178ms, max 969.182ms, min 143.805ms
BuildPhase:
- BuildRows: 365
- BuildTime: avg 67.643us, max 541.151us, min 0ns
VEXCHANGE_NODE (id=774):
- PlanInfo
- offset: 0
- BlockCount: 1
- ExecTime: avg 65.314us, max 77.500us, min 48.296us
- OutputBytes: 2.00 KB
- RowsReturned: 365
VNewOlapScanNode(lineorder) (id=756):
- PlanInfo
- TABLE: default_cluster:ssb.lineorder(lineorder), PREAGGREGATION: ON
- PREDICATES: lo_discount <= 3 AND lo_discount >= 1 AND lo_quantity < 25
- runtime filters: RF000[in_or_bloom] -> lo_orderdate
- partitions=7/7, tablets=336/336, tabletList=11074,11076,11078 ...
- cardinality=600037902, avgRowSize=130.72264, numNodes=1
- pushAggOp=NONE
- projections: lo_orderdate, lo_extendedprice, lo_discount
- project output tuple id: 3
- BlockCount: 3.575K (3575)
- ExecTime: avg 15.487ms, max 21.927ms, min 4.504ms
- OutputBytes: 174.52 MB
- RowsReturned: 11.913972M (11913972)
:
VScanner:
SegmentIterator:
Instance(8):
VDataStreamSender (dst_id=774, dst_fragments=[{"91d278557a494a01-ad7b73ac6b779c03"}]):
- BlocksSent: 1
VNewOlapScanNode(dates) (id=765):
- PlanInfo
- TABLE: default_cluster:ssb.dates(dates), PREAGGREGATION: ON
- PREDICATES: d_year = 1993
- partitions=1/1, tablets=1/1, tabletList=11778
- cardinality=2556, avgRowSize=68.56221, numNodes=1
- pushAggOp=NONE
- projections: d_datekey
- project output tuple id: 1
- BlockCount: 1
- ExecTime: avg 258.666us, max 411.533us, min 180.518us
- OutputBytes: 2.00 KB
- RowsReturned: 365
VScanner:
SegmentIterator:
```
no pipeline
```
SimpleProfile:
Instance(1):
VDataBufferSender (dst_fragment_instance_id=104524f196a845a0--7ca35c7d54b3254e):
VAGGREGATION_NODE (id=789):
- PlanInfo
- output: sum(partial_sum((lo_extendedprice * lo_discount)))[#45]
- group by:
- cardinality=1
- BlockCount: 1
- BuildTime: avg 0ns, max 0ns, min 0ns
- ExecTime: avg 245.579us, max 245.579us, min 245.579us
- OutputBytes: 4.03 KB
- RowsReturned: 1
VEXCHANGE_NODE (id=786):
- PlanInfo
- offset: 0
- BlockCount: 8
- ExecTime: avg 1s132ms, max 1s132ms, min 1s132ms
- OutputBytes: 512.00 B
- RowsReturned: 8
Instance(8):
VDataStreamSender (dst_id=786, dst_fragments=[{"104524f196a845a0-835ca382ab4cdab2"}]):
- BlocksSent: 8
VAGGREGATION_NODE (id=783):
- PlanInfo
- output: partial_sum((lo_extendedprice * lo_discount))[#44]
- group by:
- cardinality=1
- BlockCount: 8
- BuildTime: avg 81.278ms, max 109.181ms, min 25.462ms
- ExecTime: avg 122.42ms, max 163.947ms, min 38.339ms
- OutputBytes: 512.00 B
- RowsReturned: 8
VHASH_JOIN_NODE (id=777):
- PlanInfo
- join op: INNER JOIN(BROADCAST)[]
- equal join conjunct: lo_orderdate = d_datekey
- runtime filters: RF000[in_or_bloom] <- d_datekey(-1/0/2097152)
- cardinality=75,006,015
- vec output tuple id: 5
- vIntermediate tuple ids: 4
- hash output slot ids: 36 37
- BlockCount: 3.575K (3575)
- ExecTime: avg 770.537ms, max 986.275ms, min 265.347ms
- OutputBytes: 120.09 MB
- RowsReturned: 11.913972M (11913972)
ProbePhase:
- ProbeRows: 11.913972M (11913972)
- ProbeTime: avg 767.581ms, max 982.862ms, min 262.871ms
BuildPhase:
- BuildRows: 365
- BuildTime: avg 62.703us, max 501.628us, min 0ns
VEXCHANGE_NODE (id=774):
- PlanInfo
- offset: 0
- BlockCount: 1
- ExecTime: avg 245.797us, max 1.610ms, min 48.227us
- OutputBytes: 2.00 KB
- RowsReturned: 365
VNewOlapScanNode(lineorder) (id=756):
- PlanInfo
- TABLE: default_cluster:ssb.lineorder(lineorder), PREAGGREGATION: ON
- PREDICATES: lo_discount <= 3 AND lo_discount >= 1 AND lo_quantity < 25
- runtime filters: RF000[in_or_bloom] -> lo_orderdate
- partitions=7/7, tablets=336/336, tabletList=11074,11076,11078 ...
- cardinality=600037902, avgRowSize=130.72264, numNodes=1
- pushAggOp=NONE
- projections: lo_orderdate, lo_extendedprice, lo_discount
- project output tuple id: 3
- BlockCount: 3.575K (3575)
- ExecTime: avg 33.403ms, max 36.769ms, min 24.684ms
- OutputBytes: 180.14 MB
- RowsReturned: 11.913972M (11913972)
:
VScanner:
SegmentIterator:
Instance(1):
VDataStreamSender (dst_id=774, dst_fragments=[{"104524f196a845a0-835ca382ab4cdaaa", "104524f196a845a0-835ca382ab4cdaab", "104524f196a845a0-835ca382ab4cdaac", "104524f196a845a0-835ca382ab4cdaad", "104524f196a845a0-835ca382ab4cdaae", "104524f196a845a0-835ca382ab4cdaaf", "104524f196a845a0-835ca382ab4cdab0", "104524f196a845a0-835ca382ab4cdab1"}]):
- BlocksSent: 1
VNewOlapScanNode(dates) (id=765):
- PlanInfo
- TABLE: default_cluster:ssb.dates(dates), PREAGGREGATION: ON
- PREDICATES: d_year = 1993
- partitions=1/1, tablets=1/1, tabletList=11778
- cardinality=2556, avgRowSize=68.56221, numNodes=1
- pushAggOp=NONE
- projections: d_datekey
- project output tuple id: 1
- BlockCount: 1
- ExecTime: avg 1.965ms, max 1.965ms, min 1.965ms
- OutputBytes: 2.00 KB
- RowsReturned: 365
VScanner:
SegmentIterator:
```
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](profilev2) format profilev2 [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #25963:
URL: https://github.com/apache/doris/pull/25963
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](profilev2) format profilev2 [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #25963:
URL: https://github.com/apache/doris/pull/25963#issuecomment-1782184577
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Profile 1 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25963:
URL: https://github.com/apache/doris/pull/25963#issuecomment-1780933564
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.1 seconds
stream load tsv: 552 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162024157 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](profilev2) format profilev2 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #25963:
URL: https://github.com/apache/doris/pull/25963#discussion_r1374006202
##########
be/src/pipeline/exec/exchange_sink_operator.cpp:
##########
@@ -235,6 +235,14 @@ Status ExchangeSinkLocalState::open(RuntimeState* state) {
return Status::OK();
}
+std::string ExchangeSinkLocalState::id_name() {
Review Comment:
warning: method 'id_name' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/exec/exchange_sink_operator.h:161:
```diff
- std::string id_name() override;
+ static std::string id_name() override;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](profilev2) format profilev2 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25963:
URL: https://github.com/apache/doris/pull/25963#issuecomment-1782260939
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.58 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162212871 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](profilev2) format profilev2 [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #25963:
URL: https://github.com/apache/doris/pull/25963#discussion_r1373977991
##########
fe/fe-core/src/main/java/org/apache/doris/common/util/RuntimeProfile.java:
##########
@@ -690,7 +820,9 @@ public void setProfileLevel(int profileLevel) {
}
public void setPlaner(Planner planner) {
Review Comment:
not use set planner, rename to setFragmentPlanInfo(map<id,info>)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](profilev2) format profilev2 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25963:
URL: https://github.com/apache/doris/pull/25963#issuecomment-1782184991
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](profilev2) format profilev2 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25963:
URL: https://github.com/apache/doris/pull/25963#issuecomment-1782185011
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Profile 1 [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #25963:
URL: https://github.com/apache/doris/pull/25963#issuecomment-1780846731
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org