You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "924060929 (via GitHub)" <gi...@apache.org> on 2026/05/18 10:24:45 UTC
[PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
924060929 opened a new pull request, #63366:
URL: https://github.com/apache/doris/pull/63366
Previously, local exchange (LE) nodes were inserted exclusively by the
BE's `_plan_local_exchange` at pipeline build time. The FE had no
visibility into which operators needed a fan-out or shuffle before
execution, making it impossible to validate, optimize, or override LE
decisions at planning time.
This PR introduces a full FE-side local exchange planner that mirrors
BE semantics, brings several correctness fixes, and leaves the legacy
BE path fully intact behind a feature flag. See "Current architecture
notes" at the bottom for what the FE planner does and does not own.
A new `AddLocalExchange` pass runs after normal fragment assignment.
It walks each fragment's plan tree bottom-up, calling the polymorphic
`PlanNode.enforceAndDeriveLocalExchange()` on every node. Nodes
declare what distribution they require of their children; the framework
inserts `LocalExchangeNode` where needed.
`LocalExchangeNode` represents intra-fragment data redistribution and
supports the full set of exchange types: PASSTHROUGH, HASH_SHUFFLE,
BUCKET_HASH_SHUFFLE, GLOBAL_EXECUTION_HASH_SHUFFLE, BROADCAST,
PASS_TO_ONE, ADAPTIVE_PASSTHROUGH, LOCAL_MERGE_SORT, and NOOP.
The pass is guarded by `enable_local_shuffle_planner` (default true).
When disabled, BE continues to run its own `_plan_local_exchange` as
before, keeping the old path fully intact.
`maxPerBeInstances` (max pipeline instances assigned to any single BE)
is used instead of a global `instanceCount`. Planning is a no-op when
`maxPerBeInstances == 1` — inserting LE on a single-threaded pipeline
would cause task-count mismatches and pipeline starvation.
When a serial operator (e.g. OlapScanNode with a single tablet bucket)
feeds a non-serial parent without an intermediate LE, downstream tasks
starve waiting for data that never arrives. The framework detects this
case and inserts a PASSTHROUGH LE to restore N-task parallelism, exactly
matching BE's `required_data_distribution()` serial → PASSTHROUGH rule.
`LocalExchangeTypeRequire` abstracts two strategies:
- `RequireHash` — always resolves to `LOCAL_EXECUTION_HASH_SHUFFLE`
(safe for intra-fragment hash partitioning).
- `RequireSpecific` — preserves BUCKET_HASH_SHUFFLE /
GLOBAL_EXECUTION_HASH_SHUFFLE without degradation.
PR #62438 added `enable_local_exchange_before_agg`, but its BE guard
`!_needs_finalize && !enable_local_exchange_before_agg → base` conflated
two semantically different cases in AggSink and DistinctStreamingAgg:
- **AggSink**: `!finalize && hasKeys` covered both LOCAL preagg
(performance-only) and FIRST_MERGE dedup (correctness-critical).
The flag-gated early-return wrongly skipped HASH for FIRST_MERGE,
producing PASSTHROUGH-over-serial-child → wrong aggregation results.
- **DistinctStreamingAgg**: `!finalize` covered both streaming preagg
(`useStreamingPreagg=true`, performance) and non-streaming dedup
(`useStreamingPreagg=false`, correctness). Same class of bug.
FE fix:
- AggSink: restrict the flag-gated base path to `!isMerge()` LOCAL
phases. FIRST_MERGE always emits HASH regardless of the flag.
- DistinctStreamingAgg: restrict to `useStreamingPreagg=true`.
Non-streaming dedup always emits HASH.
Also add `requiresShuffleForCorrectness()` to mirror BE's
`is_shuffled_operator()`, so SetOperationNode propagates the
"downstream depends on hash" flag correctly instead of using the
coarser `parentRequire.preferType().isHashShuffle()` check that
over-inserted HASH LE on every union branch under a streaming preagg.
These fixes reduce FE/BE consistency mismatches from 8 to 3
(only pre-existing NLJ optimization differences remain).
- `enable_local_shuffle_planner` — use FE planner (default true)
- `enable_local_shuffle` — master switch for local shuffle
- `enable_local_exchange_before_agg` — HASH LE before non-final agg
(default true, mirrors apache/doris#62438)
`validateNoSerialWithoutLocalExchange()` walks the final plan tree and
logs a warning whenever a serial operator feeds a non-serial parent
without an intermediate LocalExchangeNode, catching planning gaps
before execution.
- `test_enable_local_exchange_before_agg.groovy` — 10 agg patterns
with the flag on and off; covers the FIRST_MERGE and
DistinctStreamingAgg correctness fixes.
- `test_local_shuffle_fe_be_consistency.groovy` — runs the same SQL
with `enable_local_shuffle_planner=true` and `=false` across the
full operator matrix (Agg, Sort, Analytic, HashJoin, NLJ, Set, Union,
TableFunction, AssertNumRows, RQG-derived corner cases) and asserts
result rows are identical. Only data correctness is asserted — the
two planners legitimately differ on the exact exchange counts/types
they emit, so plan-shape equality is intentionally not checked.
- `test_local_shuffle_rqg_bugs.groovy` — reproduces 20+ RQG-found
crashes and wrong-result cases.
- `test_old_coordinator_local_shuffle.groovy` — verifies the old
coordinator path is unaffected.
- `test_multilevel_join_agg_local_shuffle.groovy` — multi-level join
and aggregation plan shapes.
- `multi_version.h`: replace `atomic_load/atomic_store` (deprecated in
libstdc++ C++20 / LLVM 20) with `std::shared_mutex`-based RW locking.
- `memory.cpp`: fix `std::max` type mismatch (`long` vs `int64_t`)
on macOS.
- `bucketed_aggregation_sink_operator.h`: fix `ExchangeType::NOOP` →
`TLocalPartitionType::NOOP` after thrift enum rename.
This PR puts the FE planner in the driver's seat for LE insertion but
intentionally does NOT remove the BE-side machinery — readers should be
aware of three pieces the FE planner shares with or defers to BE:
1. **`is_serial_operator` is computed on both sides.** FE computes the
flag and writes it into Thrift, but BE's
`OperatorBase::is_serial_operator()` is still overridden per operator
in C++ and used for BE-side runtime decisions. Any future change to
the BE override needs to be mirrored on the FE side (and vice versa)
to keep the planner's view consistent with execution.
2. **The legacy BE planner stays as a fallback.**
`pipeline_fragment_context.cpp::_plan_local_exchange` is preserved
and gated by `runtime_state.h::plan_local_shuffle()`: when
`enable_local_shuffle_planner=false`, BE plans LE itself, exactly as
before. The two paths are mutually exclusive, never both running on
the same query.
3. **`_propagate_local_exchange_num_tasks` is kept as a runtime safety
net.** The two propagation passes in
`pipeline_fragment_context.cpp` fix up paired pipelines whose
`num_tasks` end up mismatched (e.g. when AGG/SORT/JOIN pipeline
splits leave a serial Exchange feeding an N-task sink). FE's
framework-level serial→non-serial fan-out (`enforceRequire` step 3)
and the `validateNoSerialWithoutLocalExchange` check aim to make
these mismatches impossible by construction, but the BE-side fixup
remains as a defensive guard.
Co-authored-by: Gabriel <li...@selectdb.com>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4479137591
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/15d92ba024b265d9441145d7bd26d93590aa9c63_15d92ba024b265d9441145d7bd26d93590aa9c63_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/15d92ba024b265d9441145d7bd26d93590aa9c63_15d92ba024b265d9441145d7bd26d93590aa9c63_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.06% (1854/2375) |
| Line Coverage | 64.52% (33325/51653) |
| Region Coverage | 65.21% (16520/25335) |
| Branch Coverage | 55.70% (8827/15848) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4485186216
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4496936392
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4491773872
# FE Regression Coverage Report
Increment line coverage ` 80.85% (477/590)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_4e05e7bfbb8aace37eb65198a1530e9325047d34_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_4e05e7bfbb8aace37eb65198a1530e9325047d34_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4489014728
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4550839649
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/a68f47c588b0124cf5474ff805c847df5dfc781e_a68f47c588b0124cf5474ff805c847df5dfc781e_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/a68f47c588b0124cf5474ff805c847df5dfc781e_a68f47c588b0124cf5474ff805c847df5dfc781e_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.06% (1854/2375) |
| Line Coverage | 64.52% (33333/51665) |
| Region Coverage | 65.18% (16519/25343) |
| Branch Coverage | 55.72% (8834/15854) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4671961348
### Correction on the [P1] throw (AddLocalExchange)
My earlier reply claimed the `throw` "never misfires" — that was wrong, and pushing it proved it. The PR's P0 / cloud_p0 runs surfaced **9 distinct legitimate patterns** where the throw fires but the plan is correct:
| pattern | tests hit |
|---|---|
| `Serial UnionNode → non-serial TableFunctionNode` | unnest_join_list, unnest_where_list, salt_join, skew_join, correlated_scalar_subquery |
| `Serial UnionNode → non-serial AggregationNode` | pull_up_predicate_set_op |
| `Serial RecursiveCteScanNode → non-serial NLJ / Agg` | rec_cte, rec_cte_from_mysql_doc |
| `Serial OlapScanNode → non-serial HashJoinNode` | test_crossjoin_inlineview_slot |
In all of these the parent runs **effectively serial** at runtime and the BE reconciles `num_tasks` across the coupled pipelines, so no LocalExchange is required and the results are correct (they all pass with the `warn`). This validation compares FE-static `isSerialOperatorOnBe`, which can't model that runtime reconciliation — so it is **not a hard invariant**.
I've reverted it to a `warn` (best-effort heuristic) with a comment explaining why a hard throw is unsafe here. Trying the throw was still worthwhile — it's exactly what surfaced these cases.
### Also fixed from the same CI run
- **`force_passthrough` NPE:** `getConnectContext()` is null under the planner unit-test mocks, so the new `enableBroadcastJoinForcePassthrough` read NPE'd in `testHashJoinNodeBranches`. Made it null-safe (treat null context as "no force").
- **clang-format:** `agg_operator_test.cpp` had a >100-col line (v16).
(The remaining P0/cloud_p0 reds are pre-existing/flaky and unrelated to local shuffle — variant `element_at` logical-planner slot errors, a json_extract `IndexOutOfBounds`, a float→bigint cast overflow, a temp-table case, and one `SUCCESS vs RUNNING` timeout. The BE-UT red is a CI git-mirror checkout failure on the agent.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4675192636
<details>
<summary>TPC-DS: <b>Total hot run time: 169893 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f056ebbed42446ea468bcc6891c05520bdfb31c3, data reload: false
query5 4328 673 482 482
query6 431 212 183 183
query7 4921 597 315 315
query8 371 221 219 219
query9 8762 4057 4044 4044
query10 466 323 261 261
query11 5922 2385 2229 2229
query12 164 107 103 103
query13 1347 616 492 492
query14 6217 5444 5041 5041
query14_1 4410 4395 4375 4375
query15 207 200 172 172
query16 1014 462 451 451
query17 1105 708 568 568
query18 2579 491 341 341
query19 205 187 139 139
query20 112 118 114 114
query21 214 141 116 116
query22 13649 13636 13522 13522
query23 17275 16513 16075 16075
query23_1 16293 16170 16295 16170
query24 7482 1826 1346 1346
query24_1 1358 1308 1325 1308
query25 602 494 416 416
query26 1314 327 167 167
query27 2695 573 352 352
query28 4460 2082 2060 2060
query29 1126 671 519 519
query30 324 244 201 201
query31 1137 1099 963 963
query32 115 72 60 60
query33 529 345 275 275
query34 1190 1230 671 671
query35 798 816 719 719
query36 1394 1364 1282 1282
query37 162 109 98 98
query38 3172 3154 3046 3046
query39 957 928 900 900
query39_1 882 867 869 867
query40 253 131 108 108
query41 73 79 74 74
query42 102 97 97 97
query43 326 334 283 283
query44
query45 202 193 190 190
query46 1141 1247 772 772
query47 2384 2407 2292 2292
query48 408 400 289 289
query49 643 511 372 372
query50 1069 372 260 260
query51 4472 4395 4419 4395
query52 92 102 79 79
query53 261 279 195 195
query54 284 239 216 216
query55 81 83 72 72
query56 264 253 233 233
query57 1434 1441 1352 1352
query58 261 230 224 224
query59 1600 1672 1434 1434
query60 300 268 280 268
query61 163 164 155 155
query62 717 655 593 593
query63 228 191 190 190
query64 2580 813 648 648
query65
query66 1779 500 357 357
query67 29676 29675 29525 29525
query68
query69 426 311 269 269
query70 963 974 948 948
query71 309 235 213 213
query72 2999 2750 2461 2461
query73 828 778 451 451
query74 5118 4968 4804 4804
query75 2794 2666 2231 2231
query76 2299 1226 813 813
query77 388 393 289 289
query78 12435 12548 11897 11897
query79 1487 1125 775 775
query80 1292 498 393 393
query81 516 290 251 251
query82 637 158 127 127
query83 361 284 249 249
query84
query85 928 540 465 465
query86 434 306 288 288
query87 3467 3330 3137 3137
query88 3765 2789 2777 2777
query89 434 387 338 338
query90 1901 185 181 181
query91 177 162 144 144
query92 60 65 58 58
query93 1596 1488 919 919
query94 738 310 308 308
query95 669 386 340 340
query96 1087 850 353 353
query97 2731 2721 2587 2587
query98 220 212 207 207
query99 1202 1208 1025 1025
Total cold run time: 253549 ms
Total hot run time: 169893 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4765222440
Codex automated review failed and did not complete.
Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jun 24th, 2026 10:56 PM.
Workflow run: https://github.com/apache/doris/actions/runs/27931683980
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4766024620
# FE UT Coverage Report
Increment line coverage ` 74.69% (419/561)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3301894165
##########
be/src/exec/exchange/local_exchange_sink_operator.cpp:
##########
@@ -37,24 +37,30 @@ std::vector<Dependency*> LocalExchangeSinkLocalState::dependencies() const {
return deps;
}
-Status LocalExchangeSinkOperatorX::init(RuntimeState* state, ExchangeType type,
- const int num_buckets, const bool use_global_hash_shuffle,
+Status LocalExchangeSinkOperatorX::init(RuntimeState* state, TLocalPartitionType::type type,
+ const int num_buckets,
const std::map<int, int>& shuffle_idx_to_instance_idx) {
+ DCHECK(!_planned_by_fe);
_name = "LOCAL_EXCHANGE_SINK_OPERATOR(" + get_exchange_type_name(type) + ")";
_type = type;
- if (_type == ExchangeType::HASH_SHUFFLE) {
- _shuffle_idx_to_instance_idx.clear();
- _use_global_shuffle = use_global_hash_shuffle;
+ if (_type == TLocalPartitionType::GLOBAL_EXECUTION_HASH_SHUFFLE) {
// For shuffle join, if data distribution has been broken by previous operator, we
// should use a HASH_SHUFFLE local exchanger to shuffle data again. To be mentioned,
// we should use map shuffle idx to instance idx because all instances will be
// distributed to all BEs. Otherwise, we should use shuffle idx directly.
- if (use_global_hash_shuffle) {
- _shuffle_idx_to_instance_idx = shuffle_idx_to_instance_idx;
+ _shuffle_idx_to_instance_idx = shuffle_idx_to_instance_idx;
+ if (state->query_options().__isset.enable_new_shuffle_hash_method &&
Review Comment:
这里和init_partitioner似乎有点冗余
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -290,6 +290,32 @@ Status PipelineFragmentContext::_build_and_prepare_full_pipeline(ThreadPool* thr
RETURN_IF_ERROR(_build_pipelines(_runtime_state->obj_pool(), *_query_ctx->desc_tbl,
&_root_op, root_pipeline));
+ // Propagate _num_instances from LOCAL_EXCHANGE pipelines to ancestor pipelines
+ // that inherited reduced num_tasks from a serial operator.
+ _propagate_local_exchange_num_tasks();
+
+ // Create deferred local exchangers now that all pipelines have final num_tasks.
+ RETURN_IF_ERROR(_create_deferred_local_exchangers());
Review Comment:
如果fe完全接管了local exchange的计算，应该就不需要be自己算required_data_distribution和算子并行度了？be不需要感知required_data_distribution，全按fe规划的来应该就行？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3413410484
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -634,7 +636,8 @@ protected void splitFragments(PhysicalPlan resultPlan) {
return;
}
- PlanTranslatorContext planTranslatorContext = new PlanTranslatorContext(cascadesContext);
+ this.planTranslatorContext = new PlanTranslatorContext(cascadesContext);
+ PlanTranslatorContext planTranslatorContext = this.planTranslatorContext;
Review Comment:
Done in `fe9a3316be0` — removed; subsequent references resolve to the field `this.planTranslatorContext`.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -657,7 +660,8 @@ protected void splitFragments(PhysicalPlan resultPlan) {
scanNodeList.addAll(planTranslatorContext.getScanNodes());
physicalRelations.addAll(planTranslatorContext.getPhysicalRelations());
descTable = planTranslatorContext.getDescTable();
- fragments = new ArrayList<>(planTranslatorContext.getPlanFragments());
+ List<PlanFragment> planFragments = planTranslatorContext.getPlanFragments();
Review Comment:
Done in `fe9a3316be0` — inlined into `fragments = new ArrayList<>(planTranslatorContext.getPlanFragments())`.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -740,6 +744,18 @@ protected void distribute(PhysicalPlan physicalPlan, ExplainLevel explainLevel)
splitFragments(physicalPlan);
doDistribute(canUseNereidsDistributePlanner, explainLevel);
+
+ SessionVariable sessionVariable = cascadesContext.getConnectContext().getSessionVariable();
+ if (sessionVariable.isEnableLocalShufflePlanner() && sessionVariable.isEnableLocalShuffle()) {
+ addLocalExchangeAfterDistribute();
+ }
Review Comment:
Done in `fe9a3316be0` — moved the `enable_local_shuffle_planner && enable_local_shuffle` gate into `addLocalExchangeAfterDistribute()`; the caller now just invokes it unconditionally.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/BucketedAggregationNode.java:
##########
@@ -97,7 +98,7 @@ public String getNodeExplainString(String detailPrefix, TExplainLevel detailLeve
}
@Override
- public boolean isSerialOperator() {
+ public boolean isSerialOperatorOnBe(ConnectContext context) {
Review Comment:
Changed in `fe9a3316be0` to override `isSerialNode()` (returning `false`) instead of `isSerialOperatorOnBe()`, matching every other node — behavior-equivalent.
One correction on the premise: `BucketedAggregationNode extends PlanNode`, not `AggregationNode`, so it never inherited `AggregationNode.isSerialNode()`; `isSerialNode()` was already the base `false`, and the old `isSerialOperatorOnBe()=false` override was effectively redundant. So there was no latent inconsistency, but overriding `isSerialNode()` is the cleaner/consistent form regardless.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/AggregationNode.java:
##########
@@ -258,4 +268,155 @@ public boolean isQueryCacheCandidate() {
public void setQueryCacheCandidate(boolean queryCacheCandidate) {
this.queryCacheCandidate = queryCacheCandidate;
}
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+
+ ConnectContext connectContext = translatorContext.getConnectContext();
+ SessionVariable sessionVariable = connectContext.getSessionVariable();
+ // PR #62438: when false, non-finalize agg falls back to BE base class.
+ boolean enableLeBeforeAgg = sessionVariable.enableLocalExchangeBeforeAgg;
+ boolean hasKeys = !aggInfo.getGroupingExprs().isEmpty();
+
+ // Each branch mirrors the corresponding BE operator's required_data_distribution()
+ // check order 1:1. The helper baseClassRequire() expands BE's base class behavior.
+ LocalExchangeTypeRequire requireChild;
+ if (canUseDistinctStreamingAgg(sessionVariable)) {
+ // DistinctStreamingAggOperatorX. Two flavors share this operator class:
+ // - streaming preagg (useStreamingPreagg=true): performance-only,
+ // flag controls
+ // - non-streaming dedup (useStreamingPreagg=false): correctness-required,
+ // always HASH regardless of flag
+ // Diverges from BE: BE's `!_needs_finalize && !enable_local_exchange_before_agg`
+ // early return catches non-streaming dedup too, causing the same family of
+ // wrong-result bug as AggSink (DORIS-25413).
+ if (needsFinalize && !hasKeys) {
+ requireChild = LocalExchangeTypeRequire.noRequire();
+ } else if (!needsFinalize && useStreamingPreagg && !enableLeBeforeAgg) {
+ requireChild = baseClassRequire(connectContext);
+ } else if (needsFinalize || (hasKeys && !useStreamingPreagg)) {
+ requireChild = AddLocalExchange.isColocated(this)
+ ? LocalExchangeTypeRequire.requireHash()
+ : parentRequire.autoRequireHash();
+ } else if (sessionVariable.enableDistinctStreamingAggForcePassthrough) {
+ requireChild = LocalExchangeTypeRequire.requirePassthrough();
+ } else {
+ requireChild = baseClassRequire(connectContext);
+ }
+ } else if (useStreamingPreagg) {
+ // StreamingAggOperatorX
+ if (children.get(0) instanceof HashJoinNode
+ && sessionVariable.enableStreamingAggHashJoinForcePassthrough) {
+ requireChild = LocalExchangeTypeRequire.requirePassthrough();
+ } else if (!needsFinalize && !enableLeBeforeAgg) {
+ requireChild = baseClassRequire(connectContext);
+ } else if (!hasKeys) {
+ requireChild = needsFinalize
+ ? LocalExchangeTypeRequire.noRequire()
+ : baseClassRequire(connectContext);
+ } else {
+ requireChild = LocalExchangeTypeRequire.requireHash();
+ }
+ } else {
+ // AggSinkOperatorX — covers finalize phase AND non-finalize phases (LOCAL
+ // preagg / FIRST_MERGE dedup). Streaming preagg goes through the StreamingAgg
+ // branch above, not here.
+ //
+ // Phase semantics for !needsFinalize:
+ // - FIRST / SECOND (LOCAL phase, !isMerge): performance-only, flag controls
+ // - FIRST_MERGE (correctness-required): always HASH regardless of flag
+ //
+ // Diverges from BE here: BE's `!_needs_finalize && !enable_local_exchange_before_agg`
+ // early return also catches FIRST_MERGE, dropping the HASH requirement and
+ // causing wrong-result (e.g. PASSTHROUGH over serial child breaks the
+ // group-by-key invariant — DORIS-25413).
+ if (!hasKeys) {
+ requireChild = needsFinalize
+ ? LocalExchangeTypeRequire.noRequire()
+ : baseClassRequire(connectContext);
+ } else if (!needsFinalize && !aggInfo.isMerge() && !enableLeBeforeAgg) {
+ // LOCAL phase (FIRST preagg / SECOND distinct local) + user opted out
+ // of pre-agg LE → base class decides: serial child → PASSTHROUGH
+ // (parallelism), non-serial child → NOOP (no LE).
+ requireChild = baseClassRequire(connectContext);
+ } else if (!needsFinalize || AddLocalExchange.isColocated(this)) {
+ // FIRST_MERGE (correctness) or finalize+colocate → HASH.
+ requireChild = parentRequire.autoRequireHash();
+ } else if (hasPartitionExprs(parentRequire)) {
+ // FE-only heuristic: finalize non-colocate with parent hash requirement
+ // → inherit parent's specific hash type.
+ requireChild = parentRequire.autoRequireHash();
+ } else {
+ // FE-only heuristic: finalize non-colocate without parent hash → skip
+ // LE (child Exchange already provides hash distribution).
+ requireChild = LocalExchangeTypeRequire.noRequire();
+ }
+ }
+
+ Pair<PlanNode, LocalExchangeType> enforceResult
+ = enforceRequire(translatorContext, children.get(0), 0, requireChild);
+ children = Lists.newArrayList(enforceResult.first);
+ return Pair.of(this, enforceResult.second);
+ }
+
+ /** BE base class required_data_distribution: serial child → PASSTHROUGH, else → NOOP. */
+ private LocalExchangeTypeRequire baseClassRequire(ConnectContext connectContext) {
+ return children.get(0).isSerialOperatorOnBe(connectContext)
+ ? LocalExchangeTypeRequire.requirePassthrough()
+ : LocalExchangeTypeRequire.noRequire();
+ }
+
+ @Override
+ protected List<Expr> getSemanticPartitionExprs() {
+ return aggInfo.getGroupingExprs();
+ }
+
+ @Override
+ protected List<Expr> getLocalExchangeDistributeExprs(int childIndex, boolean followedByShuffled) {
+ // Mirror BE AggSinkOperatorX::update_operator / StreamingAggOperatorX::update_operator:
+ // _partition_exprs = (distribute_expr_lists set && (followed_by_shuffled || has_distinct))
+ // ? distribute_expr_lists[0] : grouping_exprs
+ // The HASH LocalExchange must partition by _partition_exprs so a streaming partial preagg
+ // locally collapses same-key rows. Using child distribution (default) for a non-shuffled
+ // chain scatters same-group rows across N instances, leaving partial_preagg essentially a
+ // no-op and breaking row-arrival order at downstream merge-finalize (e.g. group_concat).
+ List<Expr> childDist = getChildDistributeExprList(childIndex);
+ boolean hasDistinct = aggInfo.getAggregateExprs().stream()
+ .map(FunctionCallExpr::getFnName)
+ .filter(name -> name != null)
+ .map(name -> name.getFunction())
+ .filter(name -> name != null)
+ .anyMatch(name -> name.startsWith("multi_distinct_"));
Review Comment:
There is no structural signal available at this layer, so the name match is intentional. Nereids rewrites `count/sum(distinct ...)` into dedicated `MultiDistinct*` functions constructed with `distinct=false` and a `multi_distinct_` name; by the time it reaches this legacy `FunctionCallExpr` / `AggregateInfo` layer, `isDistinct()` is already `false` and the function name is the only remaining signal. An `AggInfo.isMultiDistinct()` helper would still have to match the name internally — it couldn't be more "structural". (The BE `__isset.aggregate_functions` check isn't a distinct test either.) Added a comment in `fe9a3316be0` explaining this so the prefix match isn't mistaken for laziness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3413446628
##########
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##########
@@ -1130,8 +1130,10 @@ public boolean isProfileSafeStmt() {
// 1. CreateTableCommand(mainly for create as select).
// 2. LoadCommand.
// 3. InsertOverwriteTableCommand.
+ // 4. MergeIntoCommand (merge into ... using ...).
if ((plan instanceof Command) && !(plan instanceof LoadCommand)
- && !(plan instanceof CreateTableCommand) && !(plan instanceof InsertOverwriteTableCommand)) {
+ && !(plan instanceof CreateTableCommand) && !(plan instanceof InsertOverwriteTableCommand)
+ && !(plan instanceof org.apache.doris.nereids.trees.plans.commands.merge.MergeIntoCommand)) {
Review Comment:
Done in `19de375e12e` — imported `MergeIntoCommand` and use the simple name.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/MaterializationNode.java:
##########
@@ -227,7 +231,17 @@ public void setTopMaterializeNode(boolean topMaterializeNode) {
}
@Override
- public boolean isSerialOperator() {
+ public boolean isSerialNode() {
return true;
}
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ Pair<PlanNode, LocalExchangeType> enforceResult = enforceRequire(
+ translatorContext, children.get(0), 0, LocalExchangeTypeRequire.requirePassthrough());
+ children = new ArrayList<>();
Review Comment:
Done in `19de375e12e` — switched to `Lists.newArrayList()`.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -3314,7 +3349,7 @@ private boolean findOlapScanNodesByPassExchangeNode(PlanNode root) {
return false;
}
- private List<List<Expr>> getDistributeExprs(Plan ... children) {
+ private List<List<Expr>> getChildrenDistributeExprs(Plan ... children) {
Review Comment:
Renamed to `getDistributeExprs(Plan... plans)` in `19de375e12e`. I kept it taking the plans directly rather than a parent node, because the callers pass specific subsets (e.g. `hashJoin.left()`/`right()`, `child(0)`, or `children().toArray(...)`), so a single "parent" input wouldn't fit them all — the rename just drops the misleading "Children" from the name.
##########
regression-test/suites/nereids_p0/local_shuffle/test_local_shuffle_rqg_bugs.groovy:
##########
@@ -0,0 +1,1560 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/**
+ * Regression tests for bugs discovered by RQG testing on the local-exchange2 branch.
+ *
+ * These queries triggered "must set shared state" errors or incorrect results
+ * in RQG build 183992. Common conditions:
+ * - use_serial_exchange=true (makes ALL Exchanges serial, not just UNPARTITIONED)
+ * - enable_local_shuffle_planner=true (FE-planned local exchange)
+ * - parallel_pipeline_task_num > 1
+ *
+ * Error types reproduced:
+ * 1. must set shared state, in AGGREGATION_OPERATOR
+ * 2. must set shared state, in SORT_OPERATOR
+ * 3. incorrect results with GROUPING SETS + scalar subquery + window function
+ */
+suite("test_local_shuffle_rqg_bugs") {
+
+ // ============================================================
+ // Table setup — mirrors RQG table structure
+ // 10 buckets to match RQG (replication_num=1 for single-BE testing)
+ // ============================================================
+ sql "DROP TABLE IF EXISTS rqg_t1"
+ sql "DROP TABLE IF EXISTS rqg_t2"
+ sql "DROP TABLE IF EXISTS rqg_t3"
+ sql "DROP TABLE IF EXISTS rqg_t4"
+
+ sql """
+ CREATE TABLE rqg_t1 (
+ pk INT NOT NULL,
+ col_int_undef_signed INT,
+ col_int_undef_signed2 INT,
+ col_int_undef_signed_not_null INT NOT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ sql """
+ CREATE TABLE rqg_t2 (
+ pk INT NOT NULL,
+ col_int_undef_signed INT,
+ col_int_undef_signed2 INT,
+ col_bigint_undef_signed_not_null BIGINT NOT NULL,
+ col_decimal_38_10__undef_signed_not_null DECIMAL(38,10) NOT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ // Table for build 184181 GLOBAL_HASH_SHUFFLE bugs — needs varchar + bigint columns
+ sql """
+ CREATE TABLE rqg_t3 (
+ pk INT NOT NULL,
+ col_bigint_undef_signed BIGINT,
+ col_varchar_10__undef_signed VARCHAR(10),
+ col_varchar_64__undef_signed VARCHAR(64)
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ // Second table for FULL OUTER JOIN case (col_bigint_undef_signed_not_null)
+ sql """
+ CREATE TABLE rqg_t4 (
+ pk INT NOT NULL,
+ col_bigint_undef_signed_not_null BIGINT NOT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ sql """
+ INSERT INTO rqg_t3 VALUES
+ (0, -94, 'Abc', 'hello world'),
+ (1, 672609, 'Xyz', null),
+ (2, -3766684, 'Pqr', 'test string'),
+ (3, 5070261, 'abc', 'another row'),
+ (4, null, 'def', 'value four'),
+ (5, -86, 'XgpxlHBLEM', null),
+ (6, 21910, 'abc', 'they'),
+ (7, -63, 'zzzz', 'some text'),
+ (8, -8276281, 'AHlvNtoGLO', 'longer string here'),
+ (9, -101, 'mid', 'final row')
+ """
+
+ sql """
+ INSERT INTO rqg_t4 VALUES
+ (0, 0), (1, 1), (2, 2), (3, 3), (4, 4),
+ (5, 5), (6, 6), (7, 7), (8, 8), (9, 9),
+ (10, 2), (11, 2), (12, 2), (13, 3), (14, 4),
+ (15, 5), (16, 2), (17, 2), (18, 2), (19, 9)
+ """
+
+ // Insert enough rows to exercise multiple pipeline tasks
+ sql """
+ INSERT INTO rqg_t1 VALUES
+ (0, 0, 10, 0), (1, 1, 11, 1), (2, 2, 12, 2), (3, 3, 13, 3),
+ (4, 4, 14, 4), (5, 5, 15, 5), (6, 6, 16, 6), (7, 7, 17, 7),
+ (8, 8, 18, 8), (9, 9, 19, 9), (10, 0, 20, 10), (11, 1, 21, 11),
+ (12, 2, 22, 12), (13, 3, 23, 13), (14, 4, 24, 14), (15, 5, 25, 15),
+ (16, 6, 26, 16), (17, 7, 27, 17), (18, 8, 28, 18), (19, 9, 29, 19)
+ """
+
+ sql """
+ INSERT INTO rqg_t2 VALUES
+ (0, 0, 10, 100, 1.5), (1, 1, 11, 101, 2.5), (2, 2, 12, 102, 3.5),
+ (3, 3, 13, 103, 4.5), (4, 4, 14, 104, 5.5), (5, 5, 15, 105, 6.5),
+ (6, 6, 16, 106, 7.5), (7, 7, 17, 107, 8.5), (8, 8, 18, 108, 9.5),
+ (9, 9, 19, 109, 10.5), (10, 0, 20, 110, 11.5), (11, 1, 21, 111, 12.5),
+ (12, 2, 22, 112, 13.5), (13, 3, 23, 113, 14.5), (14, 4, 24, 114, 15.5),
+ (15, 5, 25, 115, 16.5), (16, 6, 26, 116, 17.5), (17, 7, 27, 117, 18.5),
+ (18, 8, 28, 118, 19.5), (19, 9, 29, 119, 20.5)
+ """
+
+ // Wait for data to be visible
+ Thread.sleep(5000)
Review Comment:
Done in `19de375e12e` — replaced `Thread.sleep(5000)` with a poll on the actual row counts (`rqg_t1`/`rqg_t2` == 20) with a timeout, so it waits on the real condition instead of a fixed delay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3413700850
##########
fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java:
##########
@@ -824,14 +828,97 @@ public void foreachDownInCurrentFragment(Consumer<PlanNode> visitor) {
});
}
- // Operators need to be executed serially. (e.g. finalized agg without key)
- public boolean isSerialOperator() {
+ /**
+ * Node-level "is this operator inherently serial" property — answers without looking
+ * at the fragment. Default false; subclasses override (e.g. finalized agg without key,
+ * UNPARTITIONED ExchangeNode with merge sort).
+ *
+ * Use ONLY in framework-internal places where we are already iterating within a
+ * fragment whose serial-source mode is fixed: {@link #shouldResetSerialFlagForChild}
+ * inputs, {@link #createLocalExchange} heavy-op gate, and child.isSerialNode() checks
+ * embedded inside an enforceRequire path. Do NOT use it when computing a
+ * {@link LocalExchangeNode.LocalExchangeTypeRequire} on a child — call
+ * {@link #isSerialOperatorOnBe} instead.
+ */
+ public boolean isSerialNode() {
Review Comment:
> @morrySnow just curious, why don't you impl it before translation and after post process in Nereids?
(Answering the review-summary question here, anchored to where it actually runs.)
It needs two inputs that only exist **after translation + distribute**, not on the Nereids physical plan:
1. **PlanNode-level info** — `AddLocalExchange` walks the translated `PlanNode` / `PlanFragment` tree via `PlanNode.enforceAndDeriveLocalExchange` (`isSerialNode()` / `isSerialOperatorOnBe()`, the distribute-expr lists serialized to BE thrift, the fragment sink type, …). Those are the legacy `PlanNode`s the translator produces — they don't exist on the Nereids physical operators.
2. **Instance counts** — whether to insert an LE (and the fan-out width) depends on the per-BE instance count: `addLocalExchange` skips fragments where `maxPerBeInstances <= 1` (mirroring BE's `_num_instances <= 1` no-op), and the PASSTHROUGH fan-out targets `_num_instances`. Instance assignment is done by `DistributePlanner`, which runs **after** translation.
So it runs right here — after `splitFragments()` + `doDistribute()` — the first point where both the BE-facing PlanNode tree and the instance assignment exist. Doing it pre-translation in Nereids would mean re-deriving fragment structure + instance counts that haven't been computed yet.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -740,6 +742,19 @@ protected void distribute(PhysicalPlan physicalPlan, ExplainLevel explainLevel)
splitFragments(physicalPlan);
doDistribute(canUseNereidsDistributePlanner, explainLevel);
+
+ addLocalExchangeAfterDistribute();
Review Comment:
> @morrySnow just curious, why don't you impl it before translation and after post process in Nereids?
(Answering the review-summary question here, anchored to where it actually runs.)
It needs two inputs that only exist **after translation + distribute**, not on the Nereids physical plan:
1. **PlanNode-level info** — `AddLocalExchange` walks the translated `PlanNode` / `PlanFragment` tree via `PlanNode.enforceAndDeriveLocalExchange` (`isSerialNode()` / `isSerialOperatorOnBe()`, the distribute-expr lists serialized to BE thrift, the fragment sink type, …). Those are the legacy `PlanNode`s the translator produces — they don't exist on the Nereids physical operators.
2. **Instance counts** — whether to insert an LE (and the fan-out width) depends on the per-BE instance count: `addLocalExchange` skips fragments where `maxPerBeInstances <= 1` (mirroring BE's `_num_instances <= 1` no-op), and the PASSTHROUGH fan-out targets `_num_instances`. Instance assignment is done by `DistributePlanner`, which runs **after** translation.
So it runs right here — after `splitFragments()` + `doDistribute()` — the first point where both the BE-facing PlanNode tree and the instance assignment exist. Doing it pre-translation in Nereids would mean re-deriving fragment structure + instance counts that haven't been computed yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4708802951
<details>
<summary>TPC-H: <b>Total hot run time: 28977 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 832ba5f0bf8ab5846bb316e94f1099ae58fe466f, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17651 4086 4052 4052
q2 q3 10758 1387 819 819
q4 4689 481 340 340
q5 7533 884 597 597
q6 188 180 139 139
q7 781 848 630 630
q8 9596 1638 1716 1638
q9 6531 4544 4542 4542
q10 6858 1830 1517 1517
q11 439 276 245 245
q12 657 420 282 282
q13 18194 3347 2786 2786
q14 268 261 245 245
q15 q16 819 774 712 712
q17 931 945 985 945
q18 6939 5658 5693 5658
q19 1473 1228 1010 1010
q20 541 408 264 264
q21 5760 2697 2254 2254
q22 443 350 302 302
Total cold run time: 101049 ms
Total hot run time: 28977 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4435 4392 4325 4325
q2 q3 4501 4982 4343 4343
q4 2076 2210 1389 1389
q5 4497 4382 4381 4381
q6 227 174 128 128
q7 1751 1952 1755 1755
q8 2561 2188 2140 2140
q9 8293 8180 8170 8170
q10 4808 4731 4301 4301
q11 587 434 382 382
q12 747 838 616 616
q13 3354 3625 3051 3051
q14 310 310 277 277
q15 q16 743 752 632 632
q17 1364 1321 1348 1321
q18 8195 7196 7104 7104
q19 1214 1108 1142 1108
q20 2246 2273 1959 1959
q21 5275 4602 4453 4453
q22 527 441 455 441
Total cold run time: 57711 ms
Total hot run time: 52276 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3412773289
##########
fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java:
##########
@@ -824,14 +828,97 @@ public void foreachDownInCurrentFragment(Consumer<PlanNode> visitor) {
});
}
- // Operators need to be executed serially. (e.g. finalized agg without key)
- public boolean isSerialOperator() {
+ /**
+ * Node-level "is this operator inherently serial" property — answers without looking
+ * at the fragment. Default false; subclasses override (e.g. finalized agg without key,
+ * UNPARTITIONED ExchangeNode with merge sort).
+ *
+ * Use ONLY in framework-internal places where we are already iterating within a
+ * fragment whose serial-source mode is fixed: {@link #shouldResetSerialFlagForChild}
+ * inputs, {@link #createLocalExchange} heavy-op gate, and child.isSerialNode() checks
+ * embedded inside an enforceRequire path. Do NOT use it when computing a
+ * {@link LocalExchangeNode.LocalExchangeTypeRequire} on a child — call
+ * {@link #isSerialOperatorOnBe} instead.
+ */
+ public boolean isSerialNode() {
Review Comment:
it should be protected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3413654832
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -740,6 +742,19 @@ protected void distribute(PhysicalPlan physicalPlan, ExplainLevel explainLevel)
splitFragments(physicalPlan);
doDistribute(canUseNereidsDistributePlanner, explainLevel);
+
+ addLocalExchangeAfterDistribute();
Review Comment:
> @morrySnow just curious, why don't you impl it before translation and after post process in Nereids?
(Answering the review-summary question here, anchored to where it actually runs.)
It needs two inputs that only exist **after translation + distribute**, not on the Nereids physical plan:
1. **PlanNode-level info** — `AddLocalExchange` walks the translated `PlanNode` / `PlanFragment` tree via `PlanNode.enforceAndDeriveLocalExchange` (`isSerialNode()` / `isSerialOperatorOnBe()`, the distribute-expr lists serialized to BE thrift, the fragment sink type, …). Those are the legacy `PlanNode`s the translator produces — they don't exist on the Nereids physical operators.
2. **Instance counts** — whether to insert an LE (and the fan-out width) depends on the per-BE instance count: `addLocalExchange` skips fragments where `maxPerBeInstances <= 1` (mirroring BE's `_num_instances <= 1` no-op), and the PASSTHROUGH fan-out targets `_num_instances`. Instance assignment is done by `DistributePlanner`, which runs **after** translation.
So it runs right here — after `splitFragments()` + `doDistribute()` — the first point where both the BE-facing PlanNode tree and the instance assignment exist. Doing it pre-translation in Nereids would mean re-deriving fragment structure + instance counts that haven't been computed yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3433158030
##########
be/src/exec/pipeline/dependency.h:
##########
@@ -832,50 +832,44 @@ struct SetSharedState : public BasicSharedState {
Status hash_table_init();
};
-enum class ExchangeType : uint8_t {
- NOOP = 0,
- // Shuffle data by Crc32CHashPartitioner
- HASH_SHUFFLE = 1,
- // Round-robin passthrough data blocks.
- PASSTHROUGH = 2,
- // Shuffle data by Crc32HashPartitioner<ShuffleChannelIds> (e.g. same as storage engine).
- BUCKET_HASH_SHUFFLE = 3,
- // Passthrough data blocks to all channels.
- BROADCAST = 4,
- // Passthrough data to channels evenly in an adaptive way.
- ADAPTIVE_PASSTHROUGH = 5,
- // Send all data to the first channel.
- PASS_TO_ONE = 6,
-};
+inline bool is_shuffled_exchange(TLocalPartitionType::type idx) {
Review Comment:
Good catch — fixed in `0436cc4cb57`. Added `#include <gen_cpp/Partitions_types.h>` to `dependency.h` (the header that defines `TLocalPartitionType`, matching e.g. `vdata_stream_sender.h`), so it no longer depends on include order in TUs like `record_batch_queue.cpp` / `group_commit_mgr.cpp`.
##########
fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java:
##########
@@ -563,6 +567,10 @@ public void update(Map<String, String> summaryInfo) {
// This method is used to display the final data status when the overall query ends.
// This can avoid recalculating some strings and so on every time during the update process.
public void queryFinished() {
+ // Mark profile collection as complete. This is written last, after all BE fragment
+ // profiles have been merged (called post-waitForFragmentsDone). Pollers can wait for
+ // this field to avoid reading a partial profile.
Review Comment:
Fixed in `0436cc4cb57` — moved the `IS_PROFILE_COLLECTION_COMPLETED` marker to the **end** of `queryFinished()`, after the optional `SPLITS_ASSIGNMENT_WEIGHT` block, so a poller can no longer observe `completed=true` on a partial profile. (The comment claimed "written last" but it was actually written first — now it genuinely is last.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4681012159
<details>
<summary>TPC-H: <b>Total hot run time: 29402 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f33ce47e1bdfae26100cf19b96a7eab06238d13, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17704 4038 4060 4038
q2 q3 10988 1496 806 806
q4 4823 477 350 350
q5 8626 880 598 598
q6 356 170 134 134
q7 928 830 636 636
q8 10949 1740 1633 1633
q9 7280 4582 4503 4503
q10 6823 1829 1504 1504
q11 457 275 248 248
q12 646 423 282 282
q13 18194 3405 2803 2803
q14 260 263 242 242
q15 q16 823 779 708 708
q17 981 968 932 932
q18 6744 5747 5651 5651
q19 1147 1238 1172 1172
q20 527 411 272 272
q21 5959 2836 2570 2570
q22 434 380 320 320
Total cold run time: 104649 ms
Total hot run time: 29402 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4918 4738 5031 4738
q2 q3 4987 5292 4679 4679
q4 2117 2190 1416 1416
q5 4981 4742 4725 4725
q6 239 195 132 132
q7 1926 1735 1531 1531
q8 2267 2031 1971 1971
q9 7623 7582 7604 7582
q10 4744 4694 4216 4216
q11 535 390 352 352
q12 734 737 525 525
q13 3024 3415 2764 2764
q14 280 280 255 255
q15 q16 680 695 618 618
q17 1295 1264 1282 1264
q18 7439 6953 6693 6693
q19 1119 1087 1106 1087
q20 2237 2256 1985 1985
q21 5275 4637 4467 4467
q22 520 446 426 426
Total cold run time: 56940 ms
Total hot run time: 51426 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4681249141
<details>
<summary>TPC-H: <b>Total hot run time: 29291 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 784476ead971eeebeb0b637f6da4a434e380ce36, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17871 4099 4070 4070
q2 q3 10706 1399 816 816
q4 4696 472 342 342
q5 7542 868 587 587
q6 187 170 133 133
q7 771 841 630 630
q8 9335 1554 1691 1554
q9 5604 4539 4522 4522
q10 6791 1812 1559 1559
q11 456 266 252 252
q12 626 426 292 292
q13 18184 3425 2825 2825
q14 273 260 238 238
q15 q16 824 777 713 713
q17 1008 888 941 888
q18 6887 5756 5552 5552
q19 2282 1315 1070 1070
q20 495 409 258 258
q21 6269 2888 2670 2670
q22 466 373 320 320
Total cold run time: 101273 ms
Total hot run time: 29291 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5027 4733 4818 4733
q2 q3 4876 5381 4647 4647
q4 2095 2189 1407 1407
q5 4975 4733 4606 4606
q6 233 189 144 144
q7 1908 1781 1531 1531
q8 2404 2101 2140 2101
q9 8021 7689 7622 7622
q10 4731 4679 4216 4216
q11 520 385 348 348
q12 723 740 526 526
q13 3003 3354 2835 2835
q14 275 277 257 257
q15 q16 674 691 608 608
q17 1289 1252 1246 1246
q18 7278 6706 6786 6706
q19 1087 1075 1110 1075
q20 2248 2202 1935 1935
q21 5284 4551 4459 4459
q22 508 439 412 412
Total cold run time: 57159 ms
Total hot run time: 51414 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3413420330
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -690,6 +716,188 @@ Status PipelineFragmentContext::_build_pipelines(ObjectPool* pool, const Descrip
return Status::OK();
}
+Status PipelineFragmentContext::_create_deferred_local_exchangers() {
+ for (auto& info : _deferred_exchangers) {
+ // DANGER ZONE — do not "fix" this line without reading the history.
+ //
+ // sender_count seeds Exchanger::_running_sink_operators, which the source side
+ // waits to reach 0 via sub_running_sink_operators on each sink LocalState close.
+ // The correct value is THIS pipeline-instance's sink task count, which is exactly
+ // info.upstream_pipe->num_tasks() — one PipelineTask per task, one close per task.
+ //
+ // Tempting wrong fix #1: `std::max(num_tasks, _num_instances)` to mirror the
+ // BE-planned path in _add_local_exchange_impl (~line 1023). THIS BREAKS the
+ // common FE-planned shape of `serial scan → LE(PT) → ...`: upstream_pipe
+ // genuinely has num_tasks=1, only 1 close arrives, but seed becomes
+ // _num_instances so _running_sink_operators never reaches 0 — downstream
+ // sources hang on SHUFFLE_DATA_DEPENDENCY (e.g. MTMV refresh from
+ // mtmv_up_down_job_p0/load.groovy stays at Status=RUNNING; build 949402
+ // regressed exactly this way). BE-planned mode uses max() because its
+ // `cur_pipe` is the source-side pipeline (always raised to _num_instances by
+ // add_pipeline) — not analogous to our `upstream_pipe` here, which is the
+ // sink-side pipeline that may legitimately stay at 1 for serial sources.
+ //
+ // Tempting wrong fix #2: multiply by _num_instances on the theory shared_state
+ // is shared across all instances. Same hang — each fragment-instance
+ // PipelineFragmentContext has its OWN _op_id_to_shared_state map, so the
+ // exchanger is per-instance, not per-BE. num_tasks() is already the right
+ // close-count for one instance.
+ //
+ // If a hang shows up with `_running_sink_operators < 0`, the bug is upstream:
+ // _propagate_local_exchange_num_tasks left num_tasks too low (or too high) for
+ // this fragment shape. Fix THAT pass, not this seed value.
+ const int sender_count = info.upstream_pipe->num_tasks();
+ switch (info.partition_type) {
+ case TLocalPartitionType::LOCAL_EXECUTION_HASH_SHUFFLE:
+ case TLocalPartitionType::GLOBAL_EXECUTION_HASH_SHUFFLE:
+ info.shared_state->exchanger = ShuffleExchanger::create_unique(
+ sender_count, _num_instances, info.num_partitions, info.free_blocks_limit,
+ info.partition_type);
+ break;
+ case TLocalPartitionType::BUCKET_HASH_SHUFFLE:
+ info.shared_state->exchanger = BucketShuffleExchanger::create_unique(
+ sender_count, _num_instances, info.num_partitions, info.free_blocks_limit);
+ break;
+ case TLocalPartitionType::PASSTHROUGH:
+ info.shared_state->exchanger = PassthroughExchanger::create_unique(
+ sender_count, _num_instances, info.free_blocks_limit);
+ break;
+ case TLocalPartitionType::BROADCAST:
+ info.shared_state->exchanger = BroadcastExchanger::create_unique(
+ sender_count, _num_instances, info.free_blocks_limit);
+ break;
+ case TLocalPartitionType::PASS_TO_ONE:
+ if (_runtime_state->enable_share_hash_table_for_broadcast_join()) {
+ info.shared_state->exchanger = PassToOneExchanger::create_unique(
+ sender_count, _num_instances, info.free_blocks_limit);
+ } else {
+ info.shared_state->exchanger = BroadcastExchanger::create_unique(
+ sender_count, _num_instances, info.free_blocks_limit);
+ }
+ break;
+ case TLocalPartitionType::ADAPTIVE_PASSTHROUGH:
+ info.shared_state->exchanger = AdaptivePassthroughExchanger::create_unique(
+ sender_count, _num_instances, info.free_blocks_limit);
+ break;
+ case TLocalPartitionType::NOOP:
+ case TLocalPartitionType::LOCAL_MERGE_SORT:
Review Comment:
Not a regression. `LOCAL_MERGE_SORT` as a `TLocalPartitionType` is only emitted by the legacy BE-side local-exchange planner. The local-merge-sort capability itself is driven by `SortNode.use_local_merge` → `LocalMergeSortSourceOperatorX` (`pipeline_fragment_context.cpp:1858`), which FE sets and works the same in both planner modes. The FE-planned local-exchange path simply never needs to emit this partition type, so the factory treats it (and NOOP) as a protocol violation. If you have a sort plan where FE-planned mode would actually need a `LOCAL_MERGE_SORT` *local exchange*, point me at it and I'll dig in.
##########
fe/fe-core/src/main/java/org/apache/doris/qe/runtime/ThriftPlansBuilder.java:
##########
@@ -277,7 +277,8 @@ private static Multiset<DistributedPlanWorker> computeInstanceNumPerWorker(
return workerCounter;
}
- private static Map<Integer, Integer> computeExchangeSenderNum(PipelineDistributedPlan distributedPlan) {
+ private static Map<Integer, Integer> computeExchangeSenderNum(
+ PipelineDistributedPlan distributedPlan) {
Review Comment:
This is just an incidental line-length reformat of the method signature (no behavior change). Happy to revert it to keep the diff minimal if you'd prefer — it crept in from formatting.
##########
be/src/exec/operator/partitioned_hash_join_probe_operator.h:
##########
@@ -228,16 +228,8 @@ class PartitionedHashJoinProbeOperatorX final
Status pull(doris::RuntimeState* state, Block* output_block, bool* eos) const override;
bool need_more_input_data(RuntimeState* state) const override;
- DataDistribution required_data_distribution(RuntimeState* /*state*/) const override {
- if (_join_op == TJoinOp::NULL_AWARE_LEFT_ANTI_JOIN) {
- return {ExchangeType::NOOP};
- }
- return (_join_distribution == TJoinDistributionType::BUCKET_SHUFFLE ||
- _join_distribution == TJoinDistributionType::COLOCATE
- ? DataDistribution(ExchangeType::BUCKET_HASH_SHUFFLE,
- _distribution_partition_exprs)
- : DataDistribution(ExchangeType::HASH_SHUFFLE,
- _distribution_partition_exprs));
+ DataDistribution required_data_distribution(RuntimeState* state) const override {
+ return _inner_probe_operator->required_data_distribution(state);
Review Comment:
Not a perf optimization — it's a consistency refactor. `PartitionedHashJoinProbeOperatorX` now delegates `required_data_distribution()` to its inner `_inner_probe_operator` instead of duplicating the join-distribution logic, so the spill path reports the same required distribution as the non-spill `HashJoinProbeOperatorX` (single source of truth). It also drops the now-unused `_join_distribution` member.
##########
be/src/udf/python/boost_process_compat.h:
##########
@@ -0,0 +1,28 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
Review Comment:
已移除（commit `1d48e9ae984`）：`boost_process_compat.h` 以及 `build.sh` / `python_server.cpp` / `python_udf_runtime.{h,cpp}` 里跟 boost/process 相关的改动都还原到 master 了，这些 mac 编译相关的改动不进这个 PR。
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -290,6 +290,32 @@ Status PipelineFragmentContext::_build_and_prepare_full_pipeline(ThreadPool* thr
RETURN_IF_ERROR(_build_pipelines(_runtime_state->obj_pool(), *_query_ctx->desc_tbl,
&_root_op, root_pipeline));
+ // Propagate _num_instances from LOCAL_EXCHANGE pipelines to ancestor pipelines
+ // that inherited reduced num_tasks from a serial operator.
+ _propagate_local_exchange_num_tasks();
+
+ // Create deferred local exchangers now that all pipelines have final num_tasks.
+ RETURN_IF_ERROR(_create_deferred_local_exchangers());
Review Comment:
是的，这两块不是为了灰度/回滚——是 FE 接管 local exchange 的**当前折中**，不是一步到位。
要让 FE 把"每个算子的并发"也规划好直接下发，本质上需要 FE 自己切 pipeline、自己控制每条 pipeline 的并发（num_tasks 是 pipeline 粒度的，只在 pipeline 边界即 local exchange / pipeline breaker 处变化），BE 就不再规划 pipeline 了。那等于把 BE 的整套 pipeline 构建（`_build_pipeline`、pipeline-breaker 判定、num_tasks 推导、dependency 接线，还要兼顾 spill / runtime filter / shared state）搬到 FE，并改 thrift 契约携带完整 pipeline 结构——战线太长、风险太大。这套 FE 近似等价实现已经做了大半年、近 1 万行代码，没法一把全做。
所以现状是：FE 规划 LE 的**位置/类型/分布**（决定 plan 正确性，过去正是 BE/FE 分歧 bug 的根源），`enable_local_shuffle_planner=true` 时 BE 的 `_plan_local_exchange` 整个跳过；但每条 pipeline 的 num_tasks 仍由 BE 的 `_propagate_local_exchange_num_tasks` 拓扑推导，`_create_deferred_local_exchangers` 因为 exchanger 的 `num_partitions` 要等 num_tasks 定了才能建、所以延后创建。终态是 FE 连 num_tasks 也算好下发、BE 的 propagate 退化成 assert（只校验）——那是后续单独的大改。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4708259972
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/14ff5b48070e26ba3f1518c301fce97c36998331_14ff5b48070e26ba3f1518c301fce97c36998331_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/14ff5b48070e26ba3f1518c301fce97c36998331_14ff5b48070e26ba3f1518c301fce97c36998331_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.32% (1889/2443) |
| Line Coverage | 64.38% (33943/52725) |
| Region Coverage | 64.77% (17455/26948) |
| Branch Coverage | 53.92% (9337/17316) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3414782162
##########
fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java:
##########
@@ -939,4 +1031,263 @@ private String mergeIcebergAccessPathsWithId(
}
return StringUtils.join(mergeDisplayAccessPaths, ", ");
}
+
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ ArrayList<PlanNode> newChildren = Lists.newArrayList();
+ for (int i = 0; i < children.size(); i++) {
+ Pair<PlanNode, LocalExchangeType> childOutput
+ = enforceRequire(translatorContext, children.get(i), i, LocalExchangeTypeRequire.noRequire());
+ newChildren.add(childOutput.first);
+ }
+ this.children = newChildren;
+ return Pair.of(this, LocalExchangeType.NOOP);
+ }
+
+ /**
+ * Unified framework method: propagate serial flag → recurse child → satisfy check → Layer 1 skip → insert LE.
+ * Replaces the old enforceChild/enforceChildExchange/forceEnforceChildExchange trio.
+ *
+ * <h3>Data flow</h3>
+ * <ul>
+ * <li><b>serial-ancestor flag</b> ({@link PlanTranslatorContext#hasSerialAncestorInPipeline})
+ * — flows root → leaf during traversal. Mirrors BE's
+ * {@code any_of(operators[idx..end], is_serial_operator)} check used by
+ * {@code _add_local_exchange} to skip LE insertion when an ancestor in the same
+ * pipeline is already serial. Reset at pipeline boundaries via
+ * {@link #shouldResetSerialFlagForChild}.</li>
+ * <li><b>shuffle-for-correctness flag</b>
+ * ({@link PlanTranslatorContext#hasShuffleForCorrectnessAncestor}) — also flows
+ * root → leaf. Mirrors BE's {@code _followed_by_shuffled_operator}: tells a
+ * child whether some downstream operator depends on hash distribution for
+ * correctness, so {@code SetOperationNode} can pre-shuffle union branches.</li>
+ * <li><b>return value</b> {@code Pair<PlanNode, LocalExchangeType>} — first is the
+ * (possibly LE-wrapped) child; second is the actual output distribution as
+ * observed by the parent. Caller's {@code require.satisfy(output)} decides
+ * whether more LE is needed.</li>
+ * <li><b>parent.require</b> describes the constraint on the child output —
+ * computed inside the parent's {@code enforceAndDeriveLocalExchange} per child.</li>
+ * </ul>
+ *
+ * <h3>Invariants</h3>
+ * <ul>
+ * <li>Where a serial → non-serial transition needs redistribution, framework step 3 inserts
+ * the LE (e.g. a serial source fanned out via PASSTHROUGH). This is not a hard invariant:
+ * a serial child feeding a parent that requires PASSTHROUGH / noRequire (TableFunction,
+ * NLJ, Agg) is already correct and needs no LE, so it is intentionally not validated by a
+ * post-pass.</li>
+ * <li>{@code LocalExchangeNode} itself is always non-serial — setting it serial
+ * would defeat its purpose of fanning a 1-task pipeline back to N tasks.</li>
+ * <li>For pipeline-breaking parents ({@code shouldResetSerialFlagForChild=true}),
+ * the child starts a fresh pipeline so {@code hasSerialAncestor} is reset; the
+ * node's own {@code isSerialNode()} still composes in for the child's view.</li>
+ * <li>{@code RequireHash} accepts any hash flavour; {@code RequireSpecific} demands
+ * an exact match (with the one PASSTHROUGH/ADAPTIVE_PASSTHROUGH compatibility).
+ * Pick the looser one whenever correctness allows — see
+ * {@link LocalExchangeNode.LocalExchangeTypeRequire}.</li>
+ * </ul>
+ *
+ * <h3>Layers</h3>
+ * Layer 1 (shouldSkipLE): mirrors BE's need_to_local_exchange — skip when this node or
+ * an ancestor in the same pipeline is serial (operators[idx..end] has serial → skip).
+ * Layer 2 (require/output): each Node declares require and output in enforceAndDeriveLocalExchange.
+ */
+ protected Pair<PlanNode, LocalExchangeType> enforceRequire(
+ PlanTranslatorContext translatorContext, PlanNode child, int childIndex,
+ LocalExchangeTypeRequire require) {
+ // 1. Propagate serial-ancestor flag to child.
+ // For pipeline-splitting operators (shouldReset=true, e.g. non-streaming AGG):
+ // Drop inherited serial flag from parent (parent is in a different pipeline),
+ // but keep this node's own serial status (child is in the same pipeline as this
+ // node's sink, e.g. Exchange is in AGG_Sink pipeline).
+ // For non-splitting operators (shouldReset=false, e.g. streaming AGG):
+ // Inherit parent's serial flag + this node's own.
+ boolean inheritedSerial = shouldResetSerialFlagForChild(childIndex)
+ ? false : translatorContext.hasSerialAncestorInPipeline(this);
+ // Use isSerialOperatorOnBe (= isSerialNode && fragment.useSerialSource) instead of the
+ // raw isSerialNode(). BE's OperatorBase reads the Thrift `is_serial_operator` flag —
+ // which is what FE writes via isSerialOperatorOnBe — so when the fragment is not in
+ // serial-source mode, BE treats this operator as non-serial regardless of isSerialNode.
+ // Using isSerialNode here would set the child's serial-ancestor flag wider than BE's
+ // view and over-skip required LocalExchanges downstream.
+ boolean childHasSerialAncestor = inheritedSerial
+ || isSerialOperatorOnBe(translatorContext.getConnectContext());
+ translatorContext.setHasSerialAncestorInPipeline(child, childHasSerialAncestor);
+
+ // 1b. Propagate shuffle-for-correctness-ancestor flag to child.
+ // Mirrors BE's _followed_by_shuffled_operator: a downstream operator needs hash
+ // distribution for correctness, and the chain to here goes through HASH or NOOP
+ // requirements (so the dependency is preserved).
+ // propagate = ((inheritedShuffled || self.requiresShuffleForCorrectness)
+ // && require is hash)
+ // || (inheritedShuffled && require is noop/passthrough)
+ boolean inheritedShuffled = translatorContext.hasShuffleForCorrectnessAncestor(this);
+ boolean selfOrInheritedShuffled = inheritedShuffled || requiresShuffleForCorrectness();
+ boolean requireIsHash = require.preferType().isHashShuffle();
+ boolean requireIsNoop = require.preferType() == LocalExchangeNode.LocalExchangeType.NOOP;
+ boolean childShuffledAncestor = (selfOrInheritedShuffled && requireIsHash)
+ || (inheritedShuffled && requireIsNoop);
+ translatorContext.setHasShuffleForCorrectnessAncestor(child, childShuffledAncestor);
+
+ // 2. Recurse child (Layer 2: child declares its own require/output)
+ Pair<PlanNode, LocalExchangeType> childOutput =
+ child.enforceAndDeriveLocalExchange(translatorContext, this, require);
+
+ // Steps 2.5 and 3 both react to a serial child but address different concerns:
+ // - Step 2.5 rewrites the OUTPUT-side view (what we tell satisfy/parent about
+ // the child's actual distribution). A serial pipeline runs with 1 task so
+ // its distribution claim is meaningless — flatten to NOOP so the satisfy
+ // check below doesn't get fooled by a stale "I output BUCKET_HASH" claim.
+ // - Step 3 rewrites the REQUIRE-side decision (what we want from the child).
+ // If we previously asked for nothing (noRequire) but the child turns out
+ // to be serial and we're not, upgrade to requirePassthrough so an LE is
+ // inserted to restore parallelism.
+
+ // 2.5. Serial child override (output side): if child is serial on BE, force its
+ // reported output to NOOP. Distribution is irrelevant when the child runs
+ // with 1 task; downstream parallelism is restored either by step 3 (LE
+ // insertion) or skipped entirely by step 4b (we're also serial).
+ if (childOutput.first.isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ childOutput = Pair.of(childOutput.first, LocalExchangeType.NOOP);
+ }
+
+ // 3. Framework-level serial child check (require side, mirrors BE base class
+ // required_data_distribution): if child will be serial on BE but this node is
+ // not serial, the pipeline has a 1-task serial child feeding an N-task non-serial
+ // parent. Without LE, pipeline splits (AGG/JOIN) create paired pipelines with
+ // mismatched num_tasks → crash. Upgrade noRequire to requirePassthrough so an
+ // LE is inserted below to restore parallelism.
+ if (require instanceof LocalExchangeNode.NoRequire
+ && childOutput.first.isSerialOperatorOnBe(translatorContext.getConnectContext())
+ && !isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ require = LocalExchangeTypeRequire.requirePassthrough();
+ }
+
+ // 4. Satisfy check: child output meets requirement → done
+ if (require.satisfy(childOutput.second)) {
+ return childOutput;
+ }
+
+ // 4. Layer 1: skip LE when serial operator or ancestor in same pipeline
+ // Equivalent to BE's need_to_local_exchange: any_of(operators[idx..end], is_serial) → skip.
+ // Use isSerialOperatorOnBe (not isSerialNode) because BE's Pipeline::need_to_local_exchange
+ // checks op->is_serial_operator() which reads the Thrift flag set from isSerialOperatorOnBe;
+ // when fragment.useSerialSource is false, BE treats this node as non-serial.
+ if (translatorContext.hasSerialAncestorInPipeline(this)
+ || isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ return childOutput;
+ }
+
+ // 5. Resolve exchange type and create LE node
+ LocalExchangeType preferType = AddLocalExchange.resolveExchangeType(require);
+ List<Expr> distributeExprs = getLocalExchangeDistributeExprs(childIndex, selfOrInheritedShuffled);
+ PlanNode leNode = createLocalExchange(translatorContext, childOutput.first, preferType, distributeExprs);
+ return Pair.of(leNode, preferType);
+ }
+
+ /**
+ * Create a LocalExchangeNode wrapping child with the given exchange type.
+ * No child-type skip — matches BE's _add_local_exchange which inserts LE for any child
+ * type without checking instanceof.
+ *
+ * Handles heavy-ops bottleneck avoidance (mirrors BE pipeline_fragment_context.cpp):
+ * when upstream has 1 task (serial source) and exchange is heavy (hash/bucket/adaptive),
+ * insert a PASSTHROUGH fan-out first to avoid single-task bottleneck on the heavy
+ * exchange sink. Only applies to local-shuffle (pooling scan) fragments.
+ */
+ protected PlanNode createLocalExchange(PlanTranslatorContext translatorContext,
+ PlanNode child, LocalExchangeType exchangeType, List<Expr> distributeExprs) {
+ if (fragment != null && fragment.useSerialSource(translatorContext.getConnectContext())
+ && exchangeType.isHeavyOperation() && child.isSerialNode()) {
+ PlanNode ptNode = new LocalExchangeNode(translatorContext.nextPlanNodeId(),
+ child, LocalExchangeType.PASSTHROUGH, null);
+ return new LocalExchangeNode(translatorContext.nextPlanNodeId(), ptNode,
+ exchangeType, distributeExprs);
+ }
+ return new LocalExchangeNode(translatorContext.nextPlanNodeId(), child,
+ exchangeType, distributeExprs);
+ }
+
+ /**
+ * Whether the child at {@code childIndex} starts a new pipeline context, causing
+ * its serial-ancestor flag to be reset to {@code false} rather than inherited from this node.
+ * Override to return {@code true} for pipeline-splitting nodes (LocalExchangeNode) and nodes
+ * whose children run in an independent pipeline segment (SortNode above analytic, etc.).
+ */
+ protected boolean shouldResetSerialFlagForChild(int childIndex) {
Review Comment:
so this must follow the impl in BE?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3412722149
##########
fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java:
##########
@@ -939,4 +1026,261 @@ private String mergeIcebergAccessPathsWithId(
}
return StringUtils.join(mergeDisplayAccessPaths, ", ");
}
+
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ ArrayList<PlanNode> newChildren = Lists.newArrayList();
+ for (int i = 0; i < children.size(); i++) {
+ Pair<PlanNode, LocalExchangeType> childOutput
+ = enforceRequire(translatorContext, children.get(i), i, LocalExchangeTypeRequire.noRequire());
+ newChildren.add(childOutput.first);
+ }
+ this.children = newChildren;
+ return Pair.of(this, LocalExchangeType.NOOP);
+ }
+
+ /**
+ * Unified framework method: propagate serial flag → recurse child → satisfy check → Layer 1 skip → insert LE.
+ * Replaces the old enforceChild/enforceChildExchange/forceEnforceChildExchange trio.
+ *
+ * <h3>Data flow</h3>
+ * <ul>
+ * <li><b>serial-ancestor flag</b> ({@link PlanTranslatorContext#hasSerialAncestorInPipeline})
+ * — flows root → leaf during traversal. Mirrors BE's
+ * {@code any_of(operators[idx..end], is_serial_operator)} check used by
+ * {@code _add_local_exchange} to skip LE insertion when an ancestor in the same
+ * pipeline is already serial. Reset at pipeline boundaries via
+ * {@link #shouldResetSerialFlagForChild}.</li>
+ * <li><b>shuffle-for-correctness flag</b>
+ * ({@link PlanTranslatorContext#hasShuffleForCorrectnessAncestor}) — also flows
+ * root → leaf. Mirrors BE's {@code _followed_by_shuffled_operator}: tells a
+ * child whether some downstream operator depends on hash distribution for
+ * correctness, so {@code SetOperationNode} can pre-shuffle union branches.</li>
+ * <li><b>return value</b> {@code Pair<PlanNode, LocalExchangeType>} — first is the
+ * (possibly LE-wrapped) child; second is the actual output distribution as
+ * observed by the parent. Caller's {@code require.satisfy(output)} decides
+ * whether more LE is needed.</li>
+ * <li><b>parent.require</b> describes the constraint on the child output —
+ * computed inside the parent's {@code enforceAndDeriveLocalExchange} per child.</li>
+ * </ul>
+ *
+ * <h3>Invariants</h3>
+ * <ul>
+ * <li>Every serial → non-serial transition has an LE somewhere between them
+ * (enforced by framework step 3 below, validated by
+ * {@link AddLocalExchange#validateNoSerialWithoutLocalExchange}).</li>
Review Comment:
[P3] Please update this invariant after removing `validateNoSerialWithoutLocalExchange`. Commit `3630071a` deleted that validation because some serial-to-non-serial transitions legitimately have no LocalExchange, but this bullet still says every transition has one and links to the now-nonexistent method. Please rewrite it to describe the narrower condition enforced by step 3, or remove the stale validation claim.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4706106002
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4706461907
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/1d7cf6dbd6095ee1fb5860efe258b0ce99242ae6_1d7cf6dbd6095ee1fb5860efe258b0ce99242ae6_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/1d7cf6dbd6095ee1fb5860efe258b0ce99242ae6_1d7cf6dbd6095ee1fb5860efe258b0ce99242ae6_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.32% (1889/2443) |
| Line Coverage | 64.40% (33954/52725) |
| Region Coverage | 64.76% (17452/26948) |
| Branch Coverage | 53.96% (9344/17316) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4731455027
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4732411052
Codex automated review failed and did not complete.
Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jun 18th, 2026 1:09 AM.
Workflow run: https://github.com/apache/doris/actions/runs/27696292624
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4733306322
<details>
<summary>ClickBench: <b>Total hot run time: 25.81 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c2c51332d7f90c80ccabf77b27c23317d636a128, data reload: false
query1 0.00 0.00 0.00
query2 0.15 0.08 0.08
query3 0.36 0.24 0.24
query4 1.61 0.24 0.24
query5 0.33 0.33 0.33
query6 1.17 0.67 0.66
query7 0.03 0.01 0.01
query8 0.09 0.08 0.07
query9 0.51 0.39 0.38
query10 0.58 0.60 0.60
query11 0.33 0.18 0.18
query12 0.32 0.19 0.19
query13 0.52 0.53 0.51
query14 0.93 0.92 0.93
query15 0.67 0.59 0.59
query16 0.38 0.39 0.39
query17 1.00 1.05 1.00
query18 0.30 0.29 0.30
query19 1.92 1.82 1.76
query20 0.02 0.02 0.02
query21 15.40 0.37 0.30
query22 4.82 0.13 0.13
query23 15.85 0.48 0.30
query24 2.60 0.56 0.42
query25 0.15 0.11 0.09
query26 0.73 0.27 0.22
query27 0.11 0.09 0.10
query28 3.40 0.96 0.53
query29 12.45 4.48 3.53
query30 0.39 0.26 0.26
query31 2.77 0.64 0.34
query32 3.23 0.60 0.48
query33 2.94 2.98 2.97
query34 15.85 4.08 3.34
query35 3.30 3.25 3.28
query36 0.64 0.52 0.50
query37 0.11 0.09 0.09
query38 0.08 0.07 0.07
query39 0.08 0.06 0.06
query40 0.19 0.18 0.17
query41 0.12 0.08 0.08
query42 0.09 0.06 0.06
query43 0.07 0.07 0.07
Total cold run time: 96.59 s
Total hot run time: 25.81 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4738167972
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4739352006
# BE Regression && UT Coverage Report
Increment line coverage `76.60% (311/406)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_0436cc4cb573b104a59408a6169bd145bcfeff6c_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_0436cc4cb573b104a59408a6169bd145bcfeff6c_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 64.09% (24560/38322) |
| Line Coverage | 47.88% (255578/533815) |
| Region Coverage | 44.59% (210879/472979) |
| Branch Coverage | 45.63% (91504/200514) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4714247339
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4765391810
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4551029809
# FE UT Coverage Report
Increment line coverage ` 73.85% (449/608)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_a68f47c588b0124cf5474ff805c847df5dfc781e/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_a68f47c588b0124cf5474ff805c847df5dfc781e/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3372699227
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java:
##########
@@ -285,4 +302,71 @@ public List<Expr> getOtherJoinConjuncts() {
public List<Expr> getMarkJoinConjuncts() {
return markJoinConjuncts;
}
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+
+ LocalExchangeTypeRequire probeSideRequire;
+ LocalExchangeTypeRequire buildSideRequire;
+ LocalExchangeType outputType = null;
+
+ if (joinOp == JoinOperator.NULL_AWARE_LEFT_ANTI_JOIN) {
+ buildSideRequire = probeSideRequire = LocalExchangeTypeRequire.noRequire();
+ outputType = LocalExchangeType.NOOP;
+ } else if (distrMode == DistributionMode.BROADCAST) {
+ // BE: _child->is_serial_operator() ? PASSTHROUGH/PASS_TO_ONE : NOOP
+ boolean probeChildSerial = children.get(0).isSerialOperatorOnBe(
+ translatorContext.getConnectContext());
+ boolean buildChildSerial = children.get(1).isSerialOperatorOnBe(
+ translatorContext.getConnectContext());
+ probeSideRequire = probeChildSerial
Review Comment:
[P2] Preserve `enable_broadcast_join_force_passthrough`
The BE planner returns `PASSTHROUGH` unconditionally for a broadcast-join probe when `state->enable_broadcast_join_force_passthrough()` is true (`HashJoinProbeOperatorX::required_data_distribution`). This FE branch only checks whether the probe child is serial. Once `enable_local_shuffle_planner` is enabled, BE local-exchange planning is skipped, so a non-serial broadcast join silently ignores this session variable and no balancing exchange is inserted. Please include the session flag in `probeSideRequire` and add a plan-shape assertion for the true/false cases.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/AddLocalExchange.java:
##########
@@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.planner;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.glue.translator.PlanTranslatorContext;
+import org.apache.doris.nereids.trees.plans.distribute.DistributedPlan;
+import org.apache.doris.nereids.trees.plans.distribute.FragmentIdMapping;
+import org.apache.doris.nereids.trees.plans.distribute.PipelineDistributedPlan;
+import org.apache.doris.planner.LocalExchangeNode.LocalExchangeType;
+import org.apache.doris.planner.LocalExchangeNode.LocalExchangeTypeRequire;
+import org.apache.doris.planner.LocalExchangeNode.RequireHash;
+
+/**
+ * FE-side local exchange planner — inserts {@link LocalExchangeNode} into each fragment's
+ * plan tree so that within-fragment data redistribution is decided at planning time
+ * instead of at BE pipeline-build time.
+ *
+ * <h3>When this runs</h3>
+ * Invoked from {@code NereidsPlanner.addLocalExchangeAfterDistribute()} right after
+ * {@code DistributePlanner} has assigned instances to fragments and before the plan is
+ * serialized to BE. Gated by session variable {@code enable_local_shuffle_planner}
+ * (default true) and {@code enable_local_shuffle}; when either is off this pass is
+ * skipped entirely and BE falls back to its own {@code _plan_local_exchange}. The two
+ * paths are mutually exclusive: BE consults {@code runtime_state.h::plan_local_shuffle()}
+ * to know whether it should plan LE itself.
+ *
+ * <h3>What it changes</h3>
+ * <ul>
+ * <li>For each fragment with {@code maxPerBeInstances > 1}, walks the plan tree
+ * bottom-up via {@link PlanNode#enforceAndDeriveLocalExchange} and inserts
+ * LocalExchangeNodes where children's output distribution doesn't satisfy the
+ * parent's requirement.</li>
+ * <li>May wrap the fragment root with an extra PASSTHROUGH LE so the data sink
+ * (DataStreamSink / OlapTableSink) runs with the full instance count even when
+ * the root operator is serial — see {@link #addLocalExchangeForFragment}.</li>
+ * <li>Does NOT modify the fragment sink itself, fragment boundaries, or instance
+ * assignment.</li>
+ * </ul>
+ *
+ * <h3>Per-BE instance semantics</h3>
+ * Skips fragments where every BE has at most 1 instance. Using a global instance count
+ * would insert LE for "2 BEs × 1 instance" cases, which BE's own
+ * {@code _plan_local_exchange} would not — leading to pipeline task-count mismatch and
+ * deadlock. See {@link #addLocalExchange}.
+ *
+ * <h3>Reading order</h3>
+ * Start with {@link PlanNode#enforceRequire} (the recursion engine), then individual
+ * {@code enforceAndDeriveLocalExchange} overrides on PlanNode subclasses.
+ */
+public class AddLocalExchange {
+ /** addLocalExchange with distributed plans, skipping single-instance fragments.
+ * BE's _plan_local_exchange checks _num_instances which is the per-BE instance count.
+ * With _num_instances<=1 all pipelines on that BE have 1 task so local exchange is a no-op.
+ * We must use the same per-BE semantics: skip when every BE has at most 1 instance.
+ * Using global instanceCount would insert LE for fragments where 2 BEs each have 1 instance
+ * (global=2, per-BE=1), causing pipeline task mismatch and deadlock. */
+ public void addLocalExchange(FragmentIdMapping<DistributedPlan> distributedPlans,
+ PlanTranslatorContext context) {
+ for (DistributedPlan plan : distributedPlans.values()) {
+ PipelineDistributedPlan pipePlan = (PipelineDistributedPlan) plan;
+ long maxPerBeInstances = pipePlan.getInstanceJobs().stream()
+ .collect(java.util.stream.Collectors.groupingBy(
+ j -> j.getAssignedWorker().id(), java.util.stream.Collectors.counting()))
+ .values().stream().mapToLong(Long::longValue).max().orElse(0);
+ if (maxPerBeInstances <= 1) {
+ continue;
+ }
+ PlanFragment fragment = pipePlan.getFragmentJob().getFragment();
+ addLocalExchangeForFragment(fragment, context);
+ }
+ }
+
+ private void addLocalExchangeForFragment(PlanFragment fragment, PlanTranslatorContext context) {
+ DataSink sink = fragment.getSink();
+ LocalExchangeTypeRequire require = sink == null
+ ? LocalExchangeTypeRequire.noRequire() : sink.getLocalExchangeTypeRequire();
+ PlanNode root = fragment.getPlanRoot();
+ context.setHasSerialAncestorInPipeline(root, false);
+ Pair<PlanNode, LocalExchangeType> output = root
+ .enforceAndDeriveLocalExchange(context, null, require);
+ PlanNode newRoot = output.first;
+ // The fragment data sink (DataStreamSink, OlapTableSink) runs in the same pipeline
+ // as the root. If the root will be serial on BE, the sink pipeline has 1 task —
+ // only instance 0 sends data, others hang or miss writes.
+ // Insert PASSTHROUGH fan-out so sink runs with _num_instances tasks.
+ // This matches BE-native's default required_data_distribution():
+ // _child->is_serial_operator() ? PASSTHROUGH : NOOP
+ if (newRoot.isSerialOperatorOnBe(context.getConnectContext())) {
+ newRoot = new LocalExchangeNode(context.nextPlanNodeId(), newRoot,
+ LocalExchangeType.PASSTHROUGH, null);
+ }
+ if (newRoot != root) {
+ fragment.setPlanRoot(newRoot);
+ }
+ validateNoSerialWithoutLocalExchange(fragment.getPlanRoot(), context.getConnectContext());
+ }
+
+ /**
+ * In a local-shuffle fragment, the root check above guarantees the root pipeline
+ * has N tasks. Any serial operator reduces its pipeline to 1 task. If this serial
+ * operator feeds into a non-serial parent without LocalExchangeNode in between,
+ * some pipelines have 1 task while others have N → shared_state mismatch, data loss.
+ *
+ * Serial→serial chains are fine (all at 1 task, consistent). Only the transition
+ * from serial to non-serial needs LE to restore parallelism.
+ */
+ private void validateNoSerialWithoutLocalExchange(PlanNode node,
+ org.apache.doris.qe.ConnectContext context) {
+ for (PlanNode child : node.getChildren()) {
+ validateNoSerialWithoutLocalExchange(child, context);
+ if (child.isSerialOperatorOnBe(context)
+ && !(child instanceof LocalExchangeNode)
+ && !(node instanceof LocalExchangeNode)
+ && !(node instanceof ExchangeNode)
+ && !node.isSerialOperatorOnBe(context)) {
+ org.apache.logging.log4j.LogManager.getLogger(AddLocalExchange.class).warn(
Review Comment:
[P1] Reject invalid serial-to-parallel plans instead of logging
This predicate identifies a definite task-count mismatch: the child pipeline runs one task while the parent runs `_num_instances`. The comments and regression cases in this PR document that executing such a shape can hang, lose rows, or initialize shared state inconsistently. Logging a warning and continuing still sends the known-invalid plan to BE, potentially returning incorrect results. Please throw a planning error/assert here (or repair the plan) rather than continue.
##########
regression-test/suites/nereids_p0/local_shuffle/test_local_shuffle_rqg_bugs.groovy:
##########
@@ -0,0 +1,1557 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/**
+ * Regression tests for bugs discovered by RQG testing on the local-exchange2 branch.
+ *
+ * These queries triggered "must set shared state" errors or incorrect results
+ * in RQG build 183992. Common conditions:
+ * - use_serial_exchange=true (makes ALL Exchanges serial, not just UNPARTITIONED)
+ * - enable_local_shuffle_planner=true (FE-planned local exchange)
+ * - parallel_pipeline_task_num > 1
+ *
+ * Error types reproduced:
+ * 1. must set shared state, in AGGREGATION_OPERATOR
+ * 2. must set shared state, in SORT_OPERATOR
+ * 3. incorrect results with GROUPING SETS + scalar subquery + window function
+ */
+suite("test_local_shuffle_rqg_bugs") {
+
+ // ============================================================
+ // Table setup — mirrors RQG table structure
+ // 10 buckets to match RQG (replication_num=1 for single-BE testing)
+ // ============================================================
+ sql "DROP TABLE IF EXISTS rqg_t1"
+ sql "DROP TABLE IF EXISTS rqg_t2"
+ sql "DROP TABLE IF EXISTS rqg_t3"
+ sql "DROP TABLE IF EXISTS rqg_t4"
+
+ sql """
+ CREATE TABLE rqg_t1 (
+ pk INT NOT NULL,
+ col_int_undef_signed INT,
+ col_int_undef_signed2 INT,
+ col_int_undef_signed_not_null INT NOT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ sql """
+ CREATE TABLE rqg_t2 (
+ pk INT NOT NULL,
+ col_int_undef_signed INT,
+ col_int_undef_signed2 INT,
+ col_bigint_undef_signed_not_null BIGINT NOT NULL,
+ col_decimal_38_10__undef_signed_not_null DECIMAL(38,10) NOT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ // Table for build 184181 GLOBAL_HASH_SHUFFLE bugs — needs varchar + bigint columns
+ sql """
+ CREATE TABLE rqg_t3 (
+ pk INT NOT NULL,
+ col_bigint_undef_signed BIGINT,
+ col_varchar_10__undef_signed VARCHAR(10),
+ col_varchar_64__undef_signed VARCHAR(64)
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ // Second table for FULL OUTER JOIN case (col_bigint_undef_signed_not_null)
+ sql """
+ CREATE TABLE rqg_t4 (
+ pk INT NOT NULL,
+ col_bigint_undef_signed_not_null BIGINT NOT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ sql """
+ INSERT INTO rqg_t3 VALUES
+ (0, -94, 'Abc', 'hello world'),
+ (1, 672609, 'Xyz', null),
+ (2, -3766684, 'Pqr', 'test string'),
+ (3, 5070261, 'abc', 'another row'),
+ (4, null, 'def', 'value four'),
+ (5, -86, 'XgpxlHBLEM', null),
+ (6, 21910, 'abc', 'they'),
+ (7, -63, 'zzzz', 'some text'),
+ (8, -8276281, 'AHlvNtoGLO', 'longer string here'),
+ (9, -101, 'mid', 'final row')
+ """
+
+ sql """
+ INSERT INTO rqg_t4 VALUES
+ (0, 0), (1, 1), (2, 2), (3, 3), (4, 4),
+ (5, 5), (6, 6), (7, 7), (8, 8), (9, 9),
+ (10, 2), (11, 2), (12, 2), (13, 3), (14, 4),
+ (15, 5), (16, 2), (17, 2), (18, 2), (19, 9)
+ """
+
+ // Insert enough rows to exercise multiple pipeline tasks
+ sql """
+ INSERT INTO rqg_t1 VALUES
+ (0, 0, 10, 0), (1, 1, 11, 1), (2, 2, 12, 2), (3, 3, 13, 3),
+ (4, 4, 14, 4), (5, 5, 15, 5), (6, 6, 16, 6), (7, 7, 17, 7),
+ (8, 8, 18, 8), (9, 9, 19, 9), (10, 0, 20, 10), (11, 1, 21, 11),
+ (12, 2, 22, 12), (13, 3, 23, 13), (14, 4, 24, 14), (15, 5, 25, 15),
+ (16, 6, 26, 16), (17, 7, 27, 17), (18, 8, 28, 18), (19, 9, 29, 19)
+ """
+
+ sql """
+ INSERT INTO rqg_t2 VALUES
+ (0, 0, 10, 100, 1.5), (1, 1, 11, 101, 2.5), (2, 2, 12, 102, 3.5),
+ (3, 3, 13, 103, 4.5), (4, 4, 14, 104, 5.5), (5, 5, 15, 105, 6.5),
+ (6, 6, 16, 106, 7.5), (7, 7, 17, 107, 8.5), (8, 8, 18, 108, 9.5),
+ (9, 9, 19, 109, 10.5), (10, 0, 20, 110, 11.5), (11, 1, 21, 111, 12.5),
+ (12, 2, 22, 112, 13.5), (13, 3, 23, 113, 14.5), (14, 4, 24, 114, 15.5),
+ (15, 5, 25, 115, 16.5), (16, 6, 26, 116, 17.5), (17, 7, 27, 117, 18.5),
+ (18, 8, 28, 118, 19.5), (19, 9, 29, 119, 20.5)
+ """
+
+ // Wait for data to be visible
+ Thread.sleep(5000)
+
+ // ============================================================
+ // Common settings
+ // ============================================================
+ sql "SET enable_nereids_planner=true"
+ sql "SET enable_fallback_to_original_planner=false"
+ sql "SET runtime_filter_mode=off"
+ sql "SET enable_profile=true"
+ sql "SET enable_sql_cache=false"
+ sql "SET enable_local_shuffle=true"
+
+ // ============================================================
+ // Bug 1: must set shared state, in AGGREGATION_OPERATOR
+ // RQG case: eliminate_group_by_uniform.case_id_11007680713
+ // Key conditions: use_serial_exchange=true, parallel_pipeline_task_num=3
+ // SQL: EXCEPT with count(*) GROUP BY on both sides
+ // ============================================================
+
+ // Test with FE planner (the buggy path)
+ logger.info("=== Bug 1a: AGG shared state - EXCEPT with serial exchange (FE planner) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=3,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_common_expr_pushdown=false,
+ disable_streaming_preaggregations=true)*/
+ col_int_undef_signed_not_null as col1,
+ col_int_undef_signed_not_null as col2,
+ 0 as col3, count(1)
+ FROM rqg_t1
+ GROUP BY col1, col2, col3
+ EXCEPT
+ SELECT col_bigint_undef_signed_not_null as col1,
+ col_decimal_38_10__undef_signed_not_null as col2,
+ 5 as col3, count(1)
+ FROM rqg_t2
+ GROUP BY col1, col2, col3
+ """
+ logger.info("Bug 1a: PASSED (no crash)")
+ } catch (Throwable t) {
+ logger.error("Bug 1a FAILED: ${t.message}")
+ assertTrue(false, "Bug 1a: must set shared state in AGGREGATION_OPERATOR: ${t.message}")
+ }
+
+ // Compare with BE native planner
+ logger.info("=== Bug 1b: AGG shared state - EXCEPT with serial exchange (BE native) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=3,
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_common_expr_pushdown=false,
+ disable_streaming_preaggregations=true)*/
+ col_int_undef_signed_not_null as col1,
+ col_int_undef_signed_not_null as col2,
+ 0 as col3, count(1)
+ FROM rqg_t1
+ GROUP BY col1, col2, col3
+ EXCEPT
+ SELECT col_bigint_undef_signed_not_null as col1,
+ col_decimal_38_10__undef_signed_not_null as col2,
+ 5 as col3, count(1)
+ FROM rqg_t2
+ GROUP BY col1, col2, col3
+ """
+ logger.info("Bug 1b: PASSED (no crash)")
+ } catch (Throwable t) {
+ logger.error("Bug 1b FAILED: ${t.message}")
+ assertTrue(false, "Bug 1b: BE native also fails: ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 2: must set shared state, in SORT_OPERATOR
+ // RQG case: grouping_set.case_id_5308471751
+ // Key conditions: use_serial_exchange=true, parallel_pipeline_task_num=5
+ // SQL: GROUPING SETS + window function (PERCENT_RANK)
+ // ============================================================
+
+ logger.info("=== Bug 2a: SORT shared state - GROUPING SETS + window (FE planner) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=5,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_share_hash_table_for_broadcast_join=false,
+ disable_streaming_preaggregations=true)*/
+ SUM(PERCENT_RANK() OVER (PARTITION BY col_int_undef_signed2 ORDER BY col_int_undef_signed2))
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed2),(pk, pk),(col_int_undef_signed))
+ """
+ logger.info("Bug 2a: PASSED (no crash)")
+ } catch (Throwable t) {
+ logger.error("Bug 2a FAILED: ${t.message}")
+ assertTrue(false, "Bug 2a: must set shared state in SORT_OPERATOR: ${t.message}")
+ }
+
+ logger.info("=== Bug 2b: SORT shared state - GROUPING SETS + window (BE native) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=5,
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_share_hash_table_for_broadcast_join=false,
+ disable_streaming_preaggregations=true)*/
+ SUM(PERCENT_RANK() OVER (PARTITION BY col_int_undef_signed2 ORDER BY col_int_undef_signed2))
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed2),(pk, pk),(col_int_undef_signed))
+ """
+ logger.info("Bug 2b: PASSED (no crash)")
+ } catch (Throwable t) {
+ logger.error("Bug 2b FAILED: ${t.message}")
+ assertTrue(false, "Bug 2b: BE native also fails: ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 3: incorrect results with GROUPING SETS + scalar subquery + window
+ // RQG case: grouping_set.case_id_5694495756
+ // Key conditions: parallel_pipeline_task_num=2, disable_streaming_preaggregations=true
+ // Expected: all rows same value; Actual: values split proportionally (1/3, 2/3)
+ // ============================================================
+
+ logger.info("=== Bug 3: incorrect results - GROUPING SETS + subquery + window ===")
+ // FE planner
+ def result_fe = sql """
+ SELECT /*+SET_VAR(parallel_pipeline_task_num=2,
+ enable_local_shuffle_planner=true,
+ disable_streaming_preaggregations=true,
+ enable_share_hash_table_for_broadcast_join=true)*/
+ SUM((SELECT MAX(col_int_undef_signed2) FROM rqg_t1))
+ OVER (PARTITION BY pk ORDER BY pk)
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed2, pk),(pk), (pk))
+ """
+ // BE native
+ def result_be = sql """
+ SELECT /*+SET_VAR(parallel_pipeline_task_num=2,
+ enable_local_shuffle_planner=false,
+ disable_streaming_preaggregations=true,
+ enable_share_hash_table_for_broadcast_join=true)*/
+ SUM((SELECT MAX(col_int_undef_signed2) FROM rqg_t1))
+ OVER (PARTITION BY pk ORDER BY pk)
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed2, pk),(pk), (pk))
+ """
+ logger.info("Bug 3 FE result rows: ${result_fe.size()}, first few: ${result_fe.take(5)}")
+ logger.info("Bug 3 BE result rows: ${result_be.size()}, first few: ${result_be.take(5)}")
+
+ // All values in both should be the same
+ if (result_fe.size() != result_be.size()) {
+ logger.warn("Bug 3: row count mismatch FE=${result_fe.size()} vs BE=${result_be.size()}")
Review Comment:
[P2] Make this regression fail the suite
When FE and BE produce different row counts this only logs a warning, so the regression still passes. The same issue appears in the Bug 4/5 catches below, which log query failures without rethrowing or asserting. These cases therefore provide no protection against the crashes and result mismatches they describe. Please use a result assertion/`qt_*`, and let unexpected SQL exceptions propagate, so a recurrence fails the suite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4670455531
<details>
<summary>TPC-H: <b>Total hot run time: 29176 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7c898c900506116cfb44b47189025b6fbfa7e44c, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17658 3995 3989 3989
q2 q3 10999 1504 823 823
q4 4801 479 341 341
q5 8599 887 620 620
q6 346 172 136 136
q7 906 850 645 645
q8 10946 1568 1584 1568
q9 7098 4488 4496 4488
q10 6873 1898 1516 1516
q11 436 275 259 259
q12 653 439 292 292
q13 18137 3404 2843 2843
q14 275 262 243 243
q15 q16 816 771 704 704
q17 946 972 918 918
q18 6758 5726 5552 5552
q19 1146 1308 1135 1135
q20 526 401 259 259
q21 5629 2616 2541 2541
q22 430 364 304 304
Total cold run time: 103978 ms
Total hot run time: 29176 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4313 4239 4227 4227
q2 q3 4564 4977 4327 4327
q4 2087 2213 1402 1402
q5 4444 4275 4279 4275
q6 224 177 130 130
q7 2308 1806 1749 1749
q8 2541 2149 2132 2132
q9 8020 8056 8101 8056
q10 4784 4756 4385 4385
q11 575 411 381 381
q12 757 754 529 529
q13 3262 3674 3028 3028
q14 317 320 276 276
q15 q16 713 753 629 629
q17 1357 1332 1435 1332
q18 8185 7332 6948 6948
q19 1109 1099 1093 1093
q20 2228 2237 1957 1957
q21 5314 4613 4545 4545
q22 514 457 409 409
Total cold run time: 57616 ms
Total hot run time: 51810 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4498451933
<details>
<summary>TPC-H: <b>Total hot run time: 31122 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3ac8525eaac942c0bbeb93cf46d4c95ecf353156, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17639 3843 3858 3843
q2 q3 11031 1433 812 812
q4 4813 481 357 357
q5 10756 2313 2096 2096
q6 389 176 138 138
q7 974 762 662 662
q8 9632 1749 1671 1671
q9 7039 5045 4978 4978
q10 6487 2095 1796 1796
q11 433 276 241 241
q12 648 424 293 293
q13 18156 3453 2819 2819
q14 271 261 237 237
q15 q16 821 773 710 710
q17 998 937 925 925
q18 7221 5856 5636 5636
q19 1165 1248 1007 1007
q20 512 405 269 269
q21 5587 2628 2325 2325
q22 450 366 307 307
Total cold run time: 105022 ms
Total hot run time: 31122 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4205 4164 4120 4120
q2 q3 4591 4969 4315 4315
q4 2120 2274 1388 1388
q5 4383 4257 5342 4257
q6 250 189 138 138
q7 1931 1774 1594 1594
q8 2415 2132 2066 2066
q9 8043 7979 7939 7939
q10 4647 4510 4073 4073
q11 594 416 369 369
q12 764 758 512 512
q13 3408 3569 2960 2960
q14 307 319 270 270
q15 q16 716 736 633 633
q17 1361 1336 1301 1301
q18 7808 7299 6807 6807
q19 1125 1114 1124 1114
q20 2213 2210 1923 1923
q21 5342 4630 4454 4454
q22 516 467 403 403
Total cold run time: 56739 ms
Total hot run time: 50636 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4484211987
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4487508895
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/9016e5a105f84231f1a535be233aaf3e595c711d_9016e5a105f84231f1a535be233aaf3e595c711d_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/9016e5a105f84231f1a535be233aaf3e595c711d_9016e5a105f84231f1a535be233aaf3e595c711d_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.06% (1854/2375) |
| Line Coverage | 64.52% (33334/51662) |
| Region Coverage | 65.20% (16523/25341) |
| Branch Coverage | 55.75% (8838/15852) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4488610712
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4738675411
# FE UT Coverage Report
Increment line coverage ` 74.73% (420/562)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_0436cc4cb573b104a59408a6169bd145bcfeff6c/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_0436cc4cb573b104a59408a6169bd145bcfeff6c/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4765629089
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4739664036
# FE Regression Coverage Report
Increment line coverage ` 76.44% (451/590)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_0436cc4cb573b104a59408a6169bd145bcfeff6c_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_0436cc4cb573b104a59408a6169bd145bcfeff6c_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4508996295
# BE Regression && UT Coverage Report
Increment line coverage `85.51% (354/414)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_846d365a88901fa32fdf44b42b6cd04b900da259_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_846d365a88901fa32fdf44b42b6cd04b900da259_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.71% (27903/37857) |
| Line Coverage | 57.61% (303202/526266) |
| Region Coverage | 54.83% (254070/463365) |
| Branch Coverage | 56.35% (109800/194854) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4551441930
<details>
<summary>TPC-H: <b>Total hot run time: 32181 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a68f47c588b0124cf5474ff805c847df5dfc781e, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17763 4185 4116 4116
q2 q3 10936 1460 814 814
q4 4814 494 342 342
q5 10582 2307 2115 2115
q6 397 176 141 141
q7 934 783 653 653
q8 9712 1754 1599 1599
q9 7000 5066 5120 5066
q10 6561 2219 1917 1917
q11 439 274 244 244
q12 657 427 303 303
q13 18158 3553 2834 2834
q14 271 260 236 236
q15 q16 829 784 715 715
q17 959 946 948 946
q18 6895 5750 6243 5750
q19 1191 1281 1077 1077
q20 536 410 277 277
q21 6005 2754 2726 2726
q22 465 360 310 310
Total cold run time: 105104 ms
Total hot run time: 32181 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5120 4798 4917 4798
q2 q3 4901 5293 4731 4731
q4 2157 2260 1408 1408
q5 4931 4717 4783 4717
q6 223 165 120 120
q7 1784 1642 1465 1465
q8 2274 1997 1975 1975
q9 7659 7641 7656 7641
q10 4742 4693 4215 4215
q11 545 394 363 363
q12 732 749 527 527
q13 3130 3444 2827 2827
q14 282 279 251 251
q15 q16 692 706 627 627
q17 1295 1267 1260 1260
q18 7394 7102 6883 6883
q19 1157 1104 1113 1104
q20 2250 2235 1939 1939
q21 5368 4688 4565 4565
q22 524 452 411 411
Total cold run time: 57160 ms
Total hot run time: 51827 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4681111504
<details>
<summary>TPC-DS: <b>Total hot run time: 168597 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9f33ce47e1bdfae26100cf19b96a7eab06238d13, data reload: false
query5 4309 613 481 481
query6 434 190 171 171
query7 4823 574 304 304
query8 369 211 183 183
query9 8784 4002 3993 3993
query10 438 307 251 251
query11 5922 2356 2106 2106
query12 158 98 96 96
query13 1255 608 449 449
query14 6377 5406 5054 5054
query14_1 4397 4360 4396 4360
query15 212 194 174 174
query16 1040 452 454 452
query17 1149 679 565 565
query18 2708 471 337 337
query19 207 175 137 137
query20 123 104 102 102
query21 220 140 110 110
query22 13637 13593 13509 13509
query23 17362 16545 16139 16139
query23_1 16288 16231 16210 16210
query24 7554 1776 1307 1307
query24_1 1343 1302 1288 1288
query25 549 429 360 360
query26 1295 306 168 168
query27 2679 564 326 326
query28 4440 2027 2025 2025
query29 1060 599 455 455
query30 302 235 195 195
query31 1103 1061 945 945
query32 107 61 58 58
query33 507 348 242 242
query34 1170 1207 659 659
query35 822 791 678 678
query36 1368 1370 1220 1220
query37 153 100 89 89
query38 3155 3128 2979 2979
query39 924 944 891 891
query39_1 871 893 864 864
query40 210 122 99 99
query41 64 61 71 61
query42 98 93 96 93
query43 319 318 273 273
query44
query45 196 188 179 179
query46 1069 1156 749 749
query47 2402 2345 2200 2200
query48 385 378 299 299
query49 636 471 351 351
query50 956 362 258 258
query51 4630 4465 4323 4323
query52 87 88 77 77
query53 249 269 191 191
query54 271 211 200 200
query55 77 76 69 69
query56 248 228 206 206
query57 1429 1408 1330 1330
query58 243 225 213 213
query59 1587 1682 1412 1412
query60 282 242 228 228
query61 154 152 151 151
query62 693 638 592 592
query63 239 190 182 182
query64 2521 816 615 615
query65
query66 1739 448 343 343
query67 29653 29614 29461 29461
query68
query69 408 307 265 265
query70 991 944 977 944
query71 316 216 206 206
query72 2928 2620 2342 2342
query73 858 771 456 456
query74 5102 4916 4772 4772
query75 2664 2590 2268 2268
query76 2371 1136 795 795
query77 370 379 279 279
query78 12398 12460 11833 11833
query79 1298 1043 750 750
query80 567 495 409 409
query81 449 285 235 235
query82 241 159 131 131
query83 277 281 261 261
query84
query85 918 585 503 503
query86 337 308 274 274
query87 3401 3343 3240 3240
query88 3580 2766 2714 2714
query89 416 383 340 340
query90 2207 191 187 187
query91 192 176 161 161
query92 67 62 71 62
query93 1422 1441 877 877
query94 570 406 310 310
query95 683 456 344 344
query96 1112 876 338 338
query97 2682 2695 2556 2556
query98 214 206 210 206
query99 1157 1179 1019 1019
Total cold run time: 250555 ms
Total hot run time: 168597 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4679263646
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/692)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_f056ebbed42446ea468bcc6891c05520bdfb31c3_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_f056ebbed42446ea468bcc6891c05520bdfb31c3_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4682262951
# FE Regression Coverage Report
Increment line coverage ` 79.51% (458/576)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_784476ead971eeebeb0b637f6da4a434e380ce36_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_784476ead971eeebeb0b637f6da4a434e380ce36_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4701737240
<details>
<summary>TPC-DS: <b>Total hot run time: 168861 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8db568b5100ed397fc6347033402642e90e5509e, data reload: false
query5 4321 623 478 478
query6 428 189 175 175
query7 4957 549 310 310
query8 365 204 196 196
query9 8786 4024 4043 4024
query10 444 312 270 270
query11 5949 2384 2177 2177
query12 156 106 98 98
query13 1290 585 448 448
query14 6412 5432 5072 5072
query14_1 4397 4404 4431 4404
query15 208 201 178 178
query16 1015 446 433 433
query17 1126 722 607 607
query18 2705 490 349 349
query19 209 183 146 146
query20 113 106 107 106
query21 222 143 114 114
query22 13706 13690 13357 13357
query23 17705 16659 16182 16182
query23_1 16389 16379 16334 16334
query24 7576 1768 1307 1307
query24_1 1311 1327 1310 1310
query25 537 423 370 370
query26 1306 326 169 169
query27 2634 573 337 337
query28 4456 2036 2021 2021
query29 1060 618 483 483
query30 309 242 200 200
query31 1119 1066 952 952
query32 112 63 56 56
query33 529 314 242 242
query34 1178 1161 668 668
query35 747 782 686 686
query36 1422 1415 1235 1235
query37 165 102 87 87
query38 3235 3154 3037 3037
query39 938 922 906 906
query39_1 895 872 863 863
query40 220 123 100 100
query41 64 65 61 61
query42 94 93 93 93
query43 333 324 284 284
query44
query45 197 189 182 182
query46 1072 1241 742 742
query47 2433 2387 2236 2236
query48 401 419 294 294
query49 620 494 348 348
query50 996 371 260 260
query51 4460 4438 4316 4316
query52 87 87 76 76
query53 242 267 188 188
query54 258 220 195 195
query55 77 75 68 68
query56 243 220 225 220
query57 1451 1414 1318 1318
query58 244 214 210 210
query59 1568 1674 1533 1533
query60 281 249 215 215
query61 152 152 150 150
query62 706 641 589 589
query63 227 186 186 186
query64 2511 788 626 626
query65
query66 1781 478 343 343
query67 29806 29748 29579 29579
query68
query69 421 302 265 265
query70 911 908 975 908
query71 306 220 209 209
query72 2914 2629 2333 2333
query73 872 804 453 453
query74 5137 4970 4815 4815
query75 2649 2597 2233 2233
query76 2344 1139 761 761
query77 345 380 289 289
query78 12271 12446 11835 11835
query79 1269 1066 770 770
query80 518 463 384 384
query81 446 282 241 241
query82 231 156 119 119
query83 270 284 244 244
query84
query85 840 515 401 401
query86 331 304 265 265
query87 3380 3423 3149 3149
query88 3636 2747 2717 2717
query89 406 394 322 322
query90 2157 192 179 179
query91 168 149 129 129
query92 66 60 59 59
query93 1465 1470 848 848
query94 534 343 318 318
query95 717 487 344 344
query96 1105 885 367 367
query97 2712 2694 2561 2561
query98 209 206 198 198
query99 1165 1157 1026 1026
Total cold run time: 250849 ms
Total hot run time: 168861 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3399953617
##########
gensrc/thrift/Partitions.thrift:
##########
@@ -119,3 +153,5 @@ struct TDataPartition {
3: optional list<TRangePartition> partition_infos
4: optional TMergePartitionInfo merge_partition_info
}
+
Review Comment:
`git diff --check` reports `gensrc/thrift/Partitions.thrift:156: new blank line at EOF` after this closing brace. Please remove the extra trailing blank line(s) so the file ends here.
##########
regression-test/data/query_p0/join/test_multilevel_join_agg_local_shuffle.out:
##########
@@ -0,0 +1,815 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !bucket_shuffle_broadcast_agg_stage_1_result_on --
+1 52 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_shuffle_broadcast_agg_stage_1_result_off --
+1 52 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_shuffle_broadcast_agg_stage_2_result_on --
+1 52 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_shuffle_broadcast_agg_stage_2_result_off --
+1 52 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_shuffle_broadcast_agg_stage_3_result_on --
+1 52 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_shuffle_broadcast_agg_stage_3_result_off --
+1 52 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !shuffle_broadcast_broadcast_agg_stage_1_result_on --
+1 10 62 10
+2 5 54 20
+3 5 38 30
+4 6 49 40
+
+-- !shuffle_broadcast_broadcast_agg_stage_1_result_off --
+1 10 62 10
+2 5 54 20
+3 5 38 30
+4 6 49 40
+
+-- !shuffle_broadcast_broadcast_agg_stage_2_result_on --
+1 10 62 10
+2 5 54 20
+3 5 38 30
+4 6 49 40
+
+-- !shuffle_broadcast_broadcast_agg_stage_2_result_off --
+1 10 62 10
+2 5 54 20
+3 5 38 30
+4 6 49 40
+
+-- !shuffle_broadcast_broadcast_agg_stage_3_result_on --
+1 10 62 10
+2 5 54 20
+3 5 38 30
+4 6 49 40
+
+-- !shuffle_broadcast_broadcast_agg_stage_3_result_off --
+1 10 62 10
+2 5 54 20
+3 5 38 30
+4 6 49 40
+
+-- !bucket_broadcast_shuffle_agg_stage_1_result_on --
+1 26 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_broadcast_shuffle_agg_stage_1_result_off --
+1 26 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_broadcast_shuffle_agg_stage_2_result_on --
+1 26 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_broadcast_shuffle_agg_stage_2_result_off --
+1 26 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_broadcast_shuffle_agg_stage_3_result_on --
+1 26 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_broadcast_shuffle_agg_stage_3_result_off --
+1 26 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !alternating_bucket_bucket_bucket_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_bucket_bucket_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_bucket_shuffle_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_bucket_shuffle_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_bucket_broadcast_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_bucket_broadcast_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_shuffle_bucket_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_shuffle_bucket_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_shuffle_shuffle_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_shuffle_shuffle_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_shuffle_broadcast_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_shuffle_broadcast_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_broadcast_bucket_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_broadcast_bucket_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_broadcast_shuffle_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_broadcast_shuffle_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_broadcast_broadcast_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_broadcast_broadcast_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_bucket_bucket_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_bucket_bucket_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_bucket_shuffle_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_bucket_shuffle_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_bucket_broadcast_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_bucket_broadcast_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_shuffle_bucket_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_shuffle_bucket_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_shuffle_shuffle_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_shuffle_shuffle_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_shuffle_broadcast_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_shuffle_broadcast_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_broadcast_bucket_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_broadcast_bucket_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_broadcast_shuffle_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_broadcast_shuffle_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_broadcast_broadcast_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_broadcast_broadcast_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_bucket_bucket_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_bucket_bucket_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_bucket_shuffle_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_bucket_shuffle_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_bucket_broadcast_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_bucket_broadcast_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_shuffle_bucket_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_shuffle_bucket_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_shuffle_shuffle_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_shuffle_shuffle_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_shuffle_broadcast_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_shuffle_broadcast_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_broadcast_bucket_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_broadcast_bucket_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_broadcast_shuffle_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_broadcast_shuffle_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_broadcast_broadcast_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_broadcast_broadcast_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !window_union_all_bucket_row_number_result_on --
+1 52 4 40
+2 21 3 60
+3 16 2 60
+4 20 2 80
+
+-- !window_union_all_bucket_row_number_result_off --
+1 52 4 40
+2 21 3 60
+3 16 2 60
+4 20 2 80
+
+-- !window_union_all_bucket_window_sum_result_on --
+1 10 13 40
+2 6 7 60
+3 3 8 60
+4 3 10 80
+
+-- !window_union_all_bucket_window_sum_result_off --
+1 10 13 40
+2 6 7 60
+3 3 8 60
+4 3 10 80
+
+-- !window_union_all_shuffle_row_number_result_on --
+1 208 8 44
+2 21 3 21
+3 16 2 16
+4 20 2 18
+
+-- !window_union_all_shuffle_row_number_result_off --
+1 208 8 44
+2 21 3 21
+3 16 2 16
+4 20 2 18
+
+-- !window_union_all_shuffle_window_sum_result_on --
+1 36 26 44
+2 6 7 21
+3 3 8 16
+4 3 10 18
+
+-- !window_union_all_shuffle_window_sum_result_off --
+1 36 26 44
+2 6 7 21
+3 3 8 16
+4 3 10 18
+
+-- !window_union_all_broadcast_row_number_result_on --
+1 52 4 40
+2 21 3 60
+3 16 2 60
+4 20 2 80
+
+-- !window_union_all_broadcast_row_number_result_off --
+1 52 4 40
+2 21 3 60
+3 16 2 60
+4 20 2 80
+
+-- !window_union_all_broadcast_window_sum_result_on --
+1 10 13 40
+2 6 7 60
+3 3 8 60
+4 3 10 80
+
+-- !window_union_all_broadcast_window_sum_result_off --
+1 10 13 40
+2 6 7 60
+3 3 8 60
+4 3 10 80
+
+-- !window_except_bucket_row_number_result_on --
+1 10 2 20
+2 10 2 40
+3 5 1 30
+4 6 1 40
+
+-- !window_except_bucket_row_number_result_off --
+1 10 2 20
+2 10 2 40
+3 5 1 30
+4 6 1 40
+
+-- !window_except_bucket_window_sum_result_on --
+1 3 5 20
+2 3 5 40
+3 1 5 30
+4 1 6 40
+
+-- !window_except_bucket_window_sum_result_off --
+1 3 5 20
+2 3 5 40
+3 1 5 30
+4 1 6 40
+
+-- !window_except_shuffle_row_number_result_on --
+1 40 4 22
+2 10 2 14
+3 5 1 8
+4 6 1 9
+
+-- !window_except_shuffle_row_number_result_off --
+1 40 4 22
+2 10 2 14
+3 5 1 8
+4 6 1 9
+
+-- !window_except_shuffle_window_sum_result_on --
+1 10 10 22
+2 3 5 14
+3 1 5 8
+4 1 6 9
+
+-- !window_except_shuffle_window_sum_result_off --
+1 10 10 22
+2 3 5 14
+3 1 5 8
+4 1 6 9
+
+-- !window_except_broadcast_row_number_result_on --
+1 10 2 20
+2 10 2 40
+3 5 1 30
+4 6 1 40
+
+-- !window_except_broadcast_row_number_result_off --
+1 10 2 20
+2 10 2 40
+3 5 1 30
+4 6 1 40
+
+-- !window_except_broadcast_window_sum_result_on --
+1 3 5 20
+2 3 5 40
+3 1 5 30
+4 1 6 40
+
+-- !window_except_broadcast_window_sum_result_off --
+1 3 5 20
+2 3 5 40
+3 1 5 30
+4 1 6 40
+
+-- !window_intersect_bucket_row_number_result_on --
+
+-- !window_intersect_bucket_row_number_result_off --
+
+-- !window_intersect_bucket_window_sum_result_on --
+
+-- !window_intersect_bucket_window_sum_result_off --
+
+-- !window_intersect_shuffle_row_number_result_on --
+
+-- !window_intersect_shuffle_row_number_result_off --
+
+-- !window_intersect_shuffle_window_sum_result_on --
+
+-- !window_intersect_shuffle_window_sum_result_off --
+
+-- !window_intersect_broadcast_row_number_result_on --
+
+-- !window_intersect_broadcast_row_number_result_off --
+
+-- !window_intersect_broadcast_window_sum_result_on --
+
+-- !window_intersect_broadcast_window_sum_result_off --
+
+-- !bucket_broadcast_agg_result_on --
+1 26 10
+2 9 20
+3 8 30
+4 10 40
+
+-- !bucket_broadcast_agg_result_off --
+1 26 10
+2 9 20
+3 8 30
+4 10 40
+
+-- !partitioned_broadcast_agg_result_on --
+1 32 10
+2 19 20
+3 13 30
+4 15 40
+
+-- !partitioned_broadcast_agg_result_off --
+1 32 10
+2 19 20
+3 13 30
+4 15 40
+
+-- !bucket_partitioned_agg_result_on --
+1 52 44
+2 9 14
+3 8 8
+4 10 9
+
+-- !bucket_partitioned_agg_result_off --
+1 52 44
+2 9 14
+3 8 8
+4 10 9
+
+-- !all_three_multilevel_agg_result_on --
+1 96 10
+2 23 20
+3 16 30
+4 19 40
+
+-- !all_three_multilevel_agg_result_off --
+1 96 10
+2 23 20
+3 16 30
+4 19 40
+
+-- !agg_join_agg_mix_result_on --
+1 10 11 10
+2 5 7 20
+3 5 8 30
+4 6 9 40
+
+-- !agg_join_agg_mix_result_off --
+1 10 11 10
+2 5 7 20
+3 5 8 30
+4 6 9 40
+
+-- !double_broadcast_after_bucket_result_on --
+1 26 20
+2 9 40
+3 8 60
+4 10 80
+
+-- !double_broadcast_after_bucket_result_off --
+1 26 20
+2 9 40
+3 8 60
+4 10 80
+
+-- !partitioned_join_between_two_aggs_then_broadcast_result_on --
+1 16 10
+2 12 20
+3 13 30
+4 15 40
+
+-- !partitioned_join_between_two_aggs_then_broadcast_result_off --
+1 16 10
+2 12 20
+3 13 30
+4 15 40
+
+-- !bucket_shuffle_broadcast_two_stage_agg_result_on --
+1 52 44 10
+2 9 14 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_shuffle_broadcast_two_stage_agg_result_off --
+1 52 44 10
+2 9 14 20
+3 8 8 30
+4 10 9 40
+
+-- !left_join_null_preserving_with_multilevel_agg_result_on --
+1 5 44
+2 5 14
+3 5 8
+4 6 9
+
+-- !left_join_null_preserving_with_multilevel_agg_result_off --
+1 5 44
+2 5 14
+3 5 8
+4 6 9
+
+-- !seven_layer_bucket_shuffle_broadcast_result_on --
+1 37 10
+2 16 20
+3 16 30
+4 19 40
+
+-- !seven_layer_bucket_shuffle_broadcast_result_off --
+1 37 10
+2 16 20
+3 16 30
+4 19 40
+
+-- !eight_layer_mixed_join_agg_chain_result_on --
+1 52 44 20
+2 9 14 40
+3 8 8 60
+4 10 9 80
+
+-- !eight_layer_mixed_join_agg_chain_result_off --
+1 52 44 20
+2 9 14 40
+3 8 8 60
+4 10 9 80
+
+-- !seven_layer_left_join_mix_result_on --
+1 10 11 10
+2 5 7 20
+3 5 8 30
+4 6 9 40
+
+-- !seven_layer_left_join_mix_result_off --
+1 10 11 10
+2 5 7 20
+3 5 8 30
+4 6 9 40
+
+-- !broadcast_shuffle_broadcast_nested_agg_result_on --
+1 26 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !broadcast_shuffle_broadcast_nested_agg_result_off --
+1 26 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !window_union_join_agg_result_on --
+1 13 14
+2 7 23
+3 8 32
+4 10 42
+
+-- !window_union_join_agg_result_off --
+1 13 14
+2 7 23
+3 8 32
+4 10 42
+
+-- !window_except_join_agg_result_on --
+1 1 10
+2 1 20
+3 1 30
+
+-- !window_except_join_agg_result_off --
+1 1 10
+2 1 20
+3 1 30
+
+-- !window_intersect_shuffle_agg_result_on --
+1 22 2
+2 7 1
+3 8 1
+4 9 1
+
+-- !window_intersect_shuffle_agg_result_off --
+1 22 2
+2 7 1
+3 8 1
+4 9 1
+
+-- !window_union_except_broadcast_agg_result_on --
+1 13 4 10
+2 7 3 20
+3 8 2 30
+4 10 2 40
+
+-- !window_union_except_broadcast_agg_result_off --
+1 13 4 10
+2 7 3 20
+3 8 2 30
+4 10 2 40
+
+-- !window_setop_join_agg_chain_result_on --
+1 20 2 22
+2 10 2 14
+3 5 1 8
+4 6 1 9
+
+-- !window_setop_join_agg_chain_result_off --
+1 20 2 22
+2 10 2 14
+3 5 1 8
+4 6 1 9
+
Review Comment:
`git diff --check` reports `regression-test/data/query_p0/join/test_multilevel_join_agg_local_shuffle.out:815: new blank line at EOF` after this result row. Please regenerate or trim the expected output so it ends here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4686337234
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/9ccb7f30b5fb41183c99a77cc3de3311715aa9b6_9ccb7f30b5fb41183c99a77cc3de3311715aa9b6_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/9ccb7f30b5fb41183c99a77cc3de3311715aa9b6_9ccb7f30b5fb41183c99a77cc3de3311715aa9b6_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.32% (1889/2443) |
| Line Coverage | 64.41% (33959/52725) |
| Region Coverage | 64.81% (17464/26948) |
| Branch Coverage | 53.97% (9346/17316) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4686601063
<details>
<summary>TPC-H: <b>Total hot run time: 29135 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9ccb7f30b5fb41183c99a77cc3de3311715aa9b6, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17616 4034 4006 4006
q2 q3 10952 1397 818 818
q4 4736 478 339 339
q5 8484 901 580 580
q6 353 180 138 138
q7 946 834 622 622
q8 10931 1633 1632 1632
q9 7224 4549 4555 4549
q10 6836 1839 1512 1512
q11 437 267 249 249
q12 645 424 286 286
q13 18172 3392 2761 2761
q14 262 260 231 231
q15 q16 818 779 716 716
q17 937 939 909 909
q18 6862 5703 5637 5637
q19 1156 1212 992 992
q20 695 437 264 264
q21 6037 2749 2589 2589
q22 451 392 305 305
Total cold run time: 104550 ms
Total hot run time: 29135 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4774 4719 4949 4719
q2 q3 4849 5270 4689 4689
q4 2138 2180 1420 1420
q5 4948 4825 4620 4620
q6 239 177 131 131
q7 1855 1768 1578 1578
q8 2378 1962 1941 1941
q9 7563 7600 7617 7600
q10 4804 4696 4209 4209
q11 538 392 363 363
q12 818 747 525 525
q13 3019 3317 2817 2817
q14 281 277 249 249
q15 q16 680 700 603 603
q17 1289 1264 1256 1256
q18 7469 6762 6991 6762
q19 1098 1120 1092 1092
q20 2268 2233 1973 1973
q21 5276 4577 4419 4419
q22 516 457 402 402
Total cold run time: 56800 ms
Total hot run time: 51368 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4714370758
@BiteTheDDDDt （放到顶层，代码行上的线程回复你那边一直没刷出来——我之前误删了发到 thread 根上的那条，留下的是回复到子评论上的、GitHub UI 渲染不出来，抱歉）
> 我没懂如果fe一开始就把所有local exchange规划好，所有算子的并行度规划好，直接下发。be为什么需要_create_deferred_local_exchangers和_propagate_local_exchange_num_tasks？
>
> 这个是现在设计出来就是要先做折中方案的，还是可以直接一把做全面？_propagate_local_exchange_num_tasks和_create_deferred_local_exchangers这两块新增的应该不是为了灰度/回滚吧。
是的，这两块不是为了灰度/回滚——是 FE 接管 local exchange 的**当前折中**，不是一步到位。
要让 FE 把"每个算子的并发"也规划好直接下发，本质上需要 FE 自己切 pipeline、自己控制每条 pipeline 的并发（`num_tasks` 是 pipeline 粒度的，只在 pipeline 边界即 local exchange / pipeline breaker 处变化），BE 就不再规划 pipeline 了。那等于把 BE 的整套 pipeline 构建（`_build_pipeline`、pipeline-breaker 判定、`num_tasks` 推导、dependency 接线，还要兼顾 spill / runtime filter / shared state）搬到 FE，并改 thrift 契约携带完整 pipeline 结构——战线太长、风险太大。这套 FE 近似等价实现已经做了大半年、近 1 万行代码，没法一把全做。
所以现状是：FE 规划 LE 的**位置/类型/分布**（决定 plan 正确性，过去正是 BE/FE 分歧 bug 的根源），`enable_local_shuffle_planner=true` 时 BE 的 `_plan_local_exchange` 整个跳过；但每条 pipeline 的 `num_tasks` 仍由 BE 的 `_propagate_local_exchange_num_tasks` 拓扑推导，`_create_deferred_local_exchangers` 因为 exchanger 的 `num_partitions` 要等 `num_tasks` 定了才能建、所以延后创建。终态是 FE 连 `num_tasks` 也算好下发、BE 的 propagate 退化成 assert（只校验）——那是后续单独的大改。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3413421247
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -290,6 +290,32 @@ Status PipelineFragmentContext::_build_and_prepare_full_pipeline(ThreadPool* thr
RETURN_IF_ERROR(_build_pipelines(_runtime_state->obj_pool(), *_query_ctx->desc_tbl,
&_root_op, root_pipeline));
+ // Propagate _num_instances from LOCAL_EXCHANGE pipelines to ancestor pipelines
+ // that inherited reduced num_tasks from a serial operator.
+ _propagate_local_exchange_num_tasks();
+
+ // Create deferred local exchangers now that all pipelines have final num_tasks.
+ RETURN_IF_ERROR(_create_deferred_local_exchangers());
Review Comment:
是的，这两块不是为了灰度/回滚——是 FE 接管 local exchange 的**当前折中**，不是一步到位。
要让 FE 把"每个算子的并发"也规划好直接下发，本质上需要 FE 自己切 pipeline、自己控制每条 pipeline 的并发（num_tasks 是 pipeline 粒度的，只在 pipeline 边界即 local exchange / pipeline breaker 处变化），BE 就不再规划 pipeline 了。那等于把 BE 的整套 pipeline 构建（`_build_pipeline`、pipeline-breaker 判定、num_tasks 推导、dependency 接线，还要兼顾 spill / runtime filter / shared state）搬到 FE，并改 thrift 契约携带完整 pipeline 结构——战线太长、风险太大。这套 FE 近似等价实现已经做了大半年、近 1 万行代码，没法一把全做。
所以现状是：FE 规划 LE 的**位置/类型/分布**（决定 plan 正确性，过去正是 BE/FE 分歧 bug 的根源），`enable_local_shuffle_planner=true` 时 BE 的 `_plan_local_exchange` 整个跳过；但每条 pipeline 的 num_tasks 仍由 BE 的 `_propagate_local_exchange_num_tasks` 拓扑推导，`_create_deferred_local_exchangers` 因为 exchanger 的 `num_partitions` 要等 num_tasks 定了才能建、所以延后创建。终态是 FE 连 num_tasks 也算好下发、BE 的 propagate 退化成 assert（只校验）——那是后续单独的大改。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4765843210
# BE UT Coverage Report
Increment line coverage `15.27% (62/406)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b_1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b_1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 54.43% (21365/39252) |
| Line Coverage | 38.06% (204394/537052) |
| Region Coverage | 34.05% (160359/470985) |
| Branch Coverage | 35.05% (70209/200333) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4765205196
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4766186970
# BE UT Coverage Report
Increment line coverage `15.27% (62/406)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b_1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b_1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 54.42% (21361/39252) |
| Line Coverage | 38.02% (204203/537052) |
| Region Coverage | 34.03% (160296/470985) |
| Branch Coverage | 35.03% (70175/200333) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4739418360
# FE Regression Coverage Report
Increment line coverage ` 79.68% (451/566)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_0436cc4cb573b104a59408a6169bd145bcfeff6c_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_0436cc4cb573b104a59408a6169bd145bcfeff6c_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4733128023
<details>
<summary>TPC-H: <b>Total hot run time: 29850 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c2c51332d7f90c80ccabf77b27c23317d636a128, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17765 4236 4324 4236
q2 2005 317 190 190
q3 10891 1509 849 849
q4 4807 474 356 356
q5 8433 882 565 565
q6 334 182 142 142
q7 835 838 627 627
q8 10582 1759 1687 1687
q9 5768 4543 4561 4543
q10 6826 1853 1501 1501
q11 452 280 239 239
q12 650 426 300 300
q13 18145 3495 2787 2787
q14 274 272 238 238
q15 q16 784 787 710 710
q17 978 885 1030 885
q18 6979 5718 5620 5620
q19 1164 1351 1209 1209
q20 504 413 266 266
q21 5475 2714 2601 2601
q22 442 361 299 299
Total cold run time: 104093 ms
Total hot run time: 29850 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4553 4485 4433 4433
q2 330 360 231 231
q3 4677 5030 4559 4559
q4 2196 2292 1398 1398
q5 4599 4401 4882 4401
q6 264 208 147 147
q7 1981 1828 1620 1620
q8 2597 2286 2294 2286
q9 8170 8178 8110 8110
q10 5050 4800 4340 4340
q11 608 417 393 393
q12 747 752 552 552
q13 3320 3703 3003 3003
q14 314 317 292 292
q15 q16 749 773 716 716
q17 1430 1389 1369 1369
q18 8250 7234 6900 6900
q19 1112 1106 1151 1106
q20 2230 2265 1954 1954
q21 5450 4784 4600 4600
q22 541 460 409 409
Total cold run time: 59168 ms
Total hot run time: 52819 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4734095192
# BE Regression && UT Coverage Report
Increment line coverage `89.41% (363/406)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_c2c51332d7f90c80ccabf77b27c23317d636a128_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_c2c51332d7f90c80ccabf77b27c23317d636a128_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 72.32% (27676/38267) |
| Line Coverage | 55.73% (296935/532849) |
| Region Coverage | 52.55% (248183/472248) |
| Branch Coverage | 53.67% (107352/200029) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4738178778
# BE Regression && UT Coverage Report
Increment line coverage `89.41% (363/406)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_c2c51332d7f90c80ccabf77b27c23317d636a128_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_c2c51332d7f90c80ccabf77b27c23317d636a128_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 72.32% (27676/38267) |
| Line Coverage | 55.73% (296941/532849) |
| Region Coverage | 52.57% (248267/472248) |
| Branch Coverage | 53.67% (107358/200029) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4738192904
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4494902599
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4505410948
<details>
<summary>TPC-H: <b>Total hot run time: 31438 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f2d43a29db5822a72153fd88cbcd8bead33cbc6d, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17614 3875 3866 3866
q2 q3 10964 1519 811 811
q4 4816 474 354 354
q5 10645 2302 2071 2071
q6 386 179 135 135
q7 943 776 657 657
q8 9591 1759 1602 1602
q9 7226 5020 5003 5003
q10 6584 2094 1819 1819
q11 441 270 242 242
q12 646 434 295 295
q13 18133 3456 2831 2831
q14 266 264 235 235
q15 q16 817 792 713 713
q17 962 995 945 945
q18 6830 5827 5655 5655
q19 1200 1355 1104 1104
q20 524 407 259 259
q21 5586 2699 2529 2529
q22 447 363 312 312
Total cold run time: 104621 ms
Total hot run time: 31438 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4183 4080 4092 4080
q2 q3 4582 4943 4277 4277
q4 2125 2228 1378 1378
q5 4435 4259 5173 4259
q6 254 194 144 144
q7 1973 1755 1584 1584
q8 2385 2131 2082 2082
q9 7865 7824 7898 7824
q10 4606 4472 4098 4098
q11 592 408 384 384
q12 733 741 537 537
q13 3328 3628 2962 2962
q14 294 294 297 294
q15 q16 724 739 643 643
q17 1362 1309 1309 1309
q18 7928 7090 6700 6700
q19 1171 1096 1113 1096
q20 2264 2224 1955 1955
q21 5334 4625 4528 4528
q22 518 465 424 424
Total cold run time: 56656 ms
Total hot run time: 50558 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4506848943
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4485864440
# BE UT Coverage Report
Increment line coverage `11.11% (46/414)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/4affd222d6550dd103b7829a6fced0b7d45d317a_4affd222d6550dd103b7829a6fced0b7d45d317a/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/4affd222d6550dd103b7829a6fced0b7d45d317a_4affd222d6550dd103b7829a6fced0b7d45d317a/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.56% (20733/38712) |
| Line Coverage | 37.18% (196066/527339) |
| Region Coverage | 33.52% (153723/458547) |
| Branch Coverage | 34.55% (66975/193832) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4550835013
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4674684335
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4675427140
# FE UT Coverage Report
Increment line coverage ` 74.48% (429/576)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_f056ebbed42446ea468bcc6891c05520bdfb31c3/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_f056ebbed42446ea468bcc6891c05520bdfb31c3/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4675118864
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/f056ebbed42446ea468bcc6891c05520bdfb31c3_f056ebbed42446ea468bcc6891c05520bdfb31c3_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/f056ebbed42446ea468bcc6891c05520bdfb31c3_f056ebbed42446ea468bcc6891c05520bdfb31c3_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.39% (1915/2443) |
| Line Coverage | 64.90% (34216/52725) |
| Region Coverage | 65.33% (17604/26948) |
| Branch Coverage | 54.01% (9352/17316) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4669961175
# FE UT Coverage Report
Increment line coverage ` 72.79% (428/588)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_7c898c900506116cfb44b47189025b6fbfa7e44c/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_7c898c900506116cfb44b47189025b6fbfa7e44c/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3411912827
##########
be/src/exec/operator/partition_sort_sink_operator.h:
##########
@@ -95,9 +95,10 @@ class PartitionSortSinkOperatorX final : public DataSinkOperatorX<PartitionSortS
Status sink_impl(RuntimeState* state, Block* in_block, bool eos) override;
DataDistribution required_data_distribution(RuntimeState* /*state*/) const override {
if (_topn_phase == TPartTopNPhase::TWO_PHASE_GLOBAL) {
- return DataDistribution(ExchangeType::HASH_SHUFFLE, _distribute_exprs);
+ return DataDistribution(TLocalPartitionType::GLOBAL_EXECUTION_HASH_SHUFFLE,
Review Comment:
why all HASH_SHUFFLE update to `GLOBAL_EXECUTION_HASH_SHUFFLE`? When will `LOCAL_EXECUTION_HASH_SHUFFLE` be generated?
##########
be/src/exec/pipeline/pipeline.h:
##########
@@ -68,16 +68,15 @@ class Pipeline : public std::enable_shared_from_this<Pipeline> {
[[nodiscard]] PipelineId id() const { return _pipeline_id; }
- static bool is_hash_exchange(ExchangeType idx) {
- return idx == ExchangeType::HASH_SHUFFLE || idx == ExchangeType::BUCKET_HASH_SHUFFLE;
+ static bool is_hash_exchange(TLocalPartitionType::type idx) {
+ return is_shuffled_exchange(idx);
}
// For HASH_SHUFFLE, BUCKET_HASH_SHUFFLE, and ADAPTIVE_PASSTHROUGH,
Review Comment:
need update this comment
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -810,52 +1018,59 @@ Status PipelineFragmentContext::_add_local_exchange_impl(
sink_id, local_exchange_id, use_global_hash_shuffle ? _total_instances : _num_instances,
data_distribution.partition_exprs, bucket_seq_to_instance_idx);
if (bucket_seq_to_instance_idx.empty() &&
- data_distribution.distribution_type == ExchangeType::BUCKET_HASH_SHUFFLE) {
- data_distribution.distribution_type = ExchangeType::HASH_SHUFFLE;
+ data_distribution.distribution_type == TLocalPartitionType::BUCKET_HASH_SHUFFLE) {
+ data_distribution.distribution_type =
+ use_global_hash_shuffle ? TLocalPartitionType::GLOBAL_EXECUTION_HASH_SHUFFLE
+ : TLocalPartitionType::LOCAL_EXECUTION_HASH_SHUFFLE;
+ }
+ if (!use_global_hash_shuffle &&
+ data_distribution.distribution_type == TLocalPartitionType::GLOBAL_EXECUTION_HASH_SHUFFLE) {
+ data_distribution.distribution_type = TLocalPartitionType::LOCAL_EXECUTION_HASH_SHUFFLE;
Review Comment:
so the operator's GLOBAL_EXECUTION_HASH_SHUFFLE will be updated according to `use_global_hash_shuffle` here?
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -690,6 +716,188 @@ Status PipelineFragmentContext::_build_pipelines(ObjectPool* pool, const Descrip
return Status::OK();
}
+Status PipelineFragmentContext::_create_deferred_local_exchangers() {
+ for (auto& info : _deferred_exchangers) {
+ // DANGER ZONE — do not "fix" this line without reading the history.
+ //
+ // sender_count seeds Exchanger::_running_sink_operators, which the source side
+ // waits to reach 0 via sub_running_sink_operators on each sink LocalState close.
+ // The correct value is THIS pipeline-instance's sink task count, which is exactly
+ // info.upstream_pipe->num_tasks() — one PipelineTask per task, one close per task.
+ //
+ // Tempting wrong fix #1: `std::max(num_tasks, _num_instances)` to mirror the
+ // BE-planned path in _add_local_exchange_impl (~line 1023). THIS BREAKS the
+ // common FE-planned shape of `serial scan → LE(PT) → ...`: upstream_pipe
+ // genuinely has num_tasks=1, only 1 close arrives, but seed becomes
+ // _num_instances so _running_sink_operators never reaches 0 — downstream
+ // sources hang on SHUFFLE_DATA_DEPENDENCY (e.g. MTMV refresh from
+ // mtmv_up_down_job_p0/load.groovy stays at Status=RUNNING; build 949402
+ // regressed exactly this way). BE-planned mode uses max() because its
+ // `cur_pipe` is the source-side pipeline (always raised to _num_instances by
+ // add_pipeline) — not analogous to our `upstream_pipe` here, which is the
+ // sink-side pipeline that may legitimately stay at 1 for serial sources.
+ //
+ // Tempting wrong fix #2: multiply by _num_instances on the theory shared_state
+ // is shared across all instances. Same hang — each fragment-instance
+ // PipelineFragmentContext has its OWN _op_id_to_shared_state map, so the
+ // exchanger is per-instance, not per-BE. num_tasks() is already the right
+ // close-count for one instance.
+ //
+ // If a hang shows up with `_running_sink_operators < 0`, the bug is upstream:
+ // _propagate_local_exchange_num_tasks left num_tasks too low (or too high) for
+ // this fragment shape. Fix THAT pass, not this seed value.
+ const int sender_count = info.upstream_pipe->num_tasks();
+ switch (info.partition_type) {
+ case TLocalPartitionType::LOCAL_EXECUTION_HASH_SHUFFLE:
+ case TLocalPartitionType::GLOBAL_EXECUTION_HASH_SHUFFLE:
+ info.shared_state->exchanger = ShuffleExchanger::create_unique(
+ sender_count, _num_instances, info.num_partitions, info.free_blocks_limit,
+ info.partition_type);
+ break;
+ case TLocalPartitionType::BUCKET_HASH_SHUFFLE:
+ info.shared_state->exchanger = BucketShuffleExchanger::create_unique(
+ sender_count, _num_instances, info.num_partitions, info.free_blocks_limit);
+ break;
+ case TLocalPartitionType::PASSTHROUGH:
+ info.shared_state->exchanger = PassthroughExchanger::create_unique(
+ sender_count, _num_instances, info.free_blocks_limit);
+ break;
+ case TLocalPartitionType::BROADCAST:
+ info.shared_state->exchanger = BroadcastExchanger::create_unique(
+ sender_count, _num_instances, info.free_blocks_limit);
+ break;
+ case TLocalPartitionType::PASS_TO_ONE:
+ if (_runtime_state->enable_share_hash_table_for_broadcast_join()) {
+ info.shared_state->exchanger = PassToOneExchanger::create_unique(
+ sender_count, _num_instances, info.free_blocks_limit);
+ } else {
+ info.shared_state->exchanger = BroadcastExchanger::create_unique(
+ sender_count, _num_instances, info.free_blocks_limit);
+ }
+ break;
+ case TLocalPartitionType::ADAPTIVE_PASSTHROUGH:
+ info.shared_state->exchanger = AdaptivePassthroughExchanger::create_unique(
+ sender_count, _num_instances, info.free_blocks_limit);
+ break;
+ case TLocalPartitionType::NOOP:
+ case TLocalPartitionType::LOCAL_MERGE_SORT:
Review Comment:
so, this is a regression for plan_by_fe? what is the reason?
##########
be/src/exec/pipeline/dependency.h:
##########
@@ -832,50 +832,44 @@ struct SetSharedState : public BasicSharedState {
Status hash_table_init();
};
-enum class ExchangeType : uint8_t {
- NOOP = 0,
- // Shuffle data by Crc32CHashPartitioner
- HASH_SHUFFLE = 1,
- // Round-robin passthrough data blocks.
- PASSTHROUGH = 2,
- // Shuffle data by Crc32HashPartitioner<ShuffleChannelIds> (e.g. same as storage engine).
- BUCKET_HASH_SHUFFLE = 3,
- // Passthrough data blocks to all channels.
- BROADCAST = 4,
- // Passthrough data to channels evenly in an adaptive way.
- ADAPTIVE_PASSTHROUGH = 5,
- // Send all data to the first channel.
- PASS_TO_ONE = 6,
-};
+inline bool is_shuffled_exchange(TLocalPartitionType::type idx) {
+ return idx == TLocalPartitionType::GLOBAL_EXECUTION_HASH_SHUFFLE ||
+ idx == TLocalPartitionType::LOCAL_EXECUTION_HASH_SHUFFLE ||
+ idx == TLocalPartitionType::BUCKET_HASH_SHUFFLE;
+}
Review Comment:
same function in `operator.cpp`?
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -723,7 +931,7 @@ Status PipelineFragmentContext::_create_tree_helper(
*root = op;
}
/**
- * `ExchangeType::HASH_SHUFFLE` should be used if an operator is followed by a shuffled operator (shuffled hash join, union operator followed by co-located operators).
+ * `TLocalPartitionType::HASH_SHUFFLE` should be used if an operator is followed by a shuffled operator (shuffled hash join, union operator followed by co-located operators).
Review Comment:
HASH_SHUFFLE no longer existed, update the comment
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -690,6 +716,188 @@ Status PipelineFragmentContext::_build_pipelines(ObjectPool* pool, const Descrip
return Status::OK();
}
+Status PipelineFragmentContext::_create_deferred_local_exchangers() {
+ for (auto& info : _deferred_exchangers) {
+ // DANGER ZONE — do not "fix" this line without reading the history.
+ //
+ // sender_count seeds Exchanger::_running_sink_operators, which the source side
+ // waits to reach 0 via sub_running_sink_operators on each sink LocalState close.
+ // The correct value is THIS pipeline-instance's sink task count, which is exactly
+ // info.upstream_pipe->num_tasks() — one PipelineTask per task, one close per task.
+ //
+ // Tempting wrong fix #1: `std::max(num_tasks, _num_instances)` to mirror the
+ // BE-planned path in _add_local_exchange_impl (~line 1023). THIS BREAKS the
+ // common FE-planned shape of `serial scan → LE(PT) → ...`: upstream_pipe
+ // genuinely has num_tasks=1, only 1 close arrives, but seed becomes
+ // _num_instances so _running_sink_operators never reaches 0 — downstream
+ // sources hang on SHUFFLE_DATA_DEPENDENCY (e.g. MTMV refresh from
+ // mtmv_up_down_job_p0/load.groovy stays at Status=RUNNING; build 949402
Review Comment:
the comment contains meaningless build id 949402
##########
be/src/exec/operator/partitioned_hash_join_probe_operator.h:
##########
@@ -228,16 +228,8 @@ class PartitionedHashJoinProbeOperatorX final
Status pull(doris::RuntimeState* state, Block* output_block, bool* eos) const override;
bool need_more_input_data(RuntimeState* state) const override;
- DataDistribution required_data_distribution(RuntimeState* /*state*/) const override {
- if (_join_op == TJoinOp::NULL_AWARE_LEFT_ANTI_JOIN) {
- return {ExchangeType::NOOP};
- }
- return (_join_distribution == TJoinDistributionType::BUCKET_SHUFFLE ||
- _join_distribution == TJoinDistributionType::COLOCATE
- ? DataDistribution(ExchangeType::BUCKET_HASH_SHUFFLE,
- _distribution_partition_exprs)
- : DataDistribution(ExchangeType::HASH_SHUFFLE,
- _distribution_partition_exprs));
+ DataDistribution required_data_distribution(RuntimeState* state) const override {
+ return _inner_probe_operator->required_data_distribution(state);
Review Comment:
this is a optimization?
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -1796,6 +2011,89 @@ Status PipelineFragmentContext::_create_operator(ObjectPool* pool, const TPlanNo
RETURN_IF_ERROR(cur_pipe->add_operator(op, _parallel_instances));
break;
}
+ case TPlanNodeType::LOCAL_EXCHANGE_NODE: {
+ op = std::make_shared<LocalExchangeSourceOperatorX>(pool, tnode, next_operator_id(), descs);
+ // The downstream pipeline (containing LocalExchangeSource) must have
+ // _num_instances tasks — matching BE-native _inherit_pipeline_properties
+ // which sets pipe_with_source.set_num_tasks(_num_instances).
+ // Without this, when the parent pipeline was reduced by a serial operator
+ // (e.g., serial Exchange with use_serial_exchange=true, or UNPARTITIONED
+ // Exchange), the downstream inherits the reduced num_tasks via
+ // add_pipeline(parent). The deferred exchanger creates _num_instances
+ // channels but only fewer source tasks initialize mem_counters — the
+ // sink round-robins to all channels and crashes on uninitialized ones.
+ auto downstream_num_tasks = _num_instances;
Review Comment:
`downstream_num_tasks` is an unnecessary local variable.
##########
regression-test/suites/nereids_p0/local_shuffle/test_local_shuffle_rqg_bugs.groovy:
##########
@@ -0,0 +1,1560 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/**
+ * Regression tests for bugs discovered by RQG testing on the local-exchange2 branch.
+ *
+ * These queries triggered "must set shared state" errors or incorrect results
+ * in RQG build 183992. Common conditions:
+ * - use_serial_exchange=true (makes ALL Exchanges serial, not just UNPARTITIONED)
+ * - enable_local_shuffle_planner=true (FE-planned local exchange)
+ * - parallel_pipeline_task_num > 1
+ *
+ * Error types reproduced:
+ * 1. must set shared state, in AGGREGATION_OPERATOR
+ * 2. must set shared state, in SORT_OPERATOR
+ * 3. incorrect results with GROUPING SETS + scalar subquery + window function
+ */
+suite("test_local_shuffle_rqg_bugs") {
+
+ // ============================================================
+ // Table setup — mirrors RQG table structure
+ // 10 buckets to match RQG (replication_num=1 for single-BE testing)
+ // ============================================================
+ sql "DROP TABLE IF EXISTS rqg_t1"
+ sql "DROP TABLE IF EXISTS rqg_t2"
+ sql "DROP TABLE IF EXISTS rqg_t3"
+ sql "DROP TABLE IF EXISTS rqg_t4"
+
+ sql """
+ CREATE TABLE rqg_t1 (
+ pk INT NOT NULL,
+ col_int_undef_signed INT,
+ col_int_undef_signed2 INT,
+ col_int_undef_signed_not_null INT NOT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ sql """
+ CREATE TABLE rqg_t2 (
+ pk INT NOT NULL,
+ col_int_undef_signed INT,
+ col_int_undef_signed2 INT,
+ col_bigint_undef_signed_not_null BIGINT NOT NULL,
+ col_decimal_38_10__undef_signed_not_null DECIMAL(38,10) NOT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ // Table for build 184181 GLOBAL_HASH_SHUFFLE bugs — needs varchar + bigint columns
+ sql """
+ CREATE TABLE rqg_t3 (
+ pk INT NOT NULL,
+ col_bigint_undef_signed BIGINT,
+ col_varchar_10__undef_signed VARCHAR(10),
+ col_varchar_64__undef_signed VARCHAR(64)
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ // Second table for FULL OUTER JOIN case (col_bigint_undef_signed_not_null)
+ sql """
+ CREATE TABLE rqg_t4 (
+ pk INT NOT NULL,
+ col_bigint_undef_signed_not_null BIGINT NOT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ sql """
+ INSERT INTO rqg_t3 VALUES
+ (0, -94, 'Abc', 'hello world'),
+ (1, 672609, 'Xyz', null),
+ (2, -3766684, 'Pqr', 'test string'),
+ (3, 5070261, 'abc', 'another row'),
+ (4, null, 'def', 'value four'),
+ (5, -86, 'XgpxlHBLEM', null),
+ (6, 21910, 'abc', 'they'),
+ (7, -63, 'zzzz', 'some text'),
+ (8, -8276281, 'AHlvNtoGLO', 'longer string here'),
+ (9, -101, 'mid', 'final row')
+ """
+
+ sql """
+ INSERT INTO rqg_t4 VALUES
+ (0, 0), (1, 1), (2, 2), (3, 3), (4, 4),
+ (5, 5), (6, 6), (7, 7), (8, 8), (9, 9),
+ (10, 2), (11, 2), (12, 2), (13, 3), (14, 4),
+ (15, 5), (16, 2), (17, 2), (18, 2), (19, 9)
+ """
+
+ // Insert enough rows to exercise multiple pipeline tasks
+ sql """
+ INSERT INTO rqg_t1 VALUES
+ (0, 0, 10, 0), (1, 1, 11, 1), (2, 2, 12, 2), (3, 3, 13, 3),
+ (4, 4, 14, 4), (5, 5, 15, 5), (6, 6, 16, 6), (7, 7, 17, 7),
+ (8, 8, 18, 8), (9, 9, 19, 9), (10, 0, 20, 10), (11, 1, 21, 11),
+ (12, 2, 22, 12), (13, 3, 23, 13), (14, 4, 24, 14), (15, 5, 25, 15),
+ (16, 6, 26, 16), (17, 7, 27, 17), (18, 8, 28, 18), (19, 9, 29, 19)
+ """
+
+ sql """
+ INSERT INTO rqg_t2 VALUES
+ (0, 0, 10, 100, 1.5), (1, 1, 11, 101, 2.5), (2, 2, 12, 102, 3.5),
+ (3, 3, 13, 103, 4.5), (4, 4, 14, 104, 5.5), (5, 5, 15, 105, 6.5),
+ (6, 6, 16, 106, 7.5), (7, 7, 17, 107, 8.5), (8, 8, 18, 108, 9.5),
+ (9, 9, 19, 109, 10.5), (10, 0, 20, 110, 11.5), (11, 1, 21, 111, 12.5),
+ (12, 2, 22, 112, 13.5), (13, 3, 23, 113, 14.5), (14, 4, 24, 114, 15.5),
+ (15, 5, 25, 115, 16.5), (16, 6, 26, 116, 17.5), (17, 7, 27, 117, 18.5),
+ (18, 8, 28, 118, 19.5), (19, 9, 29, 119, 20.5)
+ """
+
+ // Wait for data to be visible
+ Thread.sleep(5000)
Review Comment:
Use a check instead of the unstable sleep
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3412980337
##########
fe/fe-core/src/main/java/org/apache/doris/planner/AssertNumRowsNode.java:
##########
@@ -78,7 +84,17 @@ public int getNumInstances() {
}
@Override
- public boolean isSerialOperator() {
+ public boolean isSerialNode() {
return true;
}
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(PlanTranslatorContext translatorContext,
+ PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+
+ Pair<PlanNode, LocalExchangeType> enforceResult = enforceRequire(
+ translatorContext, children.get(0), 0, LocalExchangeTypeRequire.requirePassthrough());
Review Comment:
i think it should require NOOP? because it only produce one row
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4707471189
<details>
<summary>TPC-DS: <b>Total hot run time: 168358 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1d7cf6dbd6095ee1fb5860efe258b0ce99242ae6, data reload: false
query5 4344 639 493 493
query6 431 186 172 172
query7 4869 542 299 299
query8 355 210 191 191
query9 8726 4054 4040 4040
query10 454 321 258 258
query11 5915 2327 2133 2133
query12 156 101 98 98
query13 1270 627 459 459
query14 6437 5385 5018 5018
query14_1 4347 4331 4331 4331
query15 200 192 171 171
query16 1004 433 443 433
query17 1087 668 544 544
query18 2428 470 332 332
query19 196 175 140 140
query20 107 104 103 103
query21 216 138 115 115
query22 13596 13558 13306 13306
query23 17424 16590 16286 16286
query23_1 16218 16211 16282 16211
query24 7554 1748 1287 1287
query24_1 1313 1304 1318 1304
query25 565 460 400 400
query26 1319 303 174 174
query27 2692 596 326 326
query28 4476 2071 2005 2005
query29 1104 626 492 492
query30 310 236 196 196
query31 1117 1086 994 994
query32 105 64 60 60
query33 539 331 267 267
query34 1175 1153 654 654
query35 756 780 693 693
query36 1387 1414 1231 1231
query37 157 104 90 90
query38 3167 3120 3053 3053
query39 929 931 912 912
query39_1 893 885 862 862
query40 244 128 105 105
query41 70 69 66 66
query42 97 94 96 94
query43 323 320 290 290
query44
query45 199 189 183 183
query46 1100 1190 730 730
query47 2430 2414 2295 2295
query48 410 406 304 304
query49 650 484 367 367
query50 1031 341 264 264
query51 4463 4465 4407 4407
query52 90 97 78 78
query53 280 261 183 183
query54 252 208 198 198
query55 77 75 70 70
query56 233 233 205 205
query57 1448 1418 1321 1321
query58 248 213 205 205
query59 1590 1613 1384 1384
query60 275 246 227 227
query61 169 160 148 148
query62 706 670 590 590
query63 224 181 189 181
query64 2527 754 584 584
query65
query66 1780 467 343 343
query67 29845 29590 29610 29590
query68
query69 434 301 262 262
query70 954 937 947 937
query71 294 227 220 220
query72 2907 2549 2325 2325
query73 836 731 418 418
query74 5117 4919 4764 4764
query75 2668 2565 2223 2223
query76 2296 1152 776 776
query77 344 392 287 287
query78 12444 12376 11848 11848
query79 1284 1157 753 753
query80 513 462 379 379
query81 451 292 241 241
query82 234 163 122 122
query83 271 274 250 250
query84
query85 823 499 418 418
query86 336 284 277 277
query87 3389 3361 3168 3168
query88 3607 2742 2726 2726
query89 415 379 331 331
query90 2155 196 174 174
query91 173 157 134 134
query92 61 61 59 59
query93 1447 1453 849 849
query94 536 372 297 297
query95 675 466 353 353
query96 1041 806 329 329
query97 2746 2725 2606 2606
query98 218 207 205 205
query99 1163 1191 1034 1034
Total cold run time: 250415 ms
Total hot run time: 168358 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3413115610
##########
be/src/udf/python/boost_process_compat.h:
##########
@@ -0,0 +1,28 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
Review Comment:
这些无关的改动能不能单独抽出来单独提
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -290,6 +290,32 @@ Status PipelineFragmentContext::_build_and_prepare_full_pipeline(ThreadPool* thr
RETURN_IF_ERROR(_build_pipelines(_runtime_state->obj_pool(), *_query_ctx->desc_tbl,
&_root_op, root_pipeline));
+ // Propagate _num_instances from LOCAL_EXCHANGE pipelines to ancestor pipelines
+ // that inherited reduced num_tasks from a serial operator.
+ _propagate_local_exchange_num_tasks();
+
+ // Create deferred local exchangers now that all pipelines have final num_tasks.
+ RETURN_IF_ERROR(_create_deferred_local_exchangers());
Review Comment:
我没懂如果fe一开始就把所有local exchange规划好，所有算子的并行度规划好，直接下发。be为什么需要_create_deferred_local_exchangers和_propagate_local_exchange_num_tasks？
这个是现在设计出来就是要先做折中方案的，还是可以直接一把做全面？_propagate_local_exchange_num_tasks和_create_deferred_local_exchangers这两块新增的应该不是为了灰度/回滚吧。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4707892364
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4477407204
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4480171001
# FE Regression Coverage Report
Increment line coverage ` 63.93% (468/732)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_15d92ba024b265d9441145d7bd26d93590aa9c63_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_15d92ba024b265d9441145d7bd26d93590aa9c63_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4478568671
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4486711226
# BE UT Coverage Report
Increment line coverage `11.11% (46/414)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/287ef124f24ef08527f79e648ccd85f5b338d59d_287ef124f24ef08527f79e648ccd85f5b338d59d/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/287ef124f24ef08527f79e648ccd85f5b338d59d_287ef124f24ef08527f79e648ccd85f5b338d59d/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.59% (20747/38712) |
| Line Coverage | 37.21% (196215/527345) |
| Region Coverage | 33.52% (153718/458547) |
| Branch Coverage | 34.57% (67007/193832) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4496367272
# BE UT Coverage Report
Increment line coverage `11.11% (46/414)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/ef4ea66a66f075dfb3259a519599ed361cd2d371_ef4ea66a66f075dfb3259a519599ed361cd2d371/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/ef4ea66a66f075dfb3259a519599ed361cd2d371_ef4ea66a66f075dfb3259a519599ed361cd2d371/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.56% (20734/38712) |
| Line Coverage | 37.19% (196132/527348) |
| Region Coverage | 33.49% (153587/458553) |
| Branch Coverage | 34.56% (66986/193834) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4504737237
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3413517236
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -290,6 +290,32 @@ Status PipelineFragmentContext::_build_and_prepare_full_pipeline(ThreadPool* thr
RETURN_IF_ERROR(_build_pipelines(_runtime_state->obj_pool(), *_query_ctx->desc_tbl,
&_root_op, root_pipeline));
+ // Propagate _num_instances from LOCAL_EXCHANGE pipelines to ancestor pipelines
+ // that inherited reduced num_tasks from a serial operator.
+ _propagate_local_exchange_num_tasks();
+
+ // Create deferred local exchangers now that all pipelines have final num_tasks.
+ RETURN_IF_ERROR(_create_deferred_local_exchangers());
Review Comment:
是的，这两块不是为了灰度/回滚——是 FE 接管 local exchange 的**当前折中**，不是一步到位。
要让 FE 把"每个算子的并发"也规划好直接下发，本质上需要 FE 自己切 pipeline、自己控制每条 pipeline 的并发（num_tasks 是 pipeline 粒度的，只在 pipeline 边界即 local exchange / pipeline breaker 处变化），BE 就不再规划 pipeline 了。那等于把 BE 的整套 pipeline 构建（`_build_pipeline`、pipeline-breaker 判定、num_tasks 推导、dependency 接线，还要兼顾 spill / runtime filter / shared state）搬到 FE，并改 thrift 契约携带完整 pipeline 结构——战线太长、风险太大。这套 FE 近似等价实现已经做了大半年、近 1 万行代码，没法一把全做。
所以现状是：FE 规划 LE 的**位置/类型/分布**（决定 plan 正确性，过去正是 BE/FE 分歧 bug 的根源），`enable_local_shuffle_planner=true` 时 BE 的 `_plan_local_exchange` 整个跳过；但每条 pipeline 的 num_tasks 仍由 BE 的 `_propagate_local_exchange_num_tasks` 拓扑推导，`_create_deferred_local_exchangers` 因为 exchanger 的 `num_partitions` 要等 num_tasks 定了才能建、所以延后创建。终态是 FE 连 num_tasks 也算好下发、BE 的 propagate 退化成 assert（只校验）——那是后续单独的大改。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3372721774
##########
fe/fe-core/src/main/java/org/apache/doris/planner/AggregationNode.java:
##########
@@ -258,4 +268,155 @@ public boolean isQueryCacheCandidate() {
public void setQueryCacheCandidate(boolean queryCacheCandidate) {
this.queryCacheCandidate = queryCacheCandidate;
}
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+
+ ConnectContext connectContext = translatorContext.getConnectContext();
+ SessionVariable sessionVariable = connectContext.getSessionVariable();
+ // PR #62438: when false, non-finalize agg falls back to BE base class.
+ boolean enableLeBeforeAgg = sessionVariable.enableLocalExchangeBeforeAgg;
+ boolean hasKeys = !aggInfo.getGroupingExprs().isEmpty();
Review Comment:
**✅ Correctness fix**: The `!needsFinalize && !aggInfo.isMerge() && !enableLeBeforeAgg` guard correctly separates FIRST_MERGE (correctness-required, always HASH) from LOCAL phase (performance-only, flag controls). This diverges from BE's `!_needs_finalize && !enable_local_exchange_before_agg` early-return which conflates both intents — the FE planner is actually *more correct* here than BE. For the backward-compat path (legacy BE planner); consider adding a comment noting this divergence explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4708103838
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/fe9a3316be056faed18351cf64096e36790c6c89_fe9a3316be056faed18351cf64096e36790c6c89_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/fe9a3316be056faed18351cf64096e36790c6c89_fe9a3316be056faed18351cf64096e36790c6c89_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.32% (1889/2443) |
| Line Coverage | 64.41% (33961/52725) |
| Region Coverage | 64.80% (17462/26948) |
| Branch Coverage | 53.93% (9339/17316) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3411861875
##########
fe/fe-core/src/main/java/org/apache/doris/planner/AggregationNode.java:
##########
@@ -258,4 +268,155 @@ public boolean isQueryCacheCandidate() {
public void setQueryCacheCandidate(boolean queryCacheCandidate) {
this.queryCacheCandidate = queryCacheCandidate;
}
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+
+ ConnectContext connectContext = translatorContext.getConnectContext();
+ SessionVariable sessionVariable = connectContext.getSessionVariable();
+ // PR #62438: when false, non-finalize agg falls back to BE base class.
+ boolean enableLeBeforeAgg = sessionVariable.enableLocalExchangeBeforeAgg;
+ boolean hasKeys = !aggInfo.getGroupingExprs().isEmpty();
Review Comment:
Thanks. The divergence is called out in the code comment on `AggregationNode.enforceAndDeriveLocalExchange`: *"Diverges from BE: BE's `!_needs_finalize && !enable_local_exchange_before_agg` early return catches non-streaming dedup too, causing the same family of wrong-result bug as AggSink (DORIS-25413)."* The FE path separates FIRST_MERGE (always HASH, correctness) from the perf-only LOCAL phase; the legacy BE-planner path still carries DORIS-25413.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java:
##########
@@ -285,4 +302,71 @@ public List<Expr> getOtherJoinConjuncts() {
public List<Expr> getMarkJoinConjuncts() {
return markJoinConjuncts;
}
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+
+ LocalExchangeTypeRequire probeSideRequire;
+ LocalExchangeTypeRequire buildSideRequire;
+ LocalExchangeType outputType = null;
+
+ if (joinOp == JoinOperator.NULL_AWARE_LEFT_ANTI_JOIN) {
+ buildSideRequire = probeSideRequire = LocalExchangeTypeRequire.noRequire();
+ outputType = LocalExchangeType.NOOP;
+ } else if (distrMode == DistributionMode.BROADCAST) {
+ // BE: _child->is_serial_operator() ? PASSTHROUGH/PASS_TO_ONE : NOOP
+ boolean probeChildSerial = children.get(0).isSerialOperatorOnBe(
+ translatorContext.getConnectContext());
+ boolean buildChildSerial = children.get(1).isSerialOperatorOnBe(
+ translatorContext.getConnectContext());
+ probeSideRequire = probeChildSerial
+ ? LocalExchangeTypeRequire.requirePassthrough()
+ : LocalExchangeTypeRequire.noRequire();
+ buildSideRequire = buildChildSerial
+ ? LocalExchangeTypeRequire.requirePassToOne()
+ : LocalExchangeTypeRequire.noRequire();
+ // For serial probe: output is PASSTHROUGH (data from single instance).
+ // For non-serial probe: propagate probe side's actual distribution.
+ outputType = probeChildSerial ? LocalExchangeType.PASSTHROUGH : null;
+ } else if (isColocate() || isBucketShuffle()) {
+ probeSideRequire = LocalExchangeTypeRequire.requireBucketHash();
+ // For BUCKET_SHUFFLE with serial build child: use requireBucketHash() (not
+ // requirePassToOne()). Unlike BROADCAST joins, BUCKET_SHUFFLE has no shared
+ // hash table mechanism — PASS_TO_ONE routes all data to task 0 while tasks 1..N-1
+ // build empty hash tables, losing rows. BUCKET_HASH_SHUFFLE correctly distributes
+ // build data by bucket to match the probe side's bucket distribution.
+ // The serial exchange returns NOOP, so enforceRequire() will insert a
+ // BUCKET_HASH_SHUFFLE local exchange (with PASSTHROUGH fan-out for heavy-ops
Review Comment:
Added — the colocate/bucket-shuffle branch now notes that both probe and build sides require `BUCKET_HASH_SHUFFLE`, and that a serial build child is handled by the serial exchange returning NOOP → `enforceRequire` inserting the `BUCKET_HASH_SHUFFLE` LE (with PASSTHROUGH fan-out for the heavy-ops bottleneck). See `HashJoinNode` ~346-360.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/TVFScanNode.java:
##########
@@ -212,4 +217,10 @@ protected void setScanParams(TFileRangeDesc rangeDesc, Split split) {
public int getNumInstances() {
return scanRangeLocations.size();
}
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(PlanTranslatorContext translatorContext,
+ PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ return Pair.of(this, LocalExchangeType.PASSTHROUGH);
+ }
Review Comment:
Fixed in `8db568b5100` — and a bit more thoroughly than the suggestion: instead of returning `NOOP`, I removed the `enforceAndDeriveLocalExchange` override on `TVFScanNode` entirely (+ its 5 now-orphaned imports). The base `PlanNode.enforceAndDeriveLocalExchange()` already returns `Pair.of(this, NOOP)` for a leaf, so the scan no longer claims to satisfy a parent `requirePassthrough()`. A `TVFScan -> merge-sort / AssertNumRows / TableFunction`, or a force-passthrough broadcast-join probe, now gets a real `LocalExchangeNode` from `enforceRequire()`, matching BE's `Pipeline::need_to_local_exchange()` for non-hash targets. Thanks for catching this.
##########
regression-test/suites/nereids_p0/local_shuffle/test_local_shuffle_rqg_bugs.groovy:
##########
@@ -0,0 +1,1561 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/**
+ * Regression tests for bugs discovered by RQG testing on the local-exchange2 branch.
+ *
+ * These queries triggered "must set shared state" errors or incorrect results
+ * in RQG build 183992. Common conditions:
+ * - use_serial_exchange=true (makes ALL Exchanges serial, not just UNPARTITIONED)
+ * - enable_local_shuffle_planner=true (FE-planned local exchange)
+ * - parallel_pipeline_task_num > 1
+ *
+ * Error types reproduced:
+ * 1. must set shared state, in AGGREGATION_OPERATOR
+ * 2. must set shared state, in SORT_OPERATOR
+ * 3. incorrect results with GROUPING SETS + scalar subquery + window function
+ */
+suite("test_local_shuffle_rqg_bugs") {
+
+ // ============================================================
+ // Table setup — mirrors RQG table structure
+ // 10 buckets to match RQG (replication_num=1 for single-BE testing)
+ // ============================================================
+ sql "DROP TABLE IF EXISTS rqg_t1"
+ sql "DROP TABLE IF EXISTS rqg_t2"
+ sql "DROP TABLE IF EXISTS rqg_t3"
+ sql "DROP TABLE IF EXISTS rqg_t4"
+
+ sql """
+ CREATE TABLE rqg_t1 (
+ pk INT NOT NULL,
+ col_int_undef_signed INT,
+ col_int_undef_signed2 INT,
+ col_int_undef_signed_not_null INT NOT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ sql """
+ CREATE TABLE rqg_t2 (
+ pk INT NOT NULL,
+ col_int_undef_signed INT,
+ col_int_undef_signed2 INT,
+ col_bigint_undef_signed_not_null BIGINT NOT NULL,
+ col_decimal_38_10__undef_signed_not_null DECIMAL(38,10) NOT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ // Table for build 184181 GLOBAL_HASH_SHUFFLE bugs — needs varchar + bigint columns
+ sql """
+ CREATE TABLE rqg_t3 (
+ pk INT NOT NULL,
+ col_bigint_undef_signed BIGINT,
+ col_varchar_10__undef_signed VARCHAR(10),
+ col_varchar_64__undef_signed VARCHAR(64)
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ // Second table for FULL OUTER JOIN case (col_bigint_undef_signed_not_null)
+ sql """
+ CREATE TABLE rqg_t4 (
+ pk INT NOT NULL,
+ col_bigint_undef_signed_not_null BIGINT NOT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ sql """
+ INSERT INTO rqg_t3 VALUES
+ (0, -94, 'Abc', 'hello world'),
+ (1, 672609, 'Xyz', null),
+ (2, -3766684, 'Pqr', 'test string'),
+ (3, 5070261, 'abc', 'another row'),
+ (4, null, 'def', 'value four'),
+ (5, -86, 'XgpxlHBLEM', null),
+ (6, 21910, 'abc', 'they'),
+ (7, -63, 'zzzz', 'some text'),
+ (8, -8276281, 'AHlvNtoGLO', 'longer string here'),
+ (9, -101, 'mid', 'final row')
+ """
+
+ sql """
+ INSERT INTO rqg_t4 VALUES
+ (0, 0), (1, 1), (2, 2), (3, 3), (4, 4),
+ (5, 5), (6, 6), (7, 7), (8, 8), (9, 9),
+ (10, 2), (11, 2), (12, 2), (13, 3), (14, 4),
+ (15, 5), (16, 2), (17, 2), (18, 2), (19, 9)
+ """
+
+ // Insert enough rows to exercise multiple pipeline tasks
+ sql """
+ INSERT INTO rqg_t1 VALUES
+ (0, 0, 10, 0), (1, 1, 11, 1), (2, 2, 12, 2), (3, 3, 13, 3),
+ (4, 4, 14, 4), (5, 5, 15, 5), (6, 6, 16, 6), (7, 7, 17, 7),
+ (8, 8, 18, 8), (9, 9, 19, 9), (10, 0, 20, 10), (11, 1, 21, 11),
+ (12, 2, 22, 12), (13, 3, 23, 13), (14, 4, 24, 14), (15, 5, 25, 15),
+ (16, 6, 26, 16), (17, 7, 27, 17), (18, 8, 28, 18), (19, 9, 29, 19)
+ """
+
+ sql """
+ INSERT INTO rqg_t2 VALUES
+ (0, 0, 10, 100, 1.5), (1, 1, 11, 101, 2.5), (2, 2, 12, 102, 3.5),
+ (3, 3, 13, 103, 4.5), (4, 4, 14, 104, 5.5), (5, 5, 15, 105, 6.5),
+ (6, 6, 16, 106, 7.5), (7, 7, 17, 107, 8.5), (8, 8, 18, 108, 9.5),
+ (9, 9, 19, 109, 10.5), (10, 0, 20, 110, 11.5), (11, 1, 21, 111, 12.5),
+ (12, 2, 22, 112, 13.5), (13, 3, 23, 113, 14.5), (14, 4, 24, 114, 15.5),
+ (15, 5, 25, 115, 16.5), (16, 6, 26, 116, 17.5), (17, 7, 27, 117, 18.5),
+ (18, 8, 28, 118, 19.5), (19, 9, 29, 119, 20.5)
+ """
+
+ // Wait for data to be visible
+ Thread.sleep(5000)
+
+ // ============================================================
+ // Common settings
+ // ============================================================
+ sql "SET enable_nereids_planner=true"
+ sql "SET enable_fallback_to_original_planner=false"
+ sql "SET runtime_filter_mode=off"
+ sql "SET enable_profile=true"
+ sql "SET enable_sql_cache=false"
+ sql "SET enable_local_shuffle=true"
+
+ // ============================================================
+ // Bug 1: must set shared state, in AGGREGATION_OPERATOR
+ // RQG case: eliminate_group_by_uniform.case_id_11007680713
+ // Key conditions: use_serial_exchange=true, parallel_pipeline_task_num=3
+ // SQL: EXCEPT with count(*) GROUP BY on both sides
+ // ============================================================
+
+ // Test with FE planner (the buggy path)
+ logger.info("=== Bug 1a: AGG shared state - EXCEPT with serial exchange (FE planner) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=3,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_common_expr_pushdown=false,
+ disable_streaming_preaggregations=true)*/
+ col_int_undef_signed_not_null as col1,
+ col_int_undef_signed_not_null as col2,
+ 0 as col3, count(1)
+ FROM rqg_t1
+ GROUP BY col1, col2, col3
+ EXCEPT
+ SELECT col_bigint_undef_signed_not_null as col1,
+ col_decimal_38_10__undef_signed_not_null as col2,
+ 5 as col3, count(1)
+ FROM rqg_t2
+ GROUP BY col1, col2, col3
+ """
+ logger.info("Bug 1a: PASSED (no crash)")
+ } catch (Throwable t) {
+ logger.error("Bug 1a FAILED: ${t.message}")
+ assertTrue(false, "Bug 1a: must set shared state in AGGREGATION_OPERATOR: ${t.message}")
+ }
+
+ // Compare with BE native planner
+ logger.info("=== Bug 1b: AGG shared state - EXCEPT with serial exchange (BE native) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=3,
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_common_expr_pushdown=false,
+ disable_streaming_preaggregations=true)*/
+ col_int_undef_signed_not_null as col1,
+ col_int_undef_signed_not_null as col2,
+ 0 as col3, count(1)
+ FROM rqg_t1
+ GROUP BY col1, col2, col3
+ EXCEPT
+ SELECT col_bigint_undef_signed_not_null as col1,
+ col_decimal_38_10__undef_signed_not_null as col2,
+ 5 as col3, count(1)
+ FROM rqg_t2
+ GROUP BY col1, col2, col3
+ """
+ logger.info("Bug 1b: PASSED (no crash)")
+ } catch (Throwable t) {
+ logger.error("Bug 1b FAILED: ${t.message}")
+ assertTrue(false, "Bug 1b: BE native also fails: ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 2: must set shared state, in SORT_OPERATOR
+ // RQG case: grouping_set.case_id_5308471751
+ // Key conditions: use_serial_exchange=true, parallel_pipeline_task_num=5
+ // SQL: GROUPING SETS + window function (PERCENT_RANK)
+ // ============================================================
+
+ logger.info("=== Bug 2a: SORT shared state - GROUPING SETS + window (FE planner) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=5,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_share_hash_table_for_broadcast_join=false,
+ disable_streaming_preaggregations=true)*/
+ SUM(PERCENT_RANK() OVER (PARTITION BY col_int_undef_signed2 ORDER BY col_int_undef_signed2))
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed2),(pk, pk),(col_int_undef_signed))
+ """
+ logger.info("Bug 2a: PASSED (no crash)")
+ } catch (Throwable t) {
+ logger.error("Bug 2a FAILED: ${t.message}")
+ assertTrue(false, "Bug 2a: must set shared state in SORT_OPERATOR: ${t.message}")
+ }
+
+ logger.info("=== Bug 2b: SORT shared state - GROUPING SETS + window (BE native) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=5,
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_share_hash_table_for_broadcast_join=false,
+ disable_streaming_preaggregations=true)*/
+ SUM(PERCENT_RANK() OVER (PARTITION BY col_int_undef_signed2 ORDER BY col_int_undef_signed2))
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed2),(pk, pk),(col_int_undef_signed))
+ """
+ logger.info("Bug 2b: PASSED (no crash)")
+ } catch (Throwable t) {
+ logger.error("Bug 2b FAILED: ${t.message}")
+ assertTrue(false, "Bug 2b: BE native also fails: ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 3: incorrect results with GROUPING SETS + scalar subquery + window
+ // RQG case: grouping_set.case_id_5694495756
+ // Key conditions: parallel_pipeline_task_num=2, disable_streaming_preaggregations=true
+ // Expected: all rows same value; Actual: values split proportionally (1/3, 2/3)
+ // ============================================================
+
+ logger.info("=== Bug 3: incorrect results - GROUPING SETS + subquery + window ===")
+ // FE planner
+ def result_fe = sql """
+ SELECT /*+SET_VAR(parallel_pipeline_task_num=2,
+ enable_local_shuffle_planner=true,
+ disable_streaming_preaggregations=true,
+ enable_share_hash_table_for_broadcast_join=true)*/
+ SUM((SELECT MAX(col_int_undef_signed2) FROM rqg_t1))
+ OVER (PARTITION BY pk ORDER BY pk)
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed2, pk),(pk), (pk))
+ """
+ // BE native
+ def result_be = sql """
+ SELECT /*+SET_VAR(parallel_pipeline_task_num=2,
+ enable_local_shuffle_planner=false,
+ disable_streaming_preaggregations=true,
+ enable_share_hash_table_for_broadcast_join=true)*/
+ SUM((SELECT MAX(col_int_undef_signed2) FROM rqg_t1))
+ OVER (PARTITION BY pk ORDER BY pk)
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed2, pk),(pk), (pk))
+ """
+ logger.info("Bug 3 FE result rows: ${result_fe.size()}, first few: ${result_fe.take(5)}")
+ logger.info("Bug 3 BE result rows: ${result_be.size()}, first few: ${result_be.take(5)}")
+
+ // FE planner and BE native must produce identical results (the bug was values split
+ // proportionally instead of equal). Assert row count and order-insensitive content so a
+ // recurrence fails the suite.
+ assertEquals(result_be.size(), result_fe.size(), "Bug 3: FE/BE row count mismatch")
+ assertEquals(result_be.collect { it.toString() }.sort(), result_fe.collect { it.toString() }.sort(),
+ "Bug 3: FE/BE result mismatch")
+
+ // ============================================================
+ // Bug 4: Simplified AGG shared state — single table GROUP BY with serial exchange
+ // Minimal reproduction attempt
+ // ============================================================
+
+ logger.info("=== Bug 4: Simplified AGG shared state ===")
+ for (int ppt : [2, 3, 4, 5]) {
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=${ppt},
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true)*/
+ col_int_undef_signed, count(*)
+ FROM rqg_t1
+ GROUP BY col_int_undef_signed
+ UNION ALL
+ SELECT col_int_undef_signed, count(*)
+ FROM rqg_t2
+ GROUP BY col_int_undef_signed
+ """
+ logger.info("Bug 4 ppt=${ppt}: PASSED")
+ } catch (Throwable t) {
+ logger.error("Bug 4 ppt=${ppt} FAILED: ${t.message}")
+ assertTrue(false, "Bug 4 ppt=${ppt}: AGG shared state crash with serial exchange: ${t.message}")
+ }
+ }
+
+ // ============================================================
+ // Bug 5: GROUPING SETS + window variations with serial exchange
+ // More variations to find minimal repro
+ // ============================================================
+
+ logger.info("=== Bug 5: GROUPING SETS + window variations ===")
+ for (int ppt : [2, 3, 4, 5]) {
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=${ppt},
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true)*/
+ pk, col_int_undef_signed,
+ ROW_NUMBER() OVER (ORDER BY pk)
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((pk, col_int_undef_signed), (pk), ())
+ ORDER BY pk
+ """
+ logger.info("Bug 5 ppt=${ppt}: PASSED")
+ } catch (Throwable t) {
+ logger.error("Bug 5 ppt=${ppt} FAILED: ${t.message}")
+ assertTrue(false, "Bug 5 ppt=${ppt}: GROUPING SETS + window crash with serial exchange: ${t.message}")
+ }
+ }
+
+ // ============================================================
+ // Bug 6: must set shared state, in CROSS_JOIN_OPERATOR
+ // Root cause: nested NLJ + pooling scan — FE planner skipped BROADCAST
+ // local exchange on outer NLJ's build side because child was NLJ (not ScanNode).
+ // Fixed in NestedLoopJoinNode.enforceAndDeriveLocalExchange by using
+ // fragment.useSerialSource() instead of instanceof ScanNode check.
+ // This was the root cause of 989 RQG test failures (build 183677).
+ // ============================================================
+
+ logger.info("=== Bug 6: CROSS_JOIN shared state - nested NLJ + pooling scan (FE planner) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(ignore_storage_data_distribution=true,
+ parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=true,
+ disable_join_reorder=true,
+ disable_colocate_plan=true,
+ auto_broadcast_join_threshold=-1,
+ broadcast_row_count_limit=0,
+ query_timeout=60)*/
+ count(a.pk) AS cnt, a.col_int_undef_signed
+ FROM rqg_t1 a
+ LEFT JOIN rqg_t1 b ON b.col_int_undef_signed >= b.col_int_undef_signed
+ LEFT JOIN rqg_t1 c ON b.pk >= b.pk
+ WHERE a.pk IS NOT NULL
+ GROUP BY a.col_int_undef_signed
+ ORDER BY cnt, a.col_int_undef_signed
+ """
+ logger.info("Bug 6: PASSED (no CROSS_JOIN_OPERATOR shared state error)")
+ } catch (Throwable t) {
+ logger.error("Bug 6 FAILED: ${t.message}")
+ assertTrue(false, "Bug 6: must set shared state in CROSS_JOIN_OPERATOR: ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 7: DataStreamSink hang — sender fragment with pooling scan
+ // Root cause: FE planner did not insert PASSTHROUGH at the root of pooling scan
+ // sender fragments. With pooling scan, only instance 0 creates pipeline tasks,
+ // so only 1 EOS is sent. The downstream ExchangeNode expects _num_instances EOSes
+ // and hangs indefinitely.
+ // Fixed in AddLocalExchange.addLocalExchangeForFragment: insert PASSTHROUGH
+ // when isLocalShuffle && newRoot.isSerialOperator().
+ // Any NLJ + pooling scan query triggers this via the UNPARTITIONED sender fragments.
+ // ============================================================
+
+ logger.info("=== Bug 7: DataStreamSink hang - NLJ + pooling scan sender (FE planner) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(ignore_storage_data_distribution=true,
+ parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=true,
+ disable_join_reorder=true,
+ disable_colocate_plan=true,
+ auto_broadcast_join_threshold=-1,
+ broadcast_row_count_limit=0,
+ query_timeout=60)*/
+ a.col_int_undef_signed, MAX(a.pk) AS mx
+ FROM rqg_t1 a
+ LEFT JOIN rqg_t1 b ON b.col_int_undef_signed < b.col_int_undef_signed
+ WHERE a.pk IS NOT NULL
+ GROUP BY a.col_int_undef_signed
+ ORDER BY a.col_int_undef_signed, mx
+ """
+ logger.info("Bug 7: PASSED (no hang)")
+ } catch (Throwable t) {
+ logger.error("Bug 7 FAILED: ${t.message}")
+ assertTrue(false, "Bug 7: DataStreamSink hang (query timed out or crashed): ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 8: must set shared state, in SORT_OPERATOR / UNION_OPERATOR
+ // Root cause: FE planner + pooling scan + GROUPING SETS. Serial UNPARTITIONED
+ // Exchange reduces downstream pipeline num_tasks to 1. SORT and UNION operators
+ // need _num_instances tasks to inject shared state for all instances.
+ // Fixed by: (1) restoring num_tasks raise for non-scan serial operators in BE
+ // deferred exchanger creation (commit 920d43d), and (2) FE inserting PASSTHROUGH
+ // after serial ExchangeNode in pooling scan fragments (commit d2e7fa2).
+ // ============================================================
+
+ logger.info("=== Bug 8a: SORT/UNION shared state - GROUPING SETS + pooling scan (FE planner) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(ignore_storage_data_distribution=true,
+ parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=true,
+ disable_streaming_preaggregations=true,
+ query_timeout=60)*/
+ pk, col_int_undef_signed, SUM(col_int_undef_signed_not_null) AS sv
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((pk, col_int_undef_signed), (pk), ())
+ ORDER BY pk, col_int_undef_signed, sv
+ """
+ logger.info("Bug 8a: PASSED (no SORT/UNION_OPERATOR shared state error)")
+ } catch (Throwable t) {
+ logger.error("Bug 8a FAILED: ${t.message}")
+ assertTrue(false, "Bug 8a: must set shared state in SORT/UNION_OPERATOR: ${t.message}")
+ }
+
+ logger.info("=== Bug 8b: SORT shared state - window + GROUPING SETS + pooling scan (FE planner) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(ignore_storage_data_distribution=true,
+ parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=true,
+ disable_streaming_preaggregations=true,
+ query_timeout=60)*/
+ pk, SUM(col_int_undef_signed_not_null) AS sv,
+ ROW_NUMBER() OVER (ORDER BY pk) AS rn
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((pk), ())
+ ORDER BY pk, sv, rn
+ """
+ logger.info("Bug 8b: PASSED (no SORT_OPERATOR shared state error)")
+ } catch (Throwable t) {
+ logger.error("Bug 8b FAILED: ${t.message}")
+ assertTrue(false, "Bug 8b: must set shared state in SORT_OPERATOR (window+grouping_sets): ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 9: FE/BE result inconsistency — agg after NLJ + pooling scan
+ // Root cause: StreamingAgg used fragment.useSerialSource()=true to require
+ // PASSTHROUGH from child, but when child is NLJ (not directly a serial scan),
+ // NLJ outputs ADAPTIVE_PASSTHROUGH. FE wrongly inserted an extra PASSTHROUGH
+ // exchange between StreamingAgg and NLJ (5 extra LOCAL_EXCHANGE_SINK_OPERATOR
+ // entries vs BE native).
+ // Fixed in AggregationNode: only requirePassthrough when
+ // children.get(0).isSerialOperator()=true, mirroring BE _child->is_serial_operator().
+ // ============================================================
+
+ logger.info("=== Bug 9: FE/BE result consistency - agg after NLJ + pooling scan ===")
+ def bug9_fe = sql """
+ SELECT /*+SET_VAR(ignore_storage_data_distribution=true,
+ parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=true,
+ disable_join_reorder=true,
+ disable_colocate_plan=true,
+ auto_broadcast_join_threshold=-1,
+ broadcast_row_count_limit=0)*/
+ a.col_int_undef_signed, MAX(a.pk) AS mx
+ FROM rqg_t1 a LEFT JOIN rqg_t1 b ON b.col_int_undef_signed < b.col_int_undef_signed
+ WHERE a.pk IS NOT NULL
+ GROUP BY a.col_int_undef_signed
+ ORDER BY a.col_int_undef_signed, mx
+ """
+ def bug9_be = sql """
+ SELECT /*+SET_VAR(ignore_storage_data_distribution=true,
+ parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=false,
+ disable_join_reorder=true,
+ disable_colocate_plan=true,
+ auto_broadcast_join_threshold=-1,
+ broadcast_row_count_limit=0)*/
+ a.col_int_undef_signed, MAX(a.pk) AS mx
+ FROM rqg_t1 a LEFT JOIN rqg_t1 b ON b.col_int_undef_signed < b.col_int_undef_signed
+ WHERE a.pk IS NOT NULL
+ GROUP BY a.col_int_undef_signed
+ ORDER BY a.col_int_undef_signed, mx
+ """
+ logger.info("Bug 9 FE rows: ${bug9_fe.size()}, BE rows: ${bug9_be.size()}")
+ assertEquals(bug9_be.size(), bug9_fe.size(), "Bug 9: FE/BE row count mismatch")
+ assertEquals(bug9_be, bug9_fe, "Bug 9: FE/BE result mismatch for agg after NLJ + pooling scan")
+ logger.info("Bug 9: PASSED (FE/BE results match)")
+
+ // ============================================================
+ // Bug 10: GLOBAL_HASH_SHUFFLE Rows mismatched — self-join + NLJ
+ // RQG case: 906784672 (build 184181)
+ // Root cause: HashJoinNode used requireGlobalExecutionHash() → GLOBAL local exchange
+ // inserted when use_serial_exchange=true; shuffle_idx_to_instance_idx map has only
+ // 4 entries (1/BE) but GLOBAL hash needs N*dop entries → most rows unrouted (0 actual rows).
+ // Fixed: changed to requireHash() so resolveExchangeType() downgrades to LOCAL hash.
+ // SQL: self-join (table1 LEFT JOIN table1 table2 ON pk=col_bigint_undef_signed)
+ // then NLJ (LEFT JOIN table1 table3 ON pk > col_bigint_undef_signed)
+ // ============================================================
+
+ logger.info("=== Bug 10: GLOBAL_HASH_SHUFFLE Rows mismatched - self-join + NLJ (build 184181 case 906784672) ===")
+ def bug10_fe = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true, parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ disable_join_reorder=true, disable_colocate_plan=true)*/
+ table1.pk AS field1, table1.col_bigint_undef_signed AS field2
+ FROM rqg_t3 AS table1
+ LEFT JOIN rqg_t3 AS table2 ON table1.pk = table2.col_bigint_undef_signed
+ LEFT JOIN rqg_t3 AS table3 ON table1.pk > table2.col_bigint_undef_signed
+ WHERE (table1.col_varchar_10__undef_signed > 'AHlvNtoGLO'
+ AND table1.col_varchar_10__undef_signed < 'zzzz')
+ OR (table1.col_bigint_undef_signed = table1.pk AND table1.col_varchar_64__undef_signed IS NULL)
+ OR (table1.pk != table1.pk AND table1.pk <> 2)
+ GROUP BY field1, field2
+ ORDER BY field1, field2
+ """
+ def bug10_be = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true, parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ disable_join_reorder=true, disable_colocate_plan=true)*/
+ table1.pk AS field1, table1.col_bigint_undef_signed AS field2
+ FROM rqg_t3 AS table1
+ LEFT JOIN rqg_t3 AS table2 ON table1.pk = table2.col_bigint_undef_signed
+ LEFT JOIN rqg_t3 AS table3 ON table1.pk > table2.col_bigint_undef_signed
+ WHERE (table1.col_varchar_10__undef_signed > 'AHlvNtoGLO'
+ AND table1.col_varchar_10__undef_signed < 'zzzz')
+ OR (table1.col_bigint_undef_signed = table1.pk AND table1.col_varchar_64__undef_signed IS NULL)
+ OR (table1.pk != table1.pk AND table1.pk <> 2)
+ GROUP BY field1, field2
+ ORDER BY field1, field2
+ """
+ logger.info("Bug 10 FE rows: ${bug10_fe.size()}, BE rows: ${bug10_be.size()}")
+ assertEquals(bug10_be.size(), bug10_fe.size(), "Bug 10: FE/BE row count mismatch (GLOBAL_HASH_SHUFFLE Rows mismatched)")
+ assertEquals(bug10_be, bug10_fe, "Bug 10: FE/BE result mismatch for self-join + NLJ")
+ logger.info("Bug 10: PASSED")
+
+ // ============================================================
+ // Bug 11: GLOBAL_HASH_SHUFFLE Rows mismatched — FULL OUTER JOIN + GROUP BY
+ // RQG case: 11007681241 (build 184181)
+ // Same root cause as Bug 10.
+ // SQL: FULL OUTER JOIN on col_bigint_undef_signed_not_null with WHERE + GROUP BY
+ // ============================================================
+
+ logger.info("=== Bug 11: GLOBAL_HASH_SHUFFLE Rows mismatched - FULL OUTER JOIN + GROUP BY (build 184181 case 11007681241) ===")
+ def bug11_fe = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true, parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false)*/
+ t1.col_bigint_undef_signed_not_null, t2.col_bigint_undef_signed_not_null, count(1)
+ FROM rqg_t4 t1
+ FULL OUTER JOIN rqg_t2 t2
+ ON t1.col_bigint_undef_signed_not_null = t2.col_bigint_undef_signed_not_null
+ WHERE t2.col_bigint_undef_signed_not_null = 2
+ GROUP BY t1.col_bigint_undef_signed_not_null, t2.col_bigint_undef_signed_not_null
+ ORDER BY 1, 2, 3
+ """
+ def bug11_be = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true, parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false)*/
+ t1.col_bigint_undef_signed_not_null, t2.col_bigint_undef_signed_not_null, count(1)
+ FROM rqg_t4 t1
+ FULL OUTER JOIN rqg_t2 t2
+ ON t1.col_bigint_undef_signed_not_null = t2.col_bigint_undef_signed_not_null
+ WHERE t2.col_bigint_undef_signed_not_null = 2
+ GROUP BY t1.col_bigint_undef_signed_not_null, t2.col_bigint_undef_signed_not_null
+ ORDER BY 1, 2, 3
+ """
+ logger.info("Bug 11 FE rows: ${bug11_fe.size()}, BE rows: ${bug11_be.size()}")
+ assertEquals(bug11_be.size(), bug11_fe.size(), "Bug 11: FE/BE row count mismatch (GLOBAL_HASH_SHUFFLE Rows mismatched)")
+ assertEquals(bug11_be, bug11_fe, "Bug 11: FE/BE result mismatch for FULL OUTER JOIN + GROUP BY")
+ logger.info("Bug 11: PASSED")
+
+ // ============================================================
+ // Bug 12: GLOBAL_HASH_SHUFFLE Rows mismatched — LEFT JOIN + VARCHAR predicates + MIN()
+ // RQG case: 906784662 (build 184181)
+ // Same root cause as Bug 10/11.
+ // SQL: LEFT JOIN on pk with VARCHAR NOT IN / BETWEEN / IN predicates, MIN() aggregate
+ // ============================================================
+
+ logger.info("=== Bug 12: GLOBAL_HASH_SHUFFLE Rows mismatched - LEFT JOIN + VARCHAR predicates (build 184181 case 906784662) ===")
+ def bug12_fe = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true, parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ disable_join_reorder=true, disable_colocate_plan=true)*/
+ table1.pk AS field1, MIN(table1.pk) AS field2
+ FROM rqg_t3 AS table1
+ LEFT JOIN rqg_t1 AS table2 ON table2.pk = table1.pk
+ WHERE table1.col_varchar_64__undef_signed NOT IN ('they')
+ AND table1.col_varchar_10__undef_signed BETWEEN 'AHlvNtoGLO' AND 'z'
+ AND table1.pk IN (3, 6, 8, 9, 2)
+ GROUP BY field1
+ ORDER BY field1, field2 ASC
+ """
+ def bug12_be = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true, parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ disable_join_reorder=true, disable_colocate_plan=true)*/
+ table1.pk AS field1, MIN(table1.pk) AS field2
+ FROM rqg_t3 AS table1
+ LEFT JOIN rqg_t1 AS table2 ON table2.pk = table1.pk
+ WHERE table1.col_varchar_64__undef_signed NOT IN ('they')
+ AND table1.col_varchar_10__undef_signed BETWEEN 'AHlvNtoGLO' AND 'z'
+ AND table1.pk IN (3, 6, 8, 9, 2)
+ GROUP BY field1
+ ORDER BY field1, field2 ASC
+ """
+ logger.info("Bug 12 FE rows: ${bug12_fe.size()}, BE rows: ${bug12_be.size()}")
+ assertEquals(bug12_be.size(), bug12_fe.size(), "Bug 12: FE/BE row count mismatch (GLOBAL_HASH_SHUFFLE Rows mismatched)")
+ assertEquals(bug12_be, bug12_fe, "Bug 12: FE/BE result mismatch for LEFT JOIN + VARCHAR predicates")
+ logger.info("Bug 12: PASSED")
+
+ // ============================================================
+ // Bug 13: NLJ COREDUMP — serial NLJ + pooling scan + BROADCAST build side
+ // RQG build 184430, query c0dafc1bed0f4910
+ // Root cause: serial NLJ (RIGHT_OUTER) with pooling scan inserted BROADCAST
+ // local exchange on build side, inflating build pipeline num_tasks to _num_instances
+ // while probe pipeline stayed at 1 task. Instance 1+ created build tasks without
+ // corresponding probe tasks → source_deps empty → set_ready_to_read() crash.
+ // Fixed: serial NLJ sets buildSideRequire=noRequire() to match BE-native
+ // num_tasks_of_parent()<=1 skip logic.
+ // ============================================================
+
+ logger.info("=== Bug 13: NLJ COREDUMP - serial NLJ + pooling scan (FE planner) ===")
+ try {
+ def bug13_fe = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true, parallel_pipeline_task_num=0,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ enable_share_hash_table_for_broadcast_join=false,
+ disable_streaming_preaggregations=true,
+ disable_join_reorder=true)*/
+ t2.col_bigint_undef_signed_not_null AS field1
+ FROM rqg_t4 AS t1
+ RIGHT OUTER JOIN rqg_t2 AS t2 ON t1.col_bigint_undef_signed_not_null > t2.col_bigint_undef_signed_not_null
+ GROUP BY field1
+ ORDER BY field1 ASC
+ """
+ def bug13_be = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true, parallel_pipeline_task_num=0,
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ enable_share_hash_table_for_broadcast_join=false,
+ disable_streaming_preaggregations=true,
+ disable_join_reorder=true)*/
+ t2.col_bigint_undef_signed_not_null AS field1
+ FROM rqg_t4 AS t1
+ RIGHT OUTER JOIN rqg_t2 AS t2 ON t1.col_bigint_undef_signed_not_null > t2.col_bigint_undef_signed_not_null
+ GROUP BY field1
+ ORDER BY field1 ASC
+ """
+ logger.info("Bug 13 FE rows: ${bug13_fe.size()}, BE rows: ${bug13_be.size()}")
+ assertEquals(bug13_be.size(), bug13_fe.size(), "Bug 13: FE/BE row count mismatch (NLJ COREDUMP)")
+ assertEquals(bug13_be, bug13_fe, "Bug 13: FE/BE result mismatch for serial NLJ + pooling scan")
+ logger.info("Bug 13: PASSED (no crash, results match)")
+ } catch (Throwable t) {
+ logger.error("Bug 13 FAILED: ${t.message}")
+ assertTrue(false, "Bug 13: NLJ COREDUMP (serial NLJ + pooling scan): ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 14: BUCKET_SHUFFLE join + serial build Exchange — must set shared state
+ // RQG build 184563, cases 906784706/906784783/906784987/906785006
+ // Root cause: BUCKET_SHUFFLE join build side ExchangeNode marked serial in
+ // pooling scan fragment → build pipeline num_tasks reduced to 1 →
+ // instance 1+ have probe tasks without build tasks → shared state injection
+ // fails. Fixed: isBucketShuffle() branch checks buildChildSerial and uses
+ // requirePassToOne() to restore num_tasks, matching BE-native behavior.
+ // Requires replication_num=3 + [shuffle] hint to force BUCKET_SHUFFLE plan.
+ // ============================================================
+
+ logger.info("=== Bug 14: BUCKET_SHUFFLE join + serial build Exchange (FE planner) ===")
+ // Need replication_num=3 for BUCKET_SHUFFLE. Check if allow_replica_on_same_host is enabled.
+ def allowSameHost = sql "ADMIN SHOW FRONTEND CONFIG LIKE 'allow_replica_on_same_host'"
+ if (allowSameHost[0][1].toString() == "true") {
+ sql "DROP TABLE IF EXISTS rqg_t5_rep3"
+ sql "DROP TABLE IF EXISTS rqg_t6_rep3"
+ try {
+ sql """
+ CREATE TABLE rqg_t5_rep3 (
+ pk INT NULL,
+ col_varchar_10__undef_signed VARCHAR(10) NULL,
+ col_bigint_undef_signed BIGINT NULL,
+ col_varchar_64__undef_signed VARCHAR(64) NULL
+ ) DUPLICATE KEY(pk, col_varchar_10__undef_signed)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "3")
+ """
+ sql """
+ CREATE TABLE rqg_t6_rep3 (
+ pk INT NULL,
+ col_varchar_10__undef_signed VARCHAR(10) NULL,
+ col_bigint_undef_signed BIGINT NULL,
+ col_varchar_64__undef_signed VARCHAR(64) NULL
+ ) DUPLICATE KEY(pk, col_varchar_10__undef_signed)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "3")
+ """
+ sql """
+ INSERT INTO rqg_t5_rep3 VALUES
+ (0,'abc',-94,'hello'),(1,'xyz',672609,null),(2,'pqr',-3766684,'test'),
+ (3,'abc',5070261,'another'),(4,'def',null,'value'),(5,'so',-86,null),
+ (6,'abc',21910,'they'),(7,'zzzz',-63,'some'),(8,'xPLflvBEcW',-8276281,'longer'),
+ (9,'mid',-101,'final')
+ """
+ sql """
+ INSERT INTO rqg_t6_rep3 VALUES
+ (0,'aaa',100,'world'),(1,'bbb',200,null),(2,'ccc',300,'foo'),
+ (3,'ddd',400,'bar'),(4,'eee',500,'baz'),(5,'fff',600,null),
+ (6,'ggg',700,'qux'),(7,'hhh',800,'quux'),(8,'iii',900,'corge'),
+ (9,'jjj',1000,'grault')
+ """
+ Thread.sleep(3000)
+
+ def bug14_fe = sql """
+ SELECT /*+SET_VAR(enable_local_shuffle_planner=true,
+ parallel_pipeline_task_num=3,
+ disable_streaming_preaggregations=true,
+ enable_sql_cache=false,
+ disable_join_reorder=true)*/
+ table1.pk AS field1
+ FROM rqg_t5_rep3 AS table1
+ LEFT OUTER JOIN [shuffle] rqg_t6_rep3 AS table2 ON table1.pk = table2.pk
+ WHERE table1.col_varchar_10__undef_signed >= 'so'
+ GROUP BY field1
+ ORDER BY field1
+ """
+ def bug14_be = sql """
+ SELECT /*+SET_VAR(enable_local_shuffle_planner=false,
+ parallel_pipeline_task_num=3,
+ disable_streaming_preaggregations=true,
+ enable_sql_cache=false,
+ disable_join_reorder=true)*/
+ table1.pk AS field1
+ FROM rqg_t5_rep3 AS table1
+ LEFT OUTER JOIN [shuffle] rqg_t6_rep3 AS table2 ON table1.pk = table2.pk
+ WHERE table1.col_varchar_10__undef_signed >= 'so'
+ GROUP BY field1
+ ORDER BY field1
+ """
+ logger.info("Bug 14 FE rows: ${bug14_fe.size()}, BE rows: ${bug14_be.size()}")
+ assertEquals(bug14_be.size(), bug14_fe.size(), "Bug 14: FE/BE row count mismatch (BUCKET_SHUFFLE must set shared state)")
+ assertEquals(bug14_be, bug14_fe, "Bug 14: FE/BE result mismatch for BUCKET_SHUFFLE + serial build Exchange")
+ logger.info("Bug 14: PASSED (no crash, results match)")
+ } catch (Throwable t) {
+ logger.error("Bug 14 FAILED: ${t.message}")
+ assertTrue(false, "Bug 14: BUCKET_SHUFFLE must set shared state: ${t.message}")
+ }
+ } else {
+ logger.info("Bug 14: SKIPPED (allow_replica_on_same_host not enabled, cannot create replication_num=3 tables)")
+ }
+
+ // ============================================================
+ // Bug 15: BUCKET_SHUFFLE join wrong results with serial exchange — PASS_TO_ONE data loss
+ // Root cause: When serial exchange feeds BUCKET_SHUFFLE join build side,
+ // PASS_TO_ONE routes ALL build data to task 0. Unlike BROADCAST joins,
+ // BUCKET_SHUFFLE has no shared hash table mechanism — tasks 1..N-1 build
+ // empty hash tables and lose rows during probe. Fixed by using
+ // BUCKET_HASH_SHUFFLE instead of PASS_TO_ONE for BUCKET_SHUFFLE build side.
+ // Tables use 3 buckets so pptn=4 triggers serial scan on single BE (3 < 4*1).
+ // ============================================================
+
+ logger.info("=== Bug 15: BUCKET_SHUFFLE join wrong results with serial PASS_TO_ONE ===")
+
+ sql "DROP TABLE IF EXISTS rqg_t7_3bucket"
+ sql "DROP TABLE IF EXISTS rqg_t8_3bucket"
+
+ sql """
+ CREATE TABLE rqg_t7_3bucket (
+ pk INT NOT NULL,
+ col_int INT NULL,
+ col_varchar VARCHAR(64) NULL
+ ) DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 3
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ sql """
+ CREATE TABLE rqg_t8_3bucket (
+ pk INT NOT NULL,
+ col_int INT NULL,
+ col_varchar VARCHAR(64) NULL
+ ) DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 3
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ sql """
+ INSERT INTO rqg_t7_3bucket VALUES
+ (0, 10, 'aaa'), (1, 20, 'bbb'), (2, 30, 'ccc'),
+ (3, 40, 'ddd'), (4, 50, 'eee'), (5, 60, 'fff'),
+ (6, 70, 'ggg'), (7, 80, 'hhh'), (8, 90, 'iii'), (9, 100, 'jjj')
+ """
+
+ sql """
+ INSERT INTO rqg_t8_3bucket VALUES
+ (0, 10, 'aaa'), (1, 20, 'bbb'), (2, 30, 'ccc'),
+ (3, 40, 'ddd'), (4, 50, 'eee'), (5, 60, 'fff'),
+ (6, 70, 'ggg'), (7, 80, 'hhh'), (8, 90, 'iii'), (9, 100, 'jjj')
+ """
+
+ Thread.sleep(3000)
+
+ try {
+ // pptn=4 with 3 buckets on 1 BE: 3 < 4*1 → serial scan → serial exchange
+ // This triggers the PASS_TO_ONE bug for BUCKET_SHUFFLE build side.
+ // Also test with higher pptn values to cover more cases.
+ for (int ppt : [4, 6, 8]) {
+ def bug15_fe = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,
+ parallel_pipeline_task_num=${ppt},
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_share_hash_table_for_broadcast_join=false,
+ enable_sql_cache=false,
+ disable_join_reorder=true)*/
+ t1.pk, t1.col_int, t2.col_varchar
+ FROM rqg_t7_3bucket t1
+ INNER JOIN [shuffle] rqg_t8_3bucket t2 ON t1.pk = t2.pk
+ ORDER BY t1.pk
+ """
+ def bug15_be = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,
+ parallel_pipeline_task_num=${ppt},
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_share_hash_table_for_broadcast_join=false,
+ enable_sql_cache=false,
+ disable_join_reorder=true)*/
+ t1.pk, t1.col_int, t2.col_varchar
+ FROM rqg_t7_3bucket t1
+ INNER JOIN [shuffle] rqg_t8_3bucket t2 ON t1.pk = t2.pk
+ ORDER BY t1.pk
+ """
+ logger.info("Bug 15 ppt=${ppt}: FE rows=${bug15_fe.size()}, BE rows=${bug15_be.size()}")
+ assertEquals(10, bug15_fe.size(), "Bug 15 ppt=${ppt}: expected 10 rows from FE planner, got ${bug15_fe.size()}")
+ assertEquals(bug15_be, bug15_fe, "Bug 15 ppt=${ppt}: FE/BE result mismatch for BUCKET_SHUFFLE + serial exchange")
+ }
+
+ // Also test LEFT OUTER JOIN to verify no rows lost on probe side
+ def bug15_left_fe = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,
+ parallel_pipeline_task_num=6,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_share_hash_table_for_broadcast_join=false,
+ enable_sql_cache=false,
+ disable_join_reorder=true)*/
+ t1.pk, t2.col_int
+ FROM rqg_t7_3bucket t1
+ LEFT OUTER JOIN [shuffle] rqg_t8_3bucket t2 ON t1.pk = t2.pk
+ ORDER BY t1.pk
+ """
+ def bug15_left_be = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,
+ parallel_pipeline_task_num=6,
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_share_hash_table_for_broadcast_join=false,
+ enable_sql_cache=false,
+ disable_join_reorder=true)*/
+ t1.pk, t2.col_int
+ FROM rqg_t7_3bucket t1
+ LEFT OUTER JOIN [shuffle] rqg_t8_3bucket t2 ON t1.pk = t2.pk
+ ORDER BY t1.pk
+ """
+ assertEquals(10, bug15_left_fe.size(), "Bug 15 LEFT JOIN: expected 10 rows from FE planner")
+ assertEquals(bug15_left_be, bug15_left_fe, "Bug 15 LEFT JOIN: FE/BE result mismatch")
+
+ logger.info("Bug 15: PASSED (no wrong results, all pptn values correct)")
+ } catch (Throwable t) {
+ logger.error("Bug 15 FAILED: ${t.message}")
+ assertTrue(false, "Bug 15: BUCKET_SHUFFLE wrong results with serial PASS_TO_ONE: ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 16 & 17: Serial AnalyticEval crash and DataStreamSink hang
+ // with LocalShuffleAssignedJob (multiple instances on one BE)
+ //
+ // Bug 16 (crash): Exchange wraps itself with PASSTHROUGH LocalExchange.
+ // This restores AnalyticSink pipeline to _num_instances tasks while
+ // serial AnalyticSource stays at 1 task. For instance_idx > 0,
+ // source_deps is empty → DCHECK crash.
+ //
+ // Bug 17 (hang): After fixing the crash, serial AnalyticSource reduces
+ // all downstream pipeline tasks to 1 via add_pipeline() inheritance.
+ // Only instance 0 runs DataStreamSink → receiver expects _num_instances
+ // EOSes → hang.
+ //
+ // Both triggered by: OVER() with no PARTITION BY + GROUPING SETS +
+ // pptn=0 (auto-parallel) + disable_streaming_preaggregations=true
+ // RQG build 186195, query IDs: 7f3178a77c2c4b6b, 71887f7bf804c0c, 5dd9fcad234c4484
+ // ============================================================
+ sql "DROP TABLE IF EXISTS rqg_analytic_t1"
+ sql """
+ CREATE TABLE rqg_analytic_t1 (
+ pk INT NOT NULL,
+ col_int_undef_signed INT
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ INSERT INTO rqg_analytic_t1 VALUES
+ (1, 10), (2, 20), (3, 30), (4, 40), (5, 50),
+ (6, 60), (7, 70), (8, 80), (9, 90), (10, 100),
+ (11, 10), (12, 20), (13, 30), (14, 40), (15, 50),
+ (16, 60), (17, 70), (18, 80), (19, 90), (20, 100)
+ """
+
+ try {
+ logger.info("Bug 16+17: Testing serial AnalyticEval with GROUPING SETS")
+
+ // Baseline: pptn=1 (no multi-instance, no local shuffle)
+ def bug16_baseline = sql """
+ SELECT /*+SET_VAR(parallel_pipeline_task_num=1,
+ enable_sql_cache=false,
+ disable_streaming_preaggregations=true)*/
+ COUNT(MIN(col_int_undef_signed) OVER())
+ FROM rqg_analytic_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed, pk), (), (pk))
+ ORDER BY 1
+ """
+ assertEquals(41, bug16_baseline.size(), "Bug 16 baseline: expected 41 rows")
+
+ // Test with pptn=0 (auto-parallel, triggers LocalShuffleAssignedJob)
+ for (int ppt : [0, 2, 4, 8]) {
+ def bug16_result = sql """
+ SELECT /*+SET_VAR(parallel_pipeline_task_num=${ppt},
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ disable_streaming_preaggregations=true)*/
+ COUNT(MIN(col_int_undef_signed) OVER())
+ FROM rqg_analytic_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed, pk), (), (pk))
+ ORDER BY 1
+ """
+ assertEquals(bug16_baseline, bug16_result,
+ "Bug 16+17 pptn=${ppt}: result mismatch with serial AnalyticEval")
+ }
+
+ // Also test with use_serial_exchange=true (makes ALL exchanges serial)
+ def bug16_serial = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,
+ parallel_pipeline_task_num=0,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ disable_streaming_preaggregations=true)*/
+ COUNT(MIN(col_int_undef_signed) OVER())
+ FROM rqg_analytic_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed, pk), (), (pk))
+ ORDER BY 1
+ """
+ assertEquals(bug16_baseline, bug16_serial,
+ "Bug 16+17 serial_exchange: result mismatch")
+
+ logger.info("Bug 16+17: PASSED (no crash, no hang, correct results)")
+ } catch (Throwable t) {
+ logger.error("Bug 16+17 FAILED: ${t.message}")
+ assertTrue(false, "Bug 16+17: Serial AnalyticEval crash/hang: ${t.message}")
+ }
+
+ // Bug 18: DCHECK crash in Pipeline::set_num_tasks when PASSTHROUGH LE is inserted
+ // between serial NLJ and its child Exchange.
+ // Root cause: ExchangeNode.enforceAndDeriveLocalExchange wraps UNPARTITIONED serial
+ // Exchange with PASSTHROUGH LE. On BE, NLJ_PROBE (serial) sets pipeline num_tasks=1,
+ // then the LE handler's set_num_tasks(_num_instances) overrides it to N, triggering
+ // DCHECK (serial operator in pipeline with num_tasks > 1).
+ // Fix: skip PASSTHROUGH wrapping when hasSerialAncestorInPipeline is true.
+ // Query: LEFT JOIN with always-true self-ref condition (table.pk = table.pk) creates
+ // RIGHT_OUTER NLJ (serial). With pptn>1 and ignore_data_distribution, the fragment
+ // gets N instances but NLJ forces 1 task.
+ try {
+ logger.info("Bug 18: Testing serial NLJ with PASSTHROUGH LE crash")
+ // Use existing rqg_t1 table (10 rows, 10 buckets)
+ def bug18_baseline = sql """
+ SELECT /*+SET_VAR(enable_local_shuffle_planner=false,
+ enable_sql_cache=false)*/
+ table1.col_int_undef_signed AS field1
+ FROM rqg_t1 AS table1
+ LEFT JOIN rqg_t1 AS table2
+ ON table2.pk = table2.pk
+ WHERE table1.pk BETWEEN 2 AND 11
+ GROUP BY field1
+ ORDER BY 1
+ """
+
+ // Test with various pptn values — crash requires pptn > 1
+ for (int ppt : [4, 7]) {
+ def bug18_result = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=false,
+ parallel_pipeline_task_num=${ppt},
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ enable_share_hash_table_for_broadcast_join=false,
+ enable_broadcast_join_force_passthrough=true,
+ enable_parallel_scan=false)*/
+ table1.col_int_undef_signed AS field1
+ FROM rqg_t1 AS table1
+ LEFT JOIN rqg_t1 AS table2
+ ON table2.pk = table2.pk
+ WHERE table1.pk BETWEEN 2 AND 11
+ GROUP BY field1
+ ORDER BY 1
+ """
+ assertEquals(bug18_baseline, bug18_result,
+ "Bug 18 pptn=${ppt}: result mismatch with serial NLJ + local exchange")
+ }
+ logger.info("Bug 18: PASSED (no crash, correct results)")
+ } catch (Throwable t) {
+ logger.error("Bug 18 FAILED: ${t.message}")
+ assertTrue(false, "Bug 18: Serial NLJ PASSTHROUGH LE crash: ${t.message}")
+ }
+
+ // Bug 19: source_deps.size()=0 crash in NLJ build sink.
+ // Root cause: serial NLJ (RIGHT_OUTER) resets serial ancestor flag for build side.
+ // Exchange(UNPARTITIONED) on build side sees hasSerialAncestorInPipeline=false and
+ // inserts PASSTHROUGH LE. This restores build pipeline num_tasks to _num_instances
+ // while probe pipeline stays at 1. The extra build tasks have NLJ shared state with
+ // empty source_deps → crash in set_ready_to_read().
+ // Fix: shouldResetSerialFlagForChild(1) returns false when NLJ is serial.
+ // Differs from Bug 18 in fuzzy vars: enable_share_hash_table=true, broadcast_passthrough=false.
+ try {
+ logger.info("Bug 19: Testing serial NLJ build-side source_deps crash")
+ def bug19_baseline = sql """
+ SELECT /*+SET_VAR(enable_local_shuffle_planner=false,
+ enable_sql_cache=false)*/
+ table1.col_int_undef_signed AS field1
+ FROM rqg_t1 AS table1
+ LEFT JOIN rqg_t1 AS table2
+ ON table2.pk = table2.pk
+ WHERE table1.pk BETWEEN 2 AND 11
+ GROUP BY field1
+ ORDER BY 1
+ """
+
+ for (int ppt : [2, 4]) {
+ def bug19_result = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=false,
+ parallel_pipeline_task_num=${ppt},
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ enable_share_hash_table_for_broadcast_join=true,
+ enable_broadcast_join_force_passthrough=false,
+ enable_parallel_scan=true,
+ disable_streaming_preaggregations=true)*/
+ table1.col_int_undef_signed AS field1
+ FROM rqg_t1 AS table1
+ LEFT JOIN rqg_t1 AS table2
+ ON table2.pk = table2.pk
+ WHERE table1.pk BETWEEN 2 AND 11
+ GROUP BY field1
+ ORDER BY 1
+ """
+ assertEquals(bug19_baseline, bug19_result,
+ "Bug 19 pptn=${ppt}: result mismatch with serial NLJ build side crash")
+ }
+ logger.info("Bug 19: PASSED (no crash, correct results)")
+ } catch (Throwable t) {
+ logger.error("Bug 19 FAILED: ${t.message}")
+ assertTrue(false, "Bug 19: Serial NLJ build-side source_deps crash: ${t.message}")
+ }
+
+ // Bug 20: Hang (ASAN: COREDUMP source_deps.size()=0 in AggSinkOperatorX) when
+ // use_serial_exchange=true + RIGHT JOIN + GROUP BY in non-pooling fragment.
+ // Root cause: serial HASH Exchange in non-pooling fragment returned NOOP, causing FE
+ // to insert LOCAL_EXECUTION_HASH_SHUFFLE LE. On BE, serial Exchange pipeline has 1 task
+ // but LE downstream has _num_instances tasks. AggSink on instances 1+ has empty source_deps.
+ // Fix: ExchangeNode.enforceAndDeriveLocalExchange() returns actual distribution type
+ // (GLOBAL_EXECUTION_HASH_SHUFFLE/BUCKET_HASH_SHUFFLE) for serial Exchange in non-pooling
+ // fragments, preventing LE insertion.
+ // Requires 3+ BEs to reproduce (single BE has _num_instances=1, no hang).
+ try {
+ logger.info("Bug 20: Testing serial exchange + agg hang in non-pooling fragment")
+ // Baseline uses same fuzzy vars but with planner=false (BE-native).
+ // This way we compare FE-planned vs BE-native under identical conditions,
+ // not against "correct" results — use_serial_exchange=true itself may have
+ // pre-existing BE bugs with certain pptn values.
+ def bug20_baseline_sql = { int ppt -> """
+ SELECT /*+SET_VAR(use_serial_exchange=true,
+ parallel_pipeline_task_num=${ppt},
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ enable_share_hash_table_for_broadcast_join=false,
+ disable_streaming_preaggregations=true)*/
+ table1.pk AS field1
+ FROM rqg_t1 AS table1
+ RIGHT OUTER JOIN rqg_t1 AS table2 ON table1.pk = table2.pk
+ LEFT JOIN rqg_t1 AS table3 ON table3.pk = table1.pk
+ WHERE table1.col_int_undef_signed IS NOT NULL OR table1.pk <> 10
+ GROUP BY field1
+ ORDER BY 1
+ """ }
+ for (int ppt : [3, 4, 7]) {
+ def bug20_baseline = sql bug20_baseline_sql(ppt)
+ def bug20_result = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,
+ parallel_pipeline_task_num=${ppt},
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ enable_share_hash_table_for_broadcast_join=false,
+ disable_streaming_preaggregations=true)*/
+ table1.pk AS field1
+ FROM rqg_t1 AS table1
+ RIGHT OUTER JOIN rqg_t1 AS table2 ON table1.pk = table2.pk
+ LEFT JOIN rqg_t1 AS table3 ON table3.pk = table1.pk
+ WHERE table1.col_int_undef_signed IS NOT NULL OR table1.pk <> 10
+ GROUP BY field1
+ ORDER BY 1
+ """
+ assertEquals(bug20_baseline, bug20_result,
+ "Bug 20 pptn=${ppt}: result mismatch with serial exchange + agg hang")
+ }
+ logger.info("Bug 20: PASSED (no hang, correct results)")
+ } catch (Throwable t) {
+ logger.error("Bug 20 FAILED: ${t.message}")
+ assertTrue(false, "Bug 20: Serial exchange + agg hang: ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 21: Multi-distinct COUNT on many-bucket table → COREDUMP
+ // RQG build 186737/186929/186952: AggSinkOperatorX::sink → set_ready_to_read
+ // with empty source_deps.
+ //
+ // Root cause: AGG operators (streaming, distinct-streaming, serialize) requested
+ // PASSTHROUGH from non-ScanNode serial children (Exchange, AGG), inserting a
+ // PASSTHROUGH LE that created a pipeline split disconnecting AggSink↔AggSource
+ // shared state.
+ //
+ // Fix: restrict AGG PASSTHROUGH requests to ScanNode children only.
+ // Triggered by: multi-distinct COUNT/MIN with MultiCastDataSinks feeding
+ // serial UNPARTITIONED Exchanges into streaming AGG fragments.
+ // ============================================================
+ sql "DROP TABLE IF EXISTS rqg_t5_many_buckets"
+ sql """
+ CREATE TABLE rqg_t5_many_buckets (
+ pk INT NOT NULL,
+ col_int_undef_signed INT,
+ col_date_undef_signed DATE,
+ col_date_undef_signed2 DATE,
+ col_varchar_1024__undef_signed VARCHAR(1024)
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 56
+ PROPERTIES ("replication_num" = "1")
+ """
+ sql """INSERT INTO rqg_t5_many_buckets VALUES
+ (1,1,'2023-12-09','2024-06-01','s1'),(2,2,'2023-03-15','2024-01-20','s2'),
+ (3,3,'2023-07-22','2024-03-10',NULL),(4,4,'2023-12-09','2024-06-01','s4'),
+ (5,5,'2023-01-05','2024-09-15','s5'),(6,6,'2023-08-11','2024-02-28','s6'),
+ (7,7,'2023-04-18','2024-07-04',NULL),(8,8,'2023-11-25','2024-05-12','s8'),
+ (9,9,'2023-06-30','2024-11-19','s9'),(10,10,'2023-02-14','2024-08-07','s10')
+ """
+
+ try {
+ logger.info("Bug 21: Testing multi-distinct COUNT on many-bucket table (COREDUMP fix)")
+ for (int ppt : [4, 6]) {
+ // Test without use_serial_exchange
+ def bug21_baseline = sql """
+ SELECT /*+SET_VAR(enable_local_shuffle_planner=false,
+ parallel_pipeline_task_num=${ppt},
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ query_timeout=60)*/
+ MIN(distinct col_date_undef_signed),
+ COUNT(distinct col_date_undef_signed2),
+ COUNT(distinct col_int_undef_signed)
+ FROM rqg_t5_many_buckets
+ WHERE col_int_undef_signed = col_int_undef_signed
+ LIMIT 1000
+ """
+ def bug21_result = sql """
+ SELECT /*+SET_VAR(enable_local_shuffle_planner=true,
+ parallel_pipeline_task_num=${ppt},
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ query_timeout=60)*/
+ MIN(distinct col_date_undef_signed),
+ COUNT(distinct col_date_undef_signed2),
+ COUNT(distinct col_int_undef_signed)
+ FROM rqg_t5_many_buckets
+ WHERE col_int_undef_signed = col_int_undef_signed
+ LIMIT 1000
+ """
+ assertEquals(bug21_baseline, bug21_result,
+ "Bug 21 pptn=${ppt}: multi-distinct COUNT result mismatch (was COREDUMP)")
+ }
+ logger.info("Bug 21: PASSED (no crash, correct results)")
+ } catch (Throwable t) {
+ if (t.message != null && t.message.contains("query timeout")) {
+ logger.warn("Bug 21: SKIPPED (query timeout - likely BE crash from previous test, not our bug)")
+ } else {
Review Comment:
Fixed in `8db568b5100`. Removed the `catch` branch that logged `SKIPPED` on `query timeout` — a hang is one of the failure modes this case reproduces (an EOS/close-count mismatch can hang instead of coredumping), so a timeout now fails the suite like the other `catch` blocks. Added a comment noting the hang-vs-crash distinction.
##########
gensrc/thrift/Partitions.thrift:
##########
@@ -119,3 +153,5 @@ struct TDataPartition {
3: optional list<TRangePartition> partition_infos
4: optional TMergePartitionInfo merge_partition_info
}
+
Review Comment:
Fixed in `73cc6458fd1` — removed the trailing blank line(s) at EOF; the file now ends at the closing `}` (line 155). `git diff --check` is clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4734137516
# FE Regression Coverage Report
Increment line coverage ` 80.25% (451/562)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_c2c51332d7f90c80ccabf77b27c23317d636a128_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_c2c51332d7f90c80ccabf77b27c23317d636a128_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4738227700
# FE Regression Coverage Report
Increment line coverage ` 70.66% (419/593)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_c2c51332d7f90c80ccabf77b27c23317d636a128_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_c2c51332d7f90c80ccabf77b27c23317d636a128_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4507561491
# FE UT Coverage Report
Increment line coverage ` 73.97% (449/607)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_846d365a88901fa32fdf44b42b6cd04b900da259/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_846d365a88901fa32fdf44b42b6cd04b900da259/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4508959008
# BE UT Coverage Report
Increment line coverage `11.11% (46/414)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/846d365a88901fa32fdf44b42b6cd04b900da259_846d365a88901fa32fdf44b42b6cd04b900da259/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/846d365a88901fa32fdf44b42b6cd04b900da259_846d365a88901fa32fdf44b42b6cd04b900da259/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.64% (20736/38658) |
| Line Coverage | 37.21% (196345/527718) |
| Region Coverage | 33.55% (153997/459017) |
| Branch Coverage | 34.54% (67084/194197) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4766685010
# FE Regression Coverage Report
Increment line coverage ` 81.28% (456/561)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3452056959
##########
fe/fe-core/src/main/java/org/apache/doris/planner/RepeatNode.java:
##########
@@ -86,11 +90,30 @@ public String getNodeExplainString(String detailPrefix, TExplainLevel detailLeve
// Determined by its child.
@Override
- public boolean isSerialOperator() {
- return children.get(0).isSerialOperator();
+ public boolean isSerialNode() {
+ return children.get(0).isSerialNode();
}
public GroupingInfo getGroupingInfo() {
return groupingInfo;
}
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ // REPEAT (rollup/grouping sets) is NOT distribution-preserving: it NULLs grouping
+ // columns per set and produces GROUPING_ID, which is part of the downstream agg hash
+ // key but does not exist below the repeat. Forwarding the parent HASH require down
+ // would push the local exchange before the row expansion AND hash by the child
+ // distribution (a single upstream shuffle key) instead of the agg grouping_exprs,
+ // collapsing rows onto one instance (tpcds q67, +73%). Recurse with noRequire so the
+ // parent inserts its hash local exchange ABOVE the repeat using its own grouping_exprs
+ // (mirrors BE, whose RepeatOperatorX has a NOOP required_data_distribution).
+ Pair<PlanNode, LocalExchangeType> enforceResult
+ = enforceRequire(translatorContext, children.get(0), 0,
+ LocalExchangeTypeRequire.noRequire());
+ children = new java.util.ArrayList<>();
+ children.add(enforceResult.first);
Review Comment:
Thanks for the detailed analysis. I checked this against the BE native local-shuffle path and I don't believe it's a correctness issue — forwarding the child's distribution here is a deliberate mirror of BE, and the local exchange above `Repeat` is parallelism-only.
**The grouping-id correctness shuffle is a cross-fragment network exchange, not a local exchange.** `EXPLAIN` of `GROUP BY GROUPING SETS` over a bucketed table:
```
Fragment 1: VOlapScanNode -> VREPEAT_NODE (output: ..., GROUPING_ID)
-> VAGGREGATE (update serialize, STREAMING) // partial pre-agg
-> STREAM DATA SINK HASH_PARTITIONED: <keys>, GROUPING_ID // network shuffle
Fragment 0: VEXCHANGE -> VAGGREGATE (merge finalize) // final agg
```
The aggregate directly above `Repeat` is a **streaming partial pre-aggregation**; it does not require its input partitioned by the grouping keys. Global correctness (grouping by `GROUPING_ID`) is guaranteed by the network shuffle (`HASH_PARTITIONED` by the grouping exprs + `GROUPING_ID`) feeding the final merge aggregate in Fragment 0. A `LocalExchangeNode` above `Repeat` can therefore only change the partial agg's within-backend parallelism — never the result.
**BE's native path skips the exchange here too.** `Pipeline::need_to_local_exchange` (`be/src/exec/pipeline/pipeline.cpp`):
```cpp
return _data_distribution.distribution_type != target_data_distribution.distribution_type &&
!(is_hash_exchange(_data_distribution.distribution_type) &&
is_hash_exchange(target_data_distribution.distribution_type)); // both hash -> skip
```
For `Scan -> Repeat -> AggSink`, the pipeline `_data_distribution` is the scan's `required_data_distribution` = `BUCKET_HASH_SHUFFLE` (`scan_operator.h`), `RepeatOperatorX` requires `NOOP` (no LE), and the AggSink requires `GLOBAL_EXECUTION_HASH_SHUFFLE` — both hash, so BE skips the local exchange above `Repeat`.
`RequireHash.satisfy(<any hash>) == true` on the FE side is the exact mirror of that BE rule, and `RepeatNode` forwarding the child's distribution is what lets the FE satisfy-check observe the same distribution BE's `_data_distribution` holds. Returning `NOOP` would force a local exchange BE never inserts, diverging from the native path.
So I'll keep the current behavior. Happy to add a coverage case asserting that a hash-distributed child is forwarded and no LE is inserted above `Repeat`, to lock in the parity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4701476342
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/8db568b5100ed397fc6347033402642e90e5509e_8db568b5100ed397fc6347033402642e90e5509e_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/8db568b5100ed397fc6347033402642e90e5509e_8db568b5100ed397fc6347033402642e90e5509e_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.32% (1889/2443) |
| Line Coverage | 64.40% (33955/52725) |
| Region Coverage | 64.76% (17452/26948) |
| Branch Coverage | 53.94% (9341/17316) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4668784622
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4680401624
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/1f614d8cadd1619a8c17dd6f7ec7b12ba6983286_1f614d8cadd1619a8c17dd6f7ec7b12ba6983286_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/1f614d8cadd1619a8c17dd6f7ec7b12ba6983286_1f614d8cadd1619a8c17dd6f7ec7b12ba6983286_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.32% (1889/2443) |
| Line Coverage | 64.41% (33958/52725) |
| Region Coverage | 64.80% (17463/26948) |
| Branch Coverage | 53.97% (9345/17316) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4675108661
<details>
<summary>TPC-H: <b>Total hot run time: 28973 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f056ebbed42446ea468bcc6891c05520bdfb31c3, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17614 4133 4060 4060
q2 q3 10699 1418 807 807
q4 4693 475 354 354
q5 7606 866 586 586
q6 185 173 135 135
q7 790 852 630 630
q8 9393 1551 1576 1551
q9 5673 4506 4511 4506
q10 6750 1806 1545 1545
q11 442 275 250 250
q12 637 424 291 291
q13 18158 3444 2804 2804
q14 264 265 247 247
q15 q16 830 778 708 708
q17 1098 874 981 874
q18 7191 5906 5377 5377
q19 2622 1492 1115 1115
q20 543 424 286 286
q21 6567 2791 2532 2532
q22 455 384 315 315
Total cold run time: 102210 ms
Total hot run time: 28973 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5033 4887 4937 4887
q2 q3 5054 5188 4613 4613
q4 2108 2285 1396 1396
q5 4875 4779 4824 4779
q6 238 181 131 131
q7 1826 1745 1541 1541
q8 2447 2055 1910 1910
q9 7600 7635 7595 7595
q10 4731 4665 4205 4205
q11 562 414 378 378
q12 734 745 522 522
q13 3014 3376 2831 2831
q14 278 274 258 258
q15 q16 675 694 607 607
q17 1290 1260 1253 1253
q18 7423 6887 6984 6887
q19 1121 1098 1113 1098
q20 2215 2290 1938 1938
q21 5282 4641 4909 4641
q22 526 465 402 402
Total cold run time: 57032 ms
Total hot run time: 51872 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4680255375
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/9f33ce47e1bdfae26100cf19b96a7eab06238d13_9f33ce47e1bdfae26100cf19b96a7eab06238d13_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/9f33ce47e1bdfae26100cf19b96a7eab06238d13_9f33ce47e1bdfae26100cf19b96a7eab06238d13_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.32% (1889/2443) |
| Line Coverage | 64.40% (33955/52725) |
| Region Coverage | 64.77% (17454/26948) |
| Branch Coverage | 53.93% (9338/17316) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3397293736
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/TVFScanNode.java:
##########
@@ -212,4 +217,10 @@ protected void setScanParams(TFileRangeDesc rangeDesc, Split split) {
public int getNumInstances() {
return scanRangeLocations.size();
}
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(PlanTranslatorContext translatorContext,
+ PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ return Pair.of(this, LocalExchangeType.PASSTHROUGH);
+ }
Review Comment:
`PASSTHROUGH` here says the scan already satisfies a parent `requirePassthrough()`, so `PlanNode.enforceRequire()` will not insert a `LocalExchangeNode(PASSTHROUGH)` above this TVF scan. A scan leaf has not done that rebalance; BE local-exchange planning treats non-hash requirements as always needing an exchange (`Pipeline::need_to_local_exchange()` returns true for non-hash target types). A concrete miss is `TVFScan -> merge sort` / `AssertNumRows` / `TableFunction`, or a broadcast-join probe with force-passthrough: FE-planned mode skips the local exchange while BE-planned mode would insert it, leaving the parent on the scan original task distribution. Please report `NOOP` here, or actually wrap the scan in a `LocalExchangeNode` when a parent asks for passthrough, so non-hash requirements are not silently satisfied by a leaf.
```suggestion
return Pair.of(this, LocalExchangeType.NOOP);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4673785565
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4550680970
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4674228779
<details>
<summary>TPC-DS: <b>Total hot run time: 169552 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7e15d5b3b9109d275feadd1af404dcc86ee9b487, data reload: false
query5 4318 644 480 480
query6 431 192 177 177
query7 4849 558 304 304
query8 367 228 204 204
query9 8754 4108 4096 4096
query10 477 329 297 297
query11 5943 2348 2128 2128
query12 151 114 99 99
query13 1306 614 408 408
query14 6368 5411 5033 5033
query14_1 4403 4431 4371 4371
query15 214 198 190 190
query16 1049 457 435 435
query17 1120 700 602 602
query18 2732 509 362 362
query19 212 189 144 144
query20 114 109 115 109
query21 221 143 122 122
query22 13671 13566 13517 13517
query23 17367 16420 16131 16131
query23_1 16232 16212 16274 16212
query24 7429 1804 1302 1302
query24_1 1319 1317 1316 1316
query25 586 463 420 420
query26 1303 313 174 174
query27 2657 569 347 347
query28 4439 2073 2037 2037
query29 1087 637 529 529
query30 321 239 198 198
query31 1162 1087 965 965
query32 117 67 61 61
query33 535 346 268 268
query34 1170 1155 655 655
query35 773 805 677 677
query36 1395 1373 1248 1248
query37 162 111 95 95
query38 3243 3146 2993 2993
query39 943 923 891 891
query39_1 884 903 870 870
query40 223 132 110 110
query41 76 68 72 68
query42 101 100 98 98
query43 327 331 280 280
query44
query45 211 185 180 180
query46 1119 1220 779 779
query47 2372 2402 2242 2242
query48 408 415 314 314
query49 651 485 369 369
query50 1024 356 258 258
query51 4491 4489 4330 4330
query52 95 93 80 80
query53 251 272 201 201
query54 280 243 218 218
query55 86 81 77 77
query56 260 248 243 243
query57 1464 1384 1319 1319
query58 285 218 210 210
query59 1588 1672 1419 1419
query60 280 246 250 246
query61 163 157 159 157
query62 719 642 587 587
query63 232 190 185 185
query64 2532 776 629 629
query65
query66 1769 483 358 358
query67 29783 29669 29461 29461
query68
query69 419 307 269 269
query70 970 967 951 951
query71 297 216 200 200
query72 2903 2702 2400 2400
query73 864 804 441 441
query74 5152 4987 4774 4774
query75 2654 2564 2236 2236
query76 2324 1145 830 830
query77 393 369 292 292
query78 12627 12532 11853 11853
query79 1416 1025 798 798
query80 1324 477 403 403
query81 530 285 257 257
query82 571 157 119 119
query83 323 269 251 251
query84
query85 930 537 442 442
query86 430 299 275 275
query87 3428 3346 3172 3172
query88 3686 2778 2758 2758
query89 421 383 327 327
query90 1915 185 179 179
query91 180 168 139 139
query92 67 62 58 58
query93 1655 1477 863 863
query94 741 345 293 293
query95 660 472 343 343
query96 1055 781 327 327
query97 2729 2711 2548 2548
query98 213 203 202 202
query99 1133 1199 990 990
Total cold run time: 253032 ms
Total hot run time: 169552 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4687233701
# FE Regression Coverage Report
Increment line coverage ` 79.90% (461/577)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_9ccb7f30b5fb41183c99a77cc3de3311715aa9b6_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_9ccb7f30b5fb41183c99a77cc3de3311715aa9b6_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4686164925
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4733249582
<details>
<summary>TPC-DS: <b>Total hot run time: 175572 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c2c51332d7f90c80ccabf77b27c23317d636a128, data reload: false
query5 4310 624 478 478
query6 428 190 166 166
query7 4826 579 316 316
query8 361 216 201 201
query9 8745 4094 4085 4085
query10 442 300 254 254
query11 5909 2329 2129 2129
query12 148 102 95 95
query13 1241 611 430 430
query14 6383 5369 5029 5029
query14_1 4350 4358 4363 4358
query15 203 194 175 175
query16 989 453 417 417
query17 938 677 563 563
query18 2489 471 332 332
query19 192 183 143 143
query20 110 107 102 102
query21 216 142 120 120
query22 13662 13669 13397 13397
query23 17373 16523 16119 16119
query23_1 16265 16322 16299 16299
query24 7630 1808 1293 1293
query24_1 1319 1298 1315 1298
query25 536 434 375 375
query26 1323 305 164 164
query27 2707 566 344 344
query28 4501 2039 2051 2039
query29 1079 632 462 462
query30 319 249 199 199
query31 1116 1068 951 951
query32 97 61 55 55
query33 514 302 239 239
query34 1185 1123 681 681
query35 758 789 679 679
query36 1367 1385 1214 1214
query37 160 109 96 96
query38 3249 3138 3067 3067
query39 928 911 901 901
query39_1 866 876 867 867
query40 219 125 105 105
query41 71 67 67 67
query42 98 95 94 94
query43 326 331 283 283
query44 1462 778 782 778
query45 195 186 181 181
query46 1069 1235 779 779
query47 2330 2350 2243 2243
query48 392 429 309 309
query49 641 473 358 358
query50 1050 356 261 261
query51 4449 4380 4357 4357
query52 89 89 78 78
query53 253 267 201 201
query54 301 253 214 214
query55 79 76 72 72
query56 274 242 215 215
query57 1440 1387 1342 1342
query58 257 221 222 221
query59 1573 1647 1476 1476
query60 282 257 231 231
query61 172 172 173 172
query62 702 658 593 593
query63 224 193 203 193
query64 2563 818 628 628
query65 4902 4817 4817 4817
query66 1806 491 360 360
query67 29745 29676 29561 29561
query68 3177 1592 1026 1026
query69 426 306 270 270
query70 1065 959 985 959
query71 305 245 214 214
query72 2850 2611 2340 2340
query73 811 751 437 437
query74 5170 4947 4804 4804
query75 2622 2598 2218 2218
query76 2333 1184 799 799
query77 363 376 299 299
query78 12414 12506 11724 11724
query79 1426 1197 807 807
query80 1272 490 388 388
query81 510 284 235 235
query82 562 162 118 118
query83 330 273 248 248
query84 307 148 109 109
query85 902 508 409 409
query86 420 298 297 297
query87 3383 3330 3195 3195
query88 3760 2831 2798 2798
query89 432 372 340 340
query90 1959 178 179 178
query91 177 161 132 132
query92 63 58 55 55
query93 1498 1576 889 889
query94 695 391 304 304
query95 684 392 346 346
query96 1086 809 326 326
query97 2696 2671 2556 2556
query98 212 206 197 197
query99 1170 1193 1026 1026
Total cold run time: 261645 ms
Total hot run time: 175572 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4710090655
# FE Regression Coverage Report
Increment line coverage ` 80.25% (451/562)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_832ba5f0bf8ab5846bb316e94f1099ae58fe466f_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_832ba5f0bf8ab5846bb316e94f1099ae58fe466f_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4741464159
<details>
<summary>TPC-DS: <b>Total hot run time: 176552 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0436cc4cb573b104a59408a6169bd145bcfeff6c, data reload: false
query5 4318 652 501 501
query6 433 197 171 171
query7 4834 558 302 302
query8 363 219 208 208
query9 8785 4106 4122 4106
query10 482 303 253 253
query11 5955 2353 2143 2143
query12 153 103 102 102
query13 1289 611 415 415
query14 6461 5395 5052 5052
query14_1 4394 4410 4353 4353
query15 207 200 175 175
query16 1033 451 413 413
query17 1108 697 545 545
query18 2705 477 335 335
query19 221 180 139 139
query20 114 106 107 106
query21 216 142 123 123
query22 13806 13617 13455 13455
query23 17412 16587 16222 16222
query23_1 16314 16376 16297 16297
query24 7466 1785 1367 1367
query24_1 1376 1364 1317 1317
query25 571 458 407 407
query26 1343 308 178 178
query27 2628 580 343 343
query28 4440 2051 2038 2038
query29 1093 655 507 507
query30 321 241 208 208
query31 1118 1078 960 960
query32 112 64 63 63
query33 535 331 261 261
query34 1194 1149 693 693
query35 755 800 692 692
query36 1420 1402 1199 1199
query37 156 103 97 97
query38 3239 3149 3083 3083
query39 945 921 907 907
query39_1 878 911 885 885
query40 222 130 104 104
query41 70 70 70 70
query42 100 97 98 97
query43 340 344 295 295
query44 1514 788 798 788
query45 198 189 182 182
query46 1095 1248 749 749
query47 2352 2379 2240 2240
query48 400 426 298 298
query49 646 491 371 371
query50 1024 376 275 275
query51 4472 4441 4397 4397
query52 91 94 89 89
query53 262 284 199 199
query54 277 239 212 212
query55 87 78 73 73
query56 244 243 238 238
query57 1446 1442 1309 1309
query58 250 232 223 223
query59 1602 1715 1509 1509
query60 270 238 222 222
query61 154 146 151 146
query62 704 649 588 588
query63 230 191 193 191
query64 2517 747 601 601
query65 4882 4813 4805 4805
query66 1724 470 343 343
query67 29882 29803 29575 29575
query68 3548 1559 898 898
query69 421 313 267 267
query70 1093 984 975 975
query71 309 242 211 211
query72 2975 2637 2397 2397
query73 912 801 442 442
query74 5141 4991 4759 4759
query75 2671 2565 2228 2228
query76 2337 1183 811 811
query77 363 376 280 280
query78 12395 12531 11922 11922
query79 1395 1207 780 780
query80 1280 484 398 398
query81 529 284 242 242
query82 563 162 128 128
query83 326 295 265 265
query84 260 150 112 112
query85 902 512 408 408
query86 430 316 282 282
query87 3414 3352 3211 3211
query88 3735 2792 2810 2792
query89 435 381 326 326
query90 1932 188 176 176
query91 173 161 134 134
query92 66 62 53 53
query93 1466 1533 914 914
query94 700 348 346 346
query95 669 382 331 331
query96 1074 792 351 351
query97 2691 2685 2580 2580
query98 224 211 200 200
query99 1171 1148 1022 1022
Total cold run time: 263308 ms
Total hot run time: 176552 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4708799564
<details>
<summary>TPC-H: <b>Total hot run time: 28865 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 14ff5b48070e26ba3f1518c301fce97c36998331, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17792 4097 4116 4097
q2 q3 10919 1427 835 835
q4 4747 477 338 338
q5 8470 868 592 592
q6 356 175 135 135
q7 904 845 620 620
q8 10888 1658 1676 1658
q9 7158 4525 4490 4490
q10 6808 1825 1514 1514
q11 449 276 244 244
q12 644 430 290 290
q13 18276 3421 2808 2808
q14 260 255 248 248
q15 q16 819 770 705 705
q17 926 835 891 835
q18 6909 5720 5559 5559
q19 1310 1298 1100 1100
q20 521 415 265 265
q21 5507 2608 2232 2232
q22 434 355 300 300
Total cold run time: 104097 ms
Total hot run time: 28865 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4429 4376 4423 4376
q2 q3 4572 5023 4309 4309
q4 2092 2173 1377 1377
q5 4504 4361 4374 4361
q6 227 179 147 147
q7 2223 1852 1670 1670
q8 2502 2182 2291 2182
q9 8127 8092 8085 8085
q10 4841 4803 4379 4379
q11 582 428 380 380
q12 755 744 537 537
q13 3325 3630 3021 3021
q14 315 285 272 272
q15 q16 708 720 637 637
q17 1373 1374 1392 1374
q18 7973 7433 6985 6985
q19 1108 1157 1167 1157
q20 2232 2282 1943 1943
q21 5239 4594 4462 4462
q22 516 474 412 412
Total cold run time: 57643 ms
Total hot run time: 52066 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3413407415
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PlanTranslatorContext.java:
##########
@@ -109,10 +109,36 @@ public class PlanTranslatorContext {
private final Map<ScanNode, Set<SlotId>> statsUnknownColumnsMap = Maps.newHashMap();
+ // Per-node "is there a serial operator between me and the pipeline's sink" flag.
+ // Mirrors BE's any_of(operators[idx..end], is_serial_operator) check used by
+ // _add_local_exchange / need_to_local_exchange to skip LE insertion when an ancestor
+ // in the same pipeline is already serial (the whole pipeline runs with 1 task, so an
+ // extra LE would be a no-op). Written by AddLocalExchange entry + PlanNode.enforceRequire
+ // step 1 (root → leaf during traversal). Read by PlanNode.enforceRequire step 4 (Layer 1
+ // skip) and by child overrides that compute their require. Reset to false at fragment
+ // root and across pipeline boundaries (see shouldResetSerialFlagForChild).
+ private final Map<PlanNodeId, Boolean> serialAncestorInPipelineMap = Maps.newHashMap();
+
+ // Per-node "is there a downstream operator that depends on hash distribution for
+ // correctness, with HASH/NOOP path connecting it to me" flag. Mirrors BE's
+ // _followed_by_shuffled_operator propagation in pipeline_fragment_context.cpp.
+ // Written by PlanNode.enforceRequire step 1b (root → leaf). Read by SetOperationNode
+ // to decide whether to propagate hash requirement to its inputs (only when downstream
+ // needs shuffle for correctness, not just for performance like StreamingAgg pre-agg).
+ private final Map<PlanNodeId, Boolean> shuffledAncestorMap = Maps.newHashMap();
+
+ // Whether the current fragment uses LocalShuffleAssignedJob (pooling scan with
+ // ignoreDataDistribution → _parallel_instances=1 in BE). When true, serial operators
+ // indicate real pipeline bottlenecks needing PASSTHROUGH fan-out (heavy_ops).
private boolean isTopMaterializeNode = true;
private final Set<SlotId> virtualColumnIds = Sets.newHashSet();
+ // Used by AddLocalExchange: tracks whether any serial operator exists
Review Comment:
Done in `fe9a3316be0` — removed the field and its getter/setter. The live serial-ancestor propagation uses `serialAncestorInPipelineMap` (set at `AddLocalExchange:94` / `PlanNode:1108`), so this was indeed dead.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java:
##########
@@ -939,4 +1026,261 @@ private String mergeIcebergAccessPathsWithId(
}
return StringUtils.join(mergeDisplayAccessPaths, ", ");
}
+
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ ArrayList<PlanNode> newChildren = Lists.newArrayList();
+ for (int i = 0; i < children.size(); i++) {
+ Pair<PlanNode, LocalExchangeType> childOutput
+ = enforceRequire(translatorContext, children.get(i), i, LocalExchangeTypeRequire.noRequire());
+ newChildren.add(childOutput.first);
+ }
+ this.children = newChildren;
+ return Pair.of(this, LocalExchangeType.NOOP);
+ }
+
+ /**
+ * Unified framework method: propagate serial flag → recurse child → satisfy check → Layer 1 skip → insert LE.
+ * Replaces the old enforceChild/enforceChildExchange/forceEnforceChildExchange trio.
+ *
+ * <h3>Data flow</h3>
+ * <ul>
+ * <li><b>serial-ancestor flag</b> ({@link PlanTranslatorContext#hasSerialAncestorInPipeline})
+ * — flows root → leaf during traversal. Mirrors BE's
+ * {@code any_of(operators[idx..end], is_serial_operator)} check used by
+ * {@code _add_local_exchange} to skip LE insertion when an ancestor in the same
+ * pipeline is already serial. Reset at pipeline boundaries via
+ * {@link #shouldResetSerialFlagForChild}.</li>
+ * <li><b>shuffle-for-correctness flag</b>
+ * ({@link PlanTranslatorContext#hasShuffleForCorrectnessAncestor}) — also flows
+ * root → leaf. Mirrors BE's {@code _followed_by_shuffled_operator}: tells a
+ * child whether some downstream operator depends on hash distribution for
+ * correctness, so {@code SetOperationNode} can pre-shuffle union branches.</li>
+ * <li><b>return value</b> {@code Pair<PlanNode, LocalExchangeType>} — first is the
+ * (possibly LE-wrapped) child; second is the actual output distribution as
+ * observed by the parent. Caller's {@code require.satisfy(output)} decides
+ * whether more LE is needed.</li>
+ * <li><b>parent.require</b> describes the constraint on the child output —
+ * computed inside the parent's {@code enforceAndDeriveLocalExchange} per child.</li>
+ * </ul>
+ *
+ * <h3>Invariants</h3>
+ * <ul>
+ * <li>Every serial → non-serial transition has an LE somewhere between them
+ * (enforced by framework step 3 below, validated by
+ * {@link AddLocalExchange#validateNoSerialWithoutLocalExchange}).</li>
Review Comment:
Done in `fe9a3316be0`. Removed the dead `@link` to `validateNoSerialWithoutLocalExchange` and rewrote the bullet to the narrower truth: framework step 3 inserts the LE only where a serial→non-serial transition actually needs redistribution; a serial child feeding a parent that requires PASSTHROUGH/noRequire (TableFunction, NLJ, Agg) is already correct and needs none — so it is intentionally not validated by a post-pass.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java:
##########
@@ -939,4 +1026,261 @@ private String mergeIcebergAccessPathsWithId(
}
return StringUtils.join(mergeDisplayAccessPaths, ", ");
}
+
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ ArrayList<PlanNode> newChildren = Lists.newArrayList();
+ for (int i = 0; i < children.size(); i++) {
+ Pair<PlanNode, LocalExchangeType> childOutput
+ = enforceRequire(translatorContext, children.get(i), i, LocalExchangeTypeRequire.noRequire());
+ newChildren.add(childOutput.first);
+ }
+ this.children = newChildren;
+ return Pair.of(this, LocalExchangeType.NOOP);
+ }
+
+ /**
+ * Unified framework method: propagate serial flag → recurse child → satisfy check → Layer 1 skip → insert LE.
+ * Replaces the old enforceChild/enforceChildExchange/forceEnforceChildExchange trio.
+ *
+ * <h3>Data flow</h3>
+ * <ul>
+ * <li><b>serial-ancestor flag</b> ({@link PlanTranslatorContext#hasSerialAncestorInPipeline})
+ * — flows root → leaf during traversal. Mirrors BE's
+ * {@code any_of(operators[idx..end], is_serial_operator)} check used by
+ * {@code _add_local_exchange} to skip LE insertion when an ancestor in the same
+ * pipeline is already serial. Reset at pipeline boundaries via
+ * {@link #shouldResetSerialFlagForChild}.</li>
+ * <li><b>shuffle-for-correctness flag</b>
+ * ({@link PlanTranslatorContext#hasShuffleForCorrectnessAncestor}) — also flows
+ * root → leaf. Mirrors BE's {@code _followed_by_shuffled_operator}: tells a
+ * child whether some downstream operator depends on hash distribution for
+ * correctness, so {@code SetOperationNode} can pre-shuffle union branches.</li>
+ * <li><b>return value</b> {@code Pair<PlanNode, LocalExchangeType>} — first is the
+ * (possibly LE-wrapped) child; second is the actual output distribution as
+ * observed by the parent. Caller's {@code require.satisfy(output)} decides
+ * whether more LE is needed.</li>
+ * <li><b>parent.require</b> describes the constraint on the child output —
+ * computed inside the parent's {@code enforceAndDeriveLocalExchange} per child.</li>
+ * </ul>
+ *
+ * <h3>Invariants</h3>
+ * <ul>
+ * <li>Every serial → non-serial transition has an LE somewhere between them
+ * (enforced by framework step 3 below, validated by
+ * {@link AddLocalExchange#validateNoSerialWithoutLocalExchange}).</li>
Review Comment:
Done in `fe9a3316be0` (same fix as the other thread on this line): the `@link` to the removed `validateNoSerialWithoutLocalExchange` is gone and the invariant is reworded to the narrower condition rather than claiming every serial→non-serial transition has an LE.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java:
##########
@@ -142,7 +145,8 @@ public abstract class PlanNode extends TreeNode<PlanNode> {
protected int nereidsId = -1;
- private List<List<Expr>> childrenDistributeExprLists = new ArrayList<>();
+ protected List<List<Expr>> childrenDistributeExprLists = new ArrayList<>();
+ protected List<Expr> distributeExprLists = new ArrayList<>();
Review Comment:
Done in `fe9a3316be0` — added comments: `childrenDistributeExprLists.get(i)` is the per-child hash-key expr list used to repartition the i-th child's input; `distributeExprLists` is this node's own output hash-key exprs serialized to BE.
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -1796,6 +2011,89 @@ Status PipelineFragmentContext::_create_operator(ObjectPool* pool, const TPlanNo
RETURN_IF_ERROR(cur_pipe->add_operator(op, _parallel_instances));
break;
}
+ case TPlanNodeType::LOCAL_EXCHANGE_NODE: {
+ op = std::make_shared<LocalExchangeSourceOperatorX>(pool, tnode, next_operator_id(), descs);
+ // The downstream pipeline (containing LocalExchangeSource) must have
+ // _num_instances tasks — matching BE-native _inherit_pipeline_properties
+ // which sets pipe_with_source.set_num_tasks(_num_instances).
+ // Without this, when the parent pipeline was reduced by a serial operator
+ // (e.g., serial Exchange with use_serial_exchange=true, or UNPARTITIONED
+ // Exchange), the downstream inherits the reduced num_tasks via
+ // add_pipeline(parent). The deferred exchanger creates _num_instances
+ // channels but only fewer source tasks initialize mem_counters — the
+ // sink round-robins to all channels and crashes on uninitialized ones.
+ auto downstream_num_tasks = _num_instances;
Review Comment:
Done in `fe9a3316be0` — inlined to `cur_pipe->set_num_tasks(_num_instances)`.
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -690,6 +716,188 @@ Status PipelineFragmentContext::_build_pipelines(ObjectPool* pool, const Descrip
return Status::OK();
}
+Status PipelineFragmentContext::_create_deferred_local_exchangers() {
+ for (auto& info : _deferred_exchangers) {
+ // DANGER ZONE — do not "fix" this line without reading the history.
+ //
+ // sender_count seeds Exchanger::_running_sink_operators, which the source side
+ // waits to reach 0 via sub_running_sink_operators on each sink LocalState close.
+ // The correct value is THIS pipeline-instance's sink task count, which is exactly
+ // info.upstream_pipe->num_tasks() — one PipelineTask per task, one close per task.
+ //
+ // Tempting wrong fix #1: `std::max(num_tasks, _num_instances)` to mirror the
+ // BE-planned path in _add_local_exchange_impl (~line 1023). THIS BREAKS the
+ // common FE-planned shape of `serial scan → LE(PT) → ...`: upstream_pipe
+ // genuinely has num_tasks=1, only 1 close arrives, but seed becomes
+ // _num_instances so _running_sink_operators never reaches 0 — downstream
+ // sources hang on SHUFFLE_DATA_DEPENDENCY (e.g. MTMV refresh from
+ // mtmv_up_down_job_p0/load.groovy stays at Status=RUNNING; build 949402
Review Comment:
Done in `fe9a3316be0` — dropped the internal build id from the comment (kept the test reference).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3411860865
##########
fe/fe-core/src/main/java/org/apache/doris/planner/AddLocalExchange.java:
##########
@@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.planner;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.glue.translator.PlanTranslatorContext;
+import org.apache.doris.nereids.trees.plans.distribute.DistributedPlan;
+import org.apache.doris.nereids.trees.plans.distribute.FragmentIdMapping;
+import org.apache.doris.nereids.trees.plans.distribute.PipelineDistributedPlan;
+import org.apache.doris.planner.LocalExchangeNode.LocalExchangeType;
+import org.apache.doris.planner.LocalExchangeNode.LocalExchangeTypeRequire;
+import org.apache.doris.planner.LocalExchangeNode.RequireHash;
+
+/**
+ * FE-side local exchange planner — inserts {@link LocalExchangeNode} into each fragment's
+ * plan tree so that within-fragment data redistribution is decided at planning time
+ * instead of at BE pipeline-build time.
+ *
+ * <h3>When this runs</h3>
+ * Invoked from {@code NereidsPlanner.addLocalExchangeAfterDistribute()} right after
+ * {@code DistributePlanner} has assigned instances to fragments and before the plan is
+ * serialized to BE. Gated by session variable {@code enable_local_shuffle_planner}
+ * (default true) and {@code enable_local_shuffle}; when either is off this pass is
+ * skipped entirely and BE falls back to its own {@code _plan_local_exchange}. The two
+ * paths are mutually exclusive: BE consults {@code runtime_state.h::plan_local_shuffle()}
+ * to know whether it should plan LE itself.
+ *
+ * <h3>What it changes</h3>
+ * <ul>
+ * <li>For each fragment with {@code maxPerBeInstances > 1}, walks the plan tree
+ * bottom-up via {@link PlanNode#enforceAndDeriveLocalExchange} and inserts
+ * LocalExchangeNodes where children's output distribution doesn't satisfy the
+ * parent's requirement.</li>
+ * <li>May wrap the fragment root with an extra PASSTHROUGH LE so the data sink
+ * (DataStreamSink / OlapTableSink) runs with the full instance count even when
+ * the root operator is serial — see {@link #addLocalExchangeForFragment}.</li>
+ * <li>Does NOT modify the fragment sink itself, fragment boundaries, or instance
+ * assignment.</li>
+ * </ul>
+ *
+ * <h3>Per-BE instance semantics</h3>
+ * Skips fragments where every BE has at most 1 instance. Using a global instance count
+ * would insert LE for "2 BEs × 1 instance" cases, which BE's own
+ * {@code _plan_local_exchange} would not — leading to pipeline task-count mismatch and
+ * deadlock. See {@link #addLocalExchange}.
+ *
+ * <h3>Reading order</h3>
+ * Start with {@link PlanNode#enforceRequire} (the recursion engine), then individual
+ * {@code enforceAndDeriveLocalExchange} overrides on PlanNode subclasses.
+ */
+public class AddLocalExchange {
+ /** addLocalExchange with distributed plans, skipping single-instance fragments.
+ * BE's _plan_local_exchange checks _num_instances which is the per-BE instance count.
+ * With _num_instances<=1 all pipelines on that BE have 1 task so local exchange is a no-op.
+ * We must use the same per-BE semantics: skip when every BE has at most 1 instance.
+ * Using global instanceCount would insert LE for fragments where 2 BEs each have 1 instance
+ * (global=2, per-BE=1), causing pipeline task mismatch and deadlock. */
+ public void addLocalExchange(FragmentIdMapping<DistributedPlan> distributedPlans,
+ PlanTranslatorContext context) {
+ for (DistributedPlan plan : distributedPlans.values()) {
+ PipelineDistributedPlan pipePlan = (PipelineDistributedPlan) plan;
+ long maxPerBeInstances = pipePlan.getInstanceJobs().stream()
+ .collect(java.util.stream.Collectors.groupingBy(
+ j -> j.getAssignedWorker().id(), java.util.stream.Collectors.counting()))
+ .values().stream().mapToLong(Long::longValue).max().orElse(0);
+ if (maxPerBeInstances <= 1) {
+ continue;
+ }
+ PlanFragment fragment = pipePlan.getFragmentJob().getFragment();
+ addLocalExchangeForFragment(fragment, context);
+ }
+ }
+
+ private void addLocalExchangeForFragment(PlanFragment fragment, PlanTranslatorContext context) {
+ DataSink sink = fragment.getSink();
+ LocalExchangeTypeRequire require = sink == null
+ ? LocalExchangeTypeRequire.noRequire() : sink.getLocalExchangeTypeRequire();
+ PlanNode root = fragment.getPlanRoot();
+ context.setHasSerialAncestorInPipeline(root, false);
+ Pair<PlanNode, LocalExchangeType> output = root
+ .enforceAndDeriveLocalExchange(context, null, require);
+ PlanNode newRoot = output.first;
+ // The fragment data sink (DataStreamSink, OlapTableSink) runs in the same pipeline
+ // as the root. If the root will be serial on BE, the sink pipeline has 1 task —
+ // only instance 0 sends data, others hang or miss writes.
+ // Insert PASSTHROUGH fan-out so sink runs with _num_instances tasks.
+ // This matches BE-native's default required_data_distribution():
+ // _child->is_serial_operator() ? PASSTHROUGH : NOOP
+ if (newRoot.isSerialOperatorOnBe(context.getConnectContext())) {
+ newRoot = new LocalExchangeNode(context.nextPlanNodeId(), newRoot,
+ LocalExchangeType.PASSTHROUGH, null);
+ }
+ if (newRoot != root) {
+ fragment.setPlanRoot(newRoot);
+ }
+ validateNoSerialWithoutLocalExchange(fragment.getPlanRoot(), context.getConnectContext());
+ }
+
+ /**
+ * In a local-shuffle fragment, the root check above guarantees the root pipeline
+ * has N tasks. Any serial operator reduces its pipeline to 1 task. If this serial
+ * operator feeds into a non-serial parent without LocalExchangeNode in between,
+ * some pipelines have 1 task while others have N → shared_state mismatch, data loss.
+ *
+ * Serial→serial chains are fine (all at 1 task, consistent). Only the transition
+ * from serial to non-serial needs LE to restore parallelism.
+ */
+ private void validateNoSerialWithoutLocalExchange(PlanNode node,
+ org.apache.doris.qe.ConnectContext context) {
+ for (PlanNode child : node.getChildren()) {
+ validateNoSerialWithoutLocalExchange(child, context);
+ if (child.isSerialOperatorOnBe(context)
+ && !(child instanceof LocalExchangeNode)
+ && !(node instanceof LocalExchangeNode)
+ && !(node instanceof ExchangeNode)
+ && !node.isSerialOperatorOnBe(context)) {
+ org.apache.logging.log4j.LogManager.getLogger(AddLocalExchange.class).warn(
+ "Serial " + child.getClass().getSimpleName() + "(id=" + child.getId()
+ + ") feeds into non-serial " + node.getClass().getSimpleName()
+ + "(id=" + node.getId() + ") without LocalExchangeNode"
+ + " in fragment " + node.getFragment().getFragmentId()
+ + ". FE should insert LocalExchangeNode to restore parallelism.");
+ }
+ }
+ }
+
+ public static boolean isColocated(PlanNode plan) {
+ if (plan instanceof AggregationNode) {
+ return ((AggregationNode) plan).isColocate() && isColocated(plan.getChild(0));
+ } else if (plan instanceof OlapScanNode) {
+ return true;
+ } else if (plan instanceof SelectNode) {
+ return isColocated(plan.getChild(0));
+ } else if (plan instanceof HashJoinNode) {
+ return ((HashJoinNode) plan).isColocate()
+ && (isColocated(plan.getChild(0)) || isColocated(plan.getChild(1)));
+ } else if (plan instanceof SetOperationNode) {
+ if (!((SetOperationNode) plan).isColocate()) {
+ return false;
+ }
+ for (PlanNode child : plan.getChildren()) {
+ if (isColocated(child)) {
+ return true;
+ }
Review Comment:
Done — removed all three unused params (`translatorContext`/`parent`/`child`); `resolveExchangeType(LocalExchangeTypeRequire require)` now reads only `require`. Updated all call sites + the unit test (this also resolves the `SetOperationNode` null-`firstChild` note below).
##########
fe/fe-core/src/main/java/org/apache/doris/planner/AddLocalExchange.java:
##########
@@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.planner;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.glue.translator.PlanTranslatorContext;
+import org.apache.doris.nereids.trees.plans.distribute.DistributedPlan;
+import org.apache.doris.nereids.trees.plans.distribute.FragmentIdMapping;
+import org.apache.doris.nereids.trees.plans.distribute.PipelineDistributedPlan;
+import org.apache.doris.planner.LocalExchangeNode.LocalExchangeType;
+import org.apache.doris.planner.LocalExchangeNode.LocalExchangeTypeRequire;
+import org.apache.doris.planner.LocalExchangeNode.RequireHash;
+
+/**
+ * FE-side local exchange planner — inserts {@link LocalExchangeNode} into each fragment's
+ * plan tree so that within-fragment data redistribution is decided at planning time
+ * instead of at BE pipeline-build time.
+ *
+ * <h3>When this runs</h3>
+ * Invoked from {@code NereidsPlanner.addLocalExchangeAfterDistribute()} right after
+ * {@code DistributePlanner} has assigned instances to fragments and before the plan is
+ * serialized to BE. Gated by session variable {@code enable_local_shuffle_planner}
+ * (default true) and {@code enable_local_shuffle}; when either is off this pass is
+ * skipped entirely and BE falls back to its own {@code _plan_local_exchange}. The two
+ * paths are mutually exclusive: BE consults {@code runtime_state.h::plan_local_shuffle()}
+ * to know whether it should plan LE itself.
+ *
+ * <h3>What it changes</h3>
+ * <ul>
+ * <li>For each fragment with {@code maxPerBeInstances > 1}, walks the plan tree
+ * bottom-up via {@link PlanNode#enforceAndDeriveLocalExchange} and inserts
+ * LocalExchangeNodes where children's output distribution doesn't satisfy the
+ * parent's requirement.</li>
+ * <li>May wrap the fragment root with an extra PASSTHROUGH LE so the data sink
+ * (DataStreamSink / OlapTableSink) runs with the full instance count even when
+ * the root operator is serial — see {@link #addLocalExchangeForFragment}.</li>
+ * <li>Does NOT modify the fragment sink itself, fragment boundaries, or instance
+ * assignment.</li>
+ * </ul>
+ *
+ * <h3>Per-BE instance semantics</h3>
+ * Skips fragments where every BE has at most 1 instance. Using a global instance count
+ * would insert LE for "2 BEs × 1 instance" cases, which BE's own
+ * {@code _plan_local_exchange} would not — leading to pipeline task-count mismatch and
+ * deadlock. See {@link #addLocalExchange}.
+ *
+ * <h3>Reading order</h3>
+ * Start with {@link PlanNode#enforceRequire} (the recursion engine), then individual
+ * {@code enforceAndDeriveLocalExchange} overrides on PlanNode subclasses.
+ */
+public class AddLocalExchange {
+ /** addLocalExchange with distributed plans, skipping single-instance fragments.
+ * BE's _plan_local_exchange checks _num_instances which is the per-BE instance count.
+ * With _num_instances<=1 all pipelines on that BE have 1 task so local exchange is a no-op.
+ * We must use the same per-BE semantics: skip when every BE has at most 1 instance.
+ * Using global instanceCount would insert LE for fragments where 2 BEs each have 1 instance
+ * (global=2, per-BE=1), causing pipeline task mismatch and deadlock. */
+ public void addLocalExchange(FragmentIdMapping<DistributedPlan> distributedPlans,
+ PlanTranslatorContext context) {
+ for (DistributedPlan plan : distributedPlans.values()) {
+ PipelineDistributedPlan pipePlan = (PipelineDistributedPlan) plan;
+ long maxPerBeInstances = pipePlan.getInstanceJobs().stream()
+ .collect(java.util.stream.Collectors.groupingBy(
+ j -> j.getAssignedWorker().id(), java.util.stream.Collectors.counting()))
+ .values().stream().mapToLong(Long::longValue).max().orElse(0);
+ if (maxPerBeInstances <= 1) {
+ continue;
+ }
+ PlanFragment fragment = pipePlan.getFragmentJob().getFragment();
+ addLocalExchangeForFragment(fragment, context);
+ }
+ }
+
+ private void addLocalExchangeForFragment(PlanFragment fragment, PlanTranslatorContext context) {
+ DataSink sink = fragment.getSink();
+ LocalExchangeTypeRequire require = sink == null
+ ? LocalExchangeTypeRequire.noRequire() : sink.getLocalExchangeTypeRequire();
+ PlanNode root = fragment.getPlanRoot();
+ context.setHasSerialAncestorInPipeline(root, false);
+ Pair<PlanNode, LocalExchangeType> output = root
+ .enforceAndDeriveLocalExchange(context, null, require);
+ PlanNode newRoot = output.first;
+ // The fragment data sink (DataStreamSink, OlapTableSink) runs in the same pipeline
+ // as the root. If the root will be serial on BE, the sink pipeline has 1 task —
+ // only instance 0 sends data, others hang or miss writes.
+ // Insert PASSTHROUGH fan-out so sink runs with _num_instances tasks.
+ // This matches BE-native's default required_data_distribution():
+ // _child->is_serial_operator() ? PASSTHROUGH : NOOP
+ if (newRoot.isSerialOperatorOnBe(context.getConnectContext())) {
+ newRoot = new LocalExchangeNode(context.nextPlanNodeId(), newRoot,
+ LocalExchangeType.PASSTHROUGH, null);
+ }
+ if (newRoot != root) {
+ fragment.setPlanRoot(newRoot);
+ }
+ validateNoSerialWithoutLocalExchange(fragment.getPlanRoot(), context.getConnectContext());
+ }
+
+ /**
+ * In a local-shuffle fragment, the root check above guarantees the root pipeline
+ * has N tasks. Any serial operator reduces its pipeline to 1 task. If this serial
+ * operator feeds into a non-serial parent without LocalExchangeNode in between,
+ * some pipelines have 1 task while others have N → shared_state mismatch, data loss.
+ *
+ * Serial→serial chains are fine (all at 1 task, consistent). Only the transition
+ * from serial to non-serial needs LE to restore parallelism.
+ */
+ private void validateNoSerialWithoutLocalExchange(PlanNode node,
+ org.apache.doris.qe.ConnectContext context) {
+ for (PlanNode child : node.getChildren()) {
+ validateNoSerialWithoutLocalExchange(child, context);
+ if (child.isSerialOperatorOnBe(context)
+ && !(child instanceof LocalExchangeNode)
Review Comment:
Removed the method entirely (commit `3630071a131`) rather than adding a throw/DCHECK/metric — see the longer reply on the [P1] thread at `AddLocalExchange:132`. Short version: the invariant is self-referential (it reads the same `isSerialOperatorOnBe` flag it validates), so it can't catch the one bug class that matters, and it false-fires on legitimate serial→{TableFunction, NLJ, Agg} plans. The regression suite + RQG differential are the actual guard.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/SetOperationNode.java:
##########
@@ -193,4 +198,52 @@ public int getNumInstances() {
public boolean isBucketShuffle() {
return distributionMode.equals(DistributionMode.BUCKET_SHUFFLE);
}
+
+ public boolean isColocate() {
+ return isColocate;
+ }
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(PlanTranslatorContext translatorContext,
+ PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ LocalExchangeTypeRequire requireChild;
+ LocalExchangeType outputType;
+ PlanNode firstChild = children.isEmpty() ? null : children.get(0);
+ if (this instanceof UnionNode) {
+ // Propagate parent's hash requirement to children ONLY when a downstream operator
+ // requires shuffle for correctness (not just performance optimization). Matches BE's
Review Comment:
Resolved by removing the unused `child` param from `resolveExchangeType` — `SetOperationNode` no longer passes `firstChild` (only `requireChild` at `SetOperationNode.java:222/239`), so no null can reach it.
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -690,6 +716,199 @@ Status PipelineFragmentContext::_build_pipelines(ObjectPool* pool, const Descrip
return Status::OK();
}
+Status PipelineFragmentContext::_create_deferred_local_exchangers() {
+ for (auto& info : _deferred_exchangers) {
+ // DANGER ZONE — do not "fix" this line without reading the history.
+ //
+ // sender_count seeds Exchanger::_running_sink_operators, which the source side
+ // waits to reach 0 via sub_running_sink_operators on each sink LocalState close.
+ // The correct value is THIS pipeline-instance's sink task count, which is exactly
+ // info.upstream_pipe->num_tasks() — one PipelineTask per task, one close per task.
+ //
+ // Tempting wrong fix #1: `std::max(num_tasks, _num_instances)` to mirror the
+ // BE-planned path in _add_local_exchange_impl (~line 1023). THIS BREAKS the
+ // common FE-planned shape of `serial scan → LE(PT) → ...`: upstream_pipe
+ // genuinely has num_tasks=1, only 1 close arrives, but seed becomes
+ // _num_instances so _running_sink_operators never reaches 0 — downstream
+ // sources hang on SHUFFLE_DATA_DEPENDENCY (e.g. MTMV refresh from
+ // mtmv_up_down_job_p0/load.groovy stays at Status=RUNNING; build 949402
+ // regressed exactly this way). BE-planned mode uses max() because its
+ // `cur_pipe` is the source-side pipeline (always raised to _num_instances by
+ // add_pipeline) — not analogous to our `upstream_pipe` here, which is the
+ // sink-side pipeline that may legitimately stay at 1 for serial sources.
+ //
+ // Tempting wrong fix #2: multiply by _num_instances on the theory shared_state
+ // is shared across all instances. Same hang — each fragment-instance
+ // PipelineFragmentContext has its OWN _op_id_to_shared_state map, so the
+ // exchanger is per-instance, not per-BE. num_tasks() is already the right
+ // close-count for one instance.
+ //
+ // If a hang shows up with `_running_sink_operators < 0`, the bug is upstream:
+ // _propagate_local_exchange_num_tasks left num_tasks too low (or too high) for
+ // this fragment shape. Fix THAT pass, not this seed value.
+ const int sender_count = info.upstream_pipe->num_tasks();
Review Comment:
Replaced the two iterative `while (changed)` passes in `_propagate_local_exchange_num_tasks()` with a single Kahn topological sweep over the pipeline DAG (commit `a97d3347789`): each pipeline is visited once, after all of its upstreams, so `num_tasks` is set in one pass and the loop can't run unbounded. Added a cycle `DCHECK` instead of a max-iteration guard.
##########
be/src/exec/pipeline/pipeline_fragment_context.h:
##########
@@ -320,8 +322,11 @@ class PipelineFragmentContext : public TaskExecutionContext {
std::mutex _state_map_lock;
- int _operator_id = 0;
- int _sink_operator_id = 0;
+ // Start from -1 so all operator IDs are negative. This avoids collision with
Review Comment:
Thanks — confirmed. Operator IDs are only used as map keys / `dest_id` equality and printed as-is in profiles; nothing enumerates them expecting non-negative. The only hardcoded `0` was the unpaired sink `dest_id=0`, which is exactly the collision the `-1` start avoids. No change needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4707388930
<details>
<summary>TPC-H: <b>Total hot run time: 28712 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1d7cf6dbd6095ee1fb5860efe258b0ce99242ae6, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17658 4065 4030 4030
q2 q3 10957 1469 834 834
q4 4747 475 341 341
q5 8273 899 588 588
q6 321 171 136 136
q7 896 840 632 632
q8 10931 1566 1728 1566
q9 7277 4521 4519 4519
q10 6878 1794 1545 1545
q11 445 278 243 243
q12 634 430 295 295
q13 18112 3405 2723 2723
q14 267 258 239 239
q15 q16 827 776 710 710
q17 901 876 983 876
q18 6824 5756 5537 5537
q19 1144 1196 963 963
q20 518 396 256 256
q21 5593 2659 2366 2366
q22 450 352 313 313
Total cold run time: 103653 ms
Total hot run time: 28712 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4336 4257 4252 4252
q2 q3 4570 4919 4295 4295
q4 2104 2194 1396 1396
q5 4437 4323 4339 4323
q6 229 176 130 130
q7 2126 2034 1666 1666
q8 2546 2210 2219 2210
q9 8333 8174 8228 8174
q10 4853 4740 4345 4345
q11 577 417 403 403
q12 745 770 559 559
q13 3364 3605 2964 2964
q14 309 308 270 270
q15 q16 705 737 641 641
q17 1342 1456 1339 1339
q18 8011 7466 7109 7109
q19 1084 1094 1116 1094
q20 2229 2206 1937 1937
q21 5296 4575 4485 4485
q22 536 446 408 408
Total cold run time: 57732 ms
Total hot run time: 52000 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3413517236
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -290,6 +290,32 @@ Status PipelineFragmentContext::_build_and_prepare_full_pipeline(ThreadPool* thr
RETURN_IF_ERROR(_build_pipelines(_runtime_state->obj_pool(), *_query_ctx->desc_tbl,
&_root_op, root_pipeline));
+ // Propagate _num_instances from LOCAL_EXCHANGE pipelines to ancestor pipelines
+ // that inherited reduced num_tasks from a serial operator.
+ _propagate_local_exchange_num_tasks();
+
+ // Create deferred local exchangers now that all pipelines have final num_tasks.
+ RETURN_IF_ERROR(_create_deferred_local_exchangers());
Review Comment:
是的，这两块不是为了灰度/回滚——是 FE 接管 local exchange 的**当前折中**，不是一步到位。
要让 FE 把"每个算子的并发"也规划好直接下发，本质上需要 FE 自己切 pipeline、自己控制每条 pipeline 的并发（num_tasks 是 pipeline 粒度的，只在 pipeline 边界即 local exchange / pipeline breaker 处变化），BE 就不再规划 pipeline 了。那等于把 BE 的整套 pipeline 构建（`_build_pipeline`、pipeline-breaker 判定、num_tasks 推导、dependency 接线，还要兼顾 spill / runtime filter / shared state）搬到 FE，并改 thrift 契约携带完整 pipeline 结构——战线太长、风险太大。这套 FE 近似等价实现已经做了大半年、近 1 万行代码，没法一把全做。
所以现状是：FE 规划 LE 的**位置/类型/分布**（决定 plan 正确性，过去正是 BE/FE 分歧 bug 的根源），`enable_local_shuffle_planner=true` 时 BE 的 `_plan_local_exchange` 整个跳过；但每条 pipeline 的 num_tasks 仍由 BE 的 `_propagate_local_exchange_num_tasks` 拓扑推导，`_create_deferred_local_exchangers` 因为 exchanger 的 `num_partitions` 要等 num_tasks 定了才能建、所以延后创建。终态是 FE 连 num_tasks 也算好下发、BE 的 propagate 退化成 assert（只校验）——那是后续单独的大改。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3411827399
##########
gensrc/thrift/Partitions.thrift:
##########
@@ -55,6 +55,40 @@ enum TPartitionType {
MERGE_PARTITIONED = 9
}
+enum TLocalPartitionType {
+ NOOP = 0,
+ // used to resume the global hash distribution because other distribution break the global hash distribution,
+ // such as PASSTHROUGH. and then JoinNode can shuffle data by the same hash distribution.
+ //
+ // for example: look here, need resume to GLOBAL_EXECUTION_HASH_SHUFFLE
+ // ↓
+ // Node -> LocalExchangeNode(PASSTHROUGH) → JoinNode → LocalExchangeNode(GLOBAL_EXECUTION_HASH_SHUFFLE) → JoinNode
+ // ExchangeNode(BROADCAST) ↗ ↑
+ // ExchangeNode(GLOBAL_EXECUTION_HASH_SHUFFLE)
+ GLOBAL_EXECUTION_HASH_SHUFFLE = 1,
+ // used to rebalance data for rebalance data and add parallelism
+ //
+ // for example: look here, need use LOCAL_EXECUTION_HASH_SHUFFLE to rebalance data
+ // ↓
+ // Scan(hash(id)) -> LocalExchangeNode(LOCAL_EXECUTION_HASH_SHUFFLE(id, name)) → AggregationNode(group by(id,name))
Review Comment:
This example is not very rigorous. The `hash(id)` function also ensures that tuples with the same `id` and `name` are assigned to the same instance. It would be more reasonable to change it to "scan(hash(id, name))" and "agg(hash(id))".
##########
gensrc/thrift/Partitions.thrift:
##########
@@ -55,6 +55,40 @@ enum TPartitionType {
MERGE_PARTITIONED = 9
}
+enum TLocalPartitionType {
+ NOOP = 0,
+ // used to resume the global hash distribution because other distribution break the global hash distribution,
+ // such as PASSTHROUGH. and then JoinNode can shuffle data by the same hash distribution.
+ //
+ // for example: look here, need resume to GLOBAL_EXECUTION_HASH_SHUFFLE
+ // ↓
+ // Node -> LocalExchangeNode(PASSTHROUGH) → JoinNode → LocalExchangeNode(GLOBAL_EXECUTION_HASH_SHUFFLE) → JoinNode
+ // ExchangeNode(BROADCAST) ↗ ↑
+ // ExchangeNode(GLOBAL_EXECUTION_HASH_SHUFFLE)
+ GLOBAL_EXECUTION_HASH_SHUFFLE = 1,
+ // used to rebalance data for rebalance data and add parallelism
+ //
+ // for example: look here, need use LOCAL_EXECUTION_HASH_SHUFFLE to rebalance data
+ // ↓
+ // Scan(hash(id)) -> LocalExchangeNode(LOCAL_EXECUTION_HASH_SHUFFLE(id, name)) → AggregationNode(group by(id,name))
+ //
+ // the LOCAL_EXECUTION_HASH_SHUFFLE is necessary because the hash distribution of scan node is based on id,
+ // but the hash distribution of aggregation node is based on id and name, so we need to rebalance data by both
+ // id and name to make sure the data with same id and name can be sent to the same instance of aggregation node.
+ // and we can not use GLOBAL_EXECUTION_HASH_SHUFFLE(id, name) here, because
+ // `TPipelineFragmentParams.shuffle_idx_to_instance_idx` is used to mapping partial global instance index to local
+ // instance index, and discard the other backend's instance index, the data not belong to the local instance will be
+ // discarded, which cause data loss.
+ LOCAL_EXECUTION_HASH_SHUFFLE = 2,
+ BUCKET_HASH_SHUFFLE = 3,
+ // round-robin partition, used to rebalance data for rebalance data and add parallelism
+ PASSTHROUGH = 4,
+ ADAPTIVE_PASSTHROUGH = 5,
+ BROADCAST = 6,
+ PASS_TO_ONE = 7,
+ LOCAL_MERGE_SORT = 8
Review Comment:
all of these enum value should add comment to explain what it is
##########
gensrc/thrift/PlanNodes.thrift:
##########
@@ -1427,6 +1428,24 @@ struct TExchangeNode {
4: optional Partitions.TPartitionType partition_type
}
+struct TLocalExchangeNode {
+ 1: required Partitions.TLocalPartitionType partition_type
Review Comment:
all fields should be optional
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4766920552
# BE Regression && UT Coverage Report
Increment line coverage `90.15% (366/406)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 74.09% (28397/38326) |
| Line Coverage | 58.01% (309698/533900) |
| Region Coverage | 54.80% (259297/473128) |
| Branch Coverage | 56.16% (112650/200588) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4766610456
# BE Regression && UT Coverage Report
Increment line coverage `90.15% (366/406)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 74.15% (28417/38324) |
| Line Coverage | 58.03% (309827/533892) |
| Region Coverage | 54.83% (259396/473126) |
| Branch Coverage | 56.18% (112693/200588) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4680860551
# FE UT Coverage Report
Increment line coverage ` 74.48% (429/576)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_1f614d8cadd1619a8c17dd6f7ec7b12ba6983286/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_1f614d8cadd1619a8c17dd6f7ec7b12ba6983286/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4682363000
# FE Regression Coverage Report
Increment line coverage ` 79.51% (458/576)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_784476ead971eeebeb0b637f6da4a434e380ce36_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_784476ead971eeebeb0b637f6da4a434e380ce36_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4479943101
<details>
<summary>TPC-H: <b>Total hot run time: 31734 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 15d92ba024b265d9441145d7bd26d93590aa9c63, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17803 3914 3926 3914
q2 q3 10929 1434 801 801
q4 4816 481 340 340
q5 10658 2257 2105 2105
q6 397 175 139 139
q7 914 793 617 617
q8 9633 1832 1667 1667
q9 6923 5130 4985 4985
q10 6518 2090 1770 1770
q11 430 268 239 239
q12 668 423 298 298
q13 18212 3499 2774 2774
q14 260 253 239 239
q15 q16 823 779 721 721
q17 918 940 963 940
q18 7080 5849 6162 5849
q19 1241 1333 1062 1062
q20 503 407 258 258
q21 6216 2696 2699 2696
q22 458 385 320 320
Total cold run time: 105400 ms
Total hot run time: 31734 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4514 4779 4621 4621
q2 q3 4801 5222 4621 4621
q4 2181 2213 1441 1441
q5 4809 4669 4682 4669
q6 232 191 134 134
q7 1807 1659 1400 1400
q8 2178 1885 1875 1875
q9 7379 7481 7388 7388
q10 4474 4422 3979 3979
q11 540 377 348 348
q12 725 726 508 508
q13 2994 3423 2767 2767
q14 270 277 253 253
q15 q16 670 702 606 606
q17 1267 1240 1233 1233
q18 7261 6876 6773 6773
q19 1096 1105 1118 1105
q20 2217 2232 1921 1921
q21 5321 4605 4449 4449
q22 523 482 399 399
Total cold run time: 55259 ms
Total hot run time: 50490 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4484994044
<details>
<summary>TPC-H: <b>Total hot run time: 31488 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4affd222d6550dd103b7829a6fced0b7d45d317a, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17594 3891 3973 3891
q2 q3 10951 1405 844 844
q4 4808 482 346 346
q5 10383 2275 2135 2135
q6 365 183 144 144
q7 936 776 645 645
q8 9783 1852 1610 1610
q9 7005 4945 4940 4940
q10 6473 2240 1807 1807
q11 428 262 242 242
q12 653 436 297 297
q13 18141 3324 2779 2779
q14 270 250 237 237
q15 q16 811 768 721 721
q17 908 941 925 925
q18 7073 5714 5921 5714
q19 1229 1310 1127 1127
q20 519 416 272 272
q21 5733 2807 2498 2498
q22 451 371 314 314
Total cold run time: 104514 ms
Total hot run time: 31488 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4640 4749 4525 4525
q2 q3 4851 5185 4650 4650
q4 2126 2222 1435 1435
q5 4774 4715 4650 4650
q6 224 166 119 119
q7 1673 1601 1396 1396
q8 2223 1908 1912 1908
q9 7411 7410 7408 7408
q10 4480 4414 3997 3997
q11 522 382 348 348
q12 713 720 507 507
q13 3056 3419 2783 2783
q14 282 275 251 251
q15 q16 678 701 610 610
q17 1267 1251 1255 1251
q18 7354 6931 6785 6785
q19 1154 1100 1150 1100
q20 2228 2221 1933 1933
q21 5311 4653 4496 4496
q22 530 453 396 396
Total cold run time: 55497 ms
Total hot run time: 50548 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4499368773
# FE Regression Coverage Report
Increment line coverage ` 80.70% (485/601)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_3ac8525eaac942c0bbeb93cf46d4c95ecf353156_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_3ac8525eaac942c0bbeb93cf46d4c95ecf353156_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3259079337
##########
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##########
@@ -1598,6 +1600,12 @@ public enum IgnoreSplitType {
"Whether to enable local shuffle on pipelineX engine."}, needForward = true)
private boolean enableLocalShuffle = true;
+ @VarAttrDef.VarAttr(
+ name = ENABLE_LOCAL_SHUFFLE_PLANNER, fuzzy = false, varType = VariableAnnotation.EXPERIMENTAL,
+ description = {"是否在FE规划Local Shuffle",
+ "Whether to plan local shuffle in frontend"}, needForward = true)
+ private boolean enableLocalShufflePlanner = true;
+
Review Comment:
This default enables the FE planner to serialize the new `LOCAL_EXCHANGE_NODE`/`TLocalPartitionType` protocol to every BE as soon as the FE is upgraded. During a rolling upgrade, an old BE does not have `TPlanNodeType::LOCAL_EXCHANGE_NODE` handling in `_create_operator` and will reject/fail such fragments, while `RuntimeState::plan_local_shuffle()` also disables the legacy BE planner because `enable_local_shuffle_planner` is set. Please gate this path on BE capability/version (or keep the default off until all BEs support the new node) so mixed FE/BE deployments continue to execute queries during upgrade.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java:
##########
@@ -933,4 +1020,229 @@ private String mergeIcebergAccessPathsWithId(
}
return StringUtils.join(mergeDisplayAccessPaths, ", ");
}
+
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ ArrayList<PlanNode> newChildren = Lists.newArrayList();
+ for (int i = 0; i < children.size(); i++) {
+ Pair<PlanNode, LocalExchangeType> childOutput
+ = enforceRequire(translatorContext, children.get(i), i, LocalExchangeTypeRequire.noRequire());
+ newChildren.add(childOutput.first);
+ }
+ this.children = newChildren;
+ return Pair.of(this, LocalExchangeType.NOOP);
+ }
+
+ /**
+ * Unified framework method: propagate serial flag → recurse child → satisfy check → Layer 1 skip → insert LE.
+ * Replaces the old enforceChild/enforceChildExchange/forceEnforceChildExchange trio.
+ *
+ * <h3>Data flow</h3>
+ * <ul>
+ * <li><b>serial-ancestor flag</b> ({@link PlanTranslatorContext#hasSerialAncestorInPipeline})
+ * — flows root → leaf during traversal. Mirrors BE's
+ * {@code any_of(operators[idx..end], is_serial_operator)} check used by
+ * {@code _add_local_exchange} to skip LE insertion when an ancestor in the same
+ * pipeline is already serial. Reset at pipeline boundaries via
+ * {@link #shouldResetSerialFlagForChild}.</li>
+ * <li><b>shuffle-for-correctness flag</b>
+ * ({@link PlanTranslatorContext#hasShuffleForCorrectnessAncestor}) — also flows
+ * root → leaf. Mirrors BE's {@code _followed_by_shuffled_operator}: tells a
+ * child whether some downstream operator depends on hash distribution for
+ * correctness, so {@code SetOperationNode} can pre-shuffle union branches.</li>
+ * <li><b>return value</b> {@code Pair<PlanNode, LocalExchangeType>} — first is the
+ * (possibly LE-wrapped) child; second is the actual output distribution as
+ * observed by the parent. Caller's {@code require.satisfy(output)} decides
+ * whether more LE is needed.</li>
+ * <li><b>parent.require</b> describes the constraint on the child output —
+ * computed inside the parent's {@code enforceAndDeriveLocalExchange} per child.</li>
+ * </ul>
+ *
+ * <h3>Invariants</h3>
+ * <ul>
+ * <li>Every serial → non-serial transition has an LE somewhere between them
+ * (enforced by framework step 3 below, validated by
+ * {@link AddLocalExchange#validateNoSerialWithoutLocalExchange}).</li>
+ * <li>{@code LocalExchangeNode} itself is always non-serial — setting it serial
+ * would defeat its purpose of fanning a 1-task pipeline back to N tasks.</li>
+ * <li>For pipeline-breaking parents ({@code shouldResetSerialFlagForChild=true}),
+ * the child starts a fresh pipeline so {@code hasSerialAncestor} is reset; the
+ * node's own {@code isSerialNode()} still composes in for the child's view.</li>
+ * <li>{@code RequireHash} accepts any hash flavour; {@code RequireSpecific} demands
+ * an exact match (with the one PASSTHROUGH/ADAPTIVE_PASSTHROUGH compatibility).
+ * Pick the looser one whenever correctness allows — see
+ * {@link LocalExchangeNode.LocalExchangeTypeRequire}.</li>
+ * </ul>
+ *
+ * <h3>Layers</h3>
+ * Layer 1 (shouldSkipLE): mirrors BE's need_to_local_exchange — skip when this node or
+ * an ancestor in the same pipeline is serial (operators[idx..end] has serial → skip).
+ * Layer 2 (require/output): each Node declares require and output in enforceAndDeriveLocalExchange.
+ */
+ protected Pair<PlanNode, LocalExchangeType> enforceRequire(
+ PlanTranslatorContext translatorContext, PlanNode child, int childIndex,
+ LocalExchangeTypeRequire require) {
+ // 1. Propagate serial-ancestor flag to child.
+ // For pipeline-splitting operators (shouldReset=true, e.g. non-streaming AGG):
+ // Drop inherited serial flag from parent (parent is in a different pipeline),
+ // but keep this node's own serial status (child is in the same pipeline as this
+ // node's sink, e.g. Exchange is in AGG_Sink pipeline).
+ // For non-splitting operators (shouldReset=false, e.g. streaming AGG):
+ // Inherit parent's serial flag + this node's own.
+ boolean inheritedSerial = shouldResetSerialFlagForChild(childIndex)
+ ? false : translatorContext.hasSerialAncestorInPipeline(this);
+ boolean childHasSerialAncestor = inheritedSerial || isSerialNode();
+ translatorContext.setHasSerialAncestorInPipeline(child, childHasSerialAncestor);
+
+ // 1b. Propagate shuffle-for-correctness-ancestor flag to child.
+ // Mirrors BE's _followed_by_shuffled_operator: a downstream operator needs hash
+ // distribution for correctness, and the chain to here goes through HASH or NOOP
+ // requirements (so the dependency is preserved).
+ // propagate = ((inheritedShuffled || self.requiresShuffleForCorrectness)
+ // && require is hash)
+ // || (inheritedShuffled && require is noop/passthrough)
+ boolean inheritedShuffled = translatorContext.hasShuffleForCorrectnessAncestor(this);
+ boolean selfOrInheritedShuffled = inheritedShuffled || requiresShuffleForCorrectness();
+ boolean requireIsHash = require.preferType().isHashShuffle();
+ boolean requireIsNoop = require.preferType() == LocalExchangeNode.LocalExchangeType.NOOP;
+ boolean childShuffledAncestor = (selfOrInheritedShuffled && requireIsHash)
+ || (inheritedShuffled && requireIsNoop);
+ translatorContext.setHasShuffleForCorrectnessAncestor(child, childShuffledAncestor);
+
+ // 2. Recurse child (Layer 2: child declares its own require/output)
+ Pair<PlanNode, LocalExchangeType> childOutput =
+ child.enforceAndDeriveLocalExchange(translatorContext, this, require);
+
+ // Steps 2.5 and 3 both react to a serial child but address different concerns:
+ // - Step 2.5 rewrites the OUTPUT-side view (what we tell satisfy/parent about
+ // the child's actual distribution). A serial pipeline runs with 1 task so
+ // its distribution claim is meaningless — flatten to NOOP so the satisfy
+ // check below doesn't get fooled by a stale "I output BUCKET_HASH" claim.
+ // - Step 3 rewrites the REQUIRE-side decision (what we want from the child).
+ // If we previously asked for nothing (noRequire) but the child turns out
+ // to be serial and we're not, upgrade to requirePassthrough so an LE is
+ // inserted to restore parallelism.
+
+ // 2.5. Serial child override (output side): if child is serial on BE, force its
+ // reported output to NOOP. Distribution is irrelevant when the child runs
+ // with 1 task; downstream parallelism is restored either by step 3 (LE
+ // insertion) or skipped entirely by step 4b (we're also serial).
+ if (childOutput.first.isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ childOutput = Pair.of(childOutput.first, LocalExchangeType.NOOP);
+ }
+
+ // 3. Framework-level serial child check (require side, mirrors BE base class
+ // required_data_distribution): if child will be serial on BE but this node is
+ // not serial, the pipeline has a 1-task serial child feeding an N-task non-serial
+ // parent. Without LE, pipeline splits (AGG/JOIN) create paired pipelines with
+ // mismatched num_tasks → crash. Upgrade noRequire to requirePassthrough so an
+ // LE is inserted below to restore parallelism.
+ if (require instanceof LocalExchangeNode.NoRequire
+ && childOutput.first.isSerialOperatorOnBe(translatorContext.getConnectContext())
+ && !isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ require = LocalExchangeTypeRequire.requirePassthrough();
+ }
+
+ // 4. Satisfy check: child output meets requirement → done
+ if (require.satisfy(childOutput.second)) {
+ return childOutput;
+ }
+
+ // 4. Layer 1: skip LE when serial operator or ancestor in same pipeline
+ // Equivalent to BE's need_to_local_exchange: any_of(operators[idx..end], is_serial) → skip
+ if (translatorContext.hasSerialAncestorInPipeline(this) || isSerialNode()) {
+ return childOutput;
Review Comment:
This skip uses `isSerialNode()` even though the comment above `isSerialOperatorOnBe()` says an `isSerialNode()` only actually runs with one BE task when `fragment.useSerialSource(context)` is true. For fragments where `useSerialSource` is false (for example `ignore_storage_data_distribution=false`, query cache, or NAAJ), a node such as a scalar aggregate or unpartitioned exchange can still return `isSerialNode()==true` but BE will execute it with normal parallelism (`is_serial_operator=false` in thrift). In that case this branch skips a required LocalExchange even though BE would not consider the ancestor serial, so downstream hash/passthrough requirements can be silently dropped. The serial-ancestor propagation at line 1094 has the same issue. Please base these planner decisions on `isSerialOperatorOnBe(translatorContext.getConnectContext())`, not the syntactic `isSerialNode()`, except for the explicitly documented heavy-op/local-fragment cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4508171612
# FE UT Coverage Report
Increment line coverage ` 73.97% (449/607)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_846d365a88901fa32fdf44b42b6cd04b900da259/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_846d365a88901fa32fdf44b42b6cd04b900da259/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4504902749
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/f2d43a29db5822a72153fd88cbcd8bead33cbc6d_f2d43a29db5822a72153fd88cbcd8bead33cbc6d_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/f2d43a29db5822a72153fd88cbcd8bead33cbc6d_f2d43a29db5822a72153fd88cbcd8bead33cbc6d_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.06% (1854/2375) |
| Line Coverage | 64.53% (33336/51662) |
| Region Coverage | 65.23% (16531/25341) |
| Branch Coverage | 55.73% (8834/15852) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4480118080
# BE Regression && UT Coverage Report
Increment line coverage `72.45% (305/421)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_15d92ba024b265d9441145d7bd26d93590aa9c63_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_15d92ba024b265d9441145d7bd26d93590aa9c63_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 63.48% (23997/37804) |
| Line Coverage | 47.20% (247315/523994) |
| Region Coverage | 44.16% (203214/460125) |
| Branch Coverage | 45.44% (87932/193505) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4498830486
# BE UT Coverage Report
Increment line coverage `11.11% (46/414)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/3ac8525eaac942c0bbeb93cf46d4c95ecf353156_3ac8525eaac942c0bbeb93cf46d4c95ecf353156/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/3ac8525eaac942c0bbeb93cf46d4c95ecf353156_3ac8525eaac942c0bbeb93cf46d4c95ecf353156/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.56% (20747/38735) |
| Line Coverage | 37.18% (196268/527904) |
| Region Coverage | 33.48% (153729/459147) |
| Branch Coverage | 34.52% (67039/194213) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4486958816
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4497974447
# FE UT Coverage Report
Increment line coverage ` 74.54% (448/601)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_3ac8525eaac942c0bbeb93cf46d4c95ecf353156/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_3ac8525eaac942c0bbeb93cf46d4c95ecf353156/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3412495881
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -3314,7 +3349,7 @@ private boolean findOlapScanNodesByPassExchangeNode(PlanNode root) {
return false;
}
- private List<List<Expr>> getDistributeExprs(Plan ... children) {
+ private List<List<Expr>> getChildrenDistributeExprs(Plan ... children) {
Review Comment:
this function's name is not very suitable. If you want to obtain the DistributeExprs of children, the input should be the parent node rather than the children.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -634,7 +636,8 @@ protected void splitFragments(PhysicalPlan resultPlan) {
return;
}
- PlanTranslatorContext planTranslatorContext = new PlanTranslatorContext(cascadesContext);
+ this.planTranslatorContext = new PlanTranslatorContext(cascadesContext);
+ PlanTranslatorContext planTranslatorContext = this.planTranslatorContext;
Review Comment:
useless local variable
##########
fe/fe-core/src/main/java/org/apache/doris/qe/runtime/ThriftPlansBuilder.java:
##########
@@ -277,7 +277,8 @@ private static Multiset<DistributedPlanWorker> computeInstanceNumPerWorker(
return workerCounter;
}
- private static Map<Integer, Integer> computeExchangeSenderNum(PipelineDistributedPlan distributedPlan) {
+ private static Map<Integer, Integer> computeExchangeSenderNum(
+ PipelineDistributedPlan distributedPlan) {
Review Comment:
why change this?
##########
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##########
@@ -1130,8 +1130,10 @@ public boolean isProfileSafeStmt() {
// 1. CreateTableCommand(mainly for create as select).
// 2. LoadCommand.
// 3. InsertOverwriteTableCommand.
+ // 4. MergeIntoCommand (merge into ... using ...).
if ((plan instanceof Command) && !(plan instanceof LoadCommand)
- && !(plan instanceof CreateTableCommand) && !(plan instanceof InsertOverwriteTableCommand)) {
+ && !(plan instanceof CreateTableCommand) && !(plan instanceof InsertOverwriteTableCommand)
+ && !(plan instanceof org.apache.doris.nereids.trees.plans.commands.merge.MergeIntoCommand)) {
Review Comment:
use import, not use qualified name
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -740,6 +744,18 @@ protected void distribute(PhysicalPlan physicalPlan, ExplainLevel explainLevel)
splitFragments(physicalPlan);
doDistribute(canUseNereidsDistributePlanner, explainLevel);
+
+ SessionVariable sessionVariable = cascadesContext.getConnectContext().getSessionVariable();
+ if (sessionVariable.isEnableLocalShufflePlanner() && sessionVariable.isEnableLocalShuffle()) {
+ addLocalExchangeAfterDistribute();
+ }
Review Comment:
the if jugdement should move into `addLocalExchangeAfterDistribute`
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -657,7 +660,8 @@ protected void splitFragments(PhysicalPlan resultPlan) {
scanNodeList.addAll(planTranslatorContext.getScanNodes());
physicalRelations.addAll(planTranslatorContext.getPhysicalRelations());
descTable = planTranslatorContext.getDescTable();
- fragments = new ArrayList<>(planTranslatorContext.getPlanFragments());
+ List<PlanFragment> planFragments = planTranslatorContext.getPlanFragments();
Review Comment:
useless local variable
##########
fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java:
##########
@@ -142,7 +145,8 @@ public abstract class PlanNode extends TreeNode<PlanNode> {
protected int nereidsId = -1;
- private List<List<Expr>> childrenDistributeExprLists = new ArrayList<>();
+ protected List<List<Expr>> childrenDistributeExprLists = new ArrayList<>();
+ protected List<Expr> distributeExprLists = new ArrayList<>();
Review Comment:
add comment to these two variables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3411859421
##########
be/src/exec/exchange/local_exchange_sink_operator.cpp:
##########
@@ -37,24 +37,30 @@ std::vector<Dependency*> LocalExchangeSinkLocalState::dependencies() const {
return deps;
}
-Status LocalExchangeSinkOperatorX::init(RuntimeState* state, ExchangeType type,
- const int num_buckets, const bool use_global_hash_shuffle,
+Status LocalExchangeSinkOperatorX::init(RuntimeState* state, TLocalPartitionType::type type,
+ const int num_buckets,
const std::map<int, int>& shuffle_idx_to_instance_idx) {
+ DCHECK(!_planned_by_fe);
_name = "LOCAL_EXCHANGE_SINK_OPERATOR(" + get_exchange_type_name(type) + ")";
_type = type;
- if (_type == ExchangeType::HASH_SHUFFLE) {
- _shuffle_idx_to_instance_idx.clear();
- _use_global_shuffle = use_global_hash_shuffle;
+ if (_type == TLocalPartitionType::GLOBAL_EXECUTION_HASH_SHUFFLE) {
// For shuffle join, if data distribution has been broken by previous operator, we
// should use a HASH_SHUFFLE local exchanger to shuffle data again. To be mentioned,
// we should use map shuffle idx to instance idx because all instances will be
// distributed to all BEs. Otherwise, we should use shuffle idx directly.
- if (use_global_hash_shuffle) {
- _shuffle_idx_to_instance_idx = shuffle_idx_to_instance_idx;
+ _shuffle_idx_to_instance_idx = shuffle_idx_to_instance_idx;
+ if (state->query_options().__isset.enable_new_shuffle_hash_method &&
Review Comment:
已抽成公共的 `_create_partitioner(state, bucket_count)`，现在 BE-native 路径的 `init()`(`local_exchange_sink_operator.cpp:77`) 和 FE-planned 路径的 `init_partitioner()`(:84) 都复用它，去掉了三段重复的 hash-partitioner 创建代码。
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -290,6 +290,32 @@ Status PipelineFragmentContext::_build_and_prepare_full_pipeline(ThreadPool* thr
RETURN_IF_ERROR(_build_pipelines(_runtime_state->obj_pool(), *_query_ctx->desc_tbl,
&_root_op, root_pipeline));
+ // Propagate _num_instances from LOCAL_EXCHANGE pipelines to ancestor pipelines
+ // that inherited reduced num_tasks from a serial operator.
+ _propagate_local_exchange_num_tasks();
+
+ // Create deferred local exchangers now that all pipelines have final num_tasks.
+ RETURN_IF_ERROR(_create_deferred_local_exchangers());
Review Comment:
是的，方向一致。`enable_local_shuffle_planner=true` 时 BE 的 `_plan_local_exchange` 被 `RuntimeState::plan_local_shuffle()` 整个跳过，BE 完全按 FE 规划好的 `LOCAL_EXCHANGE_NODE` 来建 pipeline，不再自己算 local exchange。
目前仍保留 BE 的 `required_data_distribution()` / 并行度计算，只是给 `enable_local_shuffle_planner=false` 的 BE-native 回退路径用（保证和老行为逐字节一致，方便灰度 / 回滚）。FE-planned 路径不读它。
长期我也认同应该让 BE 这边退化成「只校验不计算」——FE 既然已经插好了正确的 LE，BE 的 propagate 理想情况下应该是一个 assert（"FE 给的分布已满足要求"），而不是再算一遍。这个 PR 先保持两条路径并存、默认走 FE。
##########
fe/fe-core/src/main/java/org/apache/doris/planner/AddLocalExchange.java:
##########
@@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.planner;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.glue.translator.PlanTranslatorContext;
+import org.apache.doris.nereids.trees.plans.distribute.DistributedPlan;
+import org.apache.doris.nereids.trees.plans.distribute.FragmentIdMapping;
+import org.apache.doris.nereids.trees.plans.distribute.PipelineDistributedPlan;
+import org.apache.doris.planner.LocalExchangeNode.LocalExchangeType;
+import org.apache.doris.planner.LocalExchangeNode.LocalExchangeTypeRequire;
+import org.apache.doris.planner.LocalExchangeNode.RequireHash;
+
+/**
+ * FE-side local exchange planner — inserts {@link LocalExchangeNode} into each fragment's
+ * plan tree so that within-fragment data redistribution is decided at planning time
+ * instead of at BE pipeline-build time.
+ *
+ * <h3>When this runs</h3>
+ * Invoked from {@code NereidsPlanner.addLocalExchangeAfterDistribute()} right after
+ * {@code DistributePlanner} has assigned instances to fragments and before the plan is
+ * serialized to BE. Gated by session variable {@code enable_local_shuffle_planner}
+ * (default true) and {@code enable_local_shuffle}; when either is off this pass is
+ * skipped entirely and BE falls back to its own {@code _plan_local_exchange}. The two
+ * paths are mutually exclusive: BE consults {@code runtime_state.h::plan_local_shuffle()}
+ * to know whether it should plan LE itself.
+ *
+ * <h3>What it changes</h3>
+ * <ul>
+ * <li>For each fragment with {@code maxPerBeInstances > 1}, walks the plan tree
+ * bottom-up via {@link PlanNode#enforceAndDeriveLocalExchange} and inserts
+ * LocalExchangeNodes where children's output distribution doesn't satisfy the
+ * parent's requirement.</li>
+ * <li>May wrap the fragment root with an extra PASSTHROUGH LE so the data sink
+ * (DataStreamSink / OlapTableSink) runs with the full instance count even when
+ * the root operator is serial — see {@link #addLocalExchangeForFragment}.</li>
+ * <li>Does NOT modify the fragment sink itself, fragment boundaries, or instance
+ * assignment.</li>
+ * </ul>
+ *
+ * <h3>Per-BE instance semantics</h3>
+ * Skips fragments where every BE has at most 1 instance. Using a global instance count
+ * would insert LE for "2 BEs × 1 instance" cases, which BE's own
+ * {@code _plan_local_exchange} would not — leading to pipeline task-count mismatch and
+ * deadlock. See {@link #addLocalExchange}.
+ *
+ * <h3>Reading order</h3>
+ * Start with {@link PlanNode#enforceRequire} (the recursion engine), then individual
+ * {@code enforceAndDeriveLocalExchange} overrides on PlanNode subclasses.
+ */
+public class AddLocalExchange {
+ /** addLocalExchange with distributed plans, skipping single-instance fragments.
+ * BE's _plan_local_exchange checks _num_instances which is the per-BE instance count.
+ * With _num_instances<=1 all pipelines on that BE have 1 task so local exchange is a no-op.
+ * We must use the same per-BE semantics: skip when every BE has at most 1 instance.
+ * Using global instanceCount would insert LE for fragments where 2 BEs each have 1 instance
+ * (global=2, per-BE=1), causing pipeline task mismatch and deadlock. */
+ public void addLocalExchange(FragmentIdMapping<DistributedPlan> distributedPlans,
+ PlanTranslatorContext context) {
+ for (DistributedPlan plan : distributedPlans.values()) {
+ PipelineDistributedPlan pipePlan = (PipelineDistributedPlan) plan;
+ long maxPerBeInstances = pipePlan.getInstanceJobs().stream()
+ .collect(java.util.stream.Collectors.groupingBy(
+ j -> j.getAssignedWorker().id(), java.util.stream.Collectors.counting()))
+ .values().stream().mapToLong(Long::longValue).max().orElse(0);
+ if (maxPerBeInstances <= 1) {
+ continue;
+ }
+ PlanFragment fragment = pipePlan.getFragmentJob().getFragment();
+ addLocalExchangeForFragment(fragment, context);
+ }
+ }
+
+ private void addLocalExchangeForFragment(PlanFragment fragment, PlanTranslatorContext context) {
+ DataSink sink = fragment.getSink();
+ LocalExchangeTypeRequire require = sink == null
+ ? LocalExchangeTypeRequire.noRequire() : sink.getLocalExchangeTypeRequire();
+ PlanNode root = fragment.getPlanRoot();
+ context.setHasSerialAncestorInPipeline(root, false);
+ Pair<PlanNode, LocalExchangeType> output = root
+ .enforceAndDeriveLocalExchange(context, null, require);
+ PlanNode newRoot = output.first;
+ // The fragment data sink (DataStreamSink, OlapTableSink) runs in the same pipeline
+ // as the root. If the root will be serial on BE, the sink pipeline has 1 task —
+ // only instance 0 sends data, others hang or miss writes.
+ // Insert PASSTHROUGH fan-out so sink runs with _num_instances tasks.
+ // This matches BE-native's default required_data_distribution():
+ // _child->is_serial_operator() ? PASSTHROUGH : NOOP
+ if (newRoot.isSerialOperatorOnBe(context.getConnectContext())) {
+ newRoot = new LocalExchangeNode(context.nextPlanNodeId(), newRoot,
+ LocalExchangeType.PASSTHROUGH, null);
+ }
+ if (newRoot != root) {
+ fragment.setPlanRoot(newRoot);
+ }
+ validateNoSerialWithoutLocalExchange(fragment.getPlanRoot(), context.getConnectContext());
+ }
+
+ /**
+ * In a local-shuffle fragment, the root check above guarantees the root pipeline
+ * has N tasks. Any serial operator reduces its pipeline to 1 task. If this serial
+ * operator feeds into a non-serial parent without LocalExchangeNode in between,
+ * some pipelines have 1 task while others have N → shared_state mismatch, data loss.
+ *
+ * Serial→serial chains are fine (all at 1 task, consistent). Only the transition
+ * from serial to non-serial needs LE to restore parallelism.
+ */
+ private void validateNoSerialWithoutLocalExchange(PlanNode node,
+ org.apache.doris.qe.ConnectContext context) {
+ for (PlanNode child : node.getChildren()) {
+ validateNoSerialWithoutLocalExchange(child, context);
+ if (child.isSerialOperatorOnBe(context)
+ && !(child instanceof LocalExchangeNode)
+ && !(node instanceof LocalExchangeNode)
+ && !(node instanceof ExchangeNode)
+ && !node.isSerialOperatorOnBe(context)) {
+ org.apache.logging.log4j.LogManager.getLogger(AddLocalExchange.class).warn(
Review Comment:
Removed this validation entirely (commit `3630071a131`) — after the CI evidence, neither `throw` nor `warn` is worth keeping:
1. **It reads the same `isSerialOperatorOnBe` flag it would validate**, so it is blind to the one bug class that actually matters — a *wrong* serial flag. The framework and the check would agree on the same wrong answer; a check can't validate an input it shares with the thing it's checking.
2. **It fires on legitimate plans.** Pushing it as a `throw` surfaced 9 distinct legitimate P0/cloud_p0 patterns where it fires but the plan is correct: serial `UnionNode → TableFunctionNode/AggregationNode`, serial `RecursiveCteScanNode → NLJ/Agg`, serial `OlapScanNode → HashJoinNode`. In all of these the parent runs effectively serial at runtime and BE reconciles `num_tasks` across the coupled pipelines, so no LE is required and results are correct.
3. **The one class it *could* catch** — a `require`→LE logic regression — is already caught end-to-end by the regression suite / RQG.
Correctness still comes from `enforceRequire` inserting the LEs, validated by `test_local_shuffle_rqg_bugs` + the RQG differential against the BE-native baseline. The principled direction is a single source of truth for the serial/distribution decision (BE reading the FE-written flag), so there is no FE-prediction-vs-BE-reality gap to police in the first place.
##########
regression-test/suites/nereids_p0/local_shuffle/test_local_shuffle_rqg_bugs.groovy:
##########
@@ -0,0 +1,1557 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/**
+ * Regression tests for bugs discovered by RQG testing on the local-exchange2 branch.
+ *
+ * These queries triggered "must set shared state" errors or incorrect results
+ * in RQG build 183992. Common conditions:
+ * - use_serial_exchange=true (makes ALL Exchanges serial, not just UNPARTITIONED)
+ * - enable_local_shuffle_planner=true (FE-planned local exchange)
+ * - parallel_pipeline_task_num > 1
+ *
+ * Error types reproduced:
+ * 1. must set shared state, in AGGREGATION_OPERATOR
+ * 2. must set shared state, in SORT_OPERATOR
+ * 3. incorrect results with GROUPING SETS + scalar subquery + window function
+ */
+suite("test_local_shuffle_rqg_bugs") {
+
+ // ============================================================
+ // Table setup — mirrors RQG table structure
+ // 10 buckets to match RQG (replication_num=1 for single-BE testing)
+ // ============================================================
+ sql "DROP TABLE IF EXISTS rqg_t1"
+ sql "DROP TABLE IF EXISTS rqg_t2"
+ sql "DROP TABLE IF EXISTS rqg_t3"
+ sql "DROP TABLE IF EXISTS rqg_t4"
+
+ sql """
+ CREATE TABLE rqg_t1 (
+ pk INT NOT NULL,
+ col_int_undef_signed INT,
+ col_int_undef_signed2 INT,
+ col_int_undef_signed_not_null INT NOT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ sql """
+ CREATE TABLE rqg_t2 (
+ pk INT NOT NULL,
+ col_int_undef_signed INT,
+ col_int_undef_signed2 INT,
+ col_bigint_undef_signed_not_null BIGINT NOT NULL,
+ col_decimal_38_10__undef_signed_not_null DECIMAL(38,10) NOT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ // Table for build 184181 GLOBAL_HASH_SHUFFLE bugs — needs varchar + bigint columns
+ sql """
+ CREATE TABLE rqg_t3 (
+ pk INT NOT NULL,
+ col_bigint_undef_signed BIGINT,
+ col_varchar_10__undef_signed VARCHAR(10),
+ col_varchar_64__undef_signed VARCHAR(64)
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ // Second table for FULL OUTER JOIN case (col_bigint_undef_signed_not_null)
+ sql """
+ CREATE TABLE rqg_t4 (
+ pk INT NOT NULL,
+ col_bigint_undef_signed_not_null BIGINT NOT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ sql """
+ INSERT INTO rqg_t3 VALUES
+ (0, -94, 'Abc', 'hello world'),
+ (1, 672609, 'Xyz', null),
+ (2, -3766684, 'Pqr', 'test string'),
+ (3, 5070261, 'abc', 'another row'),
+ (4, null, 'def', 'value four'),
+ (5, -86, 'XgpxlHBLEM', null),
+ (6, 21910, 'abc', 'they'),
+ (7, -63, 'zzzz', 'some text'),
+ (8, -8276281, 'AHlvNtoGLO', 'longer string here'),
+ (9, -101, 'mid', 'final row')
+ """
+
+ sql """
+ INSERT INTO rqg_t4 VALUES
+ (0, 0), (1, 1), (2, 2), (3, 3), (4, 4),
+ (5, 5), (6, 6), (7, 7), (8, 8), (9, 9),
+ (10, 2), (11, 2), (12, 2), (13, 3), (14, 4),
+ (15, 5), (16, 2), (17, 2), (18, 2), (19, 9)
+ """
+
+ // Insert enough rows to exercise multiple pipeline tasks
+ sql """
+ INSERT INTO rqg_t1 VALUES
+ (0, 0, 10, 0), (1, 1, 11, 1), (2, 2, 12, 2), (3, 3, 13, 3),
+ (4, 4, 14, 4), (5, 5, 15, 5), (6, 6, 16, 6), (7, 7, 17, 7),
+ (8, 8, 18, 8), (9, 9, 19, 9), (10, 0, 20, 10), (11, 1, 21, 11),
+ (12, 2, 22, 12), (13, 3, 23, 13), (14, 4, 24, 14), (15, 5, 25, 15),
+ (16, 6, 26, 16), (17, 7, 27, 17), (18, 8, 28, 18), (19, 9, 29, 19)
+ """
+
+ sql """
+ INSERT INTO rqg_t2 VALUES
+ (0, 0, 10, 100, 1.5), (1, 1, 11, 101, 2.5), (2, 2, 12, 102, 3.5),
+ (3, 3, 13, 103, 4.5), (4, 4, 14, 104, 5.5), (5, 5, 15, 105, 6.5),
+ (6, 6, 16, 106, 7.5), (7, 7, 17, 107, 8.5), (8, 8, 18, 108, 9.5),
+ (9, 9, 19, 109, 10.5), (10, 0, 20, 110, 11.5), (11, 1, 21, 111, 12.5),
+ (12, 2, 22, 112, 13.5), (13, 3, 23, 113, 14.5), (14, 4, 24, 114, 15.5),
+ (15, 5, 25, 115, 16.5), (16, 6, 26, 116, 17.5), (17, 7, 27, 117, 18.5),
+ (18, 8, 28, 118, 19.5), (19, 9, 29, 119, 20.5)
+ """
+
+ // Wait for data to be visible
+ Thread.sleep(5000)
+
+ // ============================================================
+ // Common settings
+ // ============================================================
+ sql "SET enable_nereids_planner=true"
+ sql "SET enable_fallback_to_original_planner=false"
+ sql "SET runtime_filter_mode=off"
+ sql "SET enable_profile=true"
+ sql "SET enable_sql_cache=false"
+ sql "SET enable_local_shuffle=true"
+
+ // ============================================================
+ // Bug 1: must set shared state, in AGGREGATION_OPERATOR
+ // RQG case: eliminate_group_by_uniform.case_id_11007680713
+ // Key conditions: use_serial_exchange=true, parallel_pipeline_task_num=3
+ // SQL: EXCEPT with count(*) GROUP BY on both sides
+ // ============================================================
+
+ // Test with FE planner (the buggy path)
+ logger.info("=== Bug 1a: AGG shared state - EXCEPT with serial exchange (FE planner) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=3,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_common_expr_pushdown=false,
+ disable_streaming_preaggregations=true)*/
+ col_int_undef_signed_not_null as col1,
+ col_int_undef_signed_not_null as col2,
+ 0 as col3, count(1)
+ FROM rqg_t1
+ GROUP BY col1, col2, col3
+ EXCEPT
+ SELECT col_bigint_undef_signed_not_null as col1,
+ col_decimal_38_10__undef_signed_not_null as col2,
+ 5 as col3, count(1)
+ FROM rqg_t2
+ GROUP BY col1, col2, col3
+ """
+ logger.info("Bug 1a: PASSED (no crash)")
+ } catch (Throwable t) {
+ logger.error("Bug 1a FAILED: ${t.message}")
+ assertTrue(false, "Bug 1a: must set shared state in AGGREGATION_OPERATOR: ${t.message}")
+ }
+
+ // Compare with BE native planner
+ logger.info("=== Bug 1b: AGG shared state - EXCEPT with serial exchange (BE native) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=3,
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_common_expr_pushdown=false,
+ disable_streaming_preaggregations=true)*/
+ col_int_undef_signed_not_null as col1,
+ col_int_undef_signed_not_null as col2,
+ 0 as col3, count(1)
+ FROM rqg_t1
+ GROUP BY col1, col2, col3
+ EXCEPT
+ SELECT col_bigint_undef_signed_not_null as col1,
+ col_decimal_38_10__undef_signed_not_null as col2,
+ 5 as col3, count(1)
+ FROM rqg_t2
+ GROUP BY col1, col2, col3
+ """
+ logger.info("Bug 1b: PASSED (no crash)")
+ } catch (Throwable t) {
+ logger.error("Bug 1b FAILED: ${t.message}")
+ assertTrue(false, "Bug 1b: BE native also fails: ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 2: must set shared state, in SORT_OPERATOR
+ // RQG case: grouping_set.case_id_5308471751
+ // Key conditions: use_serial_exchange=true, parallel_pipeline_task_num=5
+ // SQL: GROUPING SETS + window function (PERCENT_RANK)
+ // ============================================================
+
+ logger.info("=== Bug 2a: SORT shared state - GROUPING SETS + window (FE planner) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=5,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_share_hash_table_for_broadcast_join=false,
+ disable_streaming_preaggregations=true)*/
+ SUM(PERCENT_RANK() OVER (PARTITION BY col_int_undef_signed2 ORDER BY col_int_undef_signed2))
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed2),(pk, pk),(col_int_undef_signed))
+ """
+ logger.info("Bug 2a: PASSED (no crash)")
+ } catch (Throwable t) {
+ logger.error("Bug 2a FAILED: ${t.message}")
+ assertTrue(false, "Bug 2a: must set shared state in SORT_OPERATOR: ${t.message}")
+ }
+
+ logger.info("=== Bug 2b: SORT shared state - GROUPING SETS + window (BE native) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=5,
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_share_hash_table_for_broadcast_join=false,
+ disable_streaming_preaggregations=true)*/
+ SUM(PERCENT_RANK() OVER (PARTITION BY col_int_undef_signed2 ORDER BY col_int_undef_signed2))
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed2),(pk, pk),(col_int_undef_signed))
+ """
+ logger.info("Bug 2b: PASSED (no crash)")
+ } catch (Throwable t) {
+ logger.error("Bug 2b FAILED: ${t.message}")
+ assertTrue(false, "Bug 2b: BE native also fails: ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 3: incorrect results with GROUPING SETS + scalar subquery + window
+ // RQG case: grouping_set.case_id_5694495756
+ // Key conditions: parallel_pipeline_task_num=2, disable_streaming_preaggregations=true
+ // Expected: all rows same value; Actual: values split proportionally (1/3, 2/3)
+ // ============================================================
+
+ logger.info("=== Bug 3: incorrect results - GROUPING SETS + subquery + window ===")
+ // FE planner
+ def result_fe = sql """
+ SELECT /*+SET_VAR(parallel_pipeline_task_num=2,
+ enable_local_shuffle_planner=true,
+ disable_streaming_preaggregations=true,
+ enable_share_hash_table_for_broadcast_join=true)*/
+ SUM((SELECT MAX(col_int_undef_signed2) FROM rqg_t1))
+ OVER (PARTITION BY pk ORDER BY pk)
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed2, pk),(pk), (pk))
+ """
+ // BE native
+ def result_be = sql """
+ SELECT /*+SET_VAR(parallel_pipeline_task_num=2,
+ enable_local_shuffle_planner=false,
+ disable_streaming_preaggregations=true,
+ enable_share_hash_table_for_broadcast_join=true)*/
+ SUM((SELECT MAX(col_int_undef_signed2) FROM rqg_t1))
+ OVER (PARTITION BY pk ORDER BY pk)
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed2, pk),(pk), (pk))
+ """
+ logger.info("Bug 3 FE result rows: ${result_fe.size()}, first few: ${result_fe.take(5)}")
+ logger.info("Bug 3 BE result rows: ${result_be.size()}, first few: ${result_be.take(5)}")
+
+ // All values in both should be the same
+ if (result_fe.size() != result_be.size()) {
+ logger.warn("Bug 3: row count mismatch FE=${result_fe.size()} vs BE=${result_be.size()}")
Review Comment:
Done. Bug 3 now asserts FE==BE (row count + order-insensitive content); Bug 4/5 raise `assertTrue(false)` on an unexpected exception instead of only logging (Bug 1/2/6/7/8 already asserted in their `catch`). The Bug 21 `catch` that swallowed `query timeout` (flagged separately by the bot) was also removed, so a hang fails the suite too.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java:
##########
@@ -285,4 +302,71 @@ public List<Expr> getOtherJoinConjuncts() {
public List<Expr> getMarkJoinConjuncts() {
return markJoinConjuncts;
}
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+
+ LocalExchangeTypeRequire probeSideRequire;
+ LocalExchangeTypeRequire buildSideRequire;
+ LocalExchangeType outputType = null;
+
+ if (joinOp == JoinOperator.NULL_AWARE_LEFT_ANTI_JOIN) {
+ buildSideRequire = probeSideRequire = LocalExchangeTypeRequire.noRequire();
+ outputType = LocalExchangeType.NOOP;
+ } else if (distrMode == DistributionMode.BROADCAST) {
+ // BE: _child->is_serial_operator() ? PASSTHROUGH/PASS_TO_ONE : NOOP
+ boolean probeChildSerial = children.get(0).isSerialOperatorOnBe(
+ translatorContext.getConnectContext());
+ boolean buildChildSerial = children.get(1).isSerialOperatorOnBe(
+ translatorContext.getConnectContext());
+ probeSideRequire = probeChildSerial
Review Comment:
Mirrored — `probeSideRequire` now folds in the session flag: `probePassthrough = forcePassthrough || probeChildSerial`, where `forcePassthrough` reads `enableBroadcastJoinForcePassthrough` (`HashJoinNode` ~329-337, null-safe for the unit-test mocks).
One caveat from investigating it on a 4-BE cluster (forced non-serial probe via `ignore_storage_data_distribution=false`): it is effectively a no-op there. `enforceRequire` only inserts a PASSTHROUGH local exchange to fan a *serial* (1-task) source out to N tasks; an already-N-task source already satisfies passthrough, so no exchange is added (identical plan/results vs BE-native, no crash). The branch is kept to match BE's intent; a true rebalance of a non-serial probe (actually forcing the exchange) is a perf-only follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3411863216
##########
regression-test/data/query_p0/join/test_multilevel_join_agg_local_shuffle.out:
##########
@@ -0,0 +1,815 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !bucket_shuffle_broadcast_agg_stage_1_result_on --
+1 52 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_shuffle_broadcast_agg_stage_1_result_off --
+1 52 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_shuffle_broadcast_agg_stage_2_result_on --
+1 52 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_shuffle_broadcast_agg_stage_2_result_off --
+1 52 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_shuffle_broadcast_agg_stage_3_result_on --
+1 52 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_shuffle_broadcast_agg_stage_3_result_off --
+1 52 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !shuffle_broadcast_broadcast_agg_stage_1_result_on --
+1 10 62 10
+2 5 54 20
+3 5 38 30
+4 6 49 40
+
+-- !shuffle_broadcast_broadcast_agg_stage_1_result_off --
+1 10 62 10
+2 5 54 20
+3 5 38 30
+4 6 49 40
+
+-- !shuffle_broadcast_broadcast_agg_stage_2_result_on --
+1 10 62 10
+2 5 54 20
+3 5 38 30
+4 6 49 40
+
+-- !shuffle_broadcast_broadcast_agg_stage_2_result_off --
+1 10 62 10
+2 5 54 20
+3 5 38 30
+4 6 49 40
+
+-- !shuffle_broadcast_broadcast_agg_stage_3_result_on --
+1 10 62 10
+2 5 54 20
+3 5 38 30
+4 6 49 40
+
+-- !shuffle_broadcast_broadcast_agg_stage_3_result_off --
+1 10 62 10
+2 5 54 20
+3 5 38 30
+4 6 49 40
+
+-- !bucket_broadcast_shuffle_agg_stage_1_result_on --
+1 26 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_broadcast_shuffle_agg_stage_1_result_off --
+1 26 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_broadcast_shuffle_agg_stage_2_result_on --
+1 26 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_broadcast_shuffle_agg_stage_2_result_off --
+1 26 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_broadcast_shuffle_agg_stage_3_result_on --
+1 26 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_broadcast_shuffle_agg_stage_3_result_off --
+1 26 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !alternating_bucket_bucket_bucket_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_bucket_bucket_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_bucket_shuffle_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_bucket_shuffle_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_bucket_broadcast_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_bucket_broadcast_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_shuffle_bucket_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_shuffle_bucket_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_shuffle_shuffle_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_shuffle_shuffle_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_shuffle_broadcast_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_shuffle_broadcast_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_broadcast_bucket_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_broadcast_bucket_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_broadcast_shuffle_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_broadcast_shuffle_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_broadcast_broadcast_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_bucket_broadcast_broadcast_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_bucket_bucket_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_bucket_bucket_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_bucket_shuffle_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_bucket_shuffle_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_bucket_broadcast_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_bucket_broadcast_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_shuffle_bucket_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_shuffle_bucket_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_shuffle_shuffle_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_shuffle_shuffle_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_shuffle_broadcast_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_shuffle_broadcast_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_broadcast_bucket_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_broadcast_bucket_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_broadcast_shuffle_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_broadcast_shuffle_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_broadcast_broadcast_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_shuffle_broadcast_broadcast_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_bucket_bucket_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_bucket_bucket_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_bucket_shuffle_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_bucket_shuffle_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_bucket_broadcast_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_bucket_broadcast_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_shuffle_bucket_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_shuffle_bucket_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_shuffle_shuffle_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_shuffle_shuffle_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_shuffle_broadcast_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_shuffle_broadcast_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_broadcast_bucket_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_broadcast_bucket_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_broadcast_shuffle_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_broadcast_shuffle_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_broadcast_broadcast_result_on --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !alternating_broadcast_broadcast_broadcast_result_off --
+1 63 16 11
+2 16 24 22
+3 16 35 33
+4 19 46 44
+
+-- !window_union_all_bucket_row_number_result_on --
+1 52 4 40
+2 21 3 60
+3 16 2 60
+4 20 2 80
+
+-- !window_union_all_bucket_row_number_result_off --
+1 52 4 40
+2 21 3 60
+3 16 2 60
+4 20 2 80
+
+-- !window_union_all_bucket_window_sum_result_on --
+1 10 13 40
+2 6 7 60
+3 3 8 60
+4 3 10 80
+
+-- !window_union_all_bucket_window_sum_result_off --
+1 10 13 40
+2 6 7 60
+3 3 8 60
+4 3 10 80
+
+-- !window_union_all_shuffle_row_number_result_on --
+1 208 8 44
+2 21 3 21
+3 16 2 16
+4 20 2 18
+
+-- !window_union_all_shuffle_row_number_result_off --
+1 208 8 44
+2 21 3 21
+3 16 2 16
+4 20 2 18
+
+-- !window_union_all_shuffle_window_sum_result_on --
+1 36 26 44
+2 6 7 21
+3 3 8 16
+4 3 10 18
+
+-- !window_union_all_shuffle_window_sum_result_off --
+1 36 26 44
+2 6 7 21
+3 3 8 16
+4 3 10 18
+
+-- !window_union_all_broadcast_row_number_result_on --
+1 52 4 40
+2 21 3 60
+3 16 2 60
+4 20 2 80
+
+-- !window_union_all_broadcast_row_number_result_off --
+1 52 4 40
+2 21 3 60
+3 16 2 60
+4 20 2 80
+
+-- !window_union_all_broadcast_window_sum_result_on --
+1 10 13 40
+2 6 7 60
+3 3 8 60
+4 3 10 80
+
+-- !window_union_all_broadcast_window_sum_result_off --
+1 10 13 40
+2 6 7 60
+3 3 8 60
+4 3 10 80
+
+-- !window_except_bucket_row_number_result_on --
+1 10 2 20
+2 10 2 40
+3 5 1 30
+4 6 1 40
+
+-- !window_except_bucket_row_number_result_off --
+1 10 2 20
+2 10 2 40
+3 5 1 30
+4 6 1 40
+
+-- !window_except_bucket_window_sum_result_on --
+1 3 5 20
+2 3 5 40
+3 1 5 30
+4 1 6 40
+
+-- !window_except_bucket_window_sum_result_off --
+1 3 5 20
+2 3 5 40
+3 1 5 30
+4 1 6 40
+
+-- !window_except_shuffle_row_number_result_on --
+1 40 4 22
+2 10 2 14
+3 5 1 8
+4 6 1 9
+
+-- !window_except_shuffle_row_number_result_off --
+1 40 4 22
+2 10 2 14
+3 5 1 8
+4 6 1 9
+
+-- !window_except_shuffle_window_sum_result_on --
+1 10 10 22
+2 3 5 14
+3 1 5 8
+4 1 6 9
+
+-- !window_except_shuffle_window_sum_result_off --
+1 10 10 22
+2 3 5 14
+3 1 5 8
+4 1 6 9
+
+-- !window_except_broadcast_row_number_result_on --
+1 10 2 20
+2 10 2 40
+3 5 1 30
+4 6 1 40
+
+-- !window_except_broadcast_row_number_result_off --
+1 10 2 20
+2 10 2 40
+3 5 1 30
+4 6 1 40
+
+-- !window_except_broadcast_window_sum_result_on --
+1 3 5 20
+2 3 5 40
+3 1 5 30
+4 1 6 40
+
+-- !window_except_broadcast_window_sum_result_off --
+1 3 5 20
+2 3 5 40
+3 1 5 30
+4 1 6 40
+
+-- !window_intersect_bucket_row_number_result_on --
+
+-- !window_intersect_bucket_row_number_result_off --
+
+-- !window_intersect_bucket_window_sum_result_on --
+
+-- !window_intersect_bucket_window_sum_result_off --
+
+-- !window_intersect_shuffle_row_number_result_on --
+
+-- !window_intersect_shuffle_row_number_result_off --
+
+-- !window_intersect_shuffle_window_sum_result_on --
+
+-- !window_intersect_shuffle_window_sum_result_off --
+
+-- !window_intersect_broadcast_row_number_result_on --
+
+-- !window_intersect_broadcast_row_number_result_off --
+
+-- !window_intersect_broadcast_window_sum_result_on --
+
+-- !window_intersect_broadcast_window_sum_result_off --
+
+-- !bucket_broadcast_agg_result_on --
+1 26 10
+2 9 20
+3 8 30
+4 10 40
+
+-- !bucket_broadcast_agg_result_off --
+1 26 10
+2 9 20
+3 8 30
+4 10 40
+
+-- !partitioned_broadcast_agg_result_on --
+1 32 10
+2 19 20
+3 13 30
+4 15 40
+
+-- !partitioned_broadcast_agg_result_off --
+1 32 10
+2 19 20
+3 13 30
+4 15 40
+
+-- !bucket_partitioned_agg_result_on --
+1 52 44
+2 9 14
+3 8 8
+4 10 9
+
+-- !bucket_partitioned_agg_result_off --
+1 52 44
+2 9 14
+3 8 8
+4 10 9
+
+-- !all_three_multilevel_agg_result_on --
+1 96 10
+2 23 20
+3 16 30
+4 19 40
+
+-- !all_three_multilevel_agg_result_off --
+1 96 10
+2 23 20
+3 16 30
+4 19 40
+
+-- !agg_join_agg_mix_result_on --
+1 10 11 10
+2 5 7 20
+3 5 8 30
+4 6 9 40
+
+-- !agg_join_agg_mix_result_off --
+1 10 11 10
+2 5 7 20
+3 5 8 30
+4 6 9 40
+
+-- !double_broadcast_after_bucket_result_on --
+1 26 20
+2 9 40
+3 8 60
+4 10 80
+
+-- !double_broadcast_after_bucket_result_off --
+1 26 20
+2 9 40
+3 8 60
+4 10 80
+
+-- !partitioned_join_between_two_aggs_then_broadcast_result_on --
+1 16 10
+2 12 20
+3 13 30
+4 15 40
+
+-- !partitioned_join_between_two_aggs_then_broadcast_result_off --
+1 16 10
+2 12 20
+3 13 30
+4 15 40
+
+-- !bucket_shuffle_broadcast_two_stage_agg_result_on --
+1 52 44 10
+2 9 14 20
+3 8 8 30
+4 10 9 40
+
+-- !bucket_shuffle_broadcast_two_stage_agg_result_off --
+1 52 44 10
+2 9 14 20
+3 8 8 30
+4 10 9 40
+
+-- !left_join_null_preserving_with_multilevel_agg_result_on --
+1 5 44
+2 5 14
+3 5 8
+4 6 9
+
+-- !left_join_null_preserving_with_multilevel_agg_result_off --
+1 5 44
+2 5 14
+3 5 8
+4 6 9
+
+-- !seven_layer_bucket_shuffle_broadcast_result_on --
+1 37 10
+2 16 20
+3 16 30
+4 19 40
+
+-- !seven_layer_bucket_shuffle_broadcast_result_off --
+1 37 10
+2 16 20
+3 16 30
+4 19 40
+
+-- !eight_layer_mixed_join_agg_chain_result_on --
+1 52 44 20
+2 9 14 40
+3 8 8 60
+4 10 9 80
+
+-- !eight_layer_mixed_join_agg_chain_result_off --
+1 52 44 20
+2 9 14 40
+3 8 8 60
+4 10 9 80
+
+-- !seven_layer_left_join_mix_result_on --
+1 10 11 10
+2 5 7 20
+3 5 8 30
+4 6 9 40
+
+-- !seven_layer_left_join_mix_result_off --
+1 10 11 10
+2 5 7 20
+3 5 8 30
+4 6 9 40
+
+-- !broadcast_shuffle_broadcast_nested_agg_result_on --
+1 26 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !broadcast_shuffle_broadcast_nested_agg_result_off --
+1 26 11 10
+2 9 7 20
+3 8 8 30
+4 10 9 40
+
+-- !window_union_join_agg_result_on --
+1 13 14
+2 7 23
+3 8 32
+4 10 42
+
+-- !window_union_join_agg_result_off --
+1 13 14
+2 7 23
+3 8 32
+4 10 42
+
+-- !window_except_join_agg_result_on --
+1 1 10
+2 1 20
+3 1 30
+
+-- !window_except_join_agg_result_off --
+1 1 10
+2 1 20
+3 1 30
+
+-- !window_intersect_shuffle_agg_result_on --
+1 22 2
+2 7 1
+3 8 1
+4 9 1
+
+-- !window_intersect_shuffle_agg_result_off --
+1 22 2
+2 7 1
+3 8 1
+4 9 1
+
+-- !window_union_except_broadcast_agg_result_on --
+1 13 4 10
+2 7 3 20
+3 8 2 30
+4 10 2 40
+
+-- !window_union_except_broadcast_agg_result_off --
+1 13 4 10
+2 7 3 20
+3 8 2 30
+4 10 2 40
+
+-- !window_setop_join_agg_chain_result_on --
+1 20 2 22
+2 10 2 14
+3 5 1 8
+4 6 1 9
+
+-- !window_setop_join_agg_chain_result_off --
+1 20 2 22
+2 10 2 14
+3 5 1 8
+4 6 1 9
+
Review Comment:
Fixed in `73cc6458fd1` — trimmed the trailing blank line at EOF so the expected output ends at the last result row. `git diff --check` is clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3413491554
##########
be/src/exec/operator/partition_sort_sink_operator.h:
##########
@@ -95,9 +95,10 @@ class PartitionSortSinkOperatorX final : public DataSinkOperatorX<PartitionSortS
Status sink_impl(RuntimeState* state, Block* in_block, bool eos) override;
DataDistribution required_data_distribution(RuntimeState* /*state*/) const override {
if (_topn_phase == TPartTopNPhase::TWO_PHASE_GLOBAL) {
- return DataDistribution(ExchangeType::HASH_SHUFFLE, _distribute_exprs);
+ return DataDistribution(TLocalPartitionType::GLOBAL_EXECUTION_HASH_SHUFFLE,
Review Comment:
`HASH_SHUFFLE` was renamed to `GLOBAL_EXECUTION_HASH_SHUFFLE` in this PR — same semantics (the old single hash-shuffle type was the global, cross-instance one), so every old `HASH_SHUFFLE` became `GLOBAL_EXECUTION_HASH_SHUFFLE`. `LOCAL_EXECUTION_HASH_SHUFFLE` is the new, finer-grained type.
An operator declares the general `GLOBAL_EXECUTION_HASH_SHUFFLE` as its requirement; BE then downgrades it to `LOCAL_EXECUTION_HASH_SHUFFLE` at pipeline build (`pipeline_fragment_context.cpp:1026-1028`) when `use_global_hash_shuffle` is false — i.e. when a within-backend rebalance suffices and a cross-backend global hash isn't needed. (On the FE-planned path, `AddLocalExchange.resolveExchangeType` does the same GLOBAL→LOCAL specialization.)
##########
be/src/exec/pipeline/dependency.h:
##########
@@ -832,50 +832,44 @@ struct SetSharedState : public BasicSharedState {
Status hash_table_init();
};
-enum class ExchangeType : uint8_t {
- NOOP = 0,
- // Shuffle data by Crc32CHashPartitioner
- HASH_SHUFFLE = 1,
- // Round-robin passthrough data blocks.
- PASSTHROUGH = 2,
- // Shuffle data by Crc32HashPartitioner<ShuffleChannelIds> (e.g. same as storage engine).
- BUCKET_HASH_SHUFFLE = 3,
- // Passthrough data blocks to all channels.
- BROADCAST = 4,
- // Passthrough data to channels evenly in an adaptive way.
- ADAPTIVE_PASSTHROUGH = 5,
- // Send all data to the first channel.
- PASS_TO_ONE = 6,
-};
+inline bool is_shuffled_exchange(TLocalPartitionType::type idx) {
+ return idx == TLocalPartitionType::GLOBAL_EXECUTION_HASH_SHUFFLE ||
+ idx == TLocalPartitionType::LOCAL_EXECUTION_HASH_SHUFFLE ||
+ idx == TLocalPartitionType::BUCKET_HASH_SHUFFLE;
+}
Review Comment:
`is_shuffled_exchange` is defined once (inline here in `dependency.h`) and reused at `pipeline.h:72/79`, `local_exchange_sink_operator.cpp:120`, and `local_exchange_source_operator.cpp:34` — there is no duplicate copy in `operator.cpp`. If you spotted a separate function in `operator.cpp` doing the same hash-type check, point me at it and I'll unify them.
##########
be/src/exec/pipeline/pipeline.h:
##########
@@ -68,16 +68,15 @@ class Pipeline : public std::enable_shared_from_this<Pipeline> {
[[nodiscard]] PipelineId id() const { return _pipeline_id; }
- static bool is_hash_exchange(ExchangeType idx) {
- return idx == ExchangeType::HASH_SHUFFLE || idx == ExchangeType::BUCKET_HASH_SHUFFLE;
+ static bool is_hash_exchange(TLocalPartitionType::type idx) {
+ return is_shuffled_exchange(idx);
}
// For HASH_SHUFFLE, BUCKET_HASH_SHUFFLE, and ADAPTIVE_PASSTHROUGH,
Review Comment:
Updated in `6a48e0864df` — the comment now lists the hash-shuffle types (GLOBAL_EXECUTION_HASH_SHUFFLE / LOCAL_EXECUTION_HASH_SHUFFLE / BUCKET_HASH_SHUFFLE) plus ADAPTIVE_PASSTHROUGH, instead of the old `HASH_SHUFFLE` name.
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -723,7 +931,7 @@ Status PipelineFragmentContext::_create_tree_helper(
*root = op;
}
/**
- * `ExchangeType::HASH_SHUFFLE` should be used if an operator is followed by a shuffled operator (shuffled hash join, union operator followed by co-located operators).
+ * `TLocalPartitionType::HASH_SHUFFLE` should be used if an operator is followed by a shuffled operator (shuffled hash join, union operator followed by co-located operators).
Review Comment:
Updated in `6a48e0864df` — `HASH_SHUFFLE` → `GLOBAL_EXECUTION_HASH_SHUFFLE` in the comment.
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -810,52 +1018,59 @@ Status PipelineFragmentContext::_add_local_exchange_impl(
sink_id, local_exchange_id, use_global_hash_shuffle ? _total_instances : _num_instances,
data_distribution.partition_exprs, bucket_seq_to_instance_idx);
if (bucket_seq_to_instance_idx.empty() &&
- data_distribution.distribution_type == ExchangeType::BUCKET_HASH_SHUFFLE) {
- data_distribution.distribution_type = ExchangeType::HASH_SHUFFLE;
+ data_distribution.distribution_type == TLocalPartitionType::BUCKET_HASH_SHUFFLE) {
+ data_distribution.distribution_type =
+ use_global_hash_shuffle ? TLocalPartitionType::GLOBAL_EXECUTION_HASH_SHUFFLE
+ : TLocalPartitionType::LOCAL_EXECUTION_HASH_SHUFFLE;
+ }
+ if (!use_global_hash_shuffle &&
+ data_distribution.distribution_type == TLocalPartitionType::GLOBAL_EXECUTION_HASH_SHUFFLE) {
+ data_distribution.distribution_type = TLocalPartitionType::LOCAL_EXECUTION_HASH_SHUFFLE;
Review Comment:
Yes, exactly. The operator declares `GLOBAL_EXECUTION_HASH_SHUFFLE` as its required distribution (the general/safe hash), and here BE downgrades it to `LOCAL_EXECUTION_HASH_SHUFFLE` when `use_global_hash_shuffle` is false. That flag is `use_global_hash_shuffle = bucket_seq_to_instance_idx.empty() && !shuffle_idx_to_instance_idx.contains(-1) && followed_by_shuffled_operator && !_use_serial_source` (line 1014) — so GLOBAL is kept only when the data must align with a cross-backend hash (followed by a shuffled operator, non-bucket, non-serial); otherwise a within-backend LOCAL rebalance is enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4708136408
@morrySnow re: "why don't you impl it before translation and after post-process in Nereids?"
Local-exchange planning needs two inputs that only exist **after translation + distribute**, not on the Nereids physical plan:
1. **PlanNode-level info** — `AddLocalExchange` walks the translated `PlanNode` / `PlanFragment` tree via `PlanNode.enforceAndDeriveLocalExchange`, using per-node things like `isSerialNode()` / `isSerialOperatorOnBe()`, the distribute-expr lists serialized to BE thrift, the fragment sink type, etc. Those are the legacy `PlanNode` structures produced by the translator — they don't exist on the Nereids physical operators.
2. **Instance counts** — whether to insert an LE (and the fan-out width) depends on the per-BE instance count: `addLocalExchange` skips fragments where `maxPerBeInstances <= 1` (mirroring BE's `_num_instances <= 1` no-op), and the PASSTHROUGH fan-out targets `_num_instances`. Instance assignment is done by `DistributePlanner`, which runs **after** translation.
So it runs in `addLocalExchangeAfterDistribute()` — right after `splitFragments()` + `doDistribute()` — because that's the first point where both the BE-facing PlanNode tree and the instance assignment exist. Doing it pre-translation in Nereids would require re-deriving the fragment structure and instance counts that haven't been computed yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4741384477
<details>
<summary>TPC-H: <b>Total hot run time: 29107 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0436cc4cb573b104a59408a6169bd145bcfeff6c, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17853 4015 3993 3993
q2 2014 322 202 202
q3 10306 1462 818 818
q4 4716 473 362 362
q5 7777 860 576 576
q6 229 181 141 141
q7 799 867 629 629
q8 10254 1483 1653 1483
q9 6015 4578 4551 4551
q10 6781 1822 1511 1511
q11 442 282 247 247
q12 662 447 299 299
q13 18169 3504 2810 2810
q14 268 263 248 248
q15 q16 789 772 716 716
q17 1041 891 924 891
q18 7153 5863 5599 5599
q19 1258 1289 1100 1100
q20 494 403 259 259
q21 5699 2667 2363 2363
q22 432 365 309 309
Total cold run time: 103151 ms
Total hot run time: 29107 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4384 4296 4308 4296
q2 311 350 226 226
q3 4622 4994 4467 4467
q4 2071 2173 1361 1361
q5 4462 4325 4337 4325
q6 228 180 130 130
q7 1966 2034 1669 1669
q8 2600 2219 2215 2215
q9 8546 8266 8187 8187
q10 4834 4777 4294 4294
q11 606 428 396 396
q12 770 747 539 539
q13 3240 3596 2970 2970
q14 299 298 279 279
q15 q16 703 723 652 652
q17 1357 1474 1324 1324
q18 8063 7235 7194 7194
q19 1146 1108 1131 1108
q20 2249 2222 1936 1936
q21 5271 4609 4533 4533
q22 512 466 401 401
Total cold run time: 58240 ms
Total hot run time: 52502 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4732291285
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/c2c51332d7f90c80ccabf77b27c23317d636a128_c2c51332d7f90c80ccabf77b27c23317d636a128_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/c2c51332d7f90c80ccabf77b27c23317d636a128_c2c51332d7f90c80ccabf77b27c23317d636a128_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.32% (1889/2443) |
| Line Coverage | 64.41% (33962/52725) |
| Region Coverage | 64.80% (17463/26948) |
| Branch Coverage | 53.98% (9348/17316) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4681086984
<details>
<summary>TPC-DS: <b>Total hot run time: 168474 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1f614d8cadd1619a8c17dd6f7ec7b12ba6983286, data reload: false
query5 4358 611 456 456
query6 441 187 168 168
query7 4814 555 287 287
query8 359 216 198 198
query9 8753 3999 4011 3999
query10 445 309 278 278
query11 5898 2360 2172 2172
query12 153 101 121 101
query13 1291 596 419 419
query14 6723 5339 5002 5002
query14_1 4368 4360 4326 4326
query15 204 200 172 172
query16 1020 461 416 416
query17 1101 689 563 563
query18 2729 461 332 332
query19 224 186 136 136
query20 109 105 100 100
query21 221 139 117 117
query22 13658 13549 13457 13457
query23 17447 16409 16263 16263
query23_1 16279 16347 16340 16340
query24 7587 1793 1313 1313
query24_1 1321 1318 1285 1285
query25 557 452 390 390
query26 1298 321 176 176
query27 2546 574 354 354
query28 4367 2036 2012 2012
query29 1089 624 509 509
query30 315 247 209 209
query31 1161 1099 961 961
query32 109 69 61 61
query33 528 322 256 256
query34 1174 1126 674 674
query35 762 784 695 695
query36 1415 1434 1260 1260
query37 161 112 91 91
query38 3201 3144 3058 3058
query39 943 929 896 896
query39_1 891 898 881 881
query40 223 126 105 105
query41 68 66 71 66
query42 98 95 95 95
query43 319 366 280 280
query44
query45 188 180 190 180
query46 1076 1195 715 715
query47 2435 2369 2218 2218
query48 397 403 277 277
query49 614 481 337 337
query50 974 361 255 255
query51 4470 4446 4323 4323
query52 88 86 75 75
query53 239 250 191 191
query54 253 211 194 194
query55 77 74 68 68
query56 221 213 199 199
query57 1451 1413 1338 1338
query58 239 202 211 202
query59 1542 1626 1436 1436
query60 288 240 223 223
query61 142 145 151 145
query62 716 666 588 588
query63 237 179 181 179
query64 2444 760 607 607
query65
query66 1705 451 342 342
query67 29005 29015 29575 29015
query68
query69 422 298 264 264
query70 971 951 939 939
query71 303 219 209 209
query72 2887 2675 2390 2390
query73 871 758 440 440
query74 5105 4926 4750 4750
query75 2660 2563 2232 2232
query76 2335 1144 748 748
query77 347 362 277 277
query78 12448 12378 11937 11937
query79 1478 986 721 721
query80 578 454 380 380
query81 451 279 243 243
query82 559 158 117 117
query83 345 273 244 244
query84
query85 852 505 410 410
query86 361 298 296 296
query87 3382 3296 3223 3223
query88 3596 2711 2675 2675
query89 412 383 331 331
query90 1951 181 181 181
query91 169 156 132 132
query92 65 64 56 56
query93 1570 1490 944 944
query94 558 359 318 318
query95 697 460 346 346
query96 978 804 344 344
query97 2720 2735 2600 2600
query98 216 207 201 201
query99 1151 1182 1028 1028
Total cold run time: 250302 ms
Total hot run time: 168474 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3397293756
##########
regression-test/suites/nereids_p0/local_shuffle/test_local_shuffle_rqg_bugs.groovy:
##########
@@ -0,0 +1,1561 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+/**
+ * Regression tests for bugs discovered by RQG testing on the local-exchange2 branch.
+ *
+ * These queries triggered "must set shared state" errors or incorrect results
+ * in RQG build 183992. Common conditions:
+ * - use_serial_exchange=true (makes ALL Exchanges serial, not just UNPARTITIONED)
+ * - enable_local_shuffle_planner=true (FE-planned local exchange)
+ * - parallel_pipeline_task_num > 1
+ *
+ * Error types reproduced:
+ * 1. must set shared state, in AGGREGATION_OPERATOR
+ * 2. must set shared state, in SORT_OPERATOR
+ * 3. incorrect results with GROUPING SETS + scalar subquery + window function
+ */
+suite("test_local_shuffle_rqg_bugs") {
+
+ // ============================================================
+ // Table setup — mirrors RQG table structure
+ // 10 buckets to match RQG (replication_num=1 for single-BE testing)
+ // ============================================================
+ sql "DROP TABLE IF EXISTS rqg_t1"
+ sql "DROP TABLE IF EXISTS rqg_t2"
+ sql "DROP TABLE IF EXISTS rqg_t3"
+ sql "DROP TABLE IF EXISTS rqg_t4"
+
+ sql """
+ CREATE TABLE rqg_t1 (
+ pk INT NOT NULL,
+ col_int_undef_signed INT,
+ col_int_undef_signed2 INT,
+ col_int_undef_signed_not_null INT NOT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ sql """
+ CREATE TABLE rqg_t2 (
+ pk INT NOT NULL,
+ col_int_undef_signed INT,
+ col_int_undef_signed2 INT,
+ col_bigint_undef_signed_not_null BIGINT NOT NULL,
+ col_decimal_38_10__undef_signed_not_null DECIMAL(38,10) NOT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ // Table for build 184181 GLOBAL_HASH_SHUFFLE bugs — needs varchar + bigint columns
+ sql """
+ CREATE TABLE rqg_t3 (
+ pk INT NOT NULL,
+ col_bigint_undef_signed BIGINT,
+ col_varchar_10__undef_signed VARCHAR(10),
+ col_varchar_64__undef_signed VARCHAR(64)
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ // Second table for FULL OUTER JOIN case (col_bigint_undef_signed_not_null)
+ sql """
+ CREATE TABLE rqg_t4 (
+ pk INT NOT NULL,
+ col_bigint_undef_signed_not_null BIGINT NOT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ sql """
+ INSERT INTO rqg_t3 VALUES
+ (0, -94, 'Abc', 'hello world'),
+ (1, 672609, 'Xyz', null),
+ (2, -3766684, 'Pqr', 'test string'),
+ (3, 5070261, 'abc', 'another row'),
+ (4, null, 'def', 'value four'),
+ (5, -86, 'XgpxlHBLEM', null),
+ (6, 21910, 'abc', 'they'),
+ (7, -63, 'zzzz', 'some text'),
+ (8, -8276281, 'AHlvNtoGLO', 'longer string here'),
+ (9, -101, 'mid', 'final row')
+ """
+
+ sql """
+ INSERT INTO rqg_t4 VALUES
+ (0, 0), (1, 1), (2, 2), (3, 3), (4, 4),
+ (5, 5), (6, 6), (7, 7), (8, 8), (9, 9),
+ (10, 2), (11, 2), (12, 2), (13, 3), (14, 4),
+ (15, 5), (16, 2), (17, 2), (18, 2), (19, 9)
+ """
+
+ // Insert enough rows to exercise multiple pipeline tasks
+ sql """
+ INSERT INTO rqg_t1 VALUES
+ (0, 0, 10, 0), (1, 1, 11, 1), (2, 2, 12, 2), (3, 3, 13, 3),
+ (4, 4, 14, 4), (5, 5, 15, 5), (6, 6, 16, 6), (7, 7, 17, 7),
+ (8, 8, 18, 8), (9, 9, 19, 9), (10, 0, 20, 10), (11, 1, 21, 11),
+ (12, 2, 22, 12), (13, 3, 23, 13), (14, 4, 24, 14), (15, 5, 25, 15),
+ (16, 6, 26, 16), (17, 7, 27, 17), (18, 8, 28, 18), (19, 9, 29, 19)
+ """
+
+ sql """
+ INSERT INTO rqg_t2 VALUES
+ (0, 0, 10, 100, 1.5), (1, 1, 11, 101, 2.5), (2, 2, 12, 102, 3.5),
+ (3, 3, 13, 103, 4.5), (4, 4, 14, 104, 5.5), (5, 5, 15, 105, 6.5),
+ (6, 6, 16, 106, 7.5), (7, 7, 17, 107, 8.5), (8, 8, 18, 108, 9.5),
+ (9, 9, 19, 109, 10.5), (10, 0, 20, 110, 11.5), (11, 1, 21, 111, 12.5),
+ (12, 2, 22, 112, 13.5), (13, 3, 23, 113, 14.5), (14, 4, 24, 114, 15.5),
+ (15, 5, 25, 115, 16.5), (16, 6, 26, 116, 17.5), (17, 7, 27, 117, 18.5),
+ (18, 8, 28, 118, 19.5), (19, 9, 29, 119, 20.5)
+ """
+
+ // Wait for data to be visible
+ Thread.sleep(5000)
+
+ // ============================================================
+ // Common settings
+ // ============================================================
+ sql "SET enable_nereids_planner=true"
+ sql "SET enable_fallback_to_original_planner=false"
+ sql "SET runtime_filter_mode=off"
+ sql "SET enable_profile=true"
+ sql "SET enable_sql_cache=false"
+ sql "SET enable_local_shuffle=true"
+
+ // ============================================================
+ // Bug 1: must set shared state, in AGGREGATION_OPERATOR
+ // RQG case: eliminate_group_by_uniform.case_id_11007680713
+ // Key conditions: use_serial_exchange=true, parallel_pipeline_task_num=3
+ // SQL: EXCEPT with count(*) GROUP BY on both sides
+ // ============================================================
+
+ // Test with FE planner (the buggy path)
+ logger.info("=== Bug 1a: AGG shared state - EXCEPT with serial exchange (FE planner) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=3,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_common_expr_pushdown=false,
+ disable_streaming_preaggregations=true)*/
+ col_int_undef_signed_not_null as col1,
+ col_int_undef_signed_not_null as col2,
+ 0 as col3, count(1)
+ FROM rqg_t1
+ GROUP BY col1, col2, col3
+ EXCEPT
+ SELECT col_bigint_undef_signed_not_null as col1,
+ col_decimal_38_10__undef_signed_not_null as col2,
+ 5 as col3, count(1)
+ FROM rqg_t2
+ GROUP BY col1, col2, col3
+ """
+ logger.info("Bug 1a: PASSED (no crash)")
+ } catch (Throwable t) {
+ logger.error("Bug 1a FAILED: ${t.message}")
+ assertTrue(false, "Bug 1a: must set shared state in AGGREGATION_OPERATOR: ${t.message}")
+ }
+
+ // Compare with BE native planner
+ logger.info("=== Bug 1b: AGG shared state - EXCEPT with serial exchange (BE native) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=3,
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_common_expr_pushdown=false,
+ disable_streaming_preaggregations=true)*/
+ col_int_undef_signed_not_null as col1,
+ col_int_undef_signed_not_null as col2,
+ 0 as col3, count(1)
+ FROM rqg_t1
+ GROUP BY col1, col2, col3
+ EXCEPT
+ SELECT col_bigint_undef_signed_not_null as col1,
+ col_decimal_38_10__undef_signed_not_null as col2,
+ 5 as col3, count(1)
+ FROM rqg_t2
+ GROUP BY col1, col2, col3
+ """
+ logger.info("Bug 1b: PASSED (no crash)")
+ } catch (Throwable t) {
+ logger.error("Bug 1b FAILED: ${t.message}")
+ assertTrue(false, "Bug 1b: BE native also fails: ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 2: must set shared state, in SORT_OPERATOR
+ // RQG case: grouping_set.case_id_5308471751
+ // Key conditions: use_serial_exchange=true, parallel_pipeline_task_num=5
+ // SQL: GROUPING SETS + window function (PERCENT_RANK)
+ // ============================================================
+
+ logger.info("=== Bug 2a: SORT shared state - GROUPING SETS + window (FE planner) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=5,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_share_hash_table_for_broadcast_join=false,
+ disable_streaming_preaggregations=true)*/
+ SUM(PERCENT_RANK() OVER (PARTITION BY col_int_undef_signed2 ORDER BY col_int_undef_signed2))
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed2),(pk, pk),(col_int_undef_signed))
+ """
+ logger.info("Bug 2a: PASSED (no crash)")
+ } catch (Throwable t) {
+ logger.error("Bug 2a FAILED: ${t.message}")
+ assertTrue(false, "Bug 2a: must set shared state in SORT_OPERATOR: ${t.message}")
+ }
+
+ logger.info("=== Bug 2b: SORT shared state - GROUPING SETS + window (BE native) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=5,
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_share_hash_table_for_broadcast_join=false,
+ disable_streaming_preaggregations=true)*/
+ SUM(PERCENT_RANK() OVER (PARTITION BY col_int_undef_signed2 ORDER BY col_int_undef_signed2))
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed2),(pk, pk),(col_int_undef_signed))
+ """
+ logger.info("Bug 2b: PASSED (no crash)")
+ } catch (Throwable t) {
+ logger.error("Bug 2b FAILED: ${t.message}")
+ assertTrue(false, "Bug 2b: BE native also fails: ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 3: incorrect results with GROUPING SETS + scalar subquery + window
+ // RQG case: grouping_set.case_id_5694495756
+ // Key conditions: parallel_pipeline_task_num=2, disable_streaming_preaggregations=true
+ // Expected: all rows same value; Actual: values split proportionally (1/3, 2/3)
+ // ============================================================
+
+ logger.info("=== Bug 3: incorrect results - GROUPING SETS + subquery + window ===")
+ // FE planner
+ def result_fe = sql """
+ SELECT /*+SET_VAR(parallel_pipeline_task_num=2,
+ enable_local_shuffle_planner=true,
+ disable_streaming_preaggregations=true,
+ enable_share_hash_table_for_broadcast_join=true)*/
+ SUM((SELECT MAX(col_int_undef_signed2) FROM rqg_t1))
+ OVER (PARTITION BY pk ORDER BY pk)
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed2, pk),(pk), (pk))
+ """
+ // BE native
+ def result_be = sql """
+ SELECT /*+SET_VAR(parallel_pipeline_task_num=2,
+ enable_local_shuffle_planner=false,
+ disable_streaming_preaggregations=true,
+ enable_share_hash_table_for_broadcast_join=true)*/
+ SUM((SELECT MAX(col_int_undef_signed2) FROM rqg_t1))
+ OVER (PARTITION BY pk ORDER BY pk)
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed2, pk),(pk), (pk))
+ """
+ logger.info("Bug 3 FE result rows: ${result_fe.size()}, first few: ${result_fe.take(5)}")
+ logger.info("Bug 3 BE result rows: ${result_be.size()}, first few: ${result_be.take(5)}")
+
+ // FE planner and BE native must produce identical results (the bug was values split
+ // proportionally instead of equal). Assert row count and order-insensitive content so a
+ // recurrence fails the suite.
+ assertEquals(result_be.size(), result_fe.size(), "Bug 3: FE/BE row count mismatch")
+ assertEquals(result_be.collect { it.toString() }.sort(), result_fe.collect { it.toString() }.sort(),
+ "Bug 3: FE/BE result mismatch")
+
+ // ============================================================
+ // Bug 4: Simplified AGG shared state — single table GROUP BY with serial exchange
+ // Minimal reproduction attempt
+ // ============================================================
+
+ logger.info("=== Bug 4: Simplified AGG shared state ===")
+ for (int ppt : [2, 3, 4, 5]) {
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=${ppt},
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true)*/
+ col_int_undef_signed, count(*)
+ FROM rqg_t1
+ GROUP BY col_int_undef_signed
+ UNION ALL
+ SELECT col_int_undef_signed, count(*)
+ FROM rqg_t2
+ GROUP BY col_int_undef_signed
+ """
+ logger.info("Bug 4 ppt=${ppt}: PASSED")
+ } catch (Throwable t) {
+ logger.error("Bug 4 ppt=${ppt} FAILED: ${t.message}")
+ assertTrue(false, "Bug 4 ppt=${ppt}: AGG shared state crash with serial exchange: ${t.message}")
+ }
+ }
+
+ // ============================================================
+ // Bug 5: GROUPING SETS + window variations with serial exchange
+ // More variations to find minimal repro
+ // ============================================================
+
+ logger.info("=== Bug 5: GROUPING SETS + window variations ===")
+ for (int ppt : [2, 3, 4, 5]) {
+ try {
+ sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,parallel_pipeline_task_num=${ppt},
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true)*/
+ pk, col_int_undef_signed,
+ ROW_NUMBER() OVER (ORDER BY pk)
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((pk, col_int_undef_signed), (pk), ())
+ ORDER BY pk
+ """
+ logger.info("Bug 5 ppt=${ppt}: PASSED")
+ } catch (Throwable t) {
+ logger.error("Bug 5 ppt=${ppt} FAILED: ${t.message}")
+ assertTrue(false, "Bug 5 ppt=${ppt}: GROUPING SETS + window crash with serial exchange: ${t.message}")
+ }
+ }
+
+ // ============================================================
+ // Bug 6: must set shared state, in CROSS_JOIN_OPERATOR
+ // Root cause: nested NLJ + pooling scan — FE planner skipped BROADCAST
+ // local exchange on outer NLJ's build side because child was NLJ (not ScanNode).
+ // Fixed in NestedLoopJoinNode.enforceAndDeriveLocalExchange by using
+ // fragment.useSerialSource() instead of instanceof ScanNode check.
+ // This was the root cause of 989 RQG test failures (build 183677).
+ // ============================================================
+
+ logger.info("=== Bug 6: CROSS_JOIN shared state - nested NLJ + pooling scan (FE planner) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(ignore_storage_data_distribution=true,
+ parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=true,
+ disable_join_reorder=true,
+ disable_colocate_plan=true,
+ auto_broadcast_join_threshold=-1,
+ broadcast_row_count_limit=0,
+ query_timeout=60)*/
+ count(a.pk) AS cnt, a.col_int_undef_signed
+ FROM rqg_t1 a
+ LEFT JOIN rqg_t1 b ON b.col_int_undef_signed >= b.col_int_undef_signed
+ LEFT JOIN rqg_t1 c ON b.pk >= b.pk
+ WHERE a.pk IS NOT NULL
+ GROUP BY a.col_int_undef_signed
+ ORDER BY cnt, a.col_int_undef_signed
+ """
+ logger.info("Bug 6: PASSED (no CROSS_JOIN_OPERATOR shared state error)")
+ } catch (Throwable t) {
+ logger.error("Bug 6 FAILED: ${t.message}")
+ assertTrue(false, "Bug 6: must set shared state in CROSS_JOIN_OPERATOR: ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 7: DataStreamSink hang — sender fragment with pooling scan
+ // Root cause: FE planner did not insert PASSTHROUGH at the root of pooling scan
+ // sender fragments. With pooling scan, only instance 0 creates pipeline tasks,
+ // so only 1 EOS is sent. The downstream ExchangeNode expects _num_instances EOSes
+ // and hangs indefinitely.
+ // Fixed in AddLocalExchange.addLocalExchangeForFragment: insert PASSTHROUGH
+ // when isLocalShuffle && newRoot.isSerialOperator().
+ // Any NLJ + pooling scan query triggers this via the UNPARTITIONED sender fragments.
+ // ============================================================
+
+ logger.info("=== Bug 7: DataStreamSink hang - NLJ + pooling scan sender (FE planner) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(ignore_storage_data_distribution=true,
+ parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=true,
+ disable_join_reorder=true,
+ disable_colocate_plan=true,
+ auto_broadcast_join_threshold=-1,
+ broadcast_row_count_limit=0,
+ query_timeout=60)*/
+ a.col_int_undef_signed, MAX(a.pk) AS mx
+ FROM rqg_t1 a
+ LEFT JOIN rqg_t1 b ON b.col_int_undef_signed < b.col_int_undef_signed
+ WHERE a.pk IS NOT NULL
+ GROUP BY a.col_int_undef_signed
+ ORDER BY a.col_int_undef_signed, mx
+ """
+ logger.info("Bug 7: PASSED (no hang)")
+ } catch (Throwable t) {
+ logger.error("Bug 7 FAILED: ${t.message}")
+ assertTrue(false, "Bug 7: DataStreamSink hang (query timed out or crashed): ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 8: must set shared state, in SORT_OPERATOR / UNION_OPERATOR
+ // Root cause: FE planner + pooling scan + GROUPING SETS. Serial UNPARTITIONED
+ // Exchange reduces downstream pipeline num_tasks to 1. SORT and UNION operators
+ // need _num_instances tasks to inject shared state for all instances.
+ // Fixed by: (1) restoring num_tasks raise for non-scan serial operators in BE
+ // deferred exchanger creation (commit 920d43d), and (2) FE inserting PASSTHROUGH
+ // after serial ExchangeNode in pooling scan fragments (commit d2e7fa2).
+ // ============================================================
+
+ logger.info("=== Bug 8a: SORT/UNION shared state - GROUPING SETS + pooling scan (FE planner) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(ignore_storage_data_distribution=true,
+ parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=true,
+ disable_streaming_preaggregations=true,
+ query_timeout=60)*/
+ pk, col_int_undef_signed, SUM(col_int_undef_signed_not_null) AS sv
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((pk, col_int_undef_signed), (pk), ())
+ ORDER BY pk, col_int_undef_signed, sv
+ """
+ logger.info("Bug 8a: PASSED (no SORT/UNION_OPERATOR shared state error)")
+ } catch (Throwable t) {
+ logger.error("Bug 8a FAILED: ${t.message}")
+ assertTrue(false, "Bug 8a: must set shared state in SORT/UNION_OPERATOR: ${t.message}")
+ }
+
+ logger.info("=== Bug 8b: SORT shared state - window + GROUPING SETS + pooling scan (FE planner) ===")
+ try {
+ sql """
+ SELECT /*+SET_VAR(ignore_storage_data_distribution=true,
+ parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=true,
+ disable_streaming_preaggregations=true,
+ query_timeout=60)*/
+ pk, SUM(col_int_undef_signed_not_null) AS sv,
+ ROW_NUMBER() OVER (ORDER BY pk) AS rn
+ FROM rqg_t1
+ GROUP BY GROUPING SETS ((pk), ())
+ ORDER BY pk, sv, rn
+ """
+ logger.info("Bug 8b: PASSED (no SORT_OPERATOR shared state error)")
+ } catch (Throwable t) {
+ logger.error("Bug 8b FAILED: ${t.message}")
+ assertTrue(false, "Bug 8b: must set shared state in SORT_OPERATOR (window+grouping_sets): ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 9: FE/BE result inconsistency — agg after NLJ + pooling scan
+ // Root cause: StreamingAgg used fragment.useSerialSource()=true to require
+ // PASSTHROUGH from child, but when child is NLJ (not directly a serial scan),
+ // NLJ outputs ADAPTIVE_PASSTHROUGH. FE wrongly inserted an extra PASSTHROUGH
+ // exchange between StreamingAgg and NLJ (5 extra LOCAL_EXCHANGE_SINK_OPERATOR
+ // entries vs BE native).
+ // Fixed in AggregationNode: only requirePassthrough when
+ // children.get(0).isSerialOperator()=true, mirroring BE _child->is_serial_operator().
+ // ============================================================
+
+ logger.info("=== Bug 9: FE/BE result consistency - agg after NLJ + pooling scan ===")
+ def bug9_fe = sql """
+ SELECT /*+SET_VAR(ignore_storage_data_distribution=true,
+ parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=true,
+ disable_join_reorder=true,
+ disable_colocate_plan=true,
+ auto_broadcast_join_threshold=-1,
+ broadcast_row_count_limit=0)*/
+ a.col_int_undef_signed, MAX(a.pk) AS mx
+ FROM rqg_t1 a LEFT JOIN rqg_t1 b ON b.col_int_undef_signed < b.col_int_undef_signed
+ WHERE a.pk IS NOT NULL
+ GROUP BY a.col_int_undef_signed
+ ORDER BY a.col_int_undef_signed, mx
+ """
+ def bug9_be = sql """
+ SELECT /*+SET_VAR(ignore_storage_data_distribution=true,
+ parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=false,
+ disable_join_reorder=true,
+ disable_colocate_plan=true,
+ auto_broadcast_join_threshold=-1,
+ broadcast_row_count_limit=0)*/
+ a.col_int_undef_signed, MAX(a.pk) AS mx
+ FROM rqg_t1 a LEFT JOIN rqg_t1 b ON b.col_int_undef_signed < b.col_int_undef_signed
+ WHERE a.pk IS NOT NULL
+ GROUP BY a.col_int_undef_signed
+ ORDER BY a.col_int_undef_signed, mx
+ """
+ logger.info("Bug 9 FE rows: ${bug9_fe.size()}, BE rows: ${bug9_be.size()}")
+ assertEquals(bug9_be.size(), bug9_fe.size(), "Bug 9: FE/BE row count mismatch")
+ assertEquals(bug9_be, bug9_fe, "Bug 9: FE/BE result mismatch for agg after NLJ + pooling scan")
+ logger.info("Bug 9: PASSED (FE/BE results match)")
+
+ // ============================================================
+ // Bug 10: GLOBAL_HASH_SHUFFLE Rows mismatched — self-join + NLJ
+ // RQG case: 906784672 (build 184181)
+ // Root cause: HashJoinNode used requireGlobalExecutionHash() → GLOBAL local exchange
+ // inserted when use_serial_exchange=true; shuffle_idx_to_instance_idx map has only
+ // 4 entries (1/BE) but GLOBAL hash needs N*dop entries → most rows unrouted (0 actual rows).
+ // Fixed: changed to requireHash() so resolveExchangeType() downgrades to LOCAL hash.
+ // SQL: self-join (table1 LEFT JOIN table1 table2 ON pk=col_bigint_undef_signed)
+ // then NLJ (LEFT JOIN table1 table3 ON pk > col_bigint_undef_signed)
+ // ============================================================
+
+ logger.info("=== Bug 10: GLOBAL_HASH_SHUFFLE Rows mismatched - self-join + NLJ (build 184181 case 906784672) ===")
+ def bug10_fe = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true, parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ disable_join_reorder=true, disable_colocate_plan=true)*/
+ table1.pk AS field1, table1.col_bigint_undef_signed AS field2
+ FROM rqg_t3 AS table1
+ LEFT JOIN rqg_t3 AS table2 ON table1.pk = table2.col_bigint_undef_signed
+ LEFT JOIN rqg_t3 AS table3 ON table1.pk > table2.col_bigint_undef_signed
+ WHERE (table1.col_varchar_10__undef_signed > 'AHlvNtoGLO'
+ AND table1.col_varchar_10__undef_signed < 'zzzz')
+ OR (table1.col_bigint_undef_signed = table1.pk AND table1.col_varchar_64__undef_signed IS NULL)
+ OR (table1.pk != table1.pk AND table1.pk <> 2)
+ GROUP BY field1, field2
+ ORDER BY field1, field2
+ """
+ def bug10_be = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true, parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ disable_join_reorder=true, disable_colocate_plan=true)*/
+ table1.pk AS field1, table1.col_bigint_undef_signed AS field2
+ FROM rqg_t3 AS table1
+ LEFT JOIN rqg_t3 AS table2 ON table1.pk = table2.col_bigint_undef_signed
+ LEFT JOIN rqg_t3 AS table3 ON table1.pk > table2.col_bigint_undef_signed
+ WHERE (table1.col_varchar_10__undef_signed > 'AHlvNtoGLO'
+ AND table1.col_varchar_10__undef_signed < 'zzzz')
+ OR (table1.col_bigint_undef_signed = table1.pk AND table1.col_varchar_64__undef_signed IS NULL)
+ OR (table1.pk != table1.pk AND table1.pk <> 2)
+ GROUP BY field1, field2
+ ORDER BY field1, field2
+ """
+ logger.info("Bug 10 FE rows: ${bug10_fe.size()}, BE rows: ${bug10_be.size()}")
+ assertEquals(bug10_be.size(), bug10_fe.size(), "Bug 10: FE/BE row count mismatch (GLOBAL_HASH_SHUFFLE Rows mismatched)")
+ assertEquals(bug10_be, bug10_fe, "Bug 10: FE/BE result mismatch for self-join + NLJ")
+ logger.info("Bug 10: PASSED")
+
+ // ============================================================
+ // Bug 11: GLOBAL_HASH_SHUFFLE Rows mismatched — FULL OUTER JOIN + GROUP BY
+ // RQG case: 11007681241 (build 184181)
+ // Same root cause as Bug 10.
+ // SQL: FULL OUTER JOIN on col_bigint_undef_signed_not_null with WHERE + GROUP BY
+ // ============================================================
+
+ logger.info("=== Bug 11: GLOBAL_HASH_SHUFFLE Rows mismatched - FULL OUTER JOIN + GROUP BY (build 184181 case 11007681241) ===")
+ def bug11_fe = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true, parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false)*/
+ t1.col_bigint_undef_signed_not_null, t2.col_bigint_undef_signed_not_null, count(1)
+ FROM rqg_t4 t1
+ FULL OUTER JOIN rqg_t2 t2
+ ON t1.col_bigint_undef_signed_not_null = t2.col_bigint_undef_signed_not_null
+ WHERE t2.col_bigint_undef_signed_not_null = 2
+ GROUP BY t1.col_bigint_undef_signed_not_null, t2.col_bigint_undef_signed_not_null
+ ORDER BY 1, 2, 3
+ """
+ def bug11_be = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true, parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false)*/
+ t1.col_bigint_undef_signed_not_null, t2.col_bigint_undef_signed_not_null, count(1)
+ FROM rqg_t4 t1
+ FULL OUTER JOIN rqg_t2 t2
+ ON t1.col_bigint_undef_signed_not_null = t2.col_bigint_undef_signed_not_null
+ WHERE t2.col_bigint_undef_signed_not_null = 2
+ GROUP BY t1.col_bigint_undef_signed_not_null, t2.col_bigint_undef_signed_not_null
+ ORDER BY 1, 2, 3
+ """
+ logger.info("Bug 11 FE rows: ${bug11_fe.size()}, BE rows: ${bug11_be.size()}")
+ assertEquals(bug11_be.size(), bug11_fe.size(), "Bug 11: FE/BE row count mismatch (GLOBAL_HASH_SHUFFLE Rows mismatched)")
+ assertEquals(bug11_be, bug11_fe, "Bug 11: FE/BE result mismatch for FULL OUTER JOIN + GROUP BY")
+ logger.info("Bug 11: PASSED")
+
+ // ============================================================
+ // Bug 12: GLOBAL_HASH_SHUFFLE Rows mismatched — LEFT JOIN + VARCHAR predicates + MIN()
+ // RQG case: 906784662 (build 184181)
+ // Same root cause as Bug 10/11.
+ // SQL: LEFT JOIN on pk with VARCHAR NOT IN / BETWEEN / IN predicates, MIN() aggregate
+ // ============================================================
+
+ logger.info("=== Bug 12: GLOBAL_HASH_SHUFFLE Rows mismatched - LEFT JOIN + VARCHAR predicates (build 184181 case 906784662) ===")
+ def bug12_fe = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true, parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ disable_join_reorder=true, disable_colocate_plan=true)*/
+ table1.pk AS field1, MIN(table1.pk) AS field2
+ FROM rqg_t3 AS table1
+ LEFT JOIN rqg_t1 AS table2 ON table2.pk = table1.pk
+ WHERE table1.col_varchar_64__undef_signed NOT IN ('they')
+ AND table1.col_varchar_10__undef_signed BETWEEN 'AHlvNtoGLO' AND 'z'
+ AND table1.pk IN (3, 6, 8, 9, 2)
+ GROUP BY field1
+ ORDER BY field1, field2 ASC
+ """
+ def bug12_be = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true, parallel_pipeline_task_num=4,
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ disable_join_reorder=true, disable_colocate_plan=true)*/
+ table1.pk AS field1, MIN(table1.pk) AS field2
+ FROM rqg_t3 AS table1
+ LEFT JOIN rqg_t1 AS table2 ON table2.pk = table1.pk
+ WHERE table1.col_varchar_64__undef_signed NOT IN ('they')
+ AND table1.col_varchar_10__undef_signed BETWEEN 'AHlvNtoGLO' AND 'z'
+ AND table1.pk IN (3, 6, 8, 9, 2)
+ GROUP BY field1
+ ORDER BY field1, field2 ASC
+ """
+ logger.info("Bug 12 FE rows: ${bug12_fe.size()}, BE rows: ${bug12_be.size()}")
+ assertEquals(bug12_be.size(), bug12_fe.size(), "Bug 12: FE/BE row count mismatch (GLOBAL_HASH_SHUFFLE Rows mismatched)")
+ assertEquals(bug12_be, bug12_fe, "Bug 12: FE/BE result mismatch for LEFT JOIN + VARCHAR predicates")
+ logger.info("Bug 12: PASSED")
+
+ // ============================================================
+ // Bug 13: NLJ COREDUMP — serial NLJ + pooling scan + BROADCAST build side
+ // RQG build 184430, query c0dafc1bed0f4910
+ // Root cause: serial NLJ (RIGHT_OUTER) with pooling scan inserted BROADCAST
+ // local exchange on build side, inflating build pipeline num_tasks to _num_instances
+ // while probe pipeline stayed at 1 task. Instance 1+ created build tasks without
+ // corresponding probe tasks → source_deps empty → set_ready_to_read() crash.
+ // Fixed: serial NLJ sets buildSideRequire=noRequire() to match BE-native
+ // num_tasks_of_parent()<=1 skip logic.
+ // ============================================================
+
+ logger.info("=== Bug 13: NLJ COREDUMP - serial NLJ + pooling scan (FE planner) ===")
+ try {
+ def bug13_fe = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true, parallel_pipeline_task_num=0,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ enable_share_hash_table_for_broadcast_join=false,
+ disable_streaming_preaggregations=true,
+ disable_join_reorder=true)*/
+ t2.col_bigint_undef_signed_not_null AS field1
+ FROM rqg_t4 AS t1
+ RIGHT OUTER JOIN rqg_t2 AS t2 ON t1.col_bigint_undef_signed_not_null > t2.col_bigint_undef_signed_not_null
+ GROUP BY field1
+ ORDER BY field1 ASC
+ """
+ def bug13_be = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true, parallel_pipeline_task_num=0,
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ enable_share_hash_table_for_broadcast_join=false,
+ disable_streaming_preaggregations=true,
+ disable_join_reorder=true)*/
+ t2.col_bigint_undef_signed_not_null AS field1
+ FROM rqg_t4 AS t1
+ RIGHT OUTER JOIN rqg_t2 AS t2 ON t1.col_bigint_undef_signed_not_null > t2.col_bigint_undef_signed_not_null
+ GROUP BY field1
+ ORDER BY field1 ASC
+ """
+ logger.info("Bug 13 FE rows: ${bug13_fe.size()}, BE rows: ${bug13_be.size()}")
+ assertEquals(bug13_be.size(), bug13_fe.size(), "Bug 13: FE/BE row count mismatch (NLJ COREDUMP)")
+ assertEquals(bug13_be, bug13_fe, "Bug 13: FE/BE result mismatch for serial NLJ + pooling scan")
+ logger.info("Bug 13: PASSED (no crash, results match)")
+ } catch (Throwable t) {
+ logger.error("Bug 13 FAILED: ${t.message}")
+ assertTrue(false, "Bug 13: NLJ COREDUMP (serial NLJ + pooling scan): ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 14: BUCKET_SHUFFLE join + serial build Exchange — must set shared state
+ // RQG build 184563, cases 906784706/906784783/906784987/906785006
+ // Root cause: BUCKET_SHUFFLE join build side ExchangeNode marked serial in
+ // pooling scan fragment → build pipeline num_tasks reduced to 1 →
+ // instance 1+ have probe tasks without build tasks → shared state injection
+ // fails. Fixed: isBucketShuffle() branch checks buildChildSerial and uses
+ // requirePassToOne() to restore num_tasks, matching BE-native behavior.
+ // Requires replication_num=3 + [shuffle] hint to force BUCKET_SHUFFLE plan.
+ // ============================================================
+
+ logger.info("=== Bug 14: BUCKET_SHUFFLE join + serial build Exchange (FE planner) ===")
+ // Need replication_num=3 for BUCKET_SHUFFLE. Check if allow_replica_on_same_host is enabled.
+ def allowSameHost = sql "ADMIN SHOW FRONTEND CONFIG LIKE 'allow_replica_on_same_host'"
+ if (allowSameHost[0][1].toString() == "true") {
+ sql "DROP TABLE IF EXISTS rqg_t5_rep3"
+ sql "DROP TABLE IF EXISTS rqg_t6_rep3"
+ try {
+ sql """
+ CREATE TABLE rqg_t5_rep3 (
+ pk INT NULL,
+ col_varchar_10__undef_signed VARCHAR(10) NULL,
+ col_bigint_undef_signed BIGINT NULL,
+ col_varchar_64__undef_signed VARCHAR(64) NULL
+ ) DUPLICATE KEY(pk, col_varchar_10__undef_signed)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "3")
+ """
+ sql """
+ CREATE TABLE rqg_t6_rep3 (
+ pk INT NULL,
+ col_varchar_10__undef_signed VARCHAR(10) NULL,
+ col_bigint_undef_signed BIGINT NULL,
+ col_varchar_64__undef_signed VARCHAR(64) NULL
+ ) DUPLICATE KEY(pk, col_varchar_10__undef_signed)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "3")
+ """
+ sql """
+ INSERT INTO rqg_t5_rep3 VALUES
+ (0,'abc',-94,'hello'),(1,'xyz',672609,null),(2,'pqr',-3766684,'test'),
+ (3,'abc',5070261,'another'),(4,'def',null,'value'),(5,'so',-86,null),
+ (6,'abc',21910,'they'),(7,'zzzz',-63,'some'),(8,'xPLflvBEcW',-8276281,'longer'),
+ (9,'mid',-101,'final')
+ """
+ sql """
+ INSERT INTO rqg_t6_rep3 VALUES
+ (0,'aaa',100,'world'),(1,'bbb',200,null),(2,'ccc',300,'foo'),
+ (3,'ddd',400,'bar'),(4,'eee',500,'baz'),(5,'fff',600,null),
+ (6,'ggg',700,'qux'),(7,'hhh',800,'quux'),(8,'iii',900,'corge'),
+ (9,'jjj',1000,'grault')
+ """
+ Thread.sleep(3000)
+
+ def bug14_fe = sql """
+ SELECT /*+SET_VAR(enable_local_shuffle_planner=true,
+ parallel_pipeline_task_num=3,
+ disable_streaming_preaggregations=true,
+ enable_sql_cache=false,
+ disable_join_reorder=true)*/
+ table1.pk AS field1
+ FROM rqg_t5_rep3 AS table1
+ LEFT OUTER JOIN [shuffle] rqg_t6_rep3 AS table2 ON table1.pk = table2.pk
+ WHERE table1.col_varchar_10__undef_signed >= 'so'
+ GROUP BY field1
+ ORDER BY field1
+ """
+ def bug14_be = sql """
+ SELECT /*+SET_VAR(enable_local_shuffle_planner=false,
+ parallel_pipeline_task_num=3,
+ disable_streaming_preaggregations=true,
+ enable_sql_cache=false,
+ disable_join_reorder=true)*/
+ table1.pk AS field1
+ FROM rqg_t5_rep3 AS table1
+ LEFT OUTER JOIN [shuffle] rqg_t6_rep3 AS table2 ON table1.pk = table2.pk
+ WHERE table1.col_varchar_10__undef_signed >= 'so'
+ GROUP BY field1
+ ORDER BY field1
+ """
+ logger.info("Bug 14 FE rows: ${bug14_fe.size()}, BE rows: ${bug14_be.size()}")
+ assertEquals(bug14_be.size(), bug14_fe.size(), "Bug 14: FE/BE row count mismatch (BUCKET_SHUFFLE must set shared state)")
+ assertEquals(bug14_be, bug14_fe, "Bug 14: FE/BE result mismatch for BUCKET_SHUFFLE + serial build Exchange")
+ logger.info("Bug 14: PASSED (no crash, results match)")
+ } catch (Throwable t) {
+ logger.error("Bug 14 FAILED: ${t.message}")
+ assertTrue(false, "Bug 14: BUCKET_SHUFFLE must set shared state: ${t.message}")
+ }
+ } else {
+ logger.info("Bug 14: SKIPPED (allow_replica_on_same_host not enabled, cannot create replication_num=3 tables)")
+ }
+
+ // ============================================================
+ // Bug 15: BUCKET_SHUFFLE join wrong results with serial exchange — PASS_TO_ONE data loss
+ // Root cause: When serial exchange feeds BUCKET_SHUFFLE join build side,
+ // PASS_TO_ONE routes ALL build data to task 0. Unlike BROADCAST joins,
+ // BUCKET_SHUFFLE has no shared hash table mechanism — tasks 1..N-1 build
+ // empty hash tables and lose rows during probe. Fixed by using
+ // BUCKET_HASH_SHUFFLE instead of PASS_TO_ONE for BUCKET_SHUFFLE build side.
+ // Tables use 3 buckets so pptn=4 triggers serial scan on single BE (3 < 4*1).
+ // ============================================================
+
+ logger.info("=== Bug 15: BUCKET_SHUFFLE join wrong results with serial PASS_TO_ONE ===")
+
+ sql "DROP TABLE IF EXISTS rqg_t7_3bucket"
+ sql "DROP TABLE IF EXISTS rqg_t8_3bucket"
+
+ sql """
+ CREATE TABLE rqg_t7_3bucket (
+ pk INT NOT NULL,
+ col_int INT NULL,
+ col_varchar VARCHAR(64) NULL
+ ) DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 3
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ sql """
+ CREATE TABLE rqg_t8_3bucket (
+ pk INT NOT NULL,
+ col_int INT NULL,
+ col_varchar VARCHAR(64) NULL
+ ) DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 3
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ sql """
+ INSERT INTO rqg_t7_3bucket VALUES
+ (0, 10, 'aaa'), (1, 20, 'bbb'), (2, 30, 'ccc'),
+ (3, 40, 'ddd'), (4, 50, 'eee'), (5, 60, 'fff'),
+ (6, 70, 'ggg'), (7, 80, 'hhh'), (8, 90, 'iii'), (9, 100, 'jjj')
+ """
+
+ sql """
+ INSERT INTO rqg_t8_3bucket VALUES
+ (0, 10, 'aaa'), (1, 20, 'bbb'), (2, 30, 'ccc'),
+ (3, 40, 'ddd'), (4, 50, 'eee'), (5, 60, 'fff'),
+ (6, 70, 'ggg'), (7, 80, 'hhh'), (8, 90, 'iii'), (9, 100, 'jjj')
+ """
+
+ Thread.sleep(3000)
+
+ try {
+ // pptn=4 with 3 buckets on 1 BE: 3 < 4*1 → serial scan → serial exchange
+ // This triggers the PASS_TO_ONE bug for BUCKET_SHUFFLE build side.
+ // Also test with higher pptn values to cover more cases.
+ for (int ppt : [4, 6, 8]) {
+ def bug15_fe = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,
+ parallel_pipeline_task_num=${ppt},
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_share_hash_table_for_broadcast_join=false,
+ enable_sql_cache=false,
+ disable_join_reorder=true)*/
+ t1.pk, t1.col_int, t2.col_varchar
+ FROM rqg_t7_3bucket t1
+ INNER JOIN [shuffle] rqg_t8_3bucket t2 ON t1.pk = t2.pk
+ ORDER BY t1.pk
+ """
+ def bug15_be = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,
+ parallel_pipeline_task_num=${ppt},
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_share_hash_table_for_broadcast_join=false,
+ enable_sql_cache=false,
+ disable_join_reorder=true)*/
+ t1.pk, t1.col_int, t2.col_varchar
+ FROM rqg_t7_3bucket t1
+ INNER JOIN [shuffle] rqg_t8_3bucket t2 ON t1.pk = t2.pk
+ ORDER BY t1.pk
+ """
+ logger.info("Bug 15 ppt=${ppt}: FE rows=${bug15_fe.size()}, BE rows=${bug15_be.size()}")
+ assertEquals(10, bug15_fe.size(), "Bug 15 ppt=${ppt}: expected 10 rows from FE planner, got ${bug15_fe.size()}")
+ assertEquals(bug15_be, bug15_fe, "Bug 15 ppt=${ppt}: FE/BE result mismatch for BUCKET_SHUFFLE + serial exchange")
+ }
+
+ // Also test LEFT OUTER JOIN to verify no rows lost on probe side
+ def bug15_left_fe = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,
+ parallel_pipeline_task_num=6,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_share_hash_table_for_broadcast_join=false,
+ enable_sql_cache=false,
+ disable_join_reorder=true)*/
+ t1.pk, t2.col_int
+ FROM rqg_t7_3bucket t1
+ LEFT OUTER JOIN [shuffle] rqg_t8_3bucket t2 ON t1.pk = t2.pk
+ ORDER BY t1.pk
+ """
+ def bug15_left_be = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,
+ parallel_pipeline_task_num=6,
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_share_hash_table_for_broadcast_join=false,
+ enable_sql_cache=false,
+ disable_join_reorder=true)*/
+ t1.pk, t2.col_int
+ FROM rqg_t7_3bucket t1
+ LEFT OUTER JOIN [shuffle] rqg_t8_3bucket t2 ON t1.pk = t2.pk
+ ORDER BY t1.pk
+ """
+ assertEquals(10, bug15_left_fe.size(), "Bug 15 LEFT JOIN: expected 10 rows from FE planner")
+ assertEquals(bug15_left_be, bug15_left_fe, "Bug 15 LEFT JOIN: FE/BE result mismatch")
+
+ logger.info("Bug 15: PASSED (no wrong results, all pptn values correct)")
+ } catch (Throwable t) {
+ logger.error("Bug 15 FAILED: ${t.message}")
+ assertTrue(false, "Bug 15: BUCKET_SHUFFLE wrong results with serial PASS_TO_ONE: ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 16 & 17: Serial AnalyticEval crash and DataStreamSink hang
+ // with LocalShuffleAssignedJob (multiple instances on one BE)
+ //
+ // Bug 16 (crash): Exchange wraps itself with PASSTHROUGH LocalExchange.
+ // This restores AnalyticSink pipeline to _num_instances tasks while
+ // serial AnalyticSource stays at 1 task. For instance_idx > 0,
+ // source_deps is empty → DCHECK crash.
+ //
+ // Bug 17 (hang): After fixing the crash, serial AnalyticSource reduces
+ // all downstream pipeline tasks to 1 via add_pipeline() inheritance.
+ // Only instance 0 runs DataStreamSink → receiver expects _num_instances
+ // EOSes → hang.
+ //
+ // Both triggered by: OVER() with no PARTITION BY + GROUPING SETS +
+ // pptn=0 (auto-parallel) + disable_streaming_preaggregations=true
+ // RQG build 186195, query IDs: 7f3178a77c2c4b6b, 71887f7bf804c0c, 5dd9fcad234c4484
+ // ============================================================
+ sql "DROP TABLE IF EXISTS rqg_analytic_t1"
+ sql """
+ CREATE TABLE rqg_analytic_t1 (
+ pk INT NOT NULL,
+ col_int_undef_signed INT
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 10
+ PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ INSERT INTO rqg_analytic_t1 VALUES
+ (1, 10), (2, 20), (3, 30), (4, 40), (5, 50),
+ (6, 60), (7, 70), (8, 80), (9, 90), (10, 100),
+ (11, 10), (12, 20), (13, 30), (14, 40), (15, 50),
+ (16, 60), (17, 70), (18, 80), (19, 90), (20, 100)
+ """
+
+ try {
+ logger.info("Bug 16+17: Testing serial AnalyticEval with GROUPING SETS")
+
+ // Baseline: pptn=1 (no multi-instance, no local shuffle)
+ def bug16_baseline = sql """
+ SELECT /*+SET_VAR(parallel_pipeline_task_num=1,
+ enable_sql_cache=false,
+ disable_streaming_preaggregations=true)*/
+ COUNT(MIN(col_int_undef_signed) OVER())
+ FROM rqg_analytic_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed, pk), (), (pk))
+ ORDER BY 1
+ """
+ assertEquals(41, bug16_baseline.size(), "Bug 16 baseline: expected 41 rows")
+
+ // Test with pptn=0 (auto-parallel, triggers LocalShuffleAssignedJob)
+ for (int ppt : [0, 2, 4, 8]) {
+ def bug16_result = sql """
+ SELECT /*+SET_VAR(parallel_pipeline_task_num=${ppt},
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ disable_streaming_preaggregations=true)*/
+ COUNT(MIN(col_int_undef_signed) OVER())
+ FROM rqg_analytic_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed, pk), (), (pk))
+ ORDER BY 1
+ """
+ assertEquals(bug16_baseline, bug16_result,
+ "Bug 16+17 pptn=${ppt}: result mismatch with serial AnalyticEval")
+ }
+
+ // Also test with use_serial_exchange=true (makes ALL exchanges serial)
+ def bug16_serial = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,
+ parallel_pipeline_task_num=0,
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ disable_streaming_preaggregations=true)*/
+ COUNT(MIN(col_int_undef_signed) OVER())
+ FROM rqg_analytic_t1
+ GROUP BY GROUPING SETS ((col_int_undef_signed, pk), (), (pk))
+ ORDER BY 1
+ """
+ assertEquals(bug16_baseline, bug16_serial,
+ "Bug 16+17 serial_exchange: result mismatch")
+
+ logger.info("Bug 16+17: PASSED (no crash, no hang, correct results)")
+ } catch (Throwable t) {
+ logger.error("Bug 16+17 FAILED: ${t.message}")
+ assertTrue(false, "Bug 16+17: Serial AnalyticEval crash/hang: ${t.message}")
+ }
+
+ // Bug 18: DCHECK crash in Pipeline::set_num_tasks when PASSTHROUGH LE is inserted
+ // between serial NLJ and its child Exchange.
+ // Root cause: ExchangeNode.enforceAndDeriveLocalExchange wraps UNPARTITIONED serial
+ // Exchange with PASSTHROUGH LE. On BE, NLJ_PROBE (serial) sets pipeline num_tasks=1,
+ // then the LE handler's set_num_tasks(_num_instances) overrides it to N, triggering
+ // DCHECK (serial operator in pipeline with num_tasks > 1).
+ // Fix: skip PASSTHROUGH wrapping when hasSerialAncestorInPipeline is true.
+ // Query: LEFT JOIN with always-true self-ref condition (table.pk = table.pk) creates
+ // RIGHT_OUTER NLJ (serial). With pptn>1 and ignore_data_distribution, the fragment
+ // gets N instances but NLJ forces 1 task.
+ try {
+ logger.info("Bug 18: Testing serial NLJ with PASSTHROUGH LE crash")
+ // Use existing rqg_t1 table (10 rows, 10 buckets)
+ def bug18_baseline = sql """
+ SELECT /*+SET_VAR(enable_local_shuffle_planner=false,
+ enable_sql_cache=false)*/
+ table1.col_int_undef_signed AS field1
+ FROM rqg_t1 AS table1
+ LEFT JOIN rqg_t1 AS table2
+ ON table2.pk = table2.pk
+ WHERE table1.pk BETWEEN 2 AND 11
+ GROUP BY field1
+ ORDER BY 1
+ """
+
+ // Test with various pptn values — crash requires pptn > 1
+ for (int ppt : [4, 7]) {
+ def bug18_result = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=false,
+ parallel_pipeline_task_num=${ppt},
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ enable_share_hash_table_for_broadcast_join=false,
+ enable_broadcast_join_force_passthrough=true,
+ enable_parallel_scan=false)*/
+ table1.col_int_undef_signed AS field1
+ FROM rqg_t1 AS table1
+ LEFT JOIN rqg_t1 AS table2
+ ON table2.pk = table2.pk
+ WHERE table1.pk BETWEEN 2 AND 11
+ GROUP BY field1
+ ORDER BY 1
+ """
+ assertEquals(bug18_baseline, bug18_result,
+ "Bug 18 pptn=${ppt}: result mismatch with serial NLJ + local exchange")
+ }
+ logger.info("Bug 18: PASSED (no crash, correct results)")
+ } catch (Throwable t) {
+ logger.error("Bug 18 FAILED: ${t.message}")
+ assertTrue(false, "Bug 18: Serial NLJ PASSTHROUGH LE crash: ${t.message}")
+ }
+
+ // Bug 19: source_deps.size()=0 crash in NLJ build sink.
+ // Root cause: serial NLJ (RIGHT_OUTER) resets serial ancestor flag for build side.
+ // Exchange(UNPARTITIONED) on build side sees hasSerialAncestorInPipeline=false and
+ // inserts PASSTHROUGH LE. This restores build pipeline num_tasks to _num_instances
+ // while probe pipeline stays at 1. The extra build tasks have NLJ shared state with
+ // empty source_deps → crash in set_ready_to_read().
+ // Fix: shouldResetSerialFlagForChild(1) returns false when NLJ is serial.
+ // Differs from Bug 18 in fuzzy vars: enable_share_hash_table=true, broadcast_passthrough=false.
+ try {
+ logger.info("Bug 19: Testing serial NLJ build-side source_deps crash")
+ def bug19_baseline = sql """
+ SELECT /*+SET_VAR(enable_local_shuffle_planner=false,
+ enable_sql_cache=false)*/
+ table1.col_int_undef_signed AS field1
+ FROM rqg_t1 AS table1
+ LEFT JOIN rqg_t1 AS table2
+ ON table2.pk = table2.pk
+ WHERE table1.pk BETWEEN 2 AND 11
+ GROUP BY field1
+ ORDER BY 1
+ """
+
+ for (int ppt : [2, 4]) {
+ def bug19_result = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=false,
+ parallel_pipeline_task_num=${ppt},
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ enable_share_hash_table_for_broadcast_join=true,
+ enable_broadcast_join_force_passthrough=false,
+ enable_parallel_scan=true,
+ disable_streaming_preaggregations=true)*/
+ table1.col_int_undef_signed AS field1
+ FROM rqg_t1 AS table1
+ LEFT JOIN rqg_t1 AS table2
+ ON table2.pk = table2.pk
+ WHERE table1.pk BETWEEN 2 AND 11
+ GROUP BY field1
+ ORDER BY 1
+ """
+ assertEquals(bug19_baseline, bug19_result,
+ "Bug 19 pptn=${ppt}: result mismatch with serial NLJ build side crash")
+ }
+ logger.info("Bug 19: PASSED (no crash, correct results)")
+ } catch (Throwable t) {
+ logger.error("Bug 19 FAILED: ${t.message}")
+ assertTrue(false, "Bug 19: Serial NLJ build-side source_deps crash: ${t.message}")
+ }
+
+ // Bug 20: Hang (ASAN: COREDUMP source_deps.size()=0 in AggSinkOperatorX) when
+ // use_serial_exchange=true + RIGHT JOIN + GROUP BY in non-pooling fragment.
+ // Root cause: serial HASH Exchange in non-pooling fragment returned NOOP, causing FE
+ // to insert LOCAL_EXECUTION_HASH_SHUFFLE LE. On BE, serial Exchange pipeline has 1 task
+ // but LE downstream has _num_instances tasks. AggSink on instances 1+ has empty source_deps.
+ // Fix: ExchangeNode.enforceAndDeriveLocalExchange() returns actual distribution type
+ // (GLOBAL_EXECUTION_HASH_SHUFFLE/BUCKET_HASH_SHUFFLE) for serial Exchange in non-pooling
+ // fragments, preventing LE insertion.
+ // Requires 3+ BEs to reproduce (single BE has _num_instances=1, no hang).
+ try {
+ logger.info("Bug 20: Testing serial exchange + agg hang in non-pooling fragment")
+ // Baseline uses same fuzzy vars but with planner=false (BE-native).
+ // This way we compare FE-planned vs BE-native under identical conditions,
+ // not against "correct" results — use_serial_exchange=true itself may have
+ // pre-existing BE bugs with certain pptn values.
+ def bug20_baseline_sql = { int ppt -> """
+ SELECT /*+SET_VAR(use_serial_exchange=true,
+ parallel_pipeline_task_num=${ppt},
+ enable_local_shuffle_planner=false,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ enable_share_hash_table_for_broadcast_join=false,
+ disable_streaming_preaggregations=true)*/
+ table1.pk AS field1
+ FROM rqg_t1 AS table1
+ RIGHT OUTER JOIN rqg_t1 AS table2 ON table1.pk = table2.pk
+ LEFT JOIN rqg_t1 AS table3 ON table3.pk = table1.pk
+ WHERE table1.col_int_undef_signed IS NOT NULL OR table1.pk <> 10
+ GROUP BY field1
+ ORDER BY 1
+ """ }
+ for (int ppt : [3, 4, 7]) {
+ def bug20_baseline = sql bug20_baseline_sql(ppt)
+ def bug20_result = sql """
+ SELECT /*+SET_VAR(use_serial_exchange=true,
+ parallel_pipeline_task_num=${ppt},
+ enable_local_shuffle_planner=true,
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ enable_share_hash_table_for_broadcast_join=false,
+ disable_streaming_preaggregations=true)*/
+ table1.pk AS field1
+ FROM rqg_t1 AS table1
+ RIGHT OUTER JOIN rqg_t1 AS table2 ON table1.pk = table2.pk
+ LEFT JOIN rqg_t1 AS table3 ON table3.pk = table1.pk
+ WHERE table1.col_int_undef_signed IS NOT NULL OR table1.pk <> 10
+ GROUP BY field1
+ ORDER BY 1
+ """
+ assertEquals(bug20_baseline, bug20_result,
+ "Bug 20 pptn=${ppt}: result mismatch with serial exchange + agg hang")
+ }
+ logger.info("Bug 20: PASSED (no hang, correct results)")
+ } catch (Throwable t) {
+ logger.error("Bug 20 FAILED: ${t.message}")
+ assertTrue(false, "Bug 20: Serial exchange + agg hang: ${t.message}")
+ }
+
+ // ============================================================
+ // Bug 21: Multi-distinct COUNT on many-bucket table → COREDUMP
+ // RQG build 186737/186929/186952: AggSinkOperatorX::sink → set_ready_to_read
+ // with empty source_deps.
+ //
+ // Root cause: AGG operators (streaming, distinct-streaming, serialize) requested
+ // PASSTHROUGH from non-ScanNode serial children (Exchange, AGG), inserting a
+ // PASSTHROUGH LE that created a pipeline split disconnecting AggSink↔AggSource
+ // shared state.
+ //
+ // Fix: restrict AGG PASSTHROUGH requests to ScanNode children only.
+ // Triggered by: multi-distinct COUNT/MIN with MultiCastDataSinks feeding
+ // serial UNPARTITIONED Exchanges into streaming AGG fragments.
+ // ============================================================
+ sql "DROP TABLE IF EXISTS rqg_t5_many_buckets"
+ sql """
+ CREATE TABLE rqg_t5_many_buckets (
+ pk INT NOT NULL,
+ col_int_undef_signed INT,
+ col_date_undef_signed DATE,
+ col_date_undef_signed2 DATE,
+ col_varchar_1024__undef_signed VARCHAR(1024)
+ ) ENGINE=OLAP
+ DUPLICATE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 56
+ PROPERTIES ("replication_num" = "1")
+ """
+ sql """INSERT INTO rqg_t5_many_buckets VALUES
+ (1,1,'2023-12-09','2024-06-01','s1'),(2,2,'2023-03-15','2024-01-20','s2'),
+ (3,3,'2023-07-22','2024-03-10',NULL),(4,4,'2023-12-09','2024-06-01','s4'),
+ (5,5,'2023-01-05','2024-09-15','s5'),(6,6,'2023-08-11','2024-02-28','s6'),
+ (7,7,'2023-04-18','2024-07-04',NULL),(8,8,'2023-11-25','2024-05-12','s8'),
+ (9,9,'2023-06-30','2024-11-19','s9'),(10,10,'2023-02-14','2024-08-07','s10')
+ """
+
+ try {
+ logger.info("Bug 21: Testing multi-distinct COUNT on many-bucket table (COREDUMP fix)")
+ for (int ppt : [4, 6]) {
+ // Test without use_serial_exchange
+ def bug21_baseline = sql """
+ SELECT /*+SET_VAR(enable_local_shuffle_planner=false,
+ parallel_pipeline_task_num=${ppt},
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ query_timeout=60)*/
+ MIN(distinct col_date_undef_signed),
+ COUNT(distinct col_date_undef_signed2),
+ COUNT(distinct col_int_undef_signed)
+ FROM rqg_t5_many_buckets
+ WHERE col_int_undef_signed = col_int_undef_signed
+ LIMIT 1000
+ """
+ def bug21_result = sql """
+ SELECT /*+SET_VAR(enable_local_shuffle_planner=true,
+ parallel_pipeline_task_num=${ppt},
+ ignore_storage_data_distribution=true,
+ enable_sql_cache=false,
+ query_timeout=60)*/
+ MIN(distinct col_date_undef_signed),
+ COUNT(distinct col_date_undef_signed2),
+ COUNT(distinct col_int_undef_signed)
+ FROM rqg_t5_many_buckets
+ WHERE col_int_undef_signed = col_int_undef_signed
+ LIMIT 1000
+ """
+ assertEquals(bug21_baseline, bug21_result,
+ "Bug 21 pptn=${ppt}: multi-distinct COUNT result mismatch (was COREDUMP)")
+ }
+ logger.info("Bug 21: PASSED (no crash, correct results)")
+ } catch (Throwable t) {
+ if (t.message != null && t.message.contains("query timeout")) {
+ logger.warn("Bug 21: SKIPPED (query timeout - likely BE crash from previous test, not our bug)")
+ } else {
Review Comment:
This catch makes the regression pass when the query times out. Timeout/hang is one of the failure modes this case is meant to catch (the comment says this reproduced a COREDUMP/shared-state bug); if FE-planned local shuffle hangs here, CI logs `SKIPPED` and the suite succeeds. Other cases in this file fail from their `catch` blocks. Please let timeouts fail too, or skip before executing under an explicit environment predicate, so Bug 21 remains a real regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4680726307
# FE UT Coverage Report
Increment line coverage ` 74.48% (429/576)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_9f33ce47e1bdfae26100cf19b96a7eab06238d13/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_9f33ce47e1bdfae26100cf19b96a7eab06238d13/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4682234726
# BE Regression && UT Coverage Report
Increment line coverage `76.83% (315/410)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_784476ead971eeebeb0b637f6da4a434e380ce36_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_784476ead971eeebeb0b637f6da4a434e380ce36_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 63.86% (24432/38257) |
| Line Coverage | 47.66% (253665/532266) |
| Region Coverage | 44.41% (209410/471561) |
| Branch Coverage | 45.55% (90922/199624) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3449986962
##########
fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java:
##########
@@ -563,6 +567,10 @@ public void update(Map<String, String> summaryInfo) {
// This method is used to display the final data status when the overall query ends.
// This can avoid recalculating some strings and so on every time during the update process.
public void queryFinished() {
+ // Mark profile collection as complete. This is written last, after all BE fragment
+ // profiles have been merged (called post-waitForFragmentsDone). Pollers can wait for
+ // this field to avoid reading a partial profile.
Review Comment:
Update: after rebasing onto latest master I noticed this marker fully duplicated master's #64392 (`Profile Completion State`: RUNNING/COLLECTING/COMPLETE, wired in `Profile.java`, tested in `ProfileManagerTest`), and ours had no reader. So instead of moving it, I **removed `IS_PROFILE_COLLECTION_COMPLETED` entirely** (commit `7fdb989a3a7`) and made the profile-plan-tree test plugin poll master's `Profile Completion State: COMPLETE` instead of a fixed sleep. The ordering concern is moot now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4765707891
Codex automated review failed and did not complete.
Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jun 24th, 2026 10:56 PM.
Workflow run: https://github.com/apache/doris/actions/runs/27934785852
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4670550333
<details>
<summary>TPC-DS: <b>Total hot run time: 168769 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7c898c900506116cfb44b47189025b6fbfa7e44c, data reload: false
query5 4341 645 491 491
query6 437 199 184 184
query7 4825 533 305 305
query8 363 230 203 203
query9 8754 4043 4032 4032
query10 477 327 295 295
query11 5779 2316 2150 2150
query12 163 103 102 102
query13 1297 618 437 437
query14 6409 5407 5085 5085
query14_1 4396 4384 4424 4384
query15 209 198 190 190
query16 1016 459 446 446
query17 1150 730 608 608
query18 2489 507 378 378
query19 208 195 157 157
query20 116 108 110 108
query21 217 150 123 123
query22 13577 13564 13325 13325
query23 17449 16533 16087 16087
query23_1 16278 16241 16335 16241
query24 7489 1755 1329 1329
query24_1 1311 1324 1340 1324
query25 571 466 394 394
query26 1308 333 177 177
query27 2646 592 327 327
query28 4418 2070 2063 2063
query29 1069 609 478 478
query30 308 237 200 200
query31 1142 1071 961 961
query32 116 63 60 60
query33 527 318 251 251
query34 1185 1195 656 656
query35 766 772 686 686
query36 1393 1433 1318 1318
query37 153 102 93 93
query38 3211 3168 3031 3031
query39 917 913 902 902
query39_1 867 866 874 866
query40 229 124 104 104
query41 72 63 62 62
query42 94 103 93 93
query43 321 323 279 279
query44
query45 198 188 176 176
query46 1100 1169 718 718
query47 2384 2425 2274 2274
query48 415 424 297 297
query49 625 470 361 361
query50 1042 358 245 245
query51 4403 4426 4341 4341
query52 89 90 79 79
query53 243 270 187 187
query54 265 228 199 199
query55 79 76 82 76
query56 248 257 227 227
query57 1428 1389 1345 1345
query58 237 217 207 207
query59 1565 1656 1445 1445
query60 284 236 235 235
query61 167 159 158 158
query62 713 651 594 594
query63 232 190 185 185
query64 2570 823 629 629
query65
query66 1826 460 344 344
query67 29633 29725 28968 28968
query68
query69 415 301 271 271
query70 919 988 900 900
query71 303 227 205 205
query72 2983 2750 2406 2406
query73 840 778 439 439
query74 5125 4933 4760 4760
query75 2665 2551 2268 2268
query76 2325 1129 786 786
query77 354 392 287 287
query78 12507 12409 11876 11876
query79 1293 1008 779 779
query80 601 474 397 397
query81 454 284 240 240
query82 569 155 122 122
query83 356 273 250 250
query84
query85 911 532 445 445
query86 377 290 270 270
query87 3386 3381 3218 3218
query88 3645 2779 2754 2754
query89 420 382 332 332
query90 1999 177 182 177
query91 179 161 137 137
query92 66 62 59 59
query93 1505 1591 907 907
query94 536 358 306 306
query95 702 389 356 356
query96 1066 787 329 329
query97 2695 2704 2551 2551
query98 208 206 209 206
query99 1166 1170 1038 1038
Total cold run time: 250844 ms
Total hot run time: 168769 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4686120768
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4686649179
<details>
<summary>TPC-DS: <b>Total hot run time: 167953 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9ccb7f30b5fb41183c99a77cc3de3311715aa9b6, data reload: false
query5 4348 619 489 489
query6 440 181 168 168
query7 4962 586 302 302
query8 359 218 202 202
query9 8772 4006 4004 4004
query10 439 298 252 252
query11 5862 2352 2163 2163
query12 157 103 96 96
query13 1273 623 459 459
query14 6392 5410 5067 5067
query14_1 4383 4392 4398 4392
query15 206 202 172 172
query16 959 470 438 438
query17 1115 705 595 595
query18 2447 477 348 348
query19 205 183 150 150
query20 111 107 105 105
query21 216 143 117 117
query22 13618 13576 13402 13402
query23 17516 16556 16200 16200
query23_1 16334 16275 16192 16192
query24 7576 1790 1294 1294
query24_1 1334 1317 1330 1317
query25 558 452 389 389
query26 1337 347 167 167
query27 2625 520 351 351
query28 4449 2030 2028 2028
query29 1140 629 485 485
query30 307 225 202 202
query31 1135 1091 953 953
query32 112 63 59 59
query33 535 325 253 253
query34 1182 1142 641 641
query35 746 776 711 711
query36 1365 1421 1206 1206
query37 155 103 89 89
query38 3214 3159 3055 3055
query39 912 916 931 916
query39_1 890 899 864 864
query40 217 119 98 98
query41 64 61 59 59
query42 92 97 93 93
query43 318 317 272 272
query44
query45 192 185 178 178
query46 1125 1232 723 723
query47 2380 2393 2267 2267
query48 415 426 304 304
query49 632 450 338 338
query50 930 347 263 263
query51 4397 4396 4315 4315
query52 89 102 77 77
query53 242 270 191 191
query54 256 213 187 187
query55 79 75 69 69
query56 227 224 202 202
query57 1437 1412 1337 1337
query58 223 214 211 211
query59 1581 1636 1414 1414
query60 291 245 222 222
query61 158 149 143 143
query62 691 643 579 579
query63 244 182 203 182
query64 2527 744 589 589
query65
query66 1793 459 338 338
query67 29713 29751 28895 28895
query68
query69 400 330 253 253
query70 930 963 942 942
query71 292 213 204 204
query72 2937 2573 2354 2354
query73 892 754 453 453
query74 5135 4988 4788 4788
query75 2638 2569 2218 2218
query76 2357 1123 752 752
query77 350 375 285 285
query78 12512 12528 11917 11917
query79 1441 1086 734 734
query80 1240 471 395 395
query81 524 282 244 244
query82 588 152 119 119
query83 323 269 246 246
query84
query85 892 502 415 415
query86 425 306 297 297
query87 3395 3382 3208 3208
query88 3583 2717 2694 2694
query89 415 383 335 335
query90 1906 175 173 173
query91 173 157 133 133
query92 64 58 50 50
query93 1489 1584 943 943
query94 710 341 289 289
query95 674 377 338 338
query96 1074 782 364 364
query97 2694 2716 2568 2568
query98 211 214 203 203
query99 1136 1163 1035 1035
Total cold run time: 251624 ms
Total hot run time: 167953 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4765420988
Codex automated review failed and did not complete.
Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jun 24th, 2026 10:56 PM.
Workflow run: https://github.com/apache/doris/actions/runs/27933095561
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4767805679
<details>
<summary>TPC-DS: <b>Total hot run time: 173280 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b, data reload: false
query5 4347 625 485 485
query6 436 185 171 171
query7 4838 594 316 316
query8 358 222 216 216
query9 8753 4149 4143 4143
query10 436 307 251 251
query11 5923 2352 2114 2114
query12 152 103 108 103
query13 1246 641 473 473
query14 6397 5437 5107 5107
query14_1 4441 4438 4471 4438
query15 207 199 177 177
query16 1033 468 439 439
query17 1145 719 583 583
query18 2721 489 364 364
query19 199 175 136 136
query20 107 105 102 102
query21 218 139 118 118
query22 13683 13645 13416 13416
query23 17344 16486 16229 16229
query23_1 16282 16203 16402 16203
query24 7546 1779 1315 1315
query24_1 1341 1315 1314 1314
query25 529 427 362 362
query26 1302 300 176 176
query27 2634 551 341 341
query28 4417 2050 2047 2047
query29 1047 603 473 473
query30 309 238 201 201
query31 1105 1063 969 969
query32 100 57 59 57
query33 533 314 253 253
query34 1164 1138 641 641
query35 758 792 666 666
query36 1370 1395 1241 1241
query37 154 102 89 89
query38 1892 1734 1657 1657
query39 924 905 903 903
query39_1 879 887 868 868
query40 216 129 103 103
query41 64 63 63 63
query42 87 88 85 85
query43 334 341 287 287
query44 1437 784 792 784
query45 194 183 179 179
query46 1114 1226 785 785
query47 2397 2340 2266 2266
query48 406 418 295 295
query49 617 463 375 375
query50 1023 352 258 258
query51 4406 4408 4289 4289
query52 81 85 72 72
query53 249 272 183 183
query54 270 237 217 217
query55 75 73 67 67
query56 249 232 222 222
query57 1425 1413 1305 1305
query58 245 220 218 218
query59 1636 1662 1458 1458
query60 298 264 250 250
query61 170 176 176 176
query62 696 652 591 591
query63 239 201 202 201
query64 2540 811 652 652
query65 4891 4763 4782 4763
query66 1735 501 355 355
query67 29839 29722 29680 29680
query68 3051 1538 978 978
query69 406 301 260 260
query70 1070 981 973 973
query71 293 237 213 213
query72 2878 2676 2362 2362
query73 889 756 418 418
query74 5107 4957 4759 4759
query75 2609 2592 2213 2213
query76 2313 1220 800 800
query77 358 381 282 282
query78 12497 12516 11872 11872
query79 1420 1237 724 724
query80 1276 465 383 383
query81 540 279 241 241
query82 630 155 119 119
query83 327 276 248 248
query84 259 146 117 117
query85 889 494 417 417
query86 442 290 277 277
query87 1833 1837 1799 1799
query88 3717 2794 2785 2785
query89 424 391 337 337
query90 1914 182 184 182
query91 169 161 132 132
query92 61 64 58 58
query93 1639 1554 930 930
query94 726 334 323 323
query95 688 381 440 381
query96 1045 783 361 361
query97 2714 2680 2575 2575
query98 224 208 205 205
query99 1176 1153 1023 1023
Total cold run time: 259154 ms
Total hot run time: 173280 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 merged PR #63366:
URL: https://github.com/apache/doris/pull/63366
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4552040940
# FE Regression Coverage Report
Increment line coverage ` 66.67% (476/714)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_ccfac05f6b543ebe78f973987084a8de173759c9_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_ccfac05f6b543ebe78f973987084a8de173759c9_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4490040910
<details>
<summary>TPC-H: <b>Total hot run time: 31839 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4e05e7bfbb8aace37eb65198a1530e9325047d34, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17008 4093 4055 4055
q2 q3 10494 1424 891 891
q4 4685 481 365 365
q5 7816 2357 2142 2142
q6 267 191 146 146
q7 1026 794 642 642
q8 9380 1750 1655 1655
q9 5537 5094 5054 5054
q10 6382 2144 1768 1768
q11 429 268 244 244
q12 638 431 315 315
q13 18180 3449 2784 2784
q14 268 257 237 237
q15 q16 825 813 716 716
q17 1009 982 1006 982
q18 7125 5794 5761 5761
q19 1179 1232 1085 1085
q20 519 422 274 274
q21 5609 2650 2411 2411
q22 446 375 312 312
Total cold run time: 98822 ms
Total hot run time: 31839 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4361 4345 4290 4290
q2 q3 4528 4949 4331 4331
q4 2184 2261 1419 1419
q5 4568 4406 4368 4368
q6 311 331 177 177
q7 2301 1961 1738 1738
q8 2704 2361 2325 2325
q9 8320 7948 8070 7948
q10 4572 4454 4060 4060
q11 610 441 418 418
q12 751 745 553 553
q13 3315 3705 2988 2988
q14 292 313 297 297
q15 q16 753 755 686 686
q17 1422 1440 1554 1440
q18 8205 7423 7590 7423
q19 1195 1153 1091 1091
q20 2232 2272 1928 1928
q21 5465 4899 4652 4652
q22 549 483 401 401
Total cold run time: 58638 ms
Total hot run time: 52533 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4508592992
<details>
<summary>TPC-H: <b>Total hot run time: 31343 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 846d365a88901fa32fdf44b42b6cd04b900da259, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17644 3831 3835 3831
q2 q3 10789 1395 822 822
q4 4742 483 353 353
q5 9240 2255 2088 2088
q6 379 175 137 137
q7 975 758 633 633
q8 9721 1667 1761 1667
q9 7096 4986 5003 4986
q10 6503 2120 1824 1824
q11 431 279 239 239
q12 689 429 296 296
q13 18201 3411 2824 2824
q14 265 259 243 243
q15 q16 814 770 704 704
q17 1034 894 886 886
q18 6924 5808 5608 5608
q19 1235 1263 1135 1135
q20 536 405 262 262
q21 5735 2582 2510 2510
q22 425 354 295 295
Total cold run time: 103378 ms
Total hot run time: 31343 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4190 4102 4346 4102
q2 q3 4570 4881 4317 4317
q4 2120 2220 1410 1410
q5 4399 4304 5149 4304
q6 260 199 141 141
q7 2066 1856 1614 1614
q8 2489 2206 2175 2175
q9 8079 7922 7854 7854
q10 4549 4534 4097 4097
q11 578 411 377 377
q12 740 740 520 520
q13 3291 3544 2951 2951
q14 303 296 261 261
q15 q16 710 725 628 628
q17 1385 1484 1318 1318
q18 7821 7326 7378 7326
q19 1152 1127 1056 1056
q20 2235 2202 1932 1932
q21 5354 4664 4478 4478
q22 521 486 414 414
Total cold run time: 56812 ms
Total hot run time: 51275 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4488243884
<details>
<summary>TPC-H: <b>Total hot run time: 31328 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9016e5a105f84231f1a535be233aaf3e595c711d, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17620 4009 3866 3866
q2 q3 10989 1467 845 845
q4 4810 476 354 354
q5 10550 2299 2104 2104
q6 382 181 139 139
q7 1020 761 642 642
q8 9603 1863 1716 1716
q9 6948 5174 4937 4937
q10 6492 2085 1794 1794
q11 428 269 245 245
q12 647 422 295 295
q13 18188 3370 2776 2776
q14 268 259 238 238
q15 q16 823 778 708 708
q17 931 931 900 900
q18 6829 5674 5604 5604
q19 1154 1225 1129 1129
q20 504 412 262 262
q21 5488 2532 2467 2467
q22 434 374 307 307
Total cold run time: 104108 ms
Total hot run time: 31328 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4246 4221 4168 4168
q2 q3 4537 4942 4338 4338
q4 2110 2242 1392 1392
q5 4393 4312 5249 4312
q6 256 194 144 144
q7 1913 1816 1568 1568
q8 2442 2154 2075 2075
q9 7958 7888 7894 7888
q10 4565 4579 4065 4065
q11 565 428 374 374
q12 738 735 518 518
q13 3221 3668 2992 2992
q14 290 298 281 281
q15 q16 722 771 666 666
q17 1355 1339 1339 1339
q18 7802 7376 6840 6840
q19 1155 1076 1078 1076
q20 2217 2227 1956 1956
q21 5329 4679 4490 4490
q22 533 449 416 416
Total cold run time: 56347 ms
Total hot run time: 50898 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3414840309
##########
fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java:
##########
@@ -939,4 +1031,263 @@ private String mergeIcebergAccessPathsWithId(
}
return StringUtils.join(mergeDisplayAccessPaths, ", ");
}
+
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ ArrayList<PlanNode> newChildren = Lists.newArrayList();
+ for (int i = 0; i < children.size(); i++) {
+ Pair<PlanNode, LocalExchangeType> childOutput
+ = enforceRequire(translatorContext, children.get(i), i, LocalExchangeTypeRequire.noRequire());
+ newChildren.add(childOutput.first);
+ }
+ this.children = newChildren;
+ return Pair.of(this, LocalExchangeType.NOOP);
+ }
+
+ /**
+ * Unified framework method: propagate serial flag → recurse child → satisfy check → Layer 1 skip → insert LE.
+ * Replaces the old enforceChild/enforceChildExchange/forceEnforceChildExchange trio.
+ *
+ * <h3>Data flow</h3>
+ * <ul>
+ * <li><b>serial-ancestor flag</b> ({@link PlanTranslatorContext#hasSerialAncestorInPipeline})
+ * — flows root → leaf during traversal. Mirrors BE's
+ * {@code any_of(operators[idx..end], is_serial_operator)} check used by
+ * {@code _add_local_exchange} to skip LE insertion when an ancestor in the same
+ * pipeline is already serial. Reset at pipeline boundaries via
+ * {@link #shouldResetSerialFlagForChild}.</li>
+ * <li><b>shuffle-for-correctness flag</b>
+ * ({@link PlanTranslatorContext#hasShuffleForCorrectnessAncestor}) — also flows
+ * root → leaf. Mirrors BE's {@code _followed_by_shuffled_operator}: tells a
+ * child whether some downstream operator depends on hash distribution for
+ * correctness, so {@code SetOperationNode} can pre-shuffle union branches.</li>
+ * <li><b>return value</b> {@code Pair<PlanNode, LocalExchangeType>} — first is the
+ * (possibly LE-wrapped) child; second is the actual output distribution as
+ * observed by the parent. Caller's {@code require.satisfy(output)} decides
+ * whether more LE is needed.</li>
+ * <li><b>parent.require</b> describes the constraint on the child output —
+ * computed inside the parent's {@code enforceAndDeriveLocalExchange} per child.</li>
+ * </ul>
+ *
+ * <h3>Invariants</h3>
+ * <ul>
+ * <li>Where a serial → non-serial transition needs redistribution, framework step 3 inserts
+ * the LE (e.g. a serial source fanned out via PASSTHROUGH). This is not a hard invariant:
+ * a serial child feeding a parent that requires PASSTHROUGH / noRequire (TableFunction,
+ * NLJ, Agg) is already correct and needs no LE, so it is intentionally not validated by a
+ * post-pass.</li>
+ * <li>{@code LocalExchangeNode} itself is always non-serial — setting it serial
+ * would defeat its purpose of fanning a 1-task pipeline back to N tasks.</li>
+ * <li>For pipeline-breaking parents ({@code shouldResetSerialFlagForChild=true}),
+ * the child starts a fresh pipeline so {@code hasSerialAncestor} is reset; the
+ * node's own {@code isSerialNode()} still composes in for the child's view.</li>
+ * <li>{@code RequireHash} accepts any hash flavour; {@code RequireSpecific} demands
+ * an exact match (with the one PASSTHROUGH/ADAPTIVE_PASSTHROUGH compatibility).
+ * Pick the looser one whenever correctness allows — see
+ * {@link LocalExchangeNode.LocalExchangeTypeRequire}.</li>
+ * </ul>
+ *
+ * <h3>Layers</h3>
+ * Layer 1 (shouldSkipLE): mirrors BE's need_to_local_exchange — skip when this node or
+ * an ancestor in the same pipeline is serial (operators[idx..end] has serial → skip).
+ * Layer 2 (require/output): each Node declares require and output in enforceAndDeriveLocalExchange.
+ */
+ protected Pair<PlanNode, LocalExchangeType> enforceRequire(
+ PlanTranslatorContext translatorContext, PlanNode child, int childIndex,
+ LocalExchangeTypeRequire require) {
+ // 1. Propagate serial-ancestor flag to child.
+ // For pipeline-splitting operators (shouldReset=true, e.g. non-streaming AGG):
+ // Drop inherited serial flag from parent (parent is in a different pipeline),
+ // but keep this node's own serial status (child is in the same pipeline as this
+ // node's sink, e.g. Exchange is in AGG_Sink pipeline).
+ // For non-splitting operators (shouldReset=false, e.g. streaming AGG):
+ // Inherit parent's serial flag + this node's own.
+ boolean inheritedSerial = shouldResetSerialFlagForChild(childIndex)
+ ? false : translatorContext.hasSerialAncestorInPipeline(this);
+ // Use isSerialOperatorOnBe (= isSerialNode && fragment.useSerialSource) instead of the
+ // raw isSerialNode(). BE's OperatorBase reads the Thrift `is_serial_operator` flag —
+ // which is what FE writes via isSerialOperatorOnBe — so when the fragment is not in
+ // serial-source mode, BE treats this operator as non-serial regardless of isSerialNode.
+ // Using isSerialNode here would set the child's serial-ancestor flag wider than BE's
+ // view and over-skip required LocalExchanges downstream.
+ boolean childHasSerialAncestor = inheritedSerial
+ || isSerialOperatorOnBe(translatorContext.getConnectContext());
+ translatorContext.setHasSerialAncestorInPipeline(child, childHasSerialAncestor);
+
+ // 1b. Propagate shuffle-for-correctness-ancestor flag to child.
+ // Mirrors BE's _followed_by_shuffled_operator: a downstream operator needs hash
+ // distribution for correctness, and the chain to here goes through HASH or NOOP
+ // requirements (so the dependency is preserved).
+ // propagate = ((inheritedShuffled || self.requiresShuffleForCorrectness)
+ // && require is hash)
+ // || (inheritedShuffled && require is noop/passthrough)
+ boolean inheritedShuffled = translatorContext.hasShuffleForCorrectnessAncestor(this);
+ boolean selfOrInheritedShuffled = inheritedShuffled || requiresShuffleForCorrectness();
+ boolean requireIsHash = require.preferType().isHashShuffle();
+ boolean requireIsNoop = require.preferType() == LocalExchangeNode.LocalExchangeType.NOOP;
+ boolean childShuffledAncestor = (selfOrInheritedShuffled && requireIsHash)
+ || (inheritedShuffled && requireIsNoop);
+ translatorContext.setHasShuffleForCorrectnessAncestor(child, childShuffledAncestor);
+
+ // 2. Recurse child (Layer 2: child declares its own require/output)
+ Pair<PlanNode, LocalExchangeType> childOutput =
+ child.enforceAndDeriveLocalExchange(translatorContext, this, require);
+
+ // Steps 2.5 and 3 both react to a serial child but address different concerns:
+ // - Step 2.5 rewrites the OUTPUT-side view (what we tell satisfy/parent about
+ // the child's actual distribution). A serial pipeline runs with 1 task so
+ // its distribution claim is meaningless — flatten to NOOP so the satisfy
+ // check below doesn't get fooled by a stale "I output BUCKET_HASH" claim.
+ // - Step 3 rewrites the REQUIRE-side decision (what we want from the child).
+ // If we previously asked for nothing (noRequire) but the child turns out
+ // to be serial and we're not, upgrade to requirePassthrough so an LE is
+ // inserted to restore parallelism.
+
+ // 2.5. Serial child override (output side): if child is serial on BE, force its
+ // reported output to NOOP. Distribution is irrelevant when the child runs
+ // with 1 task; downstream parallelism is restored either by step 3 (LE
+ // insertion) or skipped entirely by step 4b (we're also serial).
+ if (childOutput.first.isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ childOutput = Pair.of(childOutput.first, LocalExchangeType.NOOP);
+ }
+
+ // 3. Framework-level serial child check (require side, mirrors BE base class
+ // required_data_distribution): if child will be serial on BE but this node is
+ // not serial, the pipeline has a 1-task serial child feeding an N-task non-serial
+ // parent. Without LE, pipeline splits (AGG/JOIN) create paired pipelines with
+ // mismatched num_tasks → crash. Upgrade noRequire to requirePassthrough so an
+ // LE is inserted below to restore parallelism.
+ if (require instanceof LocalExchangeNode.NoRequire
+ && childOutput.first.isSerialOperatorOnBe(translatorContext.getConnectContext())
+ && !isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ require = LocalExchangeTypeRequire.requirePassthrough();
+ }
+
+ // 4. Satisfy check: child output meets requirement → done
+ if (require.satisfy(childOutput.second)) {
+ return childOutput;
+ }
+
+ // 4. Layer 1: skip LE when serial operator or ancestor in same pipeline
+ // Equivalent to BE's need_to_local_exchange: any_of(operators[idx..end], is_serial) → skip.
+ // Use isSerialOperatorOnBe (not isSerialNode) because BE's Pipeline::need_to_local_exchange
+ // checks op->is_serial_operator() which reads the Thrift flag set from isSerialOperatorOnBe;
+ // when fragment.useSerialSource is false, BE treats this node as non-serial.
+ if (translatorContext.hasSerialAncestorInPipeline(this)
Review Comment:
why can add no local exchange node when `hasSerialAncestorInPipeline(this)` equals true?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4701708703
<details>
<summary>TPC-H: <b>Total hot run time: 28718 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8db568b5100ed397fc6347033402642e90e5509e, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17706 3971 4026 3971
q2 q3 10887 1446 802 802
q4 4822 490 340 340
q5 8546 870 581 581
q6 349 170 134 134
q7 947 845 637 637
q8 10865 1468 1535 1468
q9 7051 4572 4507 4507
q10 6848 1815 1539 1539
q11 434 273 248 248
q12 644 431 292 292
q13 18112 3441 2741 2741
q14 267 258 234 234
q15 q16 823 767 707 707
q17 893 830 957 830
q18 7289 5868 5602 5602
q19 1169 1370 1081 1081
q20 511 403 261 261
q21 5603 2699 2440 2440
q22 431 363 303 303
Total cold run time: 104197 ms
Total hot run time: 28718 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4327 4269 4263 4263
q2 q3 4591 4987 4353 4353
q4 2074 2199 1380 1380
q5 4479 4321 4363 4321
q6 228 171 126 126
q7 2340 1904 1636 1636
q8 2462 2203 2148 2148
q9 8165 8102 8210 8102
q10 4792 4759 4563 4563
q11 582 424 385 385
q12 763 750 530 530
q13 3236 3729 2948 2948
q14 318 323 283 283
q15 q16 719 734 644 644
q17 1353 1353 1332 1332
q18 7985 7200 6958 6958
q19 1130 1086 1131 1086
q20 2252 2234 2011 2011
q21 5338 4588 4455 4455
q22 529 471 415 415
Total cold run time: 57663 ms
Total hot run time: 51939 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3413561854
##########
be/src/exec/pipeline/dependency.h:
##########
@@ -832,50 +832,44 @@ struct SetSharedState : public BasicSharedState {
Status hash_table_init();
};
-enum class ExchangeType : uint8_t {
- NOOP = 0,
- // Shuffle data by Crc32CHashPartitioner
- HASH_SHUFFLE = 1,
- // Round-robin passthrough data blocks.
- PASSTHROUGH = 2,
- // Shuffle data by Crc32HashPartitioner<ShuffleChannelIds> (e.g. same as storage engine).
- BUCKET_HASH_SHUFFLE = 3,
- // Passthrough data blocks to all channels.
- BROADCAST = 4,
- // Passthrough data to channels evenly in an adaptive way.
- ADAPTIVE_PASSTHROUGH = 5,
- // Send all data to the first channel.
- PASS_TO_ONE = 6,
-};
+inline bool is_shuffled_exchange(TLocalPartitionType::type idx) {
+ return idx == TLocalPartitionType::GLOBAL_EXECUTION_HASH_SHUFFLE ||
+ idx == TLocalPartitionType::LOCAL_EXECUTION_HASH_SHUFFLE ||
+ idx == TLocalPartitionType::BUCKET_HASH_SHUFFLE;
+}
Review Comment:
You're right, and my earlier reply was wrong — `OperatorBase::is_hash_shuffle` (`operator.cpp:150`) is the exact same predicate. I only grepped the name `is_shuffled_exchange` and missed it; both were added in this PR. Deduped in `832ba5f0bf8`: removed `is_hash_shuffle` and routed its single caller (`child_breaks_local_key_distribution`) to the free `is_shuffled_exchange` (operator.cpp already includes dependency.h). Thanks for catching it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4708905260
<details>
<summary>TPC-DS: <b>Total hot run time: 167932 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 832ba5f0bf8ab5846bb316e94f1099ae58fe466f, data reload: false
query5 4317 637 480 480
query6 465 189 166 166
query7 4835 535 300 300
query8 386 225 199 199
query9 8757 4047 4051 4047
query10 460 312 259 259
query11 5882 2365 2139 2139
query12 153 100 124 100
query13 1235 587 390 390
query14 6412 5348 4992 4992
query14_1 4358 4357 4307 4307
query15 207 199 174 174
query16 986 446 443 443
query17 956 681 556 556
query18 2426 474 333 333
query19 199 178 136 136
query20 112 107 104 104
query21 220 144 117 117
query22 13585 13585 13318 13318
query23 17361 16549 16126 16126
query23_1 16346 16289 16267 16267
query24 7555 1834 1319 1319
query24_1 1336 1342 1333 1333
query25 572 459 411 411
query26 1303 327 171 171
query27 2756 572 328 328
query28 4517 2032 2065 2032
query29 1152 661 500 500
query30 327 235 202 202
query31 1118 1077 970 970
query32 113 62 62 62
query33 546 327 285 285
query34 1205 1170 638 638
query35 757 780 696 696
query36 1394 1400 1207 1207
query37 160 110 91 91
query38 3193 3141 3090 3090
query39 941 922 910 910
query39_1 878 892 876 876
query40 222 130 104 104
query41 68 67 67 67
query42 100 98 94 94
query43 321 325 285 285
query44
query45 203 191 181 181
query46 1089 1231 776 776
query47 2409 2407 2320 2320
query48 415 413 308 308
query49 652 486 363 363
query50 1018 369 262 262
query51 4655 4360 4344 4344
query52 95 92 78 78
query53 248 282 191 191
query54 286 232 225 225
query55 80 79 77 77
query56 245 243 245 243
query57 1424 1413 1301 1301
query58 266 229 212 212
query59 1598 1645 1442 1442
query60 351 242 232 232
query61 153 142 160 142
query62 710 658 598 598
query63 233 189 190 189
query64 2606 763 599 599
query65
query66 1811 450 347 347
query67 29701 29638 28939 28939
query68
query69 433 312 268 268
query70 1039 962 900 900
query71 296 230 209 209
query72 2911 2613 2339 2339
query73 852 746 442 442
query74 5123 4929 4784 4784
query75 2661 2565 2213 2213
query76 2312 1162 792 792
query77 368 367 290 290
query78 12372 12600 11743 11743
query79 1516 1068 810 810
query80 616 471 391 391
query81 467 284 249 249
query82 635 160 120 120
query83 342 291 253 253
query84
query85 924 545 442 442
query86 383 292 289 289
query87 3423 3361 3187 3187
query88 3729 2773 2733 2733
query89 438 388 329 329
query90 1965 183 179 179
query91 170 158 133 133
query92 63 62 57 57
query93 1495 1437 892 892
query94 617 338 293 293
query95 687 473 334 334
query96 1072 774 329 329
query97 2696 2719 2574 2574
query98 219 215 205 205
query99 1136 1166 1007 1007
Total cold run time: 251786 ms
Total hot run time: 167932 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4701410489
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4708537624
# FE UT Coverage Report
Increment line coverage ` 74.96% (428/571)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_fe9a3316be056faed18351cf64096e36790c6c89/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_fe9a3316be056faed18351cf64096e36790c6c89/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4706824885
# FE UT Coverage Report
Increment line coverage ` 74.65% (430/576)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_1d7cf6dbd6095ee1fb5860efe258b0ce99242ae6/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_1d7cf6dbd6095ee1fb5860efe258b0ce99242ae6/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4738331130
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/0436cc4cb573b104a59408a6169bd145bcfeff6c_0436cc4cb573b104a59408a6169bd145bcfeff6c_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/0436cc4cb573b104a59408a6169bd145bcfeff6c_0436cc4cb573b104a59408a6169bd145bcfeff6c_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.32% (1889/2443) |
| Line Coverage | 64.43% (33970/52725) |
| Region Coverage | 64.80% (17461/26948) |
| Branch Coverage | 53.98% (9347/17316) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4739601944
# BE Regression && UT Coverage Report
Increment line coverage `76.60% (311/406)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_0436cc4cb573b104a59408a6169bd145bcfeff6c_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_0436cc4cb573b104a59408a6169bd145bcfeff6c_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 64.09% (24560/38322) |
| Line Coverage | 47.88% (255578/533815) |
| Region Coverage | 44.59% (210879/472979) |
| Branch Coverage | 45.63% (91504/200514) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4737259524
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4731455457
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3372721760
##########
fe/fe-core/src/main/java/org/apache/doris/planner/SetOperationNode.java:
##########
@@ -193,4 +198,52 @@ public int getNumInstances() {
public boolean isBucketShuffle() {
return distributionMode.equals(DistributionMode.BUCKET_SHUFFLE);
}
+
+ public boolean isColocate() {
+ return isColocate;
+ }
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(PlanTranslatorContext translatorContext,
+ PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ LocalExchangeTypeRequire requireChild;
+ LocalExchangeType outputType;
+ PlanNode firstChild = children.isEmpty() ? null : children.get(0);
+ if (this instanceof UnionNode) {
+ // Propagate parent's hash requirement to children ONLY when a downstream operator
+ // requires shuffle for correctness (not just performance optimization). Matches BE's
Review Comment:
**🟡 Minor**: When `children.isEmpty()` (constant-output union), `firstChild` is `null`. It's then passed to `resolveExchangeType()` — safe today because `child` is unused, but fragile if that method ever starts using the parameter. Consider adding a guard or a comment noting that `resolveExchangeType` does not dereference `child`.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/AddLocalExchange.java:
##########
@@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.planner;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.glue.translator.PlanTranslatorContext;
+import org.apache.doris.nereids.trees.plans.distribute.DistributedPlan;
+import org.apache.doris.nereids.trees.plans.distribute.FragmentIdMapping;
+import org.apache.doris.nereids.trees.plans.distribute.PipelineDistributedPlan;
+import org.apache.doris.planner.LocalExchangeNode.LocalExchangeType;
+import org.apache.doris.planner.LocalExchangeNode.LocalExchangeTypeRequire;
+import org.apache.doris.planner.LocalExchangeNode.RequireHash;
+
+/**
+ * FE-side local exchange planner — inserts {@link LocalExchangeNode} into each fragment's
+ * plan tree so that within-fragment data redistribution is decided at planning time
+ * instead of at BE pipeline-build time.
+ *
+ * <h3>When this runs</h3>
+ * Invoked from {@code NereidsPlanner.addLocalExchangeAfterDistribute()} right after
+ * {@code DistributePlanner} has assigned instances to fragments and before the plan is
+ * serialized to BE. Gated by session variable {@code enable_local_shuffle_planner}
+ * (default true) and {@code enable_local_shuffle}; when either is off this pass is
+ * skipped entirely and BE falls back to its own {@code _plan_local_exchange}. The two
+ * paths are mutually exclusive: BE consults {@code runtime_state.h::plan_local_shuffle()}
+ * to know whether it should plan LE itself.
+ *
+ * <h3>What it changes</h3>
+ * <ul>
+ * <li>For each fragment with {@code maxPerBeInstances > 1}, walks the plan tree
+ * bottom-up via {@link PlanNode#enforceAndDeriveLocalExchange} and inserts
+ * LocalExchangeNodes where children's output distribution doesn't satisfy the
+ * parent's requirement.</li>
+ * <li>May wrap the fragment root with an extra PASSTHROUGH LE so the data sink
+ * (DataStreamSink / OlapTableSink) runs with the full instance count even when
+ * the root operator is serial — see {@link #addLocalExchangeForFragment}.</li>
+ * <li>Does NOT modify the fragment sink itself, fragment boundaries, or instance
+ * assignment.</li>
+ * </ul>
+ *
+ * <h3>Per-BE instance semantics</h3>
+ * Skips fragments where every BE has at most 1 instance. Using a global instance count
+ * would insert LE for "2 BEs × 1 instance" cases, which BE's own
+ * {@code _plan_local_exchange} would not — leading to pipeline task-count mismatch and
+ * deadlock. See {@link #addLocalExchange}.
+ *
+ * <h3>Reading order</h3>
+ * Start with {@link PlanNode#enforceRequire} (the recursion engine), then individual
+ * {@code enforceAndDeriveLocalExchange} overrides on PlanNode subclasses.
+ */
+public class AddLocalExchange {
+ /** addLocalExchange with distributed plans, skipping single-instance fragments.
+ * BE's _plan_local_exchange checks _num_instances which is the per-BE instance count.
+ * With _num_instances<=1 all pipelines on that BE have 1 task so local exchange is a no-op.
+ * We must use the same per-BE semantics: skip when every BE has at most 1 instance.
+ * Using global instanceCount would insert LE for fragments where 2 BEs each have 1 instance
+ * (global=2, per-BE=1), causing pipeline task mismatch and deadlock. */
+ public void addLocalExchange(FragmentIdMapping<DistributedPlan> distributedPlans,
+ PlanTranslatorContext context) {
+ for (DistributedPlan plan : distributedPlans.values()) {
+ PipelineDistributedPlan pipePlan = (PipelineDistributedPlan) plan;
+ long maxPerBeInstances = pipePlan.getInstanceJobs().stream()
+ .collect(java.util.stream.Collectors.groupingBy(
+ j -> j.getAssignedWorker().id(), java.util.stream.Collectors.counting()))
+ .values().stream().mapToLong(Long::longValue).max().orElse(0);
+ if (maxPerBeInstances <= 1) {
+ continue;
+ }
+ PlanFragment fragment = pipePlan.getFragmentJob().getFragment();
+ addLocalExchangeForFragment(fragment, context);
+ }
+ }
+
+ private void addLocalExchangeForFragment(PlanFragment fragment, PlanTranslatorContext context) {
+ DataSink sink = fragment.getSink();
+ LocalExchangeTypeRequire require = sink == null
+ ? LocalExchangeTypeRequire.noRequire() : sink.getLocalExchangeTypeRequire();
+ PlanNode root = fragment.getPlanRoot();
+ context.setHasSerialAncestorInPipeline(root, false);
+ Pair<PlanNode, LocalExchangeType> output = root
+ .enforceAndDeriveLocalExchange(context, null, require);
+ PlanNode newRoot = output.first;
+ // The fragment data sink (DataStreamSink, OlapTableSink) runs in the same pipeline
+ // as the root. If the root will be serial on BE, the sink pipeline has 1 task —
+ // only instance 0 sends data, others hang or miss writes.
+ // Insert PASSTHROUGH fan-out so sink runs with _num_instances tasks.
+ // This matches BE-native's default required_data_distribution():
+ // _child->is_serial_operator() ? PASSTHROUGH : NOOP
+ if (newRoot.isSerialOperatorOnBe(context.getConnectContext())) {
+ newRoot = new LocalExchangeNode(context.nextPlanNodeId(), newRoot,
+ LocalExchangeType.PASSTHROUGH, null);
+ }
+ if (newRoot != root) {
+ fragment.setPlanRoot(newRoot);
+ }
+ validateNoSerialWithoutLocalExchange(fragment.getPlanRoot(), context.getConnectContext());
+ }
+
+ /**
+ * In a local-shuffle fragment, the root check above guarantees the root pipeline
+ * has N tasks. Any serial operator reduces its pipeline to 1 task. If this serial
+ * operator feeds into a non-serial parent without LocalExchangeNode in between,
+ * some pipelines have 1 task while others have N → shared_state mismatch, data loss.
+ *
+ * Serial→serial chains are fine (all at 1 task, consistent). Only the transition
+ * from serial to non-serial needs LE to restore parallelism.
+ */
+ private void validateNoSerialWithoutLocalExchange(PlanNode node,
+ org.apache.doris.qe.ConnectContext context) {
+ for (PlanNode child : node.getChildren()) {
+ validateNoSerialWithoutLocalExchange(child, context);
+ if (child.isSerialOperatorOnBe(context)
+ && !(child instanceof LocalExchangeNode)
+ && !(node instanceof LocalExchangeNode)
+ && !(node instanceof ExchangeNode)
+ && !node.isSerialOperatorOnBe(context)) {
+ org.apache.logging.log4j.LogManager.getLogger(AddLocalExchange.class).warn(
+ "Serial " + child.getClass().getSimpleName() + "(id=" + child.getId()
+ + ") feeds into non-serial " + node.getClass().getSimpleName()
+ + "(id=" + node.getId() + ") without LocalExchangeNode"
+ + " in fragment " + node.getFragment().getFragmentId()
+ + ". FE should insert LocalExchangeNode to restore parallelism.");
+ }
+ }
+ }
+
+ public static boolean isColocated(PlanNode plan) {
+ if (plan instanceof AggregationNode) {
+ return ((AggregationNode) plan).isColocate() && isColocated(plan.getChild(0));
+ } else if (plan instanceof OlapScanNode) {
+ return true;
+ } else if (plan instanceof SelectNode) {
+ return isColocated(plan.getChild(0));
+ } else if (plan instanceof HashJoinNode) {
+ return ((HashJoinNode) plan).isColocate()
+ && (isColocated(plan.getChild(0)) || isColocated(plan.getChild(1)));
+ } else if (plan instanceof SetOperationNode) {
+ if (!((SetOperationNode) plan).isColocate()) {
+ return false;
+ }
+ for (PlanNode child : plan.getChildren()) {
+ if (isColocated(child)) {
+ return true;
+ }
Review Comment:
**🟡 Minor**: `resolveExchangeType()` accepts a `child` parameter that is never used. This is confusing at call sites — e.g., `SetOperationNode` passes a potentially-null `firstChild` (when `children.isEmpty()`, line 214). It's safe today because `child` is unused, but if someone later adds code that accesses `child`, it could NPE on empty-children Union nodes. Consider removing the unused parameter.
##########
be/src/exec/pipeline/pipeline_fragment_context.h:
##########
@@ -320,8 +322,11 @@ class PipelineFragmentContext : public TaskExecutionContext {
std::mutex _state_map_lock;
- int _operator_id = 0;
- int _sink_operator_id = 0;
+ // Start from -1 so all operator IDs are negative. This avoids collision with
Review Comment:
**🟢 Note**: `_operator_id` and `_sink_operator_id` now start from `-1` instead of `0` to avoid collision with unpaired sinks (e.g., `OlapTableSink`) that hardcode `dest_id=0`. This is a clean fix. Verify that no code path in the BE enumerates or displays operator IDs expecting them to be non-negative — negative IDs in logs or profiles could confuse debugging.
##########
be/src/exec/pipeline/pipeline_fragment_context.cpp:
##########
@@ -690,6 +716,199 @@ Status PipelineFragmentContext::_build_pipelines(ObjectPool* pool, const Descrip
return Status::OK();
}
+Status PipelineFragmentContext::_create_deferred_local_exchangers() {
+ for (auto& info : _deferred_exchangers) {
+ // DANGER ZONE — do not "fix" this line without reading the history.
+ //
+ // sender_count seeds Exchanger::_running_sink_operators, which the source side
+ // waits to reach 0 via sub_running_sink_operators on each sink LocalState close.
+ // The correct value is THIS pipeline-instance's sink task count, which is exactly
+ // info.upstream_pipe->num_tasks() — one PipelineTask per task, one close per task.
+ //
+ // Tempting wrong fix #1: `std::max(num_tasks, _num_instances)` to mirror the
+ // BE-planned path in _add_local_exchange_impl (~line 1023). THIS BREAKS the
+ // common FE-planned shape of `serial scan → LE(PT) → ...`: upstream_pipe
+ // genuinely has num_tasks=1, only 1 close arrives, but seed becomes
+ // _num_instances so _running_sink_operators never reaches 0 — downstream
+ // sources hang on SHUFFLE_DATA_DEPENDENCY (e.g. MTMV refresh from
+ // mtmv_up_down_job_p0/load.groovy stays at Status=RUNNING; build 949402
+ // regressed exactly this way). BE-planned mode uses max() because its
+ // `cur_pipe` is the source-side pipeline (always raised to _num_instances by
+ // add_pipeline) — not analogous to our `upstream_pipe` here, which is the
+ // sink-side pipeline that may legitimately stay at 1 for serial sources.
+ //
+ // Tempting wrong fix #2: multiply by _num_instances on the theory shared_state
+ // is shared across all instances. Same hang — each fragment-instance
+ // PipelineFragmentContext has its OWN _op_id_to_shared_state map, so the
+ // exchanger is per-instance, not per-BE. num_tasks() is already the right
+ // close-count for one instance.
+ //
+ // If a hang shows up with `_running_sink_operators < 0`, the bug is upstream:
+ // _propagate_local_exchange_num_tasks left num_tasks too low (or too high) for
+ // this fragment shape. Fix THAT pass, not this seed value.
+ const int sender_count = info.upstream_pipe->num_tasks();
Review Comment:
**🟠 Medium**: The two-pass `while (changed)` loops in `_propagate_local_exchange_num_tasks()` rely on monotonic convergence (num_tasks only increases in Pass 1, only decreases in Pass 2). If a future code change introduces a non-monotonic update, this could loop infinitely. Consider adding a max-iteration guard (`while (changed && ++iter_count < MAX_ITERATIONS)`) with a `DCHECK` or `LOG(WARNING)` on exceeded iterations, so a future regression fails loudly rather than hanging.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/AddLocalExchange.java:
##########
@@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.planner;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.glue.translator.PlanTranslatorContext;
+import org.apache.doris.nereids.trees.plans.distribute.DistributedPlan;
+import org.apache.doris.nereids.trees.plans.distribute.FragmentIdMapping;
+import org.apache.doris.nereids.trees.plans.distribute.PipelineDistributedPlan;
+import org.apache.doris.planner.LocalExchangeNode.LocalExchangeType;
+import org.apache.doris.planner.LocalExchangeNode.LocalExchangeTypeRequire;
+import org.apache.doris.planner.LocalExchangeNode.RequireHash;
+
+/**
+ * FE-side local exchange planner — inserts {@link LocalExchangeNode} into each fragment's
+ * plan tree so that within-fragment data redistribution is decided at planning time
+ * instead of at BE pipeline-build time.
+ *
+ * <h3>When this runs</h3>
+ * Invoked from {@code NereidsPlanner.addLocalExchangeAfterDistribute()} right after
+ * {@code DistributePlanner} has assigned instances to fragments and before the plan is
+ * serialized to BE. Gated by session variable {@code enable_local_shuffle_planner}
+ * (default true) and {@code enable_local_shuffle}; when either is off this pass is
+ * skipped entirely and BE falls back to its own {@code _plan_local_exchange}. The two
+ * paths are mutually exclusive: BE consults {@code runtime_state.h::plan_local_shuffle()}
+ * to know whether it should plan LE itself.
+ *
+ * <h3>What it changes</h3>
+ * <ul>
+ * <li>For each fragment with {@code maxPerBeInstances > 1}, walks the plan tree
+ * bottom-up via {@link PlanNode#enforceAndDeriveLocalExchange} and inserts
+ * LocalExchangeNodes where children's output distribution doesn't satisfy the
+ * parent's requirement.</li>
+ * <li>May wrap the fragment root with an extra PASSTHROUGH LE so the data sink
+ * (DataStreamSink / OlapTableSink) runs with the full instance count even when
+ * the root operator is serial — see {@link #addLocalExchangeForFragment}.</li>
+ * <li>Does NOT modify the fragment sink itself, fragment boundaries, or instance
+ * assignment.</li>
+ * </ul>
+ *
+ * <h3>Per-BE instance semantics</h3>
+ * Skips fragments where every BE has at most 1 instance. Using a global instance count
+ * would insert LE for "2 BEs × 1 instance" cases, which BE's own
+ * {@code _plan_local_exchange} would not — leading to pipeline task-count mismatch and
+ * deadlock. See {@link #addLocalExchange}.
+ *
+ * <h3>Reading order</h3>
+ * Start with {@link PlanNode#enforceRequire} (the recursion engine), then individual
+ * {@code enforceAndDeriveLocalExchange} overrides on PlanNode subclasses.
+ */
+public class AddLocalExchange {
+ /** addLocalExchange with distributed plans, skipping single-instance fragments.
+ * BE's _plan_local_exchange checks _num_instances which is the per-BE instance count.
+ * With _num_instances<=1 all pipelines on that BE have 1 task so local exchange is a no-op.
+ * We must use the same per-BE semantics: skip when every BE has at most 1 instance.
+ * Using global instanceCount would insert LE for fragments where 2 BEs each have 1 instance
+ * (global=2, per-BE=1), causing pipeline task mismatch and deadlock. */
+ public void addLocalExchange(FragmentIdMapping<DistributedPlan> distributedPlans,
+ PlanTranslatorContext context) {
+ for (DistributedPlan plan : distributedPlans.values()) {
+ PipelineDistributedPlan pipePlan = (PipelineDistributedPlan) plan;
+ long maxPerBeInstances = pipePlan.getInstanceJobs().stream()
+ .collect(java.util.stream.Collectors.groupingBy(
+ j -> j.getAssignedWorker().id(), java.util.stream.Collectors.counting()))
+ .values().stream().mapToLong(Long::longValue).max().orElse(0);
+ if (maxPerBeInstances <= 1) {
+ continue;
+ }
+ PlanFragment fragment = pipePlan.getFragmentJob().getFragment();
+ addLocalExchangeForFragment(fragment, context);
+ }
+ }
+
+ private void addLocalExchangeForFragment(PlanFragment fragment, PlanTranslatorContext context) {
+ DataSink sink = fragment.getSink();
+ LocalExchangeTypeRequire require = sink == null
+ ? LocalExchangeTypeRequire.noRequire() : sink.getLocalExchangeTypeRequire();
+ PlanNode root = fragment.getPlanRoot();
+ context.setHasSerialAncestorInPipeline(root, false);
+ Pair<PlanNode, LocalExchangeType> output = root
+ .enforceAndDeriveLocalExchange(context, null, require);
+ PlanNode newRoot = output.first;
+ // The fragment data sink (DataStreamSink, OlapTableSink) runs in the same pipeline
+ // as the root. If the root will be serial on BE, the sink pipeline has 1 task —
+ // only instance 0 sends data, others hang or miss writes.
+ // Insert PASSTHROUGH fan-out so sink runs with _num_instances tasks.
+ // This matches BE-native's default required_data_distribution():
+ // _child->is_serial_operator() ? PASSTHROUGH : NOOP
+ if (newRoot.isSerialOperatorOnBe(context.getConnectContext())) {
+ newRoot = new LocalExchangeNode(context.nextPlanNodeId(), newRoot,
+ LocalExchangeType.PASSTHROUGH, null);
+ }
+ if (newRoot != root) {
+ fragment.setPlanRoot(newRoot);
+ }
+ validateNoSerialWithoutLocalExchange(fragment.getPlanRoot(), context.getConnectContext());
+ }
+
+ /**
+ * In a local-shuffle fragment, the root check above guarantees the root pipeline
+ * has N tasks. Any serial operator reduces its pipeline to 1 task. If this serial
+ * operator feeds into a non-serial parent without LocalExchangeNode in between,
+ * some pipelines have 1 task while others have N → shared_state mismatch, data loss.
+ *
+ * Serial→serial chains are fine (all at 1 task, consistent). Only the transition
+ * from serial to non-serial needs LE to restore parallelism.
+ */
+ private void validateNoSerialWithoutLocalExchange(PlanNode node,
+ org.apache.doris.qe.ConnectContext context) {
+ for (PlanNode child : node.getChildren()) {
+ validateNoSerialWithoutLocalExchange(child, context);
+ if (child.isSerialOperatorOnBe(context)
+ && !(child instanceof LocalExchangeNode)
Review Comment:
**🟡 Observation**: `validateNoSerialWithoutLocalExchange()` only logs a warning when the invariant is violated — it does not throw. If a serial→non-serial transition occurs without an LE in between, queries could produce wrong results silently. Consider: (1) in debug/test builds, throw or DCHECK; (2) add a metric counter so operators can monitor this in production before it causes user-visible issues.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/AggregationNode.java:
##########
@@ -258,4 +268,155 @@ public boolean isQueryCacheCandidate() {
public void setQueryCacheCandidate(boolean queryCacheCandidate) {
this.queryCacheCandidate = queryCacheCandidate;
}
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+
+ ConnectContext connectContext = translatorContext.getConnectContext();
+ SessionVariable sessionVariable = connectContext.getSessionVariable();
+ // PR #62438: when false, non-finalize agg falls back to BE base class.
+ boolean enableLeBeforeAgg = sessionVariable.enableLocalExchangeBeforeAgg;
+ boolean hasKeys = !aggInfo.getGroupingExprs().isEmpty();
Review Comment:
**✅ Correctness fix**: The `!needsFinalize && !aggInfo.isMerge() && !enableLeBeforeAgg` guard correctly separates FIRST_MERGE (correctness-required, always HASH) from LOCAL phase (performance-only, flag controls). This diverges from BE's `!_needs_finalize && !enable_local_exchange_before_agg` early-return which conflates both intents — the FE planner is actually *more correct* here than BE. For the backward-compat path (legacy BE planner), the bug DORIS-25413 remains; consider adding a comment noting this divergence explicitly.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java:
##########
@@ -285,4 +302,71 @@ public List<Expr> getOtherJoinConjuncts() {
public List<Expr> getMarkJoinConjuncts() {
return markJoinConjuncts;
}
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+
+ LocalExchangeTypeRequire probeSideRequire;
+ LocalExchangeTypeRequire buildSideRequire;
+ LocalExchangeType outputType = null;
+
+ if (joinOp == JoinOperator.NULL_AWARE_LEFT_ANTI_JOIN) {
+ buildSideRequire = probeSideRequire = LocalExchangeTypeRequire.noRequire();
+ outputType = LocalExchangeType.NOOP;
+ } else if (distrMode == DistributionMode.BROADCAST) {
+ // BE: _child->is_serial_operator() ? PASSTHROUGH/PASS_TO_ONE : NOOP
+ boolean probeChildSerial = children.get(0).isSerialOperatorOnBe(
+ translatorContext.getConnectContext());
+ boolean buildChildSerial = children.get(1).isSerialOperatorOnBe(
+ translatorContext.getConnectContext());
+ probeSideRequire = probeChildSerial
+ ? LocalExchangeTypeRequire.requirePassthrough()
+ : LocalExchangeTypeRequire.noRequire();
+ buildSideRequire = buildChildSerial
+ ? LocalExchangeTypeRequire.requirePassToOne()
+ : LocalExchangeTypeRequire.noRequire();
+ // For serial probe: output is PASSTHROUGH (data from single instance).
+ // For non-serial probe: propagate probe side's actual distribution.
+ outputType = probeChildSerial ? LocalExchangeType.PASSTHROUGH : null;
+ } else if (isColocate() || isBucketShuffle()) {
+ probeSideRequire = LocalExchangeTypeRequire.requireBucketHash();
+ // For BUCKET_SHUFFLE with serial build child: use requireBucketHash() (not
+ // requirePassToOne()). Unlike BROADCAST joins, BUCKET_SHUFFLE has no shared
+ // hash table mechanism — PASS_TO_ONE routes all data to task 0 while tasks 1..N-1
+ // build empty hash tables, losing rows. BUCKET_HASH_SHUFFLE correctly distributes
+ // build data by bucket to match the probe side's bucket distribution.
+ // The serial exchange returns NOOP, so enforceRequire() will insert a
+ // BUCKET_HASH_SHUFFLE local exchange (with PASSTHROUGH fan-out for heavy-ops
Review Comment:
**🟡 Clarification**: For Colocate/BucketShuffle join, both probe and build sides use `requireBucketHash()`. For a serial build child (e.g., BucketShuffle join where the build Exchange is serial), the framework will insert `PASSTHROUGH → BUCKET_HASH_SHUFFLE` chain via the heavy-ops gate. This is correct but the comment only mentions the probe side — consider adding a brief note about the build side's serial handling too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4765651549
# FE UT Coverage Report
Increment line coverage ` 74.69% (419/561)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_7fdb989a3a7e4ad0c700539a335598f4978f6938/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_7fdb989a3a7e4ad0c700539a335598f4978f6938/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3450954662
##########
fe/fe-core/src/main/java/org/apache/doris/planner/RepeatNode.java:
##########
@@ -86,11 +90,30 @@ public String getNodeExplainString(String detailPrefix, TExplainLevel detailLeve
// Determined by its child.
@Override
- public boolean isSerialOperator() {
- return children.get(0).isSerialOperator();
+ public boolean isSerialNode() {
+ return children.get(0).isSerialNode();
}
public GroupingInfo getGroupingInfo() {
return groupingInfo;
}
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ // REPEAT (rollup/grouping sets) is NOT distribution-preserving: it NULLs grouping
+ // columns per set and produces GROUPING_ID, which is part of the downstream agg hash
+ // key but does not exist below the repeat. Forwarding the parent HASH require down
+ // would push the local exchange before the row expansion AND hash by the child
+ // distribution (a single upstream shuffle key) instead of the agg grouping_exprs,
+ // collapsing rows onto one instance (tpcds q67, +73%). Recurse with noRequire so the
+ // parent inserts its hash local exchange ABOVE the repeat using its own grouping_exprs
+ // (mirrors BE, whose RepeatOperatorX has a NOOP required_data_distribution).
+ Pair<PlanNode, LocalExchangeType> enforceResult
+ = enforceRequire(translatorContext, children.get(0), 0,
+ LocalExchangeTypeRequire.noRequire());
+ children = new java.util.ArrayList<>();
+ children.add(enforceResult.first);
Review Comment:
Repeat is not distribution-preserving here, but this return value forwards the child's distribution to the parent. A reduced failing shape is:
```text
Aggregation(requireHash by post-repeat grouping exprs/grouping id)
Repeat
child(output already BUCKET_HASH_SHUFFLE or GLOBAL_EXECUTION_HASH_SHUFFLE)
```
The method intentionally recurses with `noRequire()` so the parent hash requirement is not pushed below the row-expanding `RepeatNode`. However, if the child already reports a hash distribution, `enforceResult.second` is hash, and the parent `enforceRequire(..., requireHash())` can treat the Repeat output as satisfying its requirement and skip the local exchange above Repeat. Rows are then partitioned by pre-repeat child keys instead of the post-repeat grouping keys/grouping id. Please make Repeat report `NOOP` upward so any parent hash requirement is enforced above it, and add a hash-child unit case because the current test only covers a NOOP child.
```suggestion
return Pair.of(this, LocalExchangeType.NOOP);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4767800324
<details>
<summary>TPC-DS: <b>Total hot run time: 173044 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7fdb989a3a7e4ad0c700539a335598f4978f6938, data reload: false
query5 4290 624 485 485
query6 428 195 173 173
query7 4814 526 288 288
query8 357 214 208 208
query9 8767 4037 4086 4037
query10 459 300 259 259
query11 5974 2319 2127 2127
query12 156 105 97 97
query13 1341 610 430 430
query14 6378 5424 5085 5085
query14_1 4384 4358 4352 4352
query15 203 194 178 178
query16 1012 446 450 446
query17 1093 709 555 555
query18 2442 461 327 327
query19 201 189 138 138
query20 115 110 102 102
query21 221 137 116 116
query22 13736 13536 13457 13457
query23 17407 16475 16167 16167
query23_1 16299 16258 16263 16258
query24 7643 1741 1321 1321
query24_1 1332 1323 1293 1293
query25 544 436 365 365
query26 1304 335 171 171
query27 2698 531 346 346
query28 4493 2054 2031 2031
query29 1078 629 526 526
query30 319 247 196 196
query31 1123 1073 967 967
query32 110 65 70 65
query33 539 324 263 263
query34 1159 1179 661 661
query35 791 781 688 688
query36 1386 1411 1220 1220
query37 155 106 89 89
query38 1906 1737 1678 1678
query39 926 919 893 893
query39_1 873 872 900 872
query40 225 124 106 106
query41 70 68 68 68
query42 88 87 88 87
query43 320 330 279 279
query44 1457 773 785 773
query45 199 191 179 179
query46 1063 1224 786 786
query47 2351 2346 2224 2224
query48 382 467 298 298
query49 621 451 350 350
query50 995 353 254 254
query51 4389 4497 4339 4339
query52 80 80 68 68
query53 245 270 201 201
query54 261 215 188 188
query55 74 75 67 67
query56 222 234 228 228
query57 1441 1411 1301 1301
query58 237 212 206 206
query59 1616 1653 1393 1393
query60 268 248 228 228
query61 159 147 152 147
query62 706 647 578 578
query63 224 190 201 190
query64 2533 791 645 645
query65 4899 4806 4796 4796
query66 1785 469 349 349
query67 29878 29674 29636 29636
query68 3179 1684 909 909
query69 407 298 272 272
query70 1079 929 939 929
query71 299 258 212 212
query72 2871 2606 2315 2315
query73 862 819 448 448
query74 5114 4949 4777 4777
query75 2624 2579 2248 2248
query76 2328 1220 774 774
query77 362 382 276 276
query78 12612 12308 11869 11869
query79 1384 1123 773 773
query80 583 518 366 366
query81 448 282 245 245
query82 573 156 129 129
query83 351 269 247 247
query84 273 146 112 112
query85 848 507 407 407
query86 386 297 294 294
query87 1835 1834 1742 1742
query88 3704 2813 2769 2769
query89 422 377 326 326
query90 1848 186 184 184
query91 173 159 136 136
query92 66 59 59 59
query93 1512 1524 955 955
query94 546 359 333 333
query95 676 382 344 344
query96 1073 801 355 355
query97 2713 2726 2554 2554
query98 215 208 199 199
query99 1205 1146 1017 1017
Total cold run time: 258013 ms
Total hot run time: 173044 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4774904945
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4669451448
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/7c898c900506116cfb44b47189025b6fbfa7e44c_7c898c900506116cfb44b47189025b6fbfa7e44c_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/7c898c900506116cfb44b47189025b6fbfa7e44c_7c898c900506116cfb44b47189025b6fbfa7e44c_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.39% (1915/2443) |
| Line Coverage | 64.93% (34232/52725) |
| Region Coverage | 65.36% (17613/26948) |
| Branch Coverage | 54.04% (9357/17316) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4680175630
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4680988934
<details>
<summary>TPC-H: <b>Total hot run time: 29316 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1f614d8cadd1619a8c17dd6f7ec7b12ba6983286, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17641 4066 4003 4003
q2 q3 10770 1431 800 800
q4 4684 468 336 336
q5 7532 884 586 586
q6 183 174 136 136
q7 773 823 639 639
q8 9353 1531 1503 1503
q9 5585 4502 4491 4491
q10 6779 1816 1580 1580
q11 440 274 246 246
q12 625 413 286 286
q13 18136 3407 2856 2856
q14 258 257 254 254
q15 q16 794 779 715 715
q17 1002 988 1016 988
q18 6849 5908 5547 5547
q19 1172 1259 1082 1082
q20 494 411 259 259
q21 6109 2915 2702 2702
q22 464 370 307 307
Total cold run time: 99643 ms
Total hot run time: 29316 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4987 4838 4687 4687
q2 q3 4899 5293 4704 4704
q4 2168 2182 1390 1390
q5 4837 4919 4679 4679
q6 237 176 128 128
q7 1867 1743 1610 1610
q8 2447 2127 2074 2074
q9 8150 7941 7598 7598
q10 4761 4705 4259 4259
q11 534 387 359 359
q12 735 741 524 524
q13 3057 3409 2771 2771
q14 274 279 249 249
q15 q16 680 697 606 606
q17 1283 1266 1260 1260
q18 7232 6875 6786 6786
q19 1097 1102 1103 1102
q20 2240 2239 2013 2013
q21 5283 4514 4464 4464
q22 535 461 404 404
Total cold run time: 57303 ms
Total hot run time: 51667 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4686510040
# FE UT Coverage Report
Increment line coverage ` 74.61% (429/575)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_9ccb7f30b5fb41183c99a77cc3de3311715aa9b6/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_9ccb7f30b5fb41183c99a77cc3de3311715aa9b6/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3259594757
##########
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##########
@@ -1598,6 +1600,12 @@ public enum IgnoreSplitType {
"Whether to enable local shuffle on pipelineX engine."}, needForward = true)
private boolean enableLocalShuffle = true;
+ @VarAttrDef.VarAttr(
+ name = ENABLE_LOCAL_SHUFFLE_PLANNER, fuzzy = false, varType = VariableAnnotation.EXPERIMENTAL,
+ description = {"是否在FE规划Local Shuffle",
+ "Whether to plan local shuffle in frontend"}, needForward = true)
+ private boolean enableLocalShufflePlanner = true;
+
Review Comment:
Our deployment process upgrades BE before FE, so during a rolling upgrade an old BE never receives `LOCAL_EXCHANGE_NODE` from a new FE — the new thrift node type is only sent after all BEs in the cluster have been upgraded. Keeping the default `true` is intentional so we get the FE planner enabled out of the box once the rollout completes. If you have a deployment topology that flips this order, please set `enable_local_shuffle_planner=false` in fe.conf or via SQL and the legacy BE path stays bit-for-bit identical to the old behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4680904587
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3432646806
##########
be/src/exec/pipeline/dependency.h:
##########
@@ -832,50 +832,44 @@ struct SetSharedState : public BasicSharedState {
Status hash_table_init();
};
-enum class ExchangeType : uint8_t {
- NOOP = 0,
- // Shuffle data by Crc32CHashPartitioner
- HASH_SHUFFLE = 1,
- // Round-robin passthrough data blocks.
- PASSTHROUGH = 2,
- // Shuffle data by Crc32HashPartitioner<ShuffleChannelIds> (e.g. same as storage engine).
- BUCKET_HASH_SHUFFLE = 3,
- // Passthrough data blocks to all channels.
- BROADCAST = 4,
- // Passthrough data to channels evenly in an adaptive way.
- ADAPTIVE_PASSTHROUGH = 5,
- // Send all data to the first channel.
- PASS_TO_ONE = 6,
-};
+inline bool is_shuffled_exchange(TLocalPartitionType::type idx) {
Review Comment:
dependency.h now directly exposes inline helpers and DataDistribution members that use TLocalPartitionType, but this header never includes the thrift header that defines that enum. This header is included directly from several translation units that only include Types_types.h or protobuf headers first, for example runtime/record_batch_queue.cpp and load/group_commit/group_commit_mgr.cpp, so the build depends on unrelated include order and can fail with TLocalPartitionType undeclared. Please include <gen_cpp/Partitions_types.h> from dependency.h itself before using the enum.
##########
fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java:
##########
@@ -563,6 +567,10 @@ public void update(Map<String, String> summaryInfo) {
// This method is used to display the final data status when the overall query ends.
// This can avoid recalculating some strings and so on every time during the update process.
public void queryFinished() {
+ // Mark profile collection as complete. This is written last, after all BE fragment
+ // profiles have been merged (called post-waitForFragmentsDone). Pollers can wait for
+ // this field to avoid reading a partial profile.
Review Comment:
The completion marker is written before the optional SPLITS_ASSIGNMENT_WEIGHT field, but the new contract says readers can poll this marker to know the profile is fully collected. A polling reader can observe Is Profile Collection Completed=true while queryFinished() has not yet added the assignment-weight info string, and then read a partial profile. Move this marker write to the end of queryFinished() after the optional fields are added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4741498480
<details>
<summary>ClickBench: <b>Total hot run time: 25.3 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0436cc4cb573b104a59408a6169bd145bcfeff6c, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.25 0.13 0.14
query4 1.61 0.15 0.14
query5 0.26 0.21 0.22
query6 1.26 1.06 1.08
query7 0.03 0.01 0.00
query8 0.06 0.04 0.04
query9 0.38 0.32 0.32
query10 0.56 0.55 0.60
query11 0.19 0.14 0.14
query12 0.18 0.16 0.14
query13 0.48 0.49 0.47
query14 1.01 0.99 1.00
query15 0.63 0.62 0.61
query16 0.33 0.33 0.32
query17 1.12 1.11 1.10
query18 0.22 0.22 0.22
query19 2.04 2.00 1.88
query20 0.02 0.01 0.02
query21 15.41 0.23 0.13
query22 4.74 0.06 0.06
query23 16.17 0.31 0.13
query24 2.95 0.45 0.34
query25 0.11 0.06 0.04
query26 0.73 0.22 0.14
query27 0.05 0.05 0.03
query28 3.54 0.93 0.54
query29 12.52 4.34 3.46
query30 0.27 0.16 0.17
query31 2.78 0.62 0.32
query32 3.23 0.59 0.49
query33 3.16 3.15 3.31
query34 15.57 4.21 3.58
query35 3.57 3.59 3.55
query36 0.55 0.44 0.42
query37 0.09 0.07 0.06
query38 0.06 0.04 0.04
query39 0.04 0.03 0.03
query40 0.19 0.16 0.15
query41 0.08 0.04 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.04
Total cold run time: 96.63 s
Total hot run time: 25.3 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4765205084
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4765629332
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4767836895
<details>
<summary>ClickBench: <b>Total hot run time: 25.81 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7fdb989a3a7e4ad0c700539a335598f4978f6938, data reload: false
query1 0.01 0.00 0.01
query2 0.14 0.09 0.09
query3 0.36 0.24 0.27
query4 1.61 0.25 0.24
query5 0.32 0.32 0.31
query6 1.16 0.68 0.67
query7 0.03 0.01 0.01
query8 0.10 0.08 0.07
query9 0.50 0.38 0.38
query10 0.58 0.62 0.60
query11 0.32 0.19 0.19
query12 0.32 0.19 0.19
query13 0.53 0.54 0.52
query14 0.93 0.92 0.92
query15 0.68 0.60 0.61
query16 0.38 0.39 0.38
query17 1.01 1.00 1.03
query18 0.30 0.29 0.29
query19 1.93 1.75 1.83
query20 0.02 0.01 0.01
query21 15.39 0.37 0.31
query22 4.86 0.14 0.14
query23 15.85 0.50 0.30
query24 2.53 0.56 0.43
query25 0.15 0.10 0.10
query26 0.77 0.28 0.22
query27 0.10 0.09 0.10
query28 3.49 0.87 0.51
query29 12.46 4.46 3.51
query30 0.38 0.25 0.27
query31 2.76 0.62 0.35
query32 3.23 0.60 0.48
query33 2.92 3.05 2.99
query34 15.70 4.00 3.32
query35 3.25 3.24 3.28
query36 0.63 0.51 0.51
query37 0.13 0.10 0.10
query38 0.08 0.06 0.06
query39 0.07 0.07 0.06
query40 0.22 0.18 0.17
query41 0.12 0.09 0.08
query42 0.09 0.06 0.06
query43 0.08 0.07 0.08
Total cold run time: 96.49 s
Total hot run time: 25.81 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4767041113
# FE Regression Coverage Report
Increment line coverage ` 81.28% (456/561)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4766364688
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4767842287
<details>
<summary>ClickBench: <b>Total hot run time: 26.07 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b, data reload: false
query1 0.01 0.01 0.01
query2 0.15 0.09 0.09
query3 0.37 0.24 0.25
query4 1.60 0.24 0.25
query5 0.34 0.36 0.31
query6 1.15 0.68 0.67
query7 0.04 0.01 0.01
query8 0.09 0.08 0.07
query9 0.52 0.39 0.39
query10 0.58 0.57 0.59
query11 0.32 0.19 0.18
query12 0.33 0.19 0.20
query13 0.54 0.52 0.52
query14 0.93 0.92 0.92
query15 0.69 0.60 0.60
query16 0.39 0.39 0.40
query17 1.05 1.00 1.01
query18 0.33 0.30 0.30
query19 1.93 1.83 1.80
query20 0.02 0.02 0.01
query21 15.39 0.37 0.31
query22 4.88 0.14 0.14
query23 15.84 0.51 0.32
query24 2.52 0.58 0.45
query25 0.16 0.11 0.10
query26 0.76 0.28 0.22
query27 0.10 0.09 0.10
query28 3.48 0.93 0.52
query29 12.44 4.57 3.58
query30 0.39 0.25 0.26
query31 2.77 0.64 0.35
query32 3.24 0.60 0.49
query33 2.96 2.96 2.98
query34 15.78 4.13 3.40
query35 3.29 3.28 3.31
query36 0.65 0.52 0.50
query37 0.13 0.10 0.10
query38 0.08 0.07 0.07
query39 0.07 0.06 0.06
query40 0.21 0.18 0.17
query41 0.14 0.09 0.07
query42 0.08 0.07 0.07
query43 0.08 0.07 0.06
Total cold run time: 96.82 s
Total hot run time: 26.07 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4687151641
# BE Regression && UT Coverage Report
Increment line coverage `88.29% (362/410)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_9ccb7f30b5fb41183c99a77cc3de3311715aa9b6_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_9ccb7f30b5fb41183c99a77cc3de3311715aa9b6_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.99% (28307/38256) |
| Line Coverage | 57.96% (308498/532265) |
| Region Coverage | 54.86% (258701/471560) |
| Branch Coverage | 56.21% (112201/199624) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4708791066
# FE UT Coverage Report
Increment line coverage ` 74.73% (420/562)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_832ba5f0bf8ab5846bb316e94f1099ae58fe466f/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_832ba5f0bf8ab5846bb316e94f1099ae58fe466f/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3413413674
##########
fe/fe-core/src/main/java/org/apache/doris/planner/ScanNode.java:
##########
@@ -733,7 +737,7 @@ public ScanContext getScanContext() {
}
@Override
- public boolean isSerialOperator() {
+ public boolean isSerialNode() {
ConnectContext context = ConnectContext.get();
if (context == null) {
return numScanBackends() <= 0;
Review Comment:
I don't think this is a regression — I checked the actual before/after. Master's `ScanNode.isSerialOperator()` body is byte-identical to this PR's `isSerialNode()` (both: `numScanBackends() <= 0 || getScanRangeNum() < parallelExecInstanceNum * numScanBackends() || isForceToLocalShuffle()`); neither version ever contained `fragment.useSerialSource(context)`. At the decision point (`Coordinator.assignScanRanges`, ~`Coordinator.java:2995`) master called `isSerialOperator()` and this PR calls `isSerialNode()` — same body, pure rename.
The `useSerialSource()` gate that matters there is in the *caller* (`ignoreStorageDataDistribution`, ~`Coordinator.java:2939`), which is unchanged. So the legacy scan-range assignment behaves exactly as on master. Could you point me at the specific old code where `isSerialNode()`/`isSerialOperator()` was `numScanBackends() <= 0 || fragment.useSerialSource(context)`? I couldn't find that shape in master.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/AssertNumRowsNode.java:
##########
@@ -78,7 +84,17 @@ public int getNumInstances() {
}
@Override
- public boolean isSerialOperator() {
+ public boolean isSerialNode() {
return true;
}
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(PlanTranslatorContext translatorContext,
+ PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+
+ Pair<PlanNode, LocalExchangeType> enforceResult = enforceRequire(
+ translatorContext, children.get(0), 0, LocalExchangeTypeRequire.requirePassthrough());
Review Comment:
The BE operator requires PASSTHROUGH, and the FE planner mirrors BE's `required_data_distribution` 1:1, so this matches: `AssertNumRowsOperatorX::required_data_distribution()` returns `{TLocalPartitionType::PASSTHROUGH}` (`assert_num_rows_operator.h:49`). Returning NOOP here would diverge from BE-native behavior. The "only one row" observation is about output cardinality, but `required_data_distribution` describes the input side — AssertNumRows is serial (1 task) and the PASSTHROUGH fans the (possibly N-task) child into it, same as BE.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java:
##########
@@ -824,14 +828,97 @@ public void foreachDownInCurrentFragment(Consumer<PlanNode> visitor) {
});
}
- // Operators need to be executed serially. (e.g. finalized agg without key)
- public boolean isSerialOperator() {
+ /**
+ * Node-level "is this operator inherently serial" property — answers without looking
+ * at the fragment. Default false; subclasses override (e.g. finalized agg without key,
+ * UNPARTITIONED ExchangeNode with merge sort).
+ *
+ * Use ONLY in framework-internal places where we are already iterating within a
+ * fragment whose serial-source mode is fixed: {@link #shouldResetSerialFlagForChild}
+ * inputs, {@link #createLocalExchange} heavy-op gate, and child.isSerialNode() checks
+ * embedded inside an enforceRequire path. Do NOT use it when computing a
+ * {@link LocalExchangeNode.LocalExchangeTypeRequire} on a child — call
+ * {@link #isSerialOperatorOnBe} instead.
+ */
+ public boolean isSerialNode() {
Review Comment:
It has to stay `public`: `Coordinator` (package `org.apache.doris.qe`) calls `node.get().isSerialNode()` in `assignScanRanges`, so a cross-package caller needs public visibility — `protected` wouldn't compile.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/AddLocalExchange.java:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.planner;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.glue.translator.PlanTranslatorContext;
+import org.apache.doris.nereids.trees.plans.distribute.DistributedPlan;
+import org.apache.doris.nereids.trees.plans.distribute.FragmentIdMapping;
+import org.apache.doris.nereids.trees.plans.distribute.PipelineDistributedPlan;
+import org.apache.doris.planner.LocalExchangeNode.LocalExchangeType;
+import org.apache.doris.planner.LocalExchangeNode.LocalExchangeTypeRequire;
+import org.apache.doris.planner.LocalExchangeNode.RequireHash;
+
+/**
+ * FE-side local exchange planner — inserts {@link LocalExchangeNode} into each fragment's
+ * plan tree so that within-fragment data redistribution is decided at planning time
+ * instead of at BE pipeline-build time.
+ *
+ * <h3>When this runs</h3>
+ * Invoked from {@code NereidsPlanner.addLocalExchangeAfterDistribute()} right after
+ * {@code DistributePlanner} has assigned instances to fragments and before the plan is
+ * serialized to BE. Gated by session variable {@code enable_local_shuffle_planner}
+ * (default true) and {@code enable_local_shuffle}; when either is off this pass is
+ * skipped entirely and BE falls back to its own {@code _plan_local_exchange}. The two
+ * paths are mutually exclusive: BE consults {@code runtime_state.h::plan_local_shuffle()}
+ * to know whether it should plan LE itself.
+ *
+ * <h3>What it changes</h3>
+ * <ul>
+ * <li>For each fragment with {@code maxPerBeInstances > 1}, walks the plan tree
+ * bottom-up via {@link PlanNode#enforceAndDeriveLocalExchange} and inserts
+ * LocalExchangeNodes where children's output distribution doesn't satisfy the
+ * parent's requirement.</li>
+ * <li>May wrap the fragment root with an extra PASSTHROUGH LE so the data sink
+ * (DataStreamSink / OlapTableSink) runs with the full instance count even when
+ * the root operator is serial — see {@link #addLocalExchangeForFragment}.</li>
+ * <li>Does NOT modify the fragment sink itself, fragment boundaries, or instance
+ * assignment.</li>
+ * </ul>
+ *
+ * <h3>Per-BE instance semantics</h3>
+ * Skips fragments where every BE has at most 1 instance. Using a global instance count
+ * would insert LE for "2 BEs × 1 instance" cases, which BE's own
+ * {@code _plan_local_exchange} would not — leading to pipeline task-count mismatch and
+ * deadlock. See {@link #addLocalExchange}.
+ *
+ * <h3>Reading order</h3>
+ * Start with {@link PlanNode#enforceRequire} (the recursion engine), then individual
+ * {@code enforceAndDeriveLocalExchange} overrides on PlanNode subclasses.
+ */
+public class AddLocalExchange {
+ /** addLocalExchange with distributed plans, skipping single-instance fragments.
+ * BE's _plan_local_exchange checks _num_instances which is the per-BE instance count.
+ * With _num_instances<=1 all pipelines on that BE have 1 task so local exchange is a no-op.
+ * We must use the same per-BE semantics: skip when every BE has at most 1 instance.
+ * Using global instanceCount would insert LE for fragments where 2 BEs each have 1 instance
+ * (global=2, per-BE=1), causing pipeline task mismatch and deadlock. */
+ public void addLocalExchange(FragmentIdMapping<DistributedPlan> distributedPlans,
+ PlanTranslatorContext context) {
+ for (DistributedPlan plan : distributedPlans.values()) {
+ PipelineDistributedPlan pipePlan = (PipelineDistributedPlan) plan;
+ long maxPerBeInstances = pipePlan.getInstanceJobs().stream()
+ .collect(java.util.stream.Collectors.groupingBy(
+ j -> j.getAssignedWorker().id(), java.util.stream.Collectors.counting()))
+ .values().stream().mapToLong(Long::longValue).max().orElse(0);
+ if (maxPerBeInstances <= 1) {
+ continue;
+ }
+ PlanFragment fragment = pipePlan.getFragmentJob().getFragment();
+ addLocalExchangeForFragment(fragment, context);
+ }
+ }
+
+ private void addLocalExchangeForFragment(PlanFragment fragment, PlanTranslatorContext context) {
+ DataSink sink = fragment.getSink();
+ LocalExchangeTypeRequire require = sink == null
+ ? LocalExchangeTypeRequire.noRequire() : sink.getLocalExchangeTypeRequire();
+ PlanNode root = fragment.getPlanRoot();
+ context.setHasSerialAncestorInPipeline(root, false);
+ Pair<PlanNode, LocalExchangeType> output = root
+ .enforceAndDeriveLocalExchange(context, null, require);
+ PlanNode newRoot = output.first;
+ // The fragment data sink (DataStreamSink, OlapTableSink) runs in the same pipeline
+ // as the root. If the root will be serial on BE, the sink pipeline has 1 task —
+ // only instance 0 sends data, others hang or miss writes.
+ // Insert PASSTHROUGH fan-out so sink runs with _num_instances tasks.
+ // This matches BE-native's default required_data_distribution():
+ // _child->is_serial_operator() ? PASSTHROUGH : NOOP
+ if (newRoot.isSerialOperatorOnBe(context.getConnectContext())) {
+ newRoot = new LocalExchangeNode(context.nextPlanNodeId(), newRoot,
+ LocalExchangeType.PASSTHROUGH, null);
+ }
+ if (newRoot != root) {
+ fragment.setPlanRoot(newRoot);
+ }
+ }
+
+ public static boolean isColocated(PlanNode plan) {
+ if (plan instanceof AggregationNode) {
+ return ((AggregationNode) plan).isColocate() && isColocated(plan.getChild(0));
+ } else if (plan instanceof OlapScanNode) {
+ return true;
+ } else if (plan instanceof SelectNode) {
+ return isColocated(plan.getChild(0));
+ } else if (plan instanceof HashJoinNode) {
+ return ((HashJoinNode) plan).isColocate()
+ && (isColocated(plan.getChild(0)) || isColocated(plan.getChild(1)));
+ } else if (plan instanceof SetOperationNode) {
+ if (!((SetOperationNode) plan).isColocate()) {
+ return false;
+ }
+ for (PlanNode child : plan.getChildren()) {
+ if (isColocated(child)) {
+ return true;
+ }
+ }
+ return false;
+ } else {
+ return false;
+ }
+ }
+
+ public static LocalExchangeType resolveExchangeType(LocalExchangeTypeRequire require) {
Review Comment:
I'd prefer to keep `preferType()` as `GLOBAL` and not align it down to `LOCAL`. The observation is right (it's not authoritative — `resolveExchangeType()` decides the actual type, and `hasPartitionExprs()` only uses `isHashShuffle()`), but the value itself is a deliberate safe default:
`GLOBAL_EXECUTION_HASH_SHUFFLE` is the unconditionally-valid hash partition (full cross-backend redistribution). `LOCAL_EXECUTION_HASH_SHUFFLE` only rebalances within a backend, so it's correct *only if* each key's rows are already confined to one backend — it carries a precondition. For the generic `RequireHash` ("any hash"), the canonical/safe representative should be the precondition-free one (GLOBAL). `resolveExchangeType()` then specializes RequireHash→LOCAL *only* in the FE-planned intra-fragment context where that precondition holds and GLOBAL's `shuffle_idx_to_instance_idx` could be empty.
Making `preferType()` return LOCAL would make the abstract default the precondition-bearing type — a latent footgun if any future caller (or `noopTo()`) ever used `preferType()` as a real inserted type. I'll add a comment near `resolveExchangeType()`/`RequireHash` documenting that the LOCAL specialization is intentional and scoped, rather than changing the default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4707952956
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4701410428
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3412123587
##########
gensrc/thrift/Partitions.thrift:
##########
@@ -55,6 +55,40 @@ enum TPartitionType {
MERGE_PARTITIONED = 9
}
+enum TLocalPartitionType {
+ NOOP = 0,
+ // used to resume the global hash distribution because other distribution break the global hash distribution,
+ // such as PASSTHROUGH. and then JoinNode can shuffle data by the same hash distribution.
+ //
+ // for example: look here, need resume to GLOBAL_EXECUTION_HASH_SHUFFLE
+ // ↓
+ // Node -> LocalExchangeNode(PASSTHROUGH) → JoinNode → LocalExchangeNode(GLOBAL_EXECUTION_HASH_SHUFFLE) → JoinNode
+ // ExchangeNode(BROADCAST) ↗ ↑
+ // ExchangeNode(GLOBAL_EXECUTION_HASH_SHUFFLE)
+ GLOBAL_EXECUTION_HASH_SHUFFLE = 1,
+ // used to rebalance data for rebalance data and add parallelism
+ //
+ // for example: look here, need use LOCAL_EXECUTION_HASH_SHUFFLE to rebalance data
+ // ↓
+ // Scan(hash(id)) -> LocalExchangeNode(LOCAL_EXECUTION_HASH_SHUFFLE(id, name)) → AggregationNode(group by(id,name))
Review Comment:
Thanks — you're right that the original wording was imprecise, and I've reworded the comment (`603c87096bf`). The key point is that this is a **performance (parallelism)** rebalance, not a correctness one:
- `group by (id, name)` is already correct on a scan hash-distributed by `id`, because `id` is a *subset* of the grouping keys — every `(id, name)` group is fully contained in the backend that owns its `id`. So `hash(id)` already co-locates each group, exactly as you noted; no reshuffle is needed for correctness.
- What `LOCAL_EXECUTION_HASH_SHUFFLE` adds here is **parallelism**: when there are few distinct `id` values the data is concentrated on a few local instances, so we re-partition by the full key `(id, name)` to spread the aggregation across all local instances (coarse → fine).
Your suggested `scan(hash(id, name)) -> agg(group by id)` goes the other way (fine → coarse), so it wouldn't add parallelism. More importantly it isn't a `LOCAL_EXECUTION_HASH_SHUFFLE` scenario at all: under `hash(id, name)` the rows for a single `id` are scattered across *different backends* (e.g. `(5, A)` on be1, `(5, B)` on be2), and a local exchange only rebalances among the local instances of one backend — it can't gather an `id` across backends. That re-partition needs a cross-backend hash exchange (`GLOBAL_EXECUTION_HASH_SHUFFLE` / the network shuffle), which is exactly what `LOCAL_EXECUTION_HASH_SHUFFLE` is *not* — so it wouldn't showcase this type's advantage.
So I kept the original (coarse → fine) example but rewrote the explanation to make the "parallelism, not correctness" intent explicit.
##########
gensrc/thrift/Partitions.thrift:
##########
@@ -55,6 +55,40 @@ enum TPartitionType {
MERGE_PARTITIONED = 9
}
+enum TLocalPartitionType {
+ NOOP = 0,
+ // used to resume the global hash distribution because other distribution break the global hash distribution,
+ // such as PASSTHROUGH. and then JoinNode can shuffle data by the same hash distribution.
+ //
+ // for example: look here, need resume to GLOBAL_EXECUTION_HASH_SHUFFLE
+ // ↓
+ // Node -> LocalExchangeNode(PASSTHROUGH) → JoinNode → LocalExchangeNode(GLOBAL_EXECUTION_HASH_SHUFFLE) → JoinNode
+ // ExchangeNode(BROADCAST) ↗ ↑
+ // ExchangeNode(GLOBAL_EXECUTION_HASH_SHUFFLE)
+ GLOBAL_EXECUTION_HASH_SHUFFLE = 1,
+ // used to rebalance data for rebalance data and add parallelism
+ //
+ // for example: look here, need use LOCAL_EXECUTION_HASH_SHUFFLE to rebalance data
+ // ↓
+ // Scan(hash(id)) -> LocalExchangeNode(LOCAL_EXECUTION_HASH_SHUFFLE(id, name)) → AggregationNode(group by(id,name))
+ //
+ // the LOCAL_EXECUTION_HASH_SHUFFLE is necessary because the hash distribution of scan node is based on id,
+ // but the hash distribution of aggregation node is based on id and name, so we need to rebalance data by both
+ // id and name to make sure the data with same id and name can be sent to the same instance of aggregation node.
+ // and we can not use GLOBAL_EXECUTION_HASH_SHUFFLE(id, name) here, because
+ // `TPipelineFragmentParams.shuffle_idx_to_instance_idx` is used to mapping partial global instance index to local
+ // instance index, and discard the other backend's instance index, the data not belong to the local instance will be
+ // discarded, which cause data loss.
+ LOCAL_EXECUTION_HASH_SHUFFLE = 2,
+ BUCKET_HASH_SHUFFLE = 3,
+ // round-robin partition, used to rebalance data for rebalance data and add parallelism
+ PASSTHROUGH = 4,
+ ADAPTIVE_PASSTHROUGH = 5,
+ BROADCAST = 6,
+ PASS_TO_ONE = 7,
+ LOCAL_MERGE_SORT = 8
Review Comment:
Done (`603c87096bf`). Added a comment for every value, grounded in the BE exchanger implementations, with short examples where they help:
- `NOOP` — no exchange; `DataDistribution::need_local_exchange()` is false, filtered out before BE's exchanger factory.
- `BUCKET_HASH_SHUFFLE` — `BucketShuffleExchanger`; route each row to the instance owning its bucket via `bucket_seq_to_instance_idx` (colocate / bucket-shuffle join).
- `PASSTHROUGH` — round-robin *whole blocks* (`block N -> instance N % local_instance_count`), no re-partition; e.g. fan a serial source out to N instances.
- `ADAPTIVE_PASSTHROUGH` — row-level round-robin for the first `>= local_instance_count` blocks (even initial spread), then cheap whole-block PASSTHROUGH; round-robin, not a hash shuffle, switch driven by block count.
- `BROADCAST` — copy every block to all instances (broadcast-join build side).
- `PASS_TO_ONE` — funnel to instance 0; degrades to BROADCAST when `enable_share_hash_table_for_broadcast_join` is off.
- `LOCAL_MERGE_SORT` — k-way merge of already-sorted inputs (legacy BE path only; FE never emits it).
##########
gensrc/thrift/PlanNodes.thrift:
##########
@@ -1427,6 +1428,24 @@ struct TExchangeNode {
4: optional Partitions.TPartitionType partition_type
}
+struct TLocalExchangeNode {
+ 1: required Partitions.TLocalPartitionType partition_type
Review Comment:
Done (`603c87096bf`) — changed `partition_type` to `optional`. Since it was the only `required` field, making it optional drops Thrift's positional constructor, so I also updated the single FE call site to the no-arg ctor + setter: `new TLocalExchangeNode(); ...setPartitionType(exchangeType.toThrift())`. BE reads the field value directly (always set by FE), so no BE change is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4708902228
<details>
<summary>TPC-DS: <b>Total hot run time: 168821 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 14ff5b48070e26ba3f1518c301fce97c36998331, data reload: false
query5 4342 636 467 467
query6 430 182 161 161
query7 4825 574 308 308
query8 367 231 215 215
query9 8746 4101 4029 4029
query10 444 308 250 250
query11 5909 2345 2116 2116
query12 160 100 101 100
query13 1258 603 417 417
query14 6965 5360 5064 5064
query14_1 4393 4370 4408 4370
query15 205 202 178 178
query16 1035 447 464 447
query17 1120 713 590 590
query18 2712 477 349 349
query19 203 187 144 144
query20 123 106 105 105
query21 223 139 119 119
query22 13608 13555 13307 13307
query23 17468 16461 16226 16226
query23_1 16234 16158 16408 16158
query24 7615 1762 1308 1308
query24_1 1312 1328 1329 1328
query25 565 449 406 406
query26 1312 309 171 171
query27 2591 527 335 335
query28 4415 2071 2062 2062
query29 1089 643 497 497
query30 309 236 195 195
query31 1130 1088 961 961
query32 117 65 63 63
query33 534 324 251 251
query34 1203 1179 673 673
query35 761 786 667 667
query36 1371 1373 1245 1245
query37 150 100 88 88
query38 3168 3127 3052 3052
query39 934 920 912 912
query39_1 909 870 882 870
query40 221 117 99 99
query41 61 59 57 57
query42 96 91 92 91
query43 313 314 274 274
query44
query45 194 187 185 185
query46 1090 1242 743 743
query47 2386 2361 2267 2267
query48 397 393 293 293
query49 620 451 342 342
query50 1056 354 255 255
query51 4492 4383 4332 4332
query52 86 87 74 74
query53 242 271 189 189
query54 256 212 197 197
query55 76 74 68 68
query56 231 247 211 211
query57 1436 1429 1301 1301
query58 251 210 215 210
query59 1565 1617 1433 1433
query60 280 242 227 227
query61 156 149 146 146
query62 698 649 587 587
query63 229 186 191 186
query64 2461 783 604 604
query65
query66 1754 446 332 332
query67 29250 29687 29517 29517
query68
query69 406 283 264 264
query70 1000 966 938 938
query71 308 216 212 212
query72 2968 2613 2290 2290
query73 853 738 451 451
query74 5115 4922 4777 4777
query75 2657 2555 2227 2227
query76 2305 1155 855 855
query77 345 390 278 278
query78 12512 12450 11751 11751
query79 1479 1019 767 767
query80 1286 443 402 402
query81 519 284 240 240
query82 628 152 121 121
query83 334 273 239 239
query84
query85 869 494 406 406
query86 421 299 280 280
query87 3387 3300 3135 3135
query88 3642 2793 2754 2754
query89 424 379 329 329
query90 1934 182 180 180
query91 172 165 131 131
query92 60 62 58 58
query93 1692 1377 942 942
query94 730 369 310 310
query95 689 452 340 340
query96 1011 763 354 354
query97 2722 2726 2587 2587
query98 211 205 197 197
query99 1149 1166 1035 1035
Total cold run time: 251974 ms
Total hot run time: 168821 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4674684816
### Follow-up on the [P1] check (AddLocalExchange) — removed it entirely
After thinking it through (and with the CI evidence), the serial→non-serial validation isn't worth keeping in any form — neither throw nor warn:
1. **It reads the same `isSerialOperatorOnBe` flag it would validate**, so it's blind to the one class of bug that actually matters — a *wrong serial flag*. The framework and the check would agree on the same wrong answer; a check can't validate an input it shares with the thing it's checking.
2. **It fires on legitimate plans** — a serial child feeding a parent that requires PASSTHROUGH/noRequire (TableFunction, NLJ, Agg) needs no LE and is correct. So as a `warn` it cries wolf, and as a `throw` it breaks those plans (the 9 P0 patterns from the last run).
3. **The one class it *could* catch** — a `require`→LE logic regression (e.g. a broken `satisfy()`) — the regression/RQG already catch end-to-end, just without the FE fail-fast.
Net: it adds noise and false confidence, not safety. The real oracle is the end-to-end run (which is exactly what surfaced the throw's false positives — no FE self-check did). The principled direction is a single source of truth for the serial/distribution decision (BE reading the FE-written flag) so there's no FE-prediction-vs-BE-reality gap to police in the first place.
Removed the method + its call. Correctness still comes from `enforceRequire` inserting the LEs, validated by the regression/RQG.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4476727978
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4486932993
run build all
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4487791049
# BE UT Coverage Report
Increment line coverage `11.11% (46/414)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/9016e5a105f84231f1a535be233aaf3e595c711d_9016e5a105f84231f1a535be233aaf3e595c711d/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/9016e5a105f84231f1a535be233aaf3e595c711d_9016e5a105f84231f1a535be233aaf3e595c711d/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.56% (20734/38712) |
| Line Coverage | 37.19% (196094/527345) |
| Region Coverage | 33.52% (153693/458547) |
| Branch Coverage | 34.56% (66979/193832) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4505876206
# BE Regression && UT Coverage Report
Increment line coverage `16.84% (80/475)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_f2d43a29db5822a72153fd88cbcd8bead33cbc6d_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_f2d43a29db5822a72153fd88cbcd8bead33cbc6d_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 57.50% (21769/37857) |
| Line Coverage | 40.72% (214336/526315) |
| Region Coverage | 37.18% (172374/463674) |
| Branch Coverage | 37.96% (73965/194865) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4506404038
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4499305087
# BE Regression && UT Coverage Report
Increment line coverage `71.74% (297/414)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_3ac8525eaac942c0bbeb93cf46d4c95ecf353156_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_3ac8525eaac942c0bbeb93cf46d4c95ecf353156_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 63.42% (24060/37938) |
| Line Coverage | 47.17% (248369/526552) |
| Region Coverage | 44.26% (205193/463569) |
| Branch Coverage | 45.42% (88533/194940) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4495475930
# FE UT Coverage Report
Increment line coverage ` 74.54% (448/601)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_ef4ea66a66f075dfb3259a519599ed361cd2d371/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_ef4ea66a66f075dfb3259a519599ed361cd2d371/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4551497474
<details>
<summary>TPC-DS: <b>Total hot run time: 172334 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a68f47c588b0124cf5474ff805c847df5dfc781e, data reload: false
query5 4336 665 532 532
query6 326 210 198 198
query7 4272 571 287 287
query8 329 236 254 236
query9 8797 4126 4093 4093
query10 463 360 308 308
query11 5793 2654 2204 2204
query12 183 141 125 125
query13 1348 629 479 479
query14 6158 5545 5208 5208
query14_1 4535 4506 4473 4473
query15 213 207 186 186
query16 1017 470 434 434
query17 1126 731 590 590
query18 2725 488 356 356
query19 220 209 166 166
query20 138 131 135 131
query21 213 140 127 127
query22 13765 13631 13421 13421
query23 17521 16516 16123 16123
query23_1 16515 16264 16423 16264
query24 7419 1788 1353 1353
query24_1 1323 1337 1346 1337
query25 583 497 450 450
query26 1330 325 178 178
query27 2678 599 365 365
query28 4425 2009 2020 2009
query29 1005 659 538 538
query30 306 244 208 208
query31 1136 1080 954 954
query32 89 82 79 79
query33 581 369 316 316
query34 1193 1167 649 649
query35 783 822 702 702
query36 1460 1382 1249 1249
query37 157 110 98 98
query38 3237 3182 3095 3095
query39 928 937 884 884
query39_1 899 866 886 866
query40 234 159 134 134
query41 73 70 69 69
query42 114 115 116 115
query43 334 339 301 301
query44
query45 219 214 206 206
query46 1082 1250 729 729
query47 2433 2403 2246 2246
query48 423 445 315 315
query49 665 507 404 404
query50 988 359 262 262
query51 4470 4384 4332 4332
query52 110 108 97 97
query53 250 292 210 210
query54 342 290 284 284
query55 98 95 93 93
query56 326 327 329 327
query57 1451 1416 1347 1347
query58 309 280 287 280
query59 1629 1670 1468 1468
query60 342 330 330 330
query61 187 186 219 186
query62 697 646 589 589
query63 247 214 210 210
query64 2343 782 647 647
query65
query66 1667 479 359 359
query67 29756 29770 29650 29650
query68
query69 473 345 316 316
query70 1090 1034 1015 1015
query71 307 266 277 266
query72 3044 2772 2451 2451
query73 851 757 449 449
query74 5130 4992 4809 4809
query75 2712 2641 2278 2278
query76 2284 1178 770 770
query77 405 423 336 336
query78 12419 12457 11922 11922
query79 1500 1001 765 765
query80 1333 556 469 469
query81 519 281 244 244
query82 1275 164 123 123
query83 353 280 261 261
query84 269 143 118 118
query85 939 546 458 458
query86 461 331 337 331
query87 3440 3385 3238 3238
query88 3648 2769 2747 2747
query89 458 397 347 347
query90 1896 193 188 188
query91 184 174 143 143
query92 82 78 79 78
query93 1472 1608 938 938
query94 731 356 334 334
query95 680 463 378 378
query96 1108 776 343 343
query97 2753 2783 2614 2614
query98 249 230 227 227
query99 1170 1160 1025 1025
Total cold run time: 256307 ms
Total hot run time: 172334 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4551008689
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/ccfac05f6b543ebe78f973987084a8de173759c9_ccfac05f6b543ebe78f973987084a8de173759c9_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/ccfac05f6b543ebe78f973987084a8de173759c9_ccfac05f6b543ebe78f973987084a8de173759c9_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.06% (1854/2375) |
| Line Coverage | 64.51% (33328/51665) |
| Region Coverage | 65.18% (16519/25343) |
| Branch Coverage | 55.75% (8839/15854) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3259596727
##########
fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java:
##########
@@ -933,4 +1020,229 @@ private String mergeIcebergAccessPathsWithId(
}
return StringUtils.join(mergeDisplayAccessPaths, ", ");
}
+
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ ArrayList<PlanNode> newChildren = Lists.newArrayList();
+ for (int i = 0; i < children.size(); i++) {
+ Pair<PlanNode, LocalExchangeType> childOutput
+ = enforceRequire(translatorContext, children.get(i), i, LocalExchangeTypeRequire.noRequire());
+ newChildren.add(childOutput.first);
+ }
+ this.children = newChildren;
+ return Pair.of(this, LocalExchangeType.NOOP);
+ }
+
+ /**
+ * Unified framework method: propagate serial flag → recurse child → satisfy check → Layer 1 skip → insert LE.
+ * Replaces the old enforceChild/enforceChildExchange/forceEnforceChildExchange trio.
+ *
+ * <h3>Data flow</h3>
+ * <ul>
+ * <li><b>serial-ancestor flag</b> ({@link PlanTranslatorContext#hasSerialAncestorInPipeline})
+ * — flows root → leaf during traversal. Mirrors BE's
+ * {@code any_of(operators[idx..end], is_serial_operator)} check used by
+ * {@code _add_local_exchange} to skip LE insertion when an ancestor in the same
+ * pipeline is already serial. Reset at pipeline boundaries via
+ * {@link #shouldResetSerialFlagForChild}.</li>
+ * <li><b>shuffle-for-correctness flag</b>
+ * ({@link PlanTranslatorContext#hasShuffleForCorrectnessAncestor}) — also flows
+ * root → leaf. Mirrors BE's {@code _followed_by_shuffled_operator}: tells a
+ * child whether some downstream operator depends on hash distribution for
+ * correctness, so {@code SetOperationNode} can pre-shuffle union branches.</li>
+ * <li><b>return value</b> {@code Pair<PlanNode, LocalExchangeType>} — first is the
+ * (possibly LE-wrapped) child; second is the actual output distribution as
+ * observed by the parent. Caller's {@code require.satisfy(output)} decides
+ * whether more LE is needed.</li>
+ * <li><b>parent.require</b> describes the constraint on the child output —
+ * computed inside the parent's {@code enforceAndDeriveLocalExchange} per child.</li>
+ * </ul>
+ *
+ * <h3>Invariants</h3>
+ * <ul>
+ * <li>Every serial → non-serial transition has an LE somewhere between them
+ * (enforced by framework step 3 below, validated by
+ * {@link AddLocalExchange#validateNoSerialWithoutLocalExchange}).</li>
+ * <li>{@code LocalExchangeNode} itself is always non-serial — setting it serial
+ * would defeat its purpose of fanning a 1-task pipeline back to N tasks.</li>
+ * <li>For pipeline-breaking parents ({@code shouldResetSerialFlagForChild=true}),
+ * the child starts a fresh pipeline so {@code hasSerialAncestor} is reset; the
+ * node's own {@code isSerialNode()} still composes in for the child's view.</li>
+ * <li>{@code RequireHash} accepts any hash flavour; {@code RequireSpecific} demands
+ * an exact match (with the one PASSTHROUGH/ADAPTIVE_PASSTHROUGH compatibility).
+ * Pick the looser one whenever correctness allows — see
+ * {@link LocalExchangeNode.LocalExchangeTypeRequire}.</li>
+ * </ul>
+ *
+ * <h3>Layers</h3>
+ * Layer 1 (shouldSkipLE): mirrors BE's need_to_local_exchange — skip when this node or
+ * an ancestor in the same pipeline is serial (operators[idx..end] has serial → skip).
+ * Layer 2 (require/output): each Node declares require and output in enforceAndDeriveLocalExchange.
+ */
+ protected Pair<PlanNode, LocalExchangeType> enforceRequire(
+ PlanTranslatorContext translatorContext, PlanNode child, int childIndex,
+ LocalExchangeTypeRequire require) {
+ // 1. Propagate serial-ancestor flag to child.
+ // For pipeline-splitting operators (shouldReset=true, e.g. non-streaming AGG):
+ // Drop inherited serial flag from parent (parent is in a different pipeline),
+ // but keep this node's own serial status (child is in the same pipeline as this
+ // node's sink, e.g. Exchange is in AGG_Sink pipeline).
+ // For non-splitting operators (shouldReset=false, e.g. streaming AGG):
+ // Inherit parent's serial flag + this node's own.
+ boolean inheritedSerial = shouldResetSerialFlagForChild(childIndex)
+ ? false : translatorContext.hasSerialAncestorInPipeline(this);
+ boolean childHasSerialAncestor = inheritedSerial || isSerialNode();
+ translatorContext.setHasSerialAncestorInPipeline(child, childHasSerialAncestor);
+
+ // 1b. Propagate shuffle-for-correctness-ancestor flag to child.
+ // Mirrors BE's _followed_by_shuffled_operator: a downstream operator needs hash
+ // distribution for correctness, and the chain to here goes through HASH or NOOP
+ // requirements (so the dependency is preserved).
+ // propagate = ((inheritedShuffled || self.requiresShuffleForCorrectness)
+ // && require is hash)
+ // || (inheritedShuffled && require is noop/passthrough)
+ boolean inheritedShuffled = translatorContext.hasShuffleForCorrectnessAncestor(this);
+ boolean selfOrInheritedShuffled = inheritedShuffled || requiresShuffleForCorrectness();
+ boolean requireIsHash = require.preferType().isHashShuffle();
+ boolean requireIsNoop = require.preferType() == LocalExchangeNode.LocalExchangeType.NOOP;
+ boolean childShuffledAncestor = (selfOrInheritedShuffled && requireIsHash)
+ || (inheritedShuffled && requireIsNoop);
+ translatorContext.setHasShuffleForCorrectnessAncestor(child, childShuffledAncestor);
+
+ // 2. Recurse child (Layer 2: child declares its own require/output)
+ Pair<PlanNode, LocalExchangeType> childOutput =
+ child.enforceAndDeriveLocalExchange(translatorContext, this, require);
+
+ // Steps 2.5 and 3 both react to a serial child but address different concerns:
+ // - Step 2.5 rewrites the OUTPUT-side view (what we tell satisfy/parent about
+ // the child's actual distribution). A serial pipeline runs with 1 task so
+ // its distribution claim is meaningless — flatten to NOOP so the satisfy
+ // check below doesn't get fooled by a stale "I output BUCKET_HASH" claim.
+ // - Step 3 rewrites the REQUIRE-side decision (what we want from the child).
+ // If we previously asked for nothing (noRequire) but the child turns out
+ // to be serial and we're not, upgrade to requirePassthrough so an LE is
+ // inserted to restore parallelism.
+
+ // 2.5. Serial child override (output side): if child is serial on BE, force its
+ // reported output to NOOP. Distribution is irrelevant when the child runs
+ // with 1 task; downstream parallelism is restored either by step 3 (LE
+ // insertion) or skipped entirely by step 4b (we're also serial).
+ if (childOutput.first.isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ childOutput = Pair.of(childOutput.first, LocalExchangeType.NOOP);
+ }
+
+ // 3. Framework-level serial child check (require side, mirrors BE base class
+ // required_data_distribution): if child will be serial on BE but this node is
+ // not serial, the pipeline has a 1-task serial child feeding an N-task non-serial
+ // parent. Without LE, pipeline splits (AGG/JOIN) create paired pipelines with
+ // mismatched num_tasks → crash. Upgrade noRequire to requirePassthrough so an
+ // LE is inserted below to restore parallelism.
+ if (require instanceof LocalExchangeNode.NoRequire
+ && childOutput.first.isSerialOperatorOnBe(translatorContext.getConnectContext())
+ && !isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ require = LocalExchangeTypeRequire.requirePassthrough();
+ }
+
+ // 4. Satisfy check: child output meets requirement → done
+ if (require.satisfy(childOutput.second)) {
+ return childOutput;
+ }
+
+ // 4. Layer 1: skip LE when serial operator or ancestor in same pipeline
+ // Equivalent to BE's need_to_local_exchange: any_of(operators[idx..end], is_serial) → skip
+ if (translatorContext.hasSerialAncestorInPipeline(this) || isSerialNode()) {
+ return childOutput;
Review Comment:
Good catch — fixed in 15d92ba024b. Both the Layer 1 skip and the serial-ancestor propagation in `enforceRequire` now use `isSerialOperatorOnBe(translatorContext.getConnectContext())` instead of the raw `isSerialNode()`.
Verified that BE's `OperatorBase` constructs from the Thrift `is_serial_operator` flag (which FE writes via `isSerialOperatorOnBe`, not `isSerialNode`) — so `Pipeline::need_to_local_exchange`'s `op->is_serial_operator()` check returns false when `fragment.useSerialSource(ctx)` is false, even if `isSerialNode()` is true. The previous code would have over-skipped LocalExchange in exactly the scenarios you listed (`ignore_storage_data_distribution=false`, query cache, NAAJ).
Updated `LocalShuffleNodeCoverageTest.testMaterializationNode` and `testSetOperationAndAssertNumRowsNode` to reflect the corrected behavior: in the fragment-less unit-test path `isSerialOperatorOnBe` returns false (the `fragment != null` guard) so the framework no longer skips Layer 1 and inserts the required LocalExchange.
Other `isSerialNode()` call sites in PlanNode.java were audited and left as-is:
- `toThrift()` already uses `isSerialOperatorOnBe`
- `hasSerialChildren()` is a pure node-level tree walk used only for fragment-internal heuristics
- `createLocalExchange()` heavy-op gate is already inside a `fragment.useSerialSource(ctx)` branch, so `isSerialNode` and `isSerialOperatorOnBe` are equivalent there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4479135984
# FE UT Coverage Report
Increment line coverage ` 73.93% (431/583)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_15d92ba024b265d9441145d7bd26d93590aa9c63/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_15d92ba024b265d9441145d7bd26d93590aa9c63/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4476744283
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4506413620
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4508244661
# FE Regression Coverage Report
Increment line coverage ` 80.07% (486/607)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_846d365a88901fa32fdf44b42b6cd04b900da259_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_846d365a88901fa32fdf44b42b6cd04b900da259_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4679953916
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4681351704
<details>
<summary>TPC-DS: <b>Total hot run time: 169122 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 784476ead971eeebeb0b637f6da4a434e380ce36, data reload: false
query5 4308 607 472 472
query6 441 196 171 171
query7 4868 553 291 291
query8 369 219 203 203
query9 8761 3991 3991 3991
query10 445 308 264 264
query11 5985 2348 2166 2166
query12 150 102 99 99
query13 1255 609 440 440
query14 6414 5335 5013 5013
query14_1 4358 4375 4344 4344
query15 199 192 180 180
query16 1021 443 420 420
query17 1102 669 541 541
query18 2552 459 337 337
query19 202 172 136 136
query20 121 107 105 105
query21 214 137 116 116
query22 13606 13622 13502 13502
query23 17416 16526 16210 16210
query23_1 16217 16392 16323 16323
query24 7498 1774 1288 1288
query24_1 1328 1322 1313 1313
query25 560 453 384 384
query26 1304 323 161 161
query27 2676 548 336 336
query28 4444 1990 2004 1990
query29 1127 615 492 492
query30 315 239 203 203
query31 1102 1073 953 953
query32 114 61 61 61
query33 538 327 259 259
query34 1178 1158 658 658
query35 754 800 669 669
query36 1400 1408 1245 1245
query37 159 109 96 96
query38 3231 3128 3040 3040
query39 939 917 897 897
query39_1 900 903 880 880
query40 222 123 104 104
query41 73 67 68 67
query42 98 97 96 96
query43 321 328 289 289
query44
query45 201 186 181 181
query46 1050 1197 759 759
query47 2386 2440 2266 2266
query48 401 420 298 298
query49 633 514 378 378
query50 1017 374 265 265
query51 4452 4402 4298 4298
query52 91 92 80 80
query53 257 266 193 193
query54 280 235 209 209
query55 81 75 75 75
query56 268 243 239 239
query57 1466 1415 1361 1361
query58 248 210 223 210
query59 1609 1643 1446 1446
query60 293 259 248 248
query61 184 201 155 155
query62 685 648 592 592
query63 231 180 187 180
query64 2557 795 582 582
query65
query66 1798 461 339 339
query67 29668 29620 29483 29483
query68
query69 423 297 264 264
query70 971 960 950 950
query71 296 215 202 202
query72 2931 2641 2360 2360
query73 835 768 435 435
query74 5130 4967 4806 4806
query75 2655 2566 2227 2227
query76 2300 1142 780 780
query77 342 370 296 296
query78 12331 12461 11815 11815
query79 1465 1062 758 758
query80 787 477 365 365
query81 475 278 237 237
query82 563 152 117 117
query83 344 280 247 247
query84
query85 891 513 422 422
query86 411 311 273 273
query87 3372 3320 3161 3161
query88 3578 2709 2703 2703
query89 418 379 331 331
query90 1770 174 174 174
query91 178 159 165 159
query92 65 60 58 58
query93 1437 1475 968 968
query94 602 356 307 307
query95 655 384 363 363
query96 1101 881 338 338
query97 2715 2717 2607 2607
query98 221 207 201 201
query99 1156 1189 1023 1023
Total cold run time: 250999 ms
Total hot run time: 169122 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4674136542
<details>
<summary>TPC-H: <b>Total hot run time: 28685 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7e15d5b3b9109d275feadd1af404dcc86ee9b487, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17744 4275 4026 4026
q2 q3 10665 1391 828 828
q4 4690 477 347 347
q5 7608 879 596 596
q6 190 179 145 145
q7 810 847 651 651
q8 9386 1625 1553 1553
q9 5562 4467 4499 4467
q10 6741 1827 1517 1517
q11 452 280 266 266
q12 629 426 294 294
q13 18141 3438 2802 2802
q14 298 262 247 247
q15 q16 818 772 709 709
q17 5288 1050 785 785
q18 6923 5724 5634 5634
q19 2805 1352 953 953
q20 520 417 266 266
q21 5981 2592 2298 2298
q22 434 356 301 301
Total cold run time: 105685 ms
Total hot run time: 28685 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4434 4276 4314 4276
q2 q3 4514 4906 4306 4306
q4 2097 2201 1401 1401
q5 4419 4323 4695 4323
q6 254 204 145 145
q7 2120 1803 1688 1688
q8 2515 2202 2208 2202
q9 8267 8301 8383 8301
q10 4879 4916 4369 4369
q11 579 445 404 404
q12 796 761 545 545
q13 3451 3789 3147 3147
q14 319 319 290 290
q15 q16 755 769 653 653
q17 1404 1351 1314 1314
q18 7901 7330 6818 6818
q19 1122 1108 1106 1106
q20 2279 2304 1952 1952
q21 5275 4603 4497 4497
q22 519 473 422 422
Total cold run time: 57899 ms
Total hot run time: 52159 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4682187087
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4668765617
Thanks for the thorough review! Addressed in 089459fbaae (review fixes) + 7c898c90050 (num_tasks). Verified against the local-shuffle regression (`test_local_shuffle_rqg_bugs`: 25 bugs, FE==BE, no crash) + the FE planner unit tests; `force_passthrough` additionally checked on a 4-BE cluster.
### @morrySnow
- **[P1] throw instead of log (AddLocalExchange):** restored to `throw` — this was originally an assert, downgraded to a warning while debugging. Verified it never misfires across the regression + unit tests.
- **[P2] make the regression fail the suite (`test_local_shuffle_rqg_bugs`):** Bug 3 now asserts FE==BE (row count + order-insensitive content); Bug 4/5 raise `assertTrue(false)` on an unexpected exception instead of only logging. (Bug 1/2/6/7/8 already asserted in their catch.)
- **[P2] preserve `enable_broadcast_join_force_passthrough` (HashJoinNode):** now mirrored. Investigated on a 4-BE cluster with a forced non-serial probe (`experimental_ignore_storage_data_distribution=false`): it is a **no-op** there — `enforceRequire` only inserts a PASSTHROUGH local exchange to fan a *serial* (1-task) source out to N tasks; an already-N-task source satisfies passthrough, so no exchange is added (identical plan and results vs BE-native, no crash). The check is kept to match BE's intent; a true rebalance of a non-serial probe (forcing the exchange) is a perf-only follow-up.
- **🟡 `resolveExchangeType` unused `child` param:** removed all three unused params (`translatorContext`/`parent`/`child`) — only `require` is read. Updated all 6 call sites + the unit test (also resolves the SetOperationNode null-`firstChild` note).
- **🟠 `_propagate_local_exchange_num_tasks` convergence loops:** replaced the two iterative `while (changed)` passes with a single Kahn topological sweep over the pipeline DAG — each pipeline is visited once, after all its upstreams, so num_tasks is set in one pass. It can't loop infinitely; added a cycle `DCHECK` instead of a max-iteration guard. (Commit 7c898c90050.)
- **🟡 build-side comment (HashJoinNode):** added a note that both probe and build sides require `BUCKET_HASH_SHUFFLE`.
- **🟢 `_operator_id` starting from -1 / ✅ AggregationNode correctness:** thanks — acknowledged, no change needed.
### @BiteTheDDDDt
- **`local_exchange_sink_operator` redundancy with `init_partitioner`:** extracted `_create_partitioner(state, bucket_count)`, now shared by `init()` (BE-native) and `init_partitioner()` (FE-planned) — removes the three duplicated hash-partitioner blocks.
- **BE computing `required_data_distribution` / parallelism:** retained for the **BE-native fallback** path (`enable_local_shuffle_planner=false`). The FE-planned path doesn't read it, but keeping it preserves the BE-native planner as a fallback option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4708089765
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4738677003
# BE UT Coverage Report
Increment line coverage `15.27% (62/406)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/0436cc4cb573b104a59408a6169bd145bcfeff6c_0436cc4cb573b104a59408a6169bd145bcfeff6c/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/0436cc4cb573b104a59408a6169bd145bcfeff6c_0436cc4cb573b104a59408a6169bd145bcfeff6c/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 54.45% (21370/39248) |
| Line Coverage | 38.06% (204368/536967) |
| Region Coverage | 34.07% (160400/470836) |
| Branch Coverage | 35.05% (70191/200259) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3417864896
##########
fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java:
##########
@@ -939,4 +1031,263 @@ private String mergeIcebergAccessPathsWithId(
}
return StringUtils.join(mergeDisplayAccessPaths, ", ");
}
+
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ ArrayList<PlanNode> newChildren = Lists.newArrayList();
+ for (int i = 0; i < children.size(); i++) {
+ Pair<PlanNode, LocalExchangeType> childOutput
+ = enforceRequire(translatorContext, children.get(i), i, LocalExchangeTypeRequire.noRequire());
+ newChildren.add(childOutput.first);
+ }
+ this.children = newChildren;
+ return Pair.of(this, LocalExchangeType.NOOP);
+ }
+
+ /**
+ * Unified framework method: propagate serial flag → recurse child → satisfy check → Layer 1 skip → insert LE.
+ * Replaces the old enforceChild/enforceChildExchange/forceEnforceChildExchange trio.
+ *
+ * <h3>Data flow</h3>
+ * <ul>
+ * <li><b>serial-ancestor flag</b> ({@link PlanTranslatorContext#hasSerialAncestorInPipeline})
+ * — flows root → leaf during traversal. Mirrors BE's
+ * {@code any_of(operators[idx..end], is_serial_operator)} check used by
+ * {@code _add_local_exchange} to skip LE insertion when an ancestor in the same
+ * pipeline is already serial. Reset at pipeline boundaries via
+ * {@link #shouldResetSerialFlagForChild}.</li>
+ * <li><b>shuffle-for-correctness flag</b>
+ * ({@link PlanTranslatorContext#hasShuffleForCorrectnessAncestor}) — also flows
+ * root → leaf. Mirrors BE's {@code _followed_by_shuffled_operator}: tells a
+ * child whether some downstream operator depends on hash distribution for
+ * correctness, so {@code SetOperationNode} can pre-shuffle union branches.</li>
+ * <li><b>return value</b> {@code Pair<PlanNode, LocalExchangeType>} — first is the
+ * (possibly LE-wrapped) child; second is the actual output distribution as
+ * observed by the parent. Caller's {@code require.satisfy(output)} decides
+ * whether more LE is needed.</li>
+ * <li><b>parent.require</b> describes the constraint on the child output —
+ * computed inside the parent's {@code enforceAndDeriveLocalExchange} per child.</li>
+ * </ul>
+ *
+ * <h3>Invariants</h3>
+ * <ul>
+ * <li>Where a serial → non-serial transition needs redistribution, framework step 3 inserts
+ * the LE (e.g. a serial source fanned out via PASSTHROUGH). This is not a hard invariant:
+ * a serial child feeding a parent that requires PASSTHROUGH / noRequire (TableFunction,
+ * NLJ, Agg) is already correct and needs no LE, so it is intentionally not validated by a
+ * post-pass.</li>
+ * <li>{@code LocalExchangeNode} itself is always non-serial — setting it serial
+ * would defeat its purpose of fanning a 1-task pipeline back to N tasks.</li>
+ * <li>For pipeline-breaking parents ({@code shouldResetSerialFlagForChild=true}),
+ * the child starts a fresh pipeline so {@code hasSerialAncestor} is reset; the
+ * node's own {@code isSerialNode()} still composes in for the child's view.</li>
+ * <li>{@code RequireHash} accepts any hash flavour; {@code RequireSpecific} demands
+ * an exact match (with the one PASSTHROUGH/ADAPTIVE_PASSTHROUGH compatibility).
+ * Pick the looser one whenever correctness allows — see
+ * {@link LocalExchangeNode.LocalExchangeTypeRequire}.</li>
+ * </ul>
+ *
+ * <h3>Layers</h3>
+ * Layer 1 (shouldSkipLE): mirrors BE's need_to_local_exchange — skip when this node or
+ * an ancestor in the same pipeline is serial (operators[idx..end] has serial → skip).
+ * Layer 2 (require/output): each Node declares require and output in enforceAndDeriveLocalExchange.
+ */
+ protected Pair<PlanNode, LocalExchangeType> enforceRequire(
+ PlanTranslatorContext translatorContext, PlanNode child, int childIndex,
+ LocalExchangeTypeRequire require) {
+ // 1. Propagate serial-ancestor flag to child.
+ // For pipeline-splitting operators (shouldReset=true, e.g. non-streaming AGG):
+ // Drop inherited serial flag from parent (parent is in a different pipeline),
+ // but keep this node's own serial status (child is in the same pipeline as this
+ // node's sink, e.g. Exchange is in AGG_Sink pipeline).
+ // For non-splitting operators (shouldReset=false, e.g. streaming AGG):
+ // Inherit parent's serial flag + this node's own.
+ boolean inheritedSerial = shouldResetSerialFlagForChild(childIndex)
+ ? false : translatorContext.hasSerialAncestorInPipeline(this);
+ // Use isSerialOperatorOnBe (= isSerialNode && fragment.useSerialSource) instead of the
+ // raw isSerialNode(). BE's OperatorBase reads the Thrift `is_serial_operator` flag —
+ // which is what FE writes via isSerialOperatorOnBe — so when the fragment is not in
+ // serial-source mode, BE treats this operator as non-serial regardless of isSerialNode.
+ // Using isSerialNode here would set the child's serial-ancestor flag wider than BE's
+ // view and over-skip required LocalExchanges downstream.
+ boolean childHasSerialAncestor = inheritedSerial
+ || isSerialOperatorOnBe(translatorContext.getConnectContext());
+ translatorContext.setHasSerialAncestorInPipeline(child, childHasSerialAncestor);
+
+ // 1b. Propagate shuffle-for-correctness-ancestor flag to child.
+ // Mirrors BE's _followed_by_shuffled_operator: a downstream operator needs hash
+ // distribution for correctness, and the chain to here goes through HASH or NOOP
+ // requirements (so the dependency is preserved).
+ // propagate = ((inheritedShuffled || self.requiresShuffleForCorrectness)
+ // && require is hash)
+ // || (inheritedShuffled && require is noop/passthrough)
+ boolean inheritedShuffled = translatorContext.hasShuffleForCorrectnessAncestor(this);
+ boolean selfOrInheritedShuffled = inheritedShuffled || requiresShuffleForCorrectness();
+ boolean requireIsHash = require.preferType().isHashShuffle();
+ boolean requireIsNoop = require.preferType() == LocalExchangeNode.LocalExchangeType.NOOP;
+ boolean childShuffledAncestor = (selfOrInheritedShuffled && requireIsHash)
+ || (inheritedShuffled && requireIsNoop);
+ translatorContext.setHasShuffleForCorrectnessAncestor(child, childShuffledAncestor);
+
+ // 2. Recurse child (Layer 2: child declares its own require/output)
+ Pair<PlanNode, LocalExchangeType> childOutput =
+ child.enforceAndDeriveLocalExchange(translatorContext, this, require);
+
+ // Steps 2.5 and 3 both react to a serial child but address different concerns:
+ // - Step 2.5 rewrites the OUTPUT-side view (what we tell satisfy/parent about
+ // the child's actual distribution). A serial pipeline runs with 1 task so
+ // its distribution claim is meaningless — flatten to NOOP so the satisfy
+ // check below doesn't get fooled by a stale "I output BUCKET_HASH" claim.
+ // - Step 3 rewrites the REQUIRE-side decision (what we want from the child).
+ // If we previously asked for nothing (noRequire) but the child turns out
+ // to be serial and we're not, upgrade to requirePassthrough so an LE is
+ // inserted to restore parallelism.
+
+ // 2.5. Serial child override (output side): if child is serial on BE, force its
+ // reported output to NOOP. Distribution is irrelevant when the child runs
+ // with 1 task; downstream parallelism is restored either by step 3 (LE
+ // insertion) or skipped entirely by step 4b (we're also serial).
+ if (childOutput.first.isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ childOutput = Pair.of(childOutput.first, LocalExchangeType.NOOP);
+ }
+
+ // 3. Framework-level serial child check (require side, mirrors BE base class
+ // required_data_distribution): if child will be serial on BE but this node is
+ // not serial, the pipeline has a 1-task serial child feeding an N-task non-serial
+ // parent. Without LE, pipeline splits (AGG/JOIN) create paired pipelines with
+ // mismatched num_tasks → crash. Upgrade noRequire to requirePassthrough so an
+ // LE is inserted below to restore parallelism.
+ if (require instanceof LocalExchangeNode.NoRequire
+ && childOutput.first.isSerialOperatorOnBe(translatorContext.getConnectContext())
+ && !isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ require = LocalExchangeTypeRequire.requirePassthrough();
+ }
+
+ // 4. Satisfy check: child output meets requirement → done
+ if (require.satisfy(childOutput.second)) {
+ return childOutput;
+ }
+
+ // 4. Layer 1: skip LE when serial operator or ancestor in same pipeline
+ // Equivalent to BE's need_to_local_exchange: any_of(operators[idx..end], is_serial) → skip.
+ // Use isSerialOperatorOnBe (not isSerialNode) because BE's Pipeline::need_to_local_exchange
+ // checks op->is_serial_operator() which reads the Thrift flag set from isSerialOperatorOnBe;
+ // when fragment.useSerialSource is false, BE treats this node as non-serial.
+ if (translatorContext.hasSerialAncestorInPipeline(this)
Review Comment:
This mirrors BE's `Pipeline::need_to_local_exchange` exactly (`pipeline.cpp:41-45`):
```cpp
// If serial operator exists after `idx`-th operator, we should not improve parallelism.
if (std::any_of(_operators.begin() + idx, _operators.end(),
[](OperatorPtr op){ return op->is_serial_operator(); })) {
return false;
}
```
`hasSerialAncestorInPipeline(this)` = a serial operator exists between this node and the pipeline's sink (i.e. an operator after `idx`); `isSerialOperatorOnBe(this)` = the operator at `idx` is serial. Together they are `any_of(operators[idx..end], is_serial)`. The reason it skips the LE: if any operator from here toward the sink is serial, that downstream segment runs with a single task — there is no parallelism to rebalance, so a local exchange would be pointless (and BE skips it for the same reason).
##########
fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java:
##########
@@ -939,4 +1031,263 @@ private String mergeIcebergAccessPathsWithId(
}
return StringUtils.join(mergeDisplayAccessPaths, ", ");
}
+
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ ArrayList<PlanNode> newChildren = Lists.newArrayList();
+ for (int i = 0; i < children.size(); i++) {
+ Pair<PlanNode, LocalExchangeType> childOutput
+ = enforceRequire(translatorContext, children.get(i), i, LocalExchangeTypeRequire.noRequire());
+ newChildren.add(childOutput.first);
+ }
+ this.children = newChildren;
+ return Pair.of(this, LocalExchangeType.NOOP);
+ }
+
+ /**
+ * Unified framework method: propagate serial flag → recurse child → satisfy check → Layer 1 skip → insert LE.
+ * Replaces the old enforceChild/enforceChildExchange/forceEnforceChildExchange trio.
+ *
+ * <h3>Data flow</h3>
+ * <ul>
+ * <li><b>serial-ancestor flag</b> ({@link PlanTranslatorContext#hasSerialAncestorInPipeline})
+ * — flows root → leaf during traversal. Mirrors BE's
+ * {@code any_of(operators[idx..end], is_serial_operator)} check used by
+ * {@code _add_local_exchange} to skip LE insertion when an ancestor in the same
+ * pipeline is already serial. Reset at pipeline boundaries via
+ * {@link #shouldResetSerialFlagForChild}.</li>
+ * <li><b>shuffle-for-correctness flag</b>
+ * ({@link PlanTranslatorContext#hasShuffleForCorrectnessAncestor}) — also flows
+ * root → leaf. Mirrors BE's {@code _followed_by_shuffled_operator}: tells a
+ * child whether some downstream operator depends on hash distribution for
+ * correctness, so {@code SetOperationNode} can pre-shuffle union branches.</li>
+ * <li><b>return value</b> {@code Pair<PlanNode, LocalExchangeType>} — first is the
+ * (possibly LE-wrapped) child; second is the actual output distribution as
+ * observed by the parent. Caller's {@code require.satisfy(output)} decides
+ * whether more LE is needed.</li>
+ * <li><b>parent.require</b> describes the constraint on the child output —
+ * computed inside the parent's {@code enforceAndDeriveLocalExchange} per child.</li>
+ * </ul>
+ *
+ * <h3>Invariants</h3>
+ * <ul>
+ * <li>Where a serial → non-serial transition needs redistribution, framework step 3 inserts
+ * the LE (e.g. a serial source fanned out via PASSTHROUGH). This is not a hard invariant:
+ * a serial child feeding a parent that requires PASSTHROUGH / noRequire (TableFunction,
+ * NLJ, Agg) is already correct and needs no LE, so it is intentionally not validated by a
+ * post-pass.</li>
+ * <li>{@code LocalExchangeNode} itself is always non-serial — setting it serial
+ * would defeat its purpose of fanning a 1-task pipeline back to N tasks.</li>
+ * <li>For pipeline-breaking parents ({@code shouldResetSerialFlagForChild=true}),
+ * the child starts a fresh pipeline so {@code hasSerialAncestor} is reset; the
+ * node's own {@code isSerialNode()} still composes in for the child's view.</li>
+ * <li>{@code RequireHash} accepts any hash flavour; {@code RequireSpecific} demands
+ * an exact match (with the one PASSTHROUGH/ADAPTIVE_PASSTHROUGH compatibility).
+ * Pick the looser one whenever correctness allows — see
+ * {@link LocalExchangeNode.LocalExchangeTypeRequire}.</li>
+ * </ul>
+ *
+ * <h3>Layers</h3>
+ * Layer 1 (shouldSkipLE): mirrors BE's need_to_local_exchange — skip when this node or
+ * an ancestor in the same pipeline is serial (operators[idx..end] has serial → skip).
+ * Layer 2 (require/output): each Node declares require and output in enforceAndDeriveLocalExchange.
+ */
+ protected Pair<PlanNode, LocalExchangeType> enforceRequire(
+ PlanTranslatorContext translatorContext, PlanNode child, int childIndex,
+ LocalExchangeTypeRequire require) {
+ // 1. Propagate serial-ancestor flag to child.
+ // For pipeline-splitting operators (shouldReset=true, e.g. non-streaming AGG):
+ // Drop inherited serial flag from parent (parent is in a different pipeline),
+ // but keep this node's own serial status (child is in the same pipeline as this
+ // node's sink, e.g. Exchange is in AGG_Sink pipeline).
+ // For non-splitting operators (shouldReset=false, e.g. streaming AGG):
+ // Inherit parent's serial flag + this node's own.
+ boolean inheritedSerial = shouldResetSerialFlagForChild(childIndex)
+ ? false : translatorContext.hasSerialAncestorInPipeline(this);
+ // Use isSerialOperatorOnBe (= isSerialNode && fragment.useSerialSource) instead of the
+ // raw isSerialNode(). BE's OperatorBase reads the Thrift `is_serial_operator` flag —
+ // which is what FE writes via isSerialOperatorOnBe — so when the fragment is not in
+ // serial-source mode, BE treats this operator as non-serial regardless of isSerialNode.
+ // Using isSerialNode here would set the child's serial-ancestor flag wider than BE's
+ // view and over-skip required LocalExchanges downstream.
+ boolean childHasSerialAncestor = inheritedSerial
+ || isSerialOperatorOnBe(translatorContext.getConnectContext());
+ translatorContext.setHasSerialAncestorInPipeline(child, childHasSerialAncestor);
+
+ // 1b. Propagate shuffle-for-correctness-ancestor flag to child.
+ // Mirrors BE's _followed_by_shuffled_operator: a downstream operator needs hash
+ // distribution for correctness, and the chain to here goes through HASH or NOOP
+ // requirements (so the dependency is preserved).
+ // propagate = ((inheritedShuffled || self.requiresShuffleForCorrectness)
+ // && require is hash)
+ // || (inheritedShuffled && require is noop/passthrough)
+ boolean inheritedShuffled = translatorContext.hasShuffleForCorrectnessAncestor(this);
+ boolean selfOrInheritedShuffled = inheritedShuffled || requiresShuffleForCorrectness();
+ boolean requireIsHash = require.preferType().isHashShuffle();
+ boolean requireIsNoop = require.preferType() == LocalExchangeNode.LocalExchangeType.NOOP;
+ boolean childShuffledAncestor = (selfOrInheritedShuffled && requireIsHash)
+ || (inheritedShuffled && requireIsNoop);
+ translatorContext.setHasShuffleForCorrectnessAncestor(child, childShuffledAncestor);
+
+ // 2. Recurse child (Layer 2: child declares its own require/output)
+ Pair<PlanNode, LocalExchangeType> childOutput =
+ child.enforceAndDeriveLocalExchange(translatorContext, this, require);
+
+ // Steps 2.5 and 3 both react to a serial child but address different concerns:
+ // - Step 2.5 rewrites the OUTPUT-side view (what we tell satisfy/parent about
+ // the child's actual distribution). A serial pipeline runs with 1 task so
+ // its distribution claim is meaningless — flatten to NOOP so the satisfy
+ // check below doesn't get fooled by a stale "I output BUCKET_HASH" claim.
+ // - Step 3 rewrites the REQUIRE-side decision (what we want from the child).
+ // If we previously asked for nothing (noRequire) but the child turns out
+ // to be serial and we're not, upgrade to requirePassthrough so an LE is
+ // inserted to restore parallelism.
+
+ // 2.5. Serial child override (output side): if child is serial on BE, force its
+ // reported output to NOOP. Distribution is irrelevant when the child runs
+ // with 1 task; downstream parallelism is restored either by step 3 (LE
+ // insertion) or skipped entirely by step 4b (we're also serial).
+ if (childOutput.first.isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ childOutput = Pair.of(childOutput.first, LocalExchangeType.NOOP);
+ }
+
+ // 3. Framework-level serial child check (require side, mirrors BE base class
+ // required_data_distribution): if child will be serial on BE but this node is
+ // not serial, the pipeline has a 1-task serial child feeding an N-task non-serial
+ // parent. Without LE, pipeline splits (AGG/JOIN) create paired pipelines with
+ // mismatched num_tasks → crash. Upgrade noRequire to requirePassthrough so an
+ // LE is inserted below to restore parallelism.
+ if (require instanceof LocalExchangeNode.NoRequire
+ && childOutput.first.isSerialOperatorOnBe(translatorContext.getConnectContext())
+ && !isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ require = LocalExchangeTypeRequire.requirePassthrough();
+ }
+
+ // 4. Satisfy check: child output meets requirement → done
+ if (require.satisfy(childOutput.second)) {
+ return childOutput;
+ }
+
+ // 4. Layer 1: skip LE when serial operator or ancestor in same pipeline
+ // Equivalent to BE's need_to_local_exchange: any_of(operators[idx..end], is_serial) → skip.
+ // Use isSerialOperatorOnBe (not isSerialNode) because BE's Pipeline::need_to_local_exchange
+ // checks op->is_serial_operator() which reads the Thrift flag set from isSerialOperatorOnBe;
+ // when fragment.useSerialSource is false, BE treats this node as non-serial.
+ if (translatorContext.hasSerialAncestorInPipeline(this)
+ || isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ return childOutput;
+ }
+
+ // 5. Resolve exchange type and create LE node
+ LocalExchangeType preferType = AddLocalExchange.resolveExchangeType(require);
+ List<Expr> distributeExprs = getLocalExchangeDistributeExprs(childIndex, selfOrInheritedShuffled);
+ PlanNode leNode = createLocalExchange(translatorContext, childOutput.first, preferType, distributeExprs);
+ return Pair.of(leNode, preferType);
+ }
+
+ /**
+ * Create a LocalExchangeNode wrapping child with the given exchange type.
+ * No child-type skip — matches BE's _add_local_exchange which inserts LE for any child
+ * type without checking instanceof.
+ *
+ * Handles heavy-ops bottleneck avoidance (mirrors BE pipeline_fragment_context.cpp):
+ * when upstream has 1 task (serial source) and exchange is heavy (hash/bucket/adaptive),
+ * insert a PASSTHROUGH fan-out first to avoid single-task bottleneck on the heavy
+ * exchange sink. Only applies to local-shuffle (pooling scan) fragments.
+ */
+ protected PlanNode createLocalExchange(PlanTranslatorContext translatorContext,
+ PlanNode child, LocalExchangeType exchangeType, List<Expr> distributeExprs) {
+ if (fragment != null && fragment.useSerialSource(translatorContext.getConnectContext())
+ && exchangeType.isHeavyOperation() && child.isSerialNode()) {
+ PlanNode ptNode = new LocalExchangeNode(translatorContext.nextPlanNodeId(),
+ child, LocalExchangeType.PASSTHROUGH, null);
+ return new LocalExchangeNode(translatorContext.nextPlanNodeId(), ptNode,
+ exchangeType, distributeExprs);
+ }
+ return new LocalExchangeNode(translatorContext.nextPlanNodeId(), child,
+ exchangeType, distributeExprs);
+ }
+
+ /**
+ * Whether the child at {@code childIndex} starts a new pipeline context, causing
+ * its serial-ancestor flag to be reset to {@code false} rather than inherited from this node.
+ * Override to return {@code true} for pipeline-splitting nodes (LocalExchangeNode) and nodes
+ * whose children run in an independent pipeline segment (SortNode above analytic, etc.).
Review Comment:
Good catch — you're right, fixed in `8f1befd3016` (`above` → `before`). The window-input `SortNode` is the **child** of `AnalyticEvalNode`: `PhysicalPlanTranslator.visitPhysicalWindow` does `inputPlanFragment.addPlanRoot(analyticEvalNode)`, which stacks the analytic node on top of its input sort. So the SortNode is below/before the analytic, not above it.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java:
##########
@@ -939,4 +1031,263 @@ private String mergeIcebergAccessPathsWithId(
}
return StringUtils.join(mergeDisplayAccessPaths, ", ");
}
+
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ ArrayList<PlanNode> newChildren = Lists.newArrayList();
+ for (int i = 0; i < children.size(); i++) {
+ Pair<PlanNode, LocalExchangeType> childOutput
+ = enforceRequire(translatorContext, children.get(i), i, LocalExchangeTypeRequire.noRequire());
+ newChildren.add(childOutput.first);
+ }
+ this.children = newChildren;
+ return Pair.of(this, LocalExchangeType.NOOP);
+ }
+
+ /**
+ * Unified framework method: propagate serial flag → recurse child → satisfy check → Layer 1 skip → insert LE.
+ * Replaces the old enforceChild/enforceChildExchange/forceEnforceChildExchange trio.
+ *
+ * <h3>Data flow</h3>
+ * <ul>
+ * <li><b>serial-ancestor flag</b> ({@link PlanTranslatorContext#hasSerialAncestorInPipeline})
+ * — flows root → leaf during traversal. Mirrors BE's
+ * {@code any_of(operators[idx..end], is_serial_operator)} check used by
+ * {@code _add_local_exchange} to skip LE insertion when an ancestor in the same
+ * pipeline is already serial. Reset at pipeline boundaries via
+ * {@link #shouldResetSerialFlagForChild}.</li>
+ * <li><b>shuffle-for-correctness flag</b>
+ * ({@link PlanTranslatorContext#hasShuffleForCorrectnessAncestor}) — also flows
+ * root → leaf. Mirrors BE's {@code _followed_by_shuffled_operator}: tells a
+ * child whether some downstream operator depends on hash distribution for
+ * correctness, so {@code SetOperationNode} can pre-shuffle union branches.</li>
+ * <li><b>return value</b> {@code Pair<PlanNode, LocalExchangeType>} — first is the
+ * (possibly LE-wrapped) child; second is the actual output distribution as
+ * observed by the parent. Caller's {@code require.satisfy(output)} decides
+ * whether more LE is needed.</li>
+ * <li><b>parent.require</b> describes the constraint on the child output —
+ * computed inside the parent's {@code enforceAndDeriveLocalExchange} per child.</li>
+ * </ul>
+ *
+ * <h3>Invariants</h3>
+ * <ul>
+ * <li>Where a serial → non-serial transition needs redistribution, framework step 3 inserts
+ * the LE (e.g. a serial source fanned out via PASSTHROUGH). This is not a hard invariant:
+ * a serial child feeding a parent that requires PASSTHROUGH / noRequire (TableFunction,
+ * NLJ, Agg) is already correct and needs no LE, so it is intentionally not validated by a
+ * post-pass.</li>
+ * <li>{@code LocalExchangeNode} itself is always non-serial — setting it serial
+ * would defeat its purpose of fanning a 1-task pipeline back to N tasks.</li>
+ * <li>For pipeline-breaking parents ({@code shouldResetSerialFlagForChild=true}),
+ * the child starts a fresh pipeline so {@code hasSerialAncestor} is reset; the
+ * node's own {@code isSerialNode()} still composes in for the child's view.</li>
+ * <li>{@code RequireHash} accepts any hash flavour; {@code RequireSpecific} demands
+ * an exact match (with the one PASSTHROUGH/ADAPTIVE_PASSTHROUGH compatibility).
+ * Pick the looser one whenever correctness allows — see
+ * {@link LocalExchangeNode.LocalExchangeTypeRequire}.</li>
+ * </ul>
+ *
+ * <h3>Layers</h3>
+ * Layer 1 (shouldSkipLE): mirrors BE's need_to_local_exchange — skip when this node or
+ * an ancestor in the same pipeline is serial (operators[idx..end] has serial → skip).
+ * Layer 2 (require/output): each Node declares require and output in enforceAndDeriveLocalExchange.
+ */
+ protected Pair<PlanNode, LocalExchangeType> enforceRequire(
+ PlanTranslatorContext translatorContext, PlanNode child, int childIndex,
+ LocalExchangeTypeRequire require) {
+ // 1. Propagate serial-ancestor flag to child.
+ // For pipeline-splitting operators (shouldReset=true, e.g. non-streaming AGG):
+ // Drop inherited serial flag from parent (parent is in a different pipeline),
+ // but keep this node's own serial status (child is in the same pipeline as this
+ // node's sink, e.g. Exchange is in AGG_Sink pipeline).
+ // For non-splitting operators (shouldReset=false, e.g. streaming AGG):
+ // Inherit parent's serial flag + this node's own.
+ boolean inheritedSerial = shouldResetSerialFlagForChild(childIndex)
+ ? false : translatorContext.hasSerialAncestorInPipeline(this);
+ // Use isSerialOperatorOnBe (= isSerialNode && fragment.useSerialSource) instead of the
+ // raw isSerialNode(). BE's OperatorBase reads the Thrift `is_serial_operator` flag —
+ // which is what FE writes via isSerialOperatorOnBe — so when the fragment is not in
+ // serial-source mode, BE treats this operator as non-serial regardless of isSerialNode.
+ // Using isSerialNode here would set the child's serial-ancestor flag wider than BE's
+ // view and over-skip required LocalExchanges downstream.
+ boolean childHasSerialAncestor = inheritedSerial
+ || isSerialOperatorOnBe(translatorContext.getConnectContext());
+ translatorContext.setHasSerialAncestorInPipeline(child, childHasSerialAncestor);
+
+ // 1b. Propagate shuffle-for-correctness-ancestor flag to child.
+ // Mirrors BE's _followed_by_shuffled_operator: a downstream operator needs hash
+ // distribution for correctness, and the chain to here goes through HASH or NOOP
+ // requirements (so the dependency is preserved).
+ // propagate = ((inheritedShuffled || self.requiresShuffleForCorrectness)
+ // && require is hash)
+ // || (inheritedShuffled && require is noop/passthrough)
+ boolean inheritedShuffled = translatorContext.hasShuffleForCorrectnessAncestor(this);
+ boolean selfOrInheritedShuffled = inheritedShuffled || requiresShuffleForCorrectness();
+ boolean requireIsHash = require.preferType().isHashShuffle();
+ boolean requireIsNoop = require.preferType() == LocalExchangeNode.LocalExchangeType.NOOP;
+ boolean childShuffledAncestor = (selfOrInheritedShuffled && requireIsHash)
+ || (inheritedShuffled && requireIsNoop);
+ translatorContext.setHasShuffleForCorrectnessAncestor(child, childShuffledAncestor);
+
+ // 2. Recurse child (Layer 2: child declares its own require/output)
+ Pair<PlanNode, LocalExchangeType> childOutput =
+ child.enforceAndDeriveLocalExchange(translatorContext, this, require);
+
+ // Steps 2.5 and 3 both react to a serial child but address different concerns:
+ // - Step 2.5 rewrites the OUTPUT-side view (what we tell satisfy/parent about
+ // the child's actual distribution). A serial pipeline runs with 1 task so
+ // its distribution claim is meaningless — flatten to NOOP so the satisfy
+ // check below doesn't get fooled by a stale "I output BUCKET_HASH" claim.
+ // - Step 3 rewrites the REQUIRE-side decision (what we want from the child).
+ // If we previously asked for nothing (noRequire) but the child turns out
+ // to be serial and we're not, upgrade to requirePassthrough so an LE is
+ // inserted to restore parallelism.
+
+ // 2.5. Serial child override (output side): if child is serial on BE, force its
+ // reported output to NOOP. Distribution is irrelevant when the child runs
+ // with 1 task; downstream parallelism is restored either by step 3 (LE
+ // insertion) or skipped entirely by step 4b (we're also serial).
+ if (childOutput.first.isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ childOutput = Pair.of(childOutput.first, LocalExchangeType.NOOP);
+ }
+
+ // 3. Framework-level serial child check (require side, mirrors BE base class
+ // required_data_distribution): if child will be serial on BE but this node is
+ // not serial, the pipeline has a 1-task serial child feeding an N-task non-serial
+ // parent. Without LE, pipeline splits (AGG/JOIN) create paired pipelines with
+ // mismatched num_tasks → crash. Upgrade noRequire to requirePassthrough so an
+ // LE is inserted below to restore parallelism.
+ if (require instanceof LocalExchangeNode.NoRequire
+ && childOutput.first.isSerialOperatorOnBe(translatorContext.getConnectContext())
+ && !isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ require = LocalExchangeTypeRequire.requirePassthrough();
+ }
+
+ // 4. Satisfy check: child output meets requirement → done
+ if (require.satisfy(childOutput.second)) {
+ return childOutput;
+ }
+
+ // 4. Layer 1: skip LE when serial operator or ancestor in same pipeline
+ // Equivalent to BE's need_to_local_exchange: any_of(operators[idx..end], is_serial) → skip.
+ // Use isSerialOperatorOnBe (not isSerialNode) because BE's Pipeline::need_to_local_exchange
+ // checks op->is_serial_operator() which reads the Thrift flag set from isSerialOperatorOnBe;
+ // when fragment.useSerialSource is false, BE treats this node as non-serial.
+ if (translatorContext.hasSerialAncestorInPipeline(this)
+ || isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ return childOutput;
+ }
+
+ // 5. Resolve exchange type and create LE node
+ LocalExchangeType preferType = AddLocalExchange.resolveExchangeType(require);
+ List<Expr> distributeExprs = getLocalExchangeDistributeExprs(childIndex, selfOrInheritedShuffled);
+ PlanNode leNode = createLocalExchange(translatorContext, childOutput.first, preferType, distributeExprs);
+ return Pair.of(leNode, preferType);
+ }
+
+ /**
+ * Create a LocalExchangeNode wrapping child with the given exchange type.
+ * No child-type skip — matches BE's _add_local_exchange which inserts LE for any child
+ * type without checking instanceof.
+ *
+ * Handles heavy-ops bottleneck avoidance (mirrors BE pipeline_fragment_context.cpp):
+ * when upstream has 1 task (serial source) and exchange is heavy (hash/bucket/adaptive),
+ * insert a PASSTHROUGH fan-out first to avoid single-task bottleneck on the heavy
+ * exchange sink. Only applies to local-shuffle (pooling scan) fragments.
+ */
+ protected PlanNode createLocalExchange(PlanTranslatorContext translatorContext,
+ PlanNode child, LocalExchangeType exchangeType, List<Expr> distributeExprs) {
+ if (fragment != null && fragment.useSerialSource(translatorContext.getConnectContext())
+ && exchangeType.isHeavyOperation() && child.isSerialNode()) {
+ PlanNode ptNode = new LocalExchangeNode(translatorContext.nextPlanNodeId(),
+ child, LocalExchangeType.PASSTHROUGH, null);
+ return new LocalExchangeNode(translatorContext.nextPlanNodeId(), ptNode,
+ exchangeType, distributeExprs);
+ }
+ return new LocalExchangeNode(translatorContext.nextPlanNodeId(), child,
+ exchangeType, distributeExprs);
+ }
+
+ /**
+ * Whether the child at {@code childIndex} starts a new pipeline context, causing
+ * its serial-ancestor flag to be reset to {@code false} rather than inherited from this node.
+ * Override to return {@code true} for pipeline-splitting nodes (LocalExchangeNode) and nodes
+ * whose children run in an independent pipeline segment (SortNode above analytic, etc.).
+ */
+ protected boolean shouldResetSerialFlagForChild(int childIndex) {
Review Comment:
Yes — `shouldResetSerialFlagForChild` defines FE's pipeline breakers, and it must match BE's pipeline construction. The serial-ancestor flag (`hasSerialAncestorInPipeline`, which drives the Layer-1 LE skip above) is computed per pipeline, so if FE and BE disagree on where a pipeline ends, the flag — and thus LE insertion — diverges from BE. The nodes that override it (`SortNode`, `AnalyticEvalNode`, `HashJoinNode`, `NestedLoopJoinNode`, `AggregationNode`, `SetOperationNode`, `PartitionSortNode`, `RecursiveCteNode`, `LocalExchangeNode`) are exactly BE's pipeline-breaking operators (sort sink, analytic, join build, agg sink, set op, …).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4702158070
# FE Regression Coverage Report
Increment line coverage ` 80.17% (461/575)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_8db568b5100ed397fc6347033402642e90e5509e_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_8db568b5100ed397fc6347033402642e90e5509e_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4708776883
<details>
<summary>TPC-H: <b>Total hot run time: 29190 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fe9a3316be056faed18351cf64096e36790c6c89, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17590 4182 4104 4104
q2 q3 10761 1445 847 847
q4 4689 480 338 338
q5 7540 889 583 583
q6 187 174 140 140
q7 797 863 632 632
q8 9388 1579 1636 1579
q9 5661 4519 4585 4519
q10 6761 1811 1523 1523
q11 441 270 262 262
q12 636 427 288 288
q13 18098 3531 2861 2861
q14 271 255 242 242
q15 q16 785 788 712 712
q17 888 899 991 899
q18 6965 5781 5579 5579
q19 1161 1292 1103 1103
q20 494 409 261 261
q21 5240 2628 2416 2416
q22 441 360 302 302
Total cold run time: 98794 ms
Total hot run time: 29190 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4471 4343 4377 4343
q2 q3 4531 4958 4439 4439
q4 2099 2218 1386 1386
q5 4521 4323 4319 4319
q6 224 178 132 132
q7 1738 1680 1604 1604
q8 2897 2256 2165 2165
q9 8351 8373 8233 8233
q10 4833 4758 4350 4350
q11 563 443 376 376
q12 741 757 542 542
q13 3319 3686 2979 2979
q14 300 303 287 287
q15 q16 728 760 633 633
q17 1372 1348 1332 1332
q18 8095 7429 7344 7344
q19 1180 1122 1135 1122
q20 2238 2223 1953 1953
q21 5311 4569 4464 4464
q22 573 446 408 408
Total cold run time: 58085 ms
Total hot run time: 52411 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3414779422
##########
fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java:
##########
@@ -939,4 +1031,263 @@ private String mergeIcebergAccessPathsWithId(
}
return StringUtils.join(mergeDisplayAccessPaths, ", ");
}
+
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ ArrayList<PlanNode> newChildren = Lists.newArrayList();
+ for (int i = 0; i < children.size(); i++) {
+ Pair<PlanNode, LocalExchangeType> childOutput
+ = enforceRequire(translatorContext, children.get(i), i, LocalExchangeTypeRequire.noRequire());
+ newChildren.add(childOutput.first);
+ }
+ this.children = newChildren;
+ return Pair.of(this, LocalExchangeType.NOOP);
+ }
+
+ /**
+ * Unified framework method: propagate serial flag → recurse child → satisfy check → Layer 1 skip → insert LE.
+ * Replaces the old enforceChild/enforceChildExchange/forceEnforceChildExchange trio.
+ *
+ * <h3>Data flow</h3>
+ * <ul>
+ * <li><b>serial-ancestor flag</b> ({@link PlanTranslatorContext#hasSerialAncestorInPipeline})
+ * — flows root → leaf during traversal. Mirrors BE's
+ * {@code any_of(operators[idx..end], is_serial_operator)} check used by
+ * {@code _add_local_exchange} to skip LE insertion when an ancestor in the same
+ * pipeline is already serial. Reset at pipeline boundaries via
+ * {@link #shouldResetSerialFlagForChild}.</li>
+ * <li><b>shuffle-for-correctness flag</b>
+ * ({@link PlanTranslatorContext#hasShuffleForCorrectnessAncestor}) — also flows
+ * root → leaf. Mirrors BE's {@code _followed_by_shuffled_operator}: tells a
+ * child whether some downstream operator depends on hash distribution for
+ * correctness, so {@code SetOperationNode} can pre-shuffle union branches.</li>
+ * <li><b>return value</b> {@code Pair<PlanNode, LocalExchangeType>} — first is the
+ * (possibly LE-wrapped) child; second is the actual output distribution as
+ * observed by the parent. Caller's {@code require.satisfy(output)} decides
+ * whether more LE is needed.</li>
+ * <li><b>parent.require</b> describes the constraint on the child output —
+ * computed inside the parent's {@code enforceAndDeriveLocalExchange} per child.</li>
+ * </ul>
+ *
+ * <h3>Invariants</h3>
+ * <ul>
+ * <li>Where a serial → non-serial transition needs redistribution, framework step 3 inserts
+ * the LE (e.g. a serial source fanned out via PASSTHROUGH). This is not a hard invariant:
+ * a serial child feeding a parent that requires PASSTHROUGH / noRequire (TableFunction,
+ * NLJ, Agg) is already correct and needs no LE, so it is intentionally not validated by a
+ * post-pass.</li>
+ * <li>{@code LocalExchangeNode} itself is always non-serial — setting it serial
+ * would defeat its purpose of fanning a 1-task pipeline back to N tasks.</li>
+ * <li>For pipeline-breaking parents ({@code shouldResetSerialFlagForChild=true}),
+ * the child starts a fresh pipeline so {@code hasSerialAncestor} is reset; the
+ * node's own {@code isSerialNode()} still composes in for the child's view.</li>
+ * <li>{@code RequireHash} accepts any hash flavour; {@code RequireSpecific} demands
+ * an exact match (with the one PASSTHROUGH/ADAPTIVE_PASSTHROUGH compatibility).
+ * Pick the looser one whenever correctness allows — see
+ * {@link LocalExchangeNode.LocalExchangeTypeRequire}.</li>
+ * </ul>
+ *
+ * <h3>Layers</h3>
+ * Layer 1 (shouldSkipLE): mirrors BE's need_to_local_exchange — skip when this node or
+ * an ancestor in the same pipeline is serial (operators[idx..end] has serial → skip).
+ * Layer 2 (require/output): each Node declares require and output in enforceAndDeriveLocalExchange.
+ */
+ protected Pair<PlanNode, LocalExchangeType> enforceRequire(
+ PlanTranslatorContext translatorContext, PlanNode child, int childIndex,
+ LocalExchangeTypeRequire require) {
+ // 1. Propagate serial-ancestor flag to child.
+ // For pipeline-splitting operators (shouldReset=true, e.g. non-streaming AGG):
+ // Drop inherited serial flag from parent (parent is in a different pipeline),
+ // but keep this node's own serial status (child is in the same pipeline as this
+ // node's sink, e.g. Exchange is in AGG_Sink pipeline).
+ // For non-splitting operators (shouldReset=false, e.g. streaming AGG):
+ // Inherit parent's serial flag + this node's own.
+ boolean inheritedSerial = shouldResetSerialFlagForChild(childIndex)
+ ? false : translatorContext.hasSerialAncestorInPipeline(this);
+ // Use isSerialOperatorOnBe (= isSerialNode && fragment.useSerialSource) instead of the
+ // raw isSerialNode(). BE's OperatorBase reads the Thrift `is_serial_operator` flag —
+ // which is what FE writes via isSerialOperatorOnBe — so when the fragment is not in
+ // serial-source mode, BE treats this operator as non-serial regardless of isSerialNode.
+ // Using isSerialNode here would set the child's serial-ancestor flag wider than BE's
+ // view and over-skip required LocalExchanges downstream.
+ boolean childHasSerialAncestor = inheritedSerial
+ || isSerialOperatorOnBe(translatorContext.getConnectContext());
+ translatorContext.setHasSerialAncestorInPipeline(child, childHasSerialAncestor);
+
+ // 1b. Propagate shuffle-for-correctness-ancestor flag to child.
+ // Mirrors BE's _followed_by_shuffled_operator: a downstream operator needs hash
+ // distribution for correctness, and the chain to here goes through HASH or NOOP
+ // requirements (so the dependency is preserved).
+ // propagate = ((inheritedShuffled || self.requiresShuffleForCorrectness)
+ // && require is hash)
+ // || (inheritedShuffled && require is noop/passthrough)
+ boolean inheritedShuffled = translatorContext.hasShuffleForCorrectnessAncestor(this);
+ boolean selfOrInheritedShuffled = inheritedShuffled || requiresShuffleForCorrectness();
+ boolean requireIsHash = require.preferType().isHashShuffle();
+ boolean requireIsNoop = require.preferType() == LocalExchangeNode.LocalExchangeType.NOOP;
+ boolean childShuffledAncestor = (selfOrInheritedShuffled && requireIsHash)
+ || (inheritedShuffled && requireIsNoop);
+ translatorContext.setHasShuffleForCorrectnessAncestor(child, childShuffledAncestor);
+
+ // 2. Recurse child (Layer 2: child declares its own require/output)
+ Pair<PlanNode, LocalExchangeType> childOutput =
+ child.enforceAndDeriveLocalExchange(translatorContext, this, require);
+
+ // Steps 2.5 and 3 both react to a serial child but address different concerns:
+ // - Step 2.5 rewrites the OUTPUT-side view (what we tell satisfy/parent about
+ // the child's actual distribution). A serial pipeline runs with 1 task so
+ // its distribution claim is meaningless — flatten to NOOP so the satisfy
+ // check below doesn't get fooled by a stale "I output BUCKET_HASH" claim.
+ // - Step 3 rewrites the REQUIRE-side decision (what we want from the child).
+ // If we previously asked for nothing (noRequire) but the child turns out
+ // to be serial and we're not, upgrade to requirePassthrough so an LE is
+ // inserted to restore parallelism.
+
+ // 2.5. Serial child override (output side): if child is serial on BE, force its
+ // reported output to NOOP. Distribution is irrelevant when the child runs
+ // with 1 task; downstream parallelism is restored either by step 3 (LE
+ // insertion) or skipped entirely by step 4b (we're also serial).
+ if (childOutput.first.isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ childOutput = Pair.of(childOutput.first, LocalExchangeType.NOOP);
+ }
+
+ // 3. Framework-level serial child check (require side, mirrors BE base class
+ // required_data_distribution): if child will be serial on BE but this node is
+ // not serial, the pipeline has a 1-task serial child feeding an N-task non-serial
+ // parent. Without LE, pipeline splits (AGG/JOIN) create paired pipelines with
+ // mismatched num_tasks → crash. Upgrade noRequire to requirePassthrough so an
+ // LE is inserted below to restore parallelism.
+ if (require instanceof LocalExchangeNode.NoRequire
+ && childOutput.first.isSerialOperatorOnBe(translatorContext.getConnectContext())
+ && !isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ require = LocalExchangeTypeRequire.requirePassthrough();
+ }
+
+ // 4. Satisfy check: child output meets requirement → done
+ if (require.satisfy(childOutput.second)) {
+ return childOutput;
+ }
+
+ // 4. Layer 1: skip LE when serial operator or ancestor in same pipeline
+ // Equivalent to BE's need_to_local_exchange: any_of(operators[idx..end], is_serial) → skip.
+ // Use isSerialOperatorOnBe (not isSerialNode) because BE's Pipeline::need_to_local_exchange
+ // checks op->is_serial_operator() which reads the Thrift flag set from isSerialOperatorOnBe;
+ // when fragment.useSerialSource is false, BE treats this node as non-serial.
+ if (translatorContext.hasSerialAncestorInPipeline(this)
+ || isSerialOperatorOnBe(translatorContext.getConnectContext())) {
+ return childOutput;
+ }
+
+ // 5. Resolve exchange type and create LE node
+ LocalExchangeType preferType = AddLocalExchange.resolveExchangeType(require);
+ List<Expr> distributeExprs = getLocalExchangeDistributeExprs(childIndex, selfOrInheritedShuffled);
+ PlanNode leNode = createLocalExchange(translatorContext, childOutput.first, preferType, distributeExprs);
+ return Pair.of(leNode, preferType);
+ }
+
+ /**
+ * Create a LocalExchangeNode wrapping child with the given exchange type.
+ * No child-type skip — matches BE's _add_local_exchange which inserts LE for any child
+ * type without checking instanceof.
+ *
+ * Handles heavy-ops bottleneck avoidance (mirrors BE pipeline_fragment_context.cpp):
+ * when upstream has 1 task (serial source) and exchange is heavy (hash/bucket/adaptive),
+ * insert a PASSTHROUGH fan-out first to avoid single-task bottleneck on the heavy
+ * exchange sink. Only applies to local-shuffle (pooling scan) fragments.
+ */
+ protected PlanNode createLocalExchange(PlanTranslatorContext translatorContext,
+ PlanNode child, LocalExchangeType exchangeType, List<Expr> distributeExprs) {
+ if (fragment != null && fragment.useSerialSource(translatorContext.getConnectContext())
+ && exchangeType.isHeavyOperation() && child.isSerialNode()) {
+ PlanNode ptNode = new LocalExchangeNode(translatorContext.nextPlanNodeId(),
+ child, LocalExchangeType.PASSTHROUGH, null);
+ return new LocalExchangeNode(translatorContext.nextPlanNodeId(), ptNode,
+ exchangeType, distributeExprs);
+ }
+ return new LocalExchangeNode(translatorContext.nextPlanNodeId(), child,
+ exchangeType, distributeExprs);
+ }
+
+ /**
+ * Whether the child at {@code childIndex} starts a new pipeline context, causing
+ * its serial-ancestor flag to be reset to {@code false} rather than inherited from this node.
+ * Override to return {@code true} for pipeline-splitting nodes (LocalExchangeNode) and nodes
+ * whose children run in an independent pipeline segment (SortNode above analytic, etc.).
Review Comment:
above -> before?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3412891906
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PlanTranslatorContext.java:
##########
@@ -109,10 +109,36 @@ public class PlanTranslatorContext {
private final Map<ScanNode, Set<SlotId>> statsUnknownColumnsMap = Maps.newHashMap();
+ // Per-node "is there a serial operator between me and the pipeline's sink" flag.
+ // Mirrors BE's any_of(operators[idx..end], is_serial_operator) check used by
+ // _add_local_exchange / need_to_local_exchange to skip LE insertion when an ancestor
+ // in the same pipeline is already serial (the whole pipeline runs with 1 task, so an
+ // extra LE would be a no-op). Written by AddLocalExchange entry + PlanNode.enforceRequire
+ // step 1 (root → leaf during traversal). Read by PlanNode.enforceRequire step 4 (Layer 1
+ // skip) and by child overrides that compute their require. Reset to false at fragment
+ // root and across pipeline boundaries (see shouldResetSerialFlagForChild).
+ private final Map<PlanNodeId, Boolean> serialAncestorInPipelineMap = Maps.newHashMap();
+
+ // Per-node "is there a downstream operator that depends on hash distribution for
+ // correctness, with HASH/NOOP path connecting it to me" flag. Mirrors BE's
+ // _followed_by_shuffled_operator propagation in pipeline_fragment_context.cpp.
+ // Written by PlanNode.enforceRequire step 1b (root → leaf). Read by SetOperationNode
+ // to decide whether to propagate hash requirement to its inputs (only when downstream
+ // needs shuffle for correctness, not just for performance like StreamingAgg pre-agg).
+ private final Map<PlanNodeId, Boolean> shuffledAncestorMap = Maps.newHashMap();
+
+ // Whether the current fragment uses LocalShuffleAssignedJob (pooling scan with
+ // ignoreDataDistribution → _parallel_instances=1 in BE). When true, serial operators
+ // indicate real pipeline bottlenecks needing PASSTHROUGH fan-out (heavy_ops).
private boolean isTopMaterializeNode = true;
private final Set<SlotId> virtualColumnIds = Sets.newHashSet();
+ // Used by AddLocalExchange: tracks whether any serial operator exists
Review Comment:
**Dead code**: Field `serialAncestorInCurrentPipeline` and its getter/setter (L420-424) are never called by any code. The actual serial-ancestor propagation uses `serialAncestorInPipelineMap` (the per-PlanNodeId map).
Consider removing this field and its accessors to avoid future maintainers confusing the two mechanisms.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/ScanNode.java:
##########
@@ -733,7 +737,7 @@ public ScanContext getScanContext() {
}
@Override
- public boolean isSerialOperator() {
+ public boolean isSerialNode() {
ConnectContext context = ConnectContext.get();
if (context == null) {
return numScanBackends() <= 0;
Review Comment:
**Potential regression**: `isSerialNode()` removed `fragment.useSerialSource(context)` condition, but `Coordinator.assignScanRanges` (legacy path) calls `isSerialNode()` directly to decide whether to assign all scan ranges to the first instance.
Old logic: `numScanBackends() <= 0 || fragment.useSerialSource(context)` (OR semantics).
New logic: `numScanBackends() <= 0 || getScanRangeNum() < ... || isForceToLocalShuffle()`.
The `fragment.useSerialSource()` check is gone.
When a fragment is in serial-source (pooling) mode with a multi-backend scan and sufficient scan ranges, `isSerialNode()` returns `false`, causing scan ranges to be scattered across instances. But the downstream ExchangeNode is serial (1 task), expecting data from only one instance — data from non-first instances is lost.
**Suggestion**: Either add `|| fragment != null && fragment.useSerialSource(context)` back to `isSerialNode()`, or override `isSerialOperatorOnBe()` in ScanNode with OR semantics and change Coordinator to call `isSerialOperatorOnBe()`. Note: `isSerialOperatorOnBe` uses AND (`isSerialNode() && fragment.useSerialSource()`) which cannot express the old OR behavior.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/BucketedAggregationNode.java:
##########
@@ -97,7 +98,7 @@ public String getNodeExplainString(String detailPrefix, TExplainLevel detailLeve
}
@Override
- public boolean isSerialOperator() {
+ public boolean isSerialOperatorOnBe(ConnectContext context) {
Review Comment:
**Wrong method override**: This overrides `isSerialOperatorOnBe(ConnectContext context)` to directly return `false`, while every other node overrides `isSerialNode()`.
`BucketedAggregationNode extends AggregationNode`, which overrides `isSerialNode()` as `aggInfo.getGroupingExprs().isEmpty() && needsFinalize`. If a BucketedAgg without GROUP BY keys is ever introduced (despite the comment saying it won't happen), `isSerialNode()` would return `true` but `isSerialOperatorOnBe()` would still return `false`, causing inconsistent serial semantics.
No bug currently (BucketedAgg always has GROUP BY keys), but the inconsistent override pattern is a future bug waiting to happen.
**Suggestion**: Override `isSerialNode()` to return `false` instead, matching the comment ("never a serial operator") and following the same pattern as all other node types.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/MaterializationNode.java:
##########
@@ -227,7 +231,17 @@ public void setTopMaterializeNode(boolean topMaterializeNode) {
}
@Override
- public boolean isSerialOperator() {
+ public boolean isSerialNode() {
return true;
}
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ Pair<PlanNode, LocalExchangeType> enforceResult = enforceRequire(
+ translatorContext, children.get(0), 0, LocalExchangeTypeRequire.requirePassthrough());
+ children = new ArrayList<>();
Review Comment:
**Style inconsistency**: Uses `new ArrayList<>()` while other nodes (AggregationNode, HashJoinNode, AssertNumRowsNode, etc.) use `Lists.newArrayList()`. Consider using `Lists.newArrayList()` for consistency.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/AggregationNode.java:
##########
@@ -258,4 +268,155 @@ public boolean isQueryCacheCandidate() {
public void setQueryCacheCandidate(boolean queryCacheCandidate) {
this.queryCacheCandidate = queryCacheCandidate;
}
+
+ @Override
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+
+ ConnectContext connectContext = translatorContext.getConnectContext();
+ SessionVariable sessionVariable = connectContext.getSessionVariable();
+ // PR #62438: when false, non-finalize agg falls back to BE base class.
+ boolean enableLeBeforeAgg = sessionVariable.enableLocalExchangeBeforeAgg;
+ boolean hasKeys = !aggInfo.getGroupingExprs().isEmpty();
+
+ // Each branch mirrors the corresponding BE operator's required_data_distribution()
+ // check order 1:1. The helper baseClassRequire() expands BE's base class behavior.
+ LocalExchangeTypeRequire requireChild;
+ if (canUseDistinctStreamingAgg(sessionVariable)) {
+ // DistinctStreamingAggOperatorX. Two flavors share this operator class:
+ // - streaming preagg (useStreamingPreagg=true): performance-only,
+ // flag controls
+ // - non-streaming dedup (useStreamingPreagg=false): correctness-required,
+ // always HASH regardless of flag
+ // Diverges from BE: BE's `!_needs_finalize && !enable_local_exchange_before_agg`
+ // early return catches non-streaming dedup too, causing the same family of
+ // wrong-result bug as AggSink (DORIS-25413).
+ if (needsFinalize && !hasKeys) {
+ requireChild = LocalExchangeTypeRequire.noRequire();
+ } else if (!needsFinalize && useStreamingPreagg && !enableLeBeforeAgg) {
+ requireChild = baseClassRequire(connectContext);
+ } else if (needsFinalize || (hasKeys && !useStreamingPreagg)) {
+ requireChild = AddLocalExchange.isColocated(this)
+ ? LocalExchangeTypeRequire.requireHash()
+ : parentRequire.autoRequireHash();
+ } else if (sessionVariable.enableDistinctStreamingAggForcePassthrough) {
+ requireChild = LocalExchangeTypeRequire.requirePassthrough();
+ } else {
+ requireChild = baseClassRequire(connectContext);
+ }
+ } else if (useStreamingPreagg) {
+ // StreamingAggOperatorX
+ if (children.get(0) instanceof HashJoinNode
+ && sessionVariable.enableStreamingAggHashJoinForcePassthrough) {
+ requireChild = LocalExchangeTypeRequire.requirePassthrough();
+ } else if (!needsFinalize && !enableLeBeforeAgg) {
+ requireChild = baseClassRequire(connectContext);
+ } else if (!hasKeys) {
+ requireChild = needsFinalize
+ ? LocalExchangeTypeRequire.noRequire()
+ : baseClassRequire(connectContext);
+ } else {
+ requireChild = LocalExchangeTypeRequire.requireHash();
+ }
+ } else {
+ // AggSinkOperatorX — covers finalize phase AND non-finalize phases (LOCAL
+ // preagg / FIRST_MERGE dedup). Streaming preagg goes through the StreamingAgg
+ // branch above, not here.
+ //
+ // Phase semantics for !needsFinalize:
+ // - FIRST / SECOND (LOCAL phase, !isMerge): performance-only, flag controls
+ // - FIRST_MERGE (correctness-required): always HASH regardless of flag
+ //
+ // Diverges from BE here: BE's `!_needs_finalize && !enable_local_exchange_before_agg`
+ // early return also catches FIRST_MERGE, dropping the HASH requirement and
+ // causing wrong-result (e.g. PASSTHROUGH over serial child breaks the
+ // group-by-key invariant — DORIS-25413).
+ if (!hasKeys) {
+ requireChild = needsFinalize
+ ? LocalExchangeTypeRequire.noRequire()
+ : baseClassRequire(connectContext);
+ } else if (!needsFinalize && !aggInfo.isMerge() && !enableLeBeforeAgg) {
+ // LOCAL phase (FIRST preagg / SECOND distinct local) + user opted out
+ // of pre-agg LE → base class decides: serial child → PASSTHROUGH
+ // (parallelism), non-serial child → NOOP (no LE).
+ requireChild = baseClassRequire(connectContext);
+ } else if (!needsFinalize || AddLocalExchange.isColocated(this)) {
+ // FIRST_MERGE (correctness) or finalize+colocate → HASH.
+ requireChild = parentRequire.autoRequireHash();
+ } else if (hasPartitionExprs(parentRequire)) {
+ // FE-only heuristic: finalize non-colocate with parent hash requirement
+ // → inherit parent's specific hash type.
+ requireChild = parentRequire.autoRequireHash();
+ } else {
+ // FE-only heuristic: finalize non-colocate without parent hash → skip
+ // LE (child Exchange already provides hash distribution).
+ requireChild = LocalExchangeTypeRequire.noRequire();
+ }
+ }
+
+ Pair<PlanNode, LocalExchangeType> enforceResult
+ = enforceRequire(translatorContext, children.get(0), 0, requireChild);
+ children = Lists.newArrayList(enforceResult.first);
+ return Pair.of(this, enforceResult.second);
+ }
+
+ /** BE base class required_data_distribution: serial child → PASSTHROUGH, else → NOOP. */
+ private LocalExchangeTypeRequire baseClassRequire(ConnectContext connectContext) {
+ return children.get(0).isSerialOperatorOnBe(connectContext)
+ ? LocalExchangeTypeRequire.requirePassthrough()
+ : LocalExchangeTypeRequire.noRequire();
+ }
+
+ @Override
+ protected List<Expr> getSemanticPartitionExprs() {
+ return aggInfo.getGroupingExprs();
+ }
+
+ @Override
+ protected List<Expr> getLocalExchangeDistributeExprs(int childIndex, boolean followedByShuffled) {
+ // Mirror BE AggSinkOperatorX::update_operator / StreamingAggOperatorX::update_operator:
+ // _partition_exprs = (distribute_expr_lists set && (followed_by_shuffled || has_distinct))
+ // ? distribute_expr_lists[0] : grouping_exprs
+ // The HASH LocalExchange must partition by _partition_exprs so a streaming partial preagg
+ // locally collapses same-key rows. Using child distribution (default) for a non-shuffled
+ // chain scatters same-group rows across N instances, leaving partial_preagg essentially a
+ // no-op and breaking row-arrival order at downstream merge-finalize (e.g. group_concat).
+ List<Expr> childDist = getChildDistributeExprList(childIndex);
+ boolean hasDistinct = aggInfo.getAggregateExprs().stream()
+ .map(FunctionCallExpr::getFnName)
+ .filter(name -> name != null)
+ .map(name -> name.getFunction())
+ .filter(name -> name != null)
+ .anyMatch(name -> name.startsWith("multi_distinct_"));
Review Comment:
**Fragile string-based detection**: `hasDistinct` uses `name.startsWith("multi_distinct_")` to detect multi-distinct aggregate functions. This prefix-matching on function names is fragile — if Doris renames multi-distinct functions or adds new functions with the same prefix but different semantics, this check silently breaks.
The BE equivalent uses structural checks (`tnode.agg_node.__isset.aggregate_functions`), which is more robust.
**Suggestion**: Add an API on `AggInfo` (e.g., `hasDistinctAggregates()` or `isMultiDistinct()`) that exposes the presence of multi-distinct aggregates structurally, rather than having callers infer it from function name strings.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/AddLocalExchange.java:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.planner;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.glue.translator.PlanTranslatorContext;
+import org.apache.doris.nereids.trees.plans.distribute.DistributedPlan;
+import org.apache.doris.nereids.trees.plans.distribute.FragmentIdMapping;
+import org.apache.doris.nereids.trees.plans.distribute.PipelineDistributedPlan;
+import org.apache.doris.planner.LocalExchangeNode.LocalExchangeType;
+import org.apache.doris.planner.LocalExchangeNode.LocalExchangeTypeRequire;
+import org.apache.doris.planner.LocalExchangeNode.RequireHash;
+
+/**
+ * FE-side local exchange planner — inserts {@link LocalExchangeNode} into each fragment's
+ * plan tree so that within-fragment data redistribution is decided at planning time
+ * instead of at BE pipeline-build time.
+ *
+ * <h3>When this runs</h3>
+ * Invoked from {@code NereidsPlanner.addLocalExchangeAfterDistribute()} right after
+ * {@code DistributePlanner} has assigned instances to fragments and before the plan is
+ * serialized to BE. Gated by session variable {@code enable_local_shuffle_planner}
+ * (default true) and {@code enable_local_shuffle}; when either is off this pass is
+ * skipped entirely and BE falls back to its own {@code _plan_local_exchange}. The two
+ * paths are mutually exclusive: BE consults {@code runtime_state.h::plan_local_shuffle()}
+ * to know whether it should plan LE itself.
+ *
+ * <h3>What it changes</h3>
+ * <ul>
+ * <li>For each fragment with {@code maxPerBeInstances > 1}, walks the plan tree
+ * bottom-up via {@link PlanNode#enforceAndDeriveLocalExchange} and inserts
+ * LocalExchangeNodes where children's output distribution doesn't satisfy the
+ * parent's requirement.</li>
+ * <li>May wrap the fragment root with an extra PASSTHROUGH LE so the data sink
+ * (DataStreamSink / OlapTableSink) runs with the full instance count even when
+ * the root operator is serial — see {@link #addLocalExchangeForFragment}.</li>
+ * <li>Does NOT modify the fragment sink itself, fragment boundaries, or instance
+ * assignment.</li>
+ * </ul>
+ *
+ * <h3>Per-BE instance semantics</h3>
+ * Skips fragments where every BE has at most 1 instance. Using a global instance count
+ * would insert LE for "2 BEs × 1 instance" cases, which BE's own
+ * {@code _plan_local_exchange} would not — leading to pipeline task-count mismatch and
+ * deadlock. See {@link #addLocalExchange}.
+ *
+ * <h3>Reading order</h3>
+ * Start with {@link PlanNode#enforceRequire} (the recursion engine), then individual
+ * {@code enforceAndDeriveLocalExchange} overrides on PlanNode subclasses.
+ */
+public class AddLocalExchange {
+ /** addLocalExchange with distributed plans, skipping single-instance fragments.
+ * BE's _plan_local_exchange checks _num_instances which is the per-BE instance count.
+ * With _num_instances<=1 all pipelines on that BE have 1 task so local exchange is a no-op.
+ * We must use the same per-BE semantics: skip when every BE has at most 1 instance.
+ * Using global instanceCount would insert LE for fragments where 2 BEs each have 1 instance
+ * (global=2, per-BE=1), causing pipeline task mismatch and deadlock. */
+ public void addLocalExchange(FragmentIdMapping<DistributedPlan> distributedPlans,
+ PlanTranslatorContext context) {
+ for (DistributedPlan plan : distributedPlans.values()) {
+ PipelineDistributedPlan pipePlan = (PipelineDistributedPlan) plan;
+ long maxPerBeInstances = pipePlan.getInstanceJobs().stream()
+ .collect(java.util.stream.Collectors.groupingBy(
+ j -> j.getAssignedWorker().id(), java.util.stream.Collectors.counting()))
+ .values().stream().mapToLong(Long::longValue).max().orElse(0);
+ if (maxPerBeInstances <= 1) {
+ continue;
+ }
+ PlanFragment fragment = pipePlan.getFragmentJob().getFragment();
+ addLocalExchangeForFragment(fragment, context);
+ }
+ }
+
+ private void addLocalExchangeForFragment(PlanFragment fragment, PlanTranslatorContext context) {
+ DataSink sink = fragment.getSink();
+ LocalExchangeTypeRequire require = sink == null
+ ? LocalExchangeTypeRequire.noRequire() : sink.getLocalExchangeTypeRequire();
+ PlanNode root = fragment.getPlanRoot();
+ context.setHasSerialAncestorInPipeline(root, false);
+ Pair<PlanNode, LocalExchangeType> output = root
+ .enforceAndDeriveLocalExchange(context, null, require);
+ PlanNode newRoot = output.first;
+ // The fragment data sink (DataStreamSink, OlapTableSink) runs in the same pipeline
+ // as the root. If the root will be serial on BE, the sink pipeline has 1 task —
+ // only instance 0 sends data, others hang or miss writes.
+ // Insert PASSTHROUGH fan-out so sink runs with _num_instances tasks.
+ // This matches BE-native's default required_data_distribution():
+ // _child->is_serial_operator() ? PASSTHROUGH : NOOP
+ if (newRoot.isSerialOperatorOnBe(context.getConnectContext())) {
+ newRoot = new LocalExchangeNode(context.nextPlanNodeId(), newRoot,
+ LocalExchangeType.PASSTHROUGH, null);
+ }
+ if (newRoot != root) {
+ fragment.setPlanRoot(newRoot);
+ }
+ }
+
+ public static boolean isColocated(PlanNode plan) {
+ if (plan instanceof AggregationNode) {
+ return ((AggregationNode) plan).isColocate() && isColocated(plan.getChild(0));
+ } else if (plan instanceof OlapScanNode) {
+ return true;
+ } else if (plan instanceof SelectNode) {
+ return isColocated(plan.getChild(0));
+ } else if (plan instanceof HashJoinNode) {
+ return ((HashJoinNode) plan).isColocate()
+ && (isColocated(plan.getChild(0)) || isColocated(plan.getChild(1)));
+ } else if (plan instanceof SetOperationNode) {
+ if (!((SetOperationNode) plan).isColocate()) {
+ return false;
+ }
+ for (PlanNode child : plan.getChildren()) {
+ if (isColocated(child)) {
+ return true;
+ }
+ }
+ return false;
+ } else {
+ return false;
+ }
+ }
+
+ public static LocalExchangeType resolveExchangeType(LocalExchangeTypeRequire require) {
Review Comment:
**`preferType()` / `resolveExchangeType()` inconsistency**: `RequireHash.preferType()` (at LocalExchangeNode L240-241) returns `GLOBAL_EXECUTION_HASH_SHUFFLE`, but `resolveExchangeType()` always returns `LOCAL_EXECUTION_HASH_SHUFFLE` for `RequireHash`. This means `preferType()` is not authoritative — the actual exchange type inserted is controlled by `resolveExchangeType()`.
Currently `hasPartitionExprs()` uses `preferType().isHashShuffle()` which works correctly (both GLOBAL and LOCAL return true), but the discrepancy could confuse future maintainers.
**Suggestion**: Change `RequireHash.preferType()` to return `LOCAL_EXECUTION_HASH_SHUFFLE` to align with `resolveExchangeType()`.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java:
##########
@@ -939,4 +1026,261 @@ private String mergeIcebergAccessPathsWithId(
}
return StringUtils.join(mergeDisplayAccessPaths, ", ");
}
+
+ public Pair<PlanNode, LocalExchangeType> enforceAndDeriveLocalExchange(
+ PlanTranslatorContext translatorContext, PlanNode parent, LocalExchangeTypeRequire parentRequire) {
+ ArrayList<PlanNode> newChildren = Lists.newArrayList();
+ for (int i = 0; i < children.size(); i++) {
+ Pair<PlanNode, LocalExchangeType> childOutput
+ = enforceRequire(translatorContext, children.get(i), i, LocalExchangeTypeRequire.noRequire());
+ newChildren.add(childOutput.first);
+ }
+ this.children = newChildren;
+ return Pair.of(this, LocalExchangeType.NOOP);
+ }
+
+ /**
+ * Unified framework method: propagate serial flag → recurse child → satisfy check → Layer 1 skip → insert LE.
+ * Replaces the old enforceChild/enforceChildExchange/forceEnforceChildExchange trio.
+ *
+ * <h3>Data flow</h3>
+ * <ul>
+ * <li><b>serial-ancestor flag</b> ({@link PlanTranslatorContext#hasSerialAncestorInPipeline})
+ * — flows root → leaf during traversal. Mirrors BE's
+ * {@code any_of(operators[idx..end], is_serial_operator)} check used by
+ * {@code _add_local_exchange} to skip LE insertion when an ancestor in the same
+ * pipeline is already serial. Reset at pipeline boundaries via
+ * {@link #shouldResetSerialFlagForChild}.</li>
+ * <li><b>shuffle-for-correctness flag</b>
+ * ({@link PlanTranslatorContext#hasShuffleForCorrectnessAncestor}) — also flows
+ * root → leaf. Mirrors BE's {@code _followed_by_shuffled_operator}: tells a
+ * child whether some downstream operator depends on hash distribution for
+ * correctness, so {@code SetOperationNode} can pre-shuffle union branches.</li>
+ * <li><b>return value</b> {@code Pair<PlanNode, LocalExchangeType>} — first is the
+ * (possibly LE-wrapped) child; second is the actual output distribution as
+ * observed by the parent. Caller's {@code require.satisfy(output)} decides
+ * whether more LE is needed.</li>
+ * <li><b>parent.require</b> describes the constraint on the child output —
+ * computed inside the parent's {@code enforceAndDeriveLocalExchange} per child.</li>
+ * </ul>
+ *
+ * <h3>Invariants</h3>
+ * <ul>
+ * <li>Every serial → non-serial transition has an LE somewhere between them
+ * (enforced by framework step 3 below, validated by
+ * {@link AddLocalExchange#validateNoSerialWithoutLocalExchange}).</li>
Review Comment:
**Missing method referenced in javadoc**: The javadoc references `AddLocalExchange#validateNoSerialWithoutLocalExchange` but this method doesn't exist in `AddLocalExchange`. The framework cannot enforce the invariant "every serial to non-serial transition has an LE somewhere between them" without this validation method.
Suggestion: either implement the method (a post-pass validation walk over the plan tree after `addLocalExchange` completes), or remove the reference from the javadoc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4708881365
<details>
<summary>TPC-DS: <b>Total hot run time: 168379 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fe9a3316be056faed18351cf64096e36790c6c89, data reload: false
query5 4318 629 495 495
query6 450 199 177 177
query7 4801 586 305 305
query8 371 218 197 197
query9 8764 4098 4072 4072
query10 436 303 260 260
query11 5924 2332 2143 2143
query12 159 99 96 96
query13 1245 593 384 384
query14 6421 5861 5078 5078
query14_1 4439 4492 4305 4305
query15 207 196 176 176
query16 995 456 402 402
query17 934 714 548 548
query18 2429 486 329 329
query19 196 180 134 134
query20 109 106 104 104
query21 218 136 117 117
query22 13678 13585 13396 13396
query23 17365 16540 16236 16236
query23_1 16363 16255 16259 16255
query24 7500 1825 1299 1299
query24_1 1295 1328 1339 1328
query25 532 434 368 368
query26 1308 327 169 169
query27 2684 567 330 330
query28 4497 2031 2016 2016
query29 1064 635 497 497
query30 315 268 207 207
query31 1112 1079 977 977
query32 108 64 61 61
query33 532 330 271 271
query34 1220 1196 674 674
query35 774 796 696 696
query36 1379 1415 1257 1257
query37 161 110 97 97
query38 3233 3179 3070 3070
query39 953 912 895 895
query39_1 901 903 890 890
query40 224 127 107 107
query41 71 67 67 67
query42 98 96 94 94
query43 318 334 283 283
query44
query45 195 187 184 184
query46 1101 1244 743 743
query47 2395 2374 2210 2210
query48 402 421 310 310
query49 663 492 357 357
query50 1031 364 258 258
query51 4445 4403 4242 4242
query52 92 93 81 81
query53 246 287 189 189
query54 292 234 234 234
query55 86 79 74 74
query56 236 247 230 230
query57 1416 1406 1340 1340
query58 255 218 227 218
query59 1618 1659 1463 1463
query60 304 260 257 257
query61 201 187 173 173
query62 718 661 584 584
query63 232 193 193 193
query64 2582 776 583 583
query65
query66 1785 460 336 336
query67 29729 29127 29627 29127
query68
query69 432 302 266 266
query70 988 958 914 914
query71 306 220 207 207
query72 2904 2604 2337 2337
query73 862 758 453 453
query74 5109 4902 4779 4779
query75 2684 2582 2225 2225
query76 2343 1241 810 810
query77 394 376 269 269
query78 12447 12508 11947 11947
query79 1549 1058 792 792
query80 1265 481 391 391
query81 500 281 243 243
query82 575 163 122 122
query83 346 272 251 251
query84
query85 896 497 411 411
query86 405 294 271 271
query87 3421 3361 3228 3228
query88 3690 2718 2695 2695
query89 434 390 334 334
query90 1849 179 180 179
query91 171 162 130 130
query92 66 63 57 57
query93 1523 1495 892 892
query94 706 366 323 323
query95 673 463 349 349
query96 1117 876 353 353
query97 2679 2756 2558 2558
query98 211 212 201 201
query99 1155 1179 1027 1027
Total cold run time: 252218 ms
Total hot run time: 168379 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #63366:
URL: https://github.com/apache/doris/pull/63366#discussion_r3413491799
##########
be/src/exec/pipeline/dependency.h:
##########
@@ -832,50 +832,44 @@ struct SetSharedState : public BasicSharedState {
Status hash_table_init();
};
-enum class ExchangeType : uint8_t {
- NOOP = 0,
- // Shuffle data by Crc32CHashPartitioner
- HASH_SHUFFLE = 1,
- // Round-robin passthrough data blocks.
- PASSTHROUGH = 2,
- // Shuffle data by Crc32HashPartitioner<ShuffleChannelIds> (e.g. same as storage engine).
- BUCKET_HASH_SHUFFLE = 3,
- // Passthrough data blocks to all channels.
- BROADCAST = 4,
- // Passthrough data to channels evenly in an adaptive way.
- ADAPTIVE_PASSTHROUGH = 5,
- // Send all data to the first channel.
- PASS_TO_ONE = 6,
-};
+inline bool is_shuffled_exchange(TLocalPartitionType::type idx) {
+ return idx == TLocalPartitionType::GLOBAL_EXECUTION_HASH_SHUFFLE ||
+ idx == TLocalPartitionType::LOCAL_EXECUTION_HASH_SHUFFLE ||
+ idx == TLocalPartitionType::BUCKET_HASH_SHUFFLE;
+}
Review Comment:
`is_shuffled_exchange` is defined once (inline here in `dependency.h`) and reused at `pipeline.h:72/79`, `local_exchange_sink_operator.cpp:120`, and `local_exchange_source_operator.cpp:34` — there is no duplicate copy in `operator.cpp`. If you spotted a separate function in `operator.cpp` doing the same hash-type check, point me at it and I'll unify them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4765391971
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4766384189
# FE UT Coverage Report
Increment line coverage ` 74.69% (419/561)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63366_1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63366_1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4767719239
<details>
<summary>TPC-H: <b>Total hot run time: 29079 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7fdb989a3a7e4ad0c700539a335598f4978f6938, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17593 4008 3986 3986
q2 2010 324 188 188
q3 10864 1453 811 811
q4 4777 475 341 341
q5 8475 863 577 577
q6 340 169 137 137
q7 878 845 624 624
q8 10673 1636 1581 1581
q9 5779 4482 4511 4482
q10 6890 1798 1516 1516
q11 455 275 246 246
q12 639 423 300 300
q13 18123 3327 2764 2764
q14 266 261 243 243
q15 q16 785 781 712 712
q17 973 971 936 936
q18 7290 5881 5572 5572
q19 1191 1254 1012 1012
q20 499 397 267 267
q21 5650 2639 2481 2481
q22 435 368 303 303
Total cold run time: 104585 ms
Total hot run time: 29079 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4326 4266 4229 4229
q2 309 341 216 216
q3 4575 4968 4446 4446
q4 2061 2157 1358 1358
q5 4435 4291 4280 4280
q6 233 180 133 133
q7 2047 1905 1571 1571
q8 2482 2134 2092 2092
q9 8057 7995 8080 7995
q10 4814 4756 4270 4270
q11 574 433 509 433
q12 730 766 538 538
q13 3312 3586 2964 2964
q14 291 326 275 275
q15 q16 716 722 663 663
q17 1363 1331 1321 1321
q18 8074 7348 7112 7112
q19 1092 1063 1077 1063
q20 2233 2287 1956 1956
q21 5223 4513 4422 4422
q22 517 444 401 401
Total cold run time: 57464 ms
Total hot run time: 51738 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4767723647
<details>
<summary>TPC-H: <b>Total hot run time: 29190 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1f47a2e9dcf76c032362ef2feeb9d24a55a18c2b, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17690 4024 4100 4024
q2 2007 310 186 186
q3 10908 1437 814 814
q4 4761 475 341 341
q5 8448 854 566 566
q6 363 173 141 141
q7 852 850 640 640
q8 10668 1645 1579 1579
q9 5683 4570 4501 4501
q10 6853 1820 1534 1534
q11 451 274 247 247
q12 652 425 289 289
q13 18163 3947 2789 2789
q14 267 263 232 232
q15 q16 791 788 709 709
q17 1041 999 948 948
q18 6796 5789 5638 5638
q19 1167 1332 1051 1051
q20 485 415 259 259
q21 5505 2650 2407 2407
q22 428 366 295 295
Total cold run time: 103979 ms
Total hot run time: 29190 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4422 4342 4433 4342
q2 318 367 241 241
q3 4674 5020 4414 4414
q4 2082 2169 1384 1384
q5 4492 4353 4350 4350
q6 239 217 238 217
q7 2147 1836 1601 1601
q8 2548 2196 2352 2196
q9 8197 8025 8183 8025
q10 4836 4789 4391 4391
q11 589 415 399 399
q12 753 758 534 534
q13 3331 3747 3022 3022
q14 283 303 272 272
q15 q16 743 732 631 631
q17 1399 1399 1354 1354
q18 7904 7300 6826 6826
q19 1116 1080 1107 1080
q20 2229 2227 1978 1978
q21 5359 4651 4551 4551
q22 526 458 407 407
Total cold run time: 58187 ms
Total hot run time: 52215 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4551487122
<details>
<summary>TPC-DS: <b>Total hot run time: 171534 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ccfac05f6b543ebe78f973987084a8de173759c9, data reload: false
query5 4311 657 522 522
query6 324 218 195 195
query7 4288 580 322 322
query8 336 233 221 221
query9 8798 4009 3999 3999
query10 466 346 323 323
query11 5782 2428 2370 2370
query12 181 133 123 123
query13 1277 626 447 447
query14 6157 5506 5179 5179
query14_1 4479 4494 4454 4454
query15 217 208 188 188
query16 1001 485 456 456
query17 1167 758 631 631
query18 2596 480 358 358
query19 211 198 157 157
query20 136 132 130 130
query21 218 137 114 114
query22 13726 13683 13370 13370
query23 17436 16487 16233 16233
query23_1 16332 16261 16202 16202
query24 7477 1774 1305 1305
query24_1 1288 1335 1308 1308
query25 551 466 404 404
query26 1306 315 164 164
query27 2711 592 338 338
query28 4408 1981 1973 1973
query29 963 615 485 485
query30 299 232 195 195
query31 1137 1087 954 954
query32 89 76 73 73
query33 582 345 285 285
query34 1165 1137 679 679
query35 780 788 712 712
query36 1382 1380 1215 1215
query37 156 104 93 93
query38 3217 3180 3055 3055
query39 957 918 897 897
query39_1 884 893 883 883
query40 226 147 124 124
query41 64 61 61 61
query42 109 111 111 111
query43 334 337 293 293
query44
query45 214 202 196 196
query46 1115 1193 748 748
query47 2325 2336 2239 2239
query48 415 410 297 297
query49 635 532 402 402
query50 1059 366 256 256
query51 4474 4416 4328 4328
query52 107 107 98 98
query53 254 284 201 201
query54 327 287 270 270
query55 95 93 87 87
query56 313 326 326 326
query57 1453 1425 1319 1319
query58 315 290 282 282
query59 1600 1692 1408 1408
query60 320 322 298 298
query61 159 157 160 157
query62 713 646 590 590
query63 252 194 196 194
query64 2406 782 641 641
query65
query66 1704 479 353 353
query67 29840 29683 29073 29073
query68
query69 448 344 310 310
query70 1049 956 930 930
query71 314 272 271 271
query72 3048 2490 2407 2407
query73 843 734 428 428
query74 5137 5006 4745 4745
query75 2690 2626 2259 2259
query76 2281 1172 771 771
query77 414 424 345 345
query78 12488 12287 11832 11832
query79 1492 1075 796 796
query80 676 574 494 494
query81 451 285 247 247
query82 1376 164 131 131
query83 363 291 265 265
query84 274 148 124 124
query85 998 645 470 470
query86 420 349 346 346
query87 3434 3393 3245 3245
query88 3722 2749 2794 2749
query89 447 392 345 345
query90 1965 193 184 184
query91 189 172 140 140
query92 77 81 76 76
query93 1459 1555 922 922
query94 574 343 303 303
query95 693 383 459 383
query96 1109 823 362 362
query97 2711 2723 2632 2632
query98 237 238 228 228
query99 1145 1177 1035 1035
Total cold run time: 255136 ms
Total hot run time: 171534 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](local shuffle) Move local exchange planning from BE to FE [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63366:
URL: https://github.com/apache/doris/pull/63366#issuecomment-4551434262
<details>
<summary>TPC-H: <b>Total hot run time: 31459 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ccfac05f6b543ebe78f973987084a8de173759c9, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17809 4007 3989 3989
q2 q3 10694 1384 835 835
q4 4690 474 351 351
q5 7571 2290 2157 2157
q6 247 178 141 141
q7 941 813 645 645
q8 9380 1653 1609 1609
q9 5552 5056 5080 5056
q10 6406 2229 1883 1883
q11 435 268 247 247
q12 650 437 298 298
q13 18211 3389 2780 2780
q14 266 261 235 235
q15 q16 821 772 706 706
q17 993 935 845 845
q18 6908 5717 5522 5522
q19 1183 1312 1105 1105
q20 511 423 269 269
q21 5606 2649 2468 2468
q22 441 366 318 318
Total cold run time: 99315 ms
Total hot run time: 31459 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4370 4296 4280 4280
q2 q3 4525 4981 4343 4343
q4 2067 2201 1391 1391
q5 4430 4266 4336 4266
q6 231 172 128 128
q7 2449 1929 1707 1707
q8 2542 2238 2208 2208
q9 8346 8164 8306 8164
q10 4847 4792 4315 4315
q11 568 429 384 384
q12 740 776 570 570
q13 3315 3673 2928 2928
q14 303 313 286 286
q15 q16 734 776 644 644
q17 1388 1356 1488 1356
q18 7982 7459 7354 7354
q19 1242 1190 1093 1093
q20 2215 2235 1962 1962
q21 5276 4582 4464 4464
q22 506 471 407 407
Total cold run time: 58076 ms
Total hot run time: 52250 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org