You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by hu...@apache.org on 2026/03/29 02:26:43 UTC
(doris) branch fe_local_shuffle updated: [fix](local shuffle) restore deferred exchanger num_tasks raise for non-scan serial operators
This is an automated email from the ASF dual-hosted git repository.
huajianlan pushed a commit to branch fe_local_shuffle
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/fe_local_shuffle by this push:
new 920d43dbabd [fix](local shuffle) restore deferred exchanger num_tasks raise for non-scan serial operators
920d43dbabd is described below
commit 920d43dbabdd3635f44ab9203595a0957c445459
Author: 924060929 <la...@selectdb.com>
AuthorDate: Sun Mar 29 10:26:21 2026 +0800
[fix](local shuffle) restore deferred exchanger num_tasks raise for non-scan serial operators
Restore the upstream pipeline num_tasks raise in _create_deferred_local_exchangers
that was removed in 80541b0dd93. The raise is needed because serial non-scan
operators (e.g., UNPARTITIONED Exchange) reduce the pipeline's num_tasks to
_parallel_instances (typically 1), but downstream operators (Agg, Sort, Union,
Join build sinks) need _num_instances tasks so every fragment instance can
create and inject shared state. Without the raise, instances 1+ fail with
"must set shared state, in AGGREGATION_OPERATOR/SORT_OPERATOR/UNION_OPERATOR".
The raise is now conditional: skip when the serial operator is a scan source
(pooling scan), where 1 sender is correct and PassthroughExchanger(1, N)
handles the fan-out properly. This avoids the profile count inflation that
motivated the original removal.
Co-Authored-By: Claude Opus 4.6 (1M context) <no...@anthropic.com>
---
be/src/exec/pipeline/pipeline_fragment_context.cpp | 27 ++++++++++++++++++----
1 file changed, 23 insertions(+), 4 deletions(-)
diff --git a/be/src/exec/pipeline/pipeline_fragment_context.cpp b/be/src/exec/pipeline/pipeline_fragment_context.cpp
index 29a9a04c477..148422b61ef 100644
--- a/be/src/exec/pipeline/pipeline_fragment_context.cpp
+++ b/be/src/exec/pipeline/pipeline_fragment_context.cpp
@@ -668,10 +668,29 @@ Status PipelineFragmentContext::_build_pipelines(ObjectPool* pool, const Descrip
Status PipelineFragmentContext::_create_deferred_local_exchangers() {
for (auto& info : _deferred_exchangers) {
- // upstream_pipe->num_tasks() is already the correct sender count:
- // _create_tree_helper sets it via add_operator (serial operators call set_num_tasks with
- // _parallel_instances, non-serial operators leave it at the value inherited from
- // add_pipeline). No adjustment is needed here.
+ // Raise upstream pipeline's num_tasks to _num_instances when a serial
+ // non-scan operator (e.g., UNPARTITIONED Exchange) reduced it below
+ // _num_instances. This is needed because downstream pipelines
+ // (join build, agg, sort, union, etc.) need _num_instances tasks so
+ // every fragment instance can create/inject shared state. Without the
+ // raise, instances 1+ have no upstream task and operators fail with
+ // "must set shared state".
+ //
+ // Exception: do NOT raise when the serial operator is a scan source
+ // (pooling scan). For pooling scan, 1 sender is correct —
+ // PassthroughExchanger(1, N) handles the 1→N fan-out properly.
+ if (info.upstream_pipe->num_tasks() < _num_instances) {
+ bool has_serial_scan = false;
+ for (auto& op : info.upstream_pipe->operators()) {
+ if (op->is_serial_operator() && op->is_source()) {
+ has_serial_scan = true;
+ break;
+ }
+ }
+ if (!has_serial_scan) {
+ info.upstream_pipe->set_num_tasks(_num_instances);
+ }
+ }
const int sender_count = info.upstream_pipe->num_tasks();
switch (info.partition_type) {
case TLocalPartitionType::LOCAL_EXECUTION_HASH_SHUFFLE:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org