You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "BiteTheDDDDt (via GitHub)" <gi...@apache.org> on 2026/04/01 05:19:10 UTC
[PR] [Improvement](shuffle) enlarge bucket number to fullBucketNum [doris]
BiteTheDDDDt opened a new pull request, #61988:
URL: https://github.com/apache/doris/pull/61988
This pull request addresses the scenario where the number of local shuffle instances exceeds the number of physical buckets in bucketed OLAP table scans. The changes ensure that all instances are properly assigned data by introducing the concept of virtual buckets and expanding the bucket space accordingly. The most important changes are:
**Bucket assignment logic improvements:**
* In `UnassignedScanBucketOlapTableJob.java`, when the number of instances exceeds the number of physical buckets, virtual bucket indexes are assigned to the extra instances. This guarantees that every instance receives data during local shuffle, preventing idle instances.
**Thrift parameter updates:**
* In `ThriftPlansBuilder.java`, the `num_buckets` parameter is set to the maximum of the physical bucket count and the number of worker instances. This ensures that the hash distribution covers both physical and virtual buckets, so data can be routed to every instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](shuffle) enlarge bucket number to fullBucketNum [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #61988:
URL: https://github.com/apache/doris/pull/61988#issuecomment-4167529148
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](shuffle) enlarge bucket number to fullBucketNum [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #61988:
URL: https://github.com/apache/doris/pull/61988#discussion_r3019849310
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/distribute/worker/job/UnassignedScanBucketOlapTableJob.java:
##########
@@ -254,13 +254,18 @@ protected void assignLocalShuffleJobs(ScanSource scanSource, int instanceNum, Li
}
}
+ // When instanceNum > numBuckets (physical bucket count), extra instances would get
+ // no data through BUCKET_HASH_SHUFFLE local exchange because hash % numBuckets never
+ // maps to them. Fix: assign virtual bucket indexes (beyond physical range) to extra
+ // instances so the expanded bucket space covers all instances evenly.
int thisWorkerInstanceNum = instances.size() - existsInstanceNum;
+ int numBuckets = fullBucketNum();
for (int i = thisWorkerInstanceNum; i < instanceNum; ++i) {
+ int virtualBucketIndex = numBuckets + (i - thisWorkerInstanceNum);
LocalShuffleBucketJoinAssignedJob instance = new LocalShuffleBucketJoinAssignedJob(
instances.size(), shareScanId, context.nextInstanceId(),
this, worker, emptyShareScanSource,
- // these instance not need to join, because no any bucket assign to it
- ImmutableSet.of()
+ ImmutableSet.of(virtualBucketIndex)
);
Review Comment:
The computed virtualBucketIndex can exceed the valid bucket-id range used by BE local exchange. Here numBuckets = fullBucketNum() (table physical bucket count), so if this worker scans fewer than fullBucketNum() buckets (common) and instanceNum > thisWorkerInstanceNum, virtualBucketIndex may become >= params.num_buckets (and even > params.num_buckets-1), which in BE is used to index arrays sized (num_buckets+1) and can lead to out-of-bounds / crash. Consider deriving a per-worker “expanded bucket space” N (e.g., N = max(physicalBucketNum, instanceNum) or another explicit full/virtual bucket count) and ensure all assignedJoinBucketIndexes are within [0, N).
##########
fe/fe-core/src/main/java/org/apache/doris/qe/runtime/ThriftPlansBuilder.java:
##########
@@ -411,11 +411,16 @@ private static TPipelineFragmentParams fragmentToThriftIfAbsent(
params.setFileScanParams(fileScanRangeParamsMap);
if (fragmentPlan.getFragmentJob() instanceof UnassignedScanBucketOlapTableJob) {
- int bucketNum = ((UnassignedScanBucketOlapTableJob) fragmentPlan.getFragmentJob())
+ int physicalBucketNum = ((UnassignedScanBucketOlapTableJob) fragmentPlan.getFragmentJob())
.getOlapScanNodes()
.get(0)
.getBucketNum();
- params.setNumBuckets(bucketNum);
+ // When local shuffle creates more instances than physical buckets,
+ // virtual bucket indexes are assigned beyond the physical range.
+ // Expand num_buckets to cover all virtual buckets so that
+ // hash % num_buckets can route data to every instance.
+ int workerInstanceNum = instancesPerWorker.get(worker).size();
+ params.setNumBuckets(Math.max(physicalBucketNum, workerInstanceNum));
Review Comment:
Setting params.num_buckets to max(physicalBucketNum, workerInstanceNum) does not necessarily “cover all virtual buckets” assigned by UnassignedScanBucketOlapTableJob. That job currently assigns virtual bucket ids starting at physicalBucketNum and can produce bucket ids > workerInstanceNum-1 when this worker scans fewer physical buckets than fullBucketNum(), even when physicalBucketNum >= workerInstanceNum (so num_buckets stays physicalBucketNum). In BE, bucket_seq_to_instance_idx keys are treated as bucket ids in [0, num_buckets), so any key >= num_buckets can trigger out-of-bounds. Suggest computing num_buckets from the maximum bucket id actually used in bucketSeqToInstanceIdx (+1), or adjusting the virtual bucket id scheme so all assigned bucket ids are < num_buckets for every worker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](shuffle) enlarge bucket number to fullBucketNum [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #61988:
URL: https://github.com/apache/doris/pull/61988#issuecomment-4167529434
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](shuffle) enlarge bucket number to fullBucketNum [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt closed pull request #61988: [Improvement](shuffle) enlarge bucket number to fullBucketNum
URL: https://github.com/apache/doris/pull/61988
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](shuffle) enlarge bucket number to fullBucketNum [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61988:
URL: https://github.com/apache/doris/pull/61988#issuecomment-4167893220
# FE UT Coverage Report
Increment line coverage ` 50.00% (3/6)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61988_378ce49b1c0b959b74d42c493026b1c74cb461ad/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61988_378ce49b1c0b959b74d42c493026b1c74cb461ad/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](shuffle) enlarge bucket number to fullBucketNum [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #61988:
URL: https://github.com/apache/doris/pull/61988#issuecomment-4167545181
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](shuffle) enlarge bucket number to fullBucketNum [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #61988:
URL: https://github.com/apache/doris/pull/61988#discussion_r3019848501
##########
fe/fe-core/src/main/java/org/apache/doris/qe/runtime/ThriftPlansBuilder.java:
##########
@@ -411,11 +411,16 @@ private static TPipelineFragmentParams fragmentToThriftIfAbsent(
params.setFileScanParams(fileScanRangeParamsMap);
if (fragmentPlan.getFragmentJob() instanceof UnassignedScanBucketOlapTableJob) {
- int bucketNum = ((UnassignedScanBucketOlapTableJob) fragmentPlan.getFragmentJob())
+ int physicalBucketNum = ((UnassignedScanBucketOlapTableJob) fragmentPlan.getFragmentJob())
.getOlapScanNodes()
.get(0)
.getBucketNum();
- params.setNumBuckets(bucketNum);
+ // When local shuffle creates more instances than physical buckets,
+ // virtual bucket indexes are assigned beyond the physical range.
+ // Expand num_buckets to cover all virtual buckets so that
+ // hash % num_buckets can route data to every instance.
+ int workerInstanceNum = instancesPerWorker.get(worker).size();
+ params.setNumBuckets(Math.max(physicalBucketNum, workerInstanceNum));
Review Comment:
This FE-side `num_buckets` expansion is not enough for the distributed case. `computeBucketIdToInstanceId()` still sends only this worker's sparse physical bucket ids plus any virtual ids from `assignLocalShuffleJobs()`, while BE hashes rows modulo `num_buckets` and requires every produced bucket id to be present in that map (`ShuffleExchanger::_split_rows()` returns `Rows mismatched! Data may be lost.` otherwise).
A concrete failure case is a 16-bucket table where one worker only scans buckets `{3, 11}` but local shuffle wants 8 instances: after this change the worker still does not publish a complete destination map for the expanded bucket space, so some hash results have no target instance. This only works when a worker already owns the full dense bucket range locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org