You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "sollhui (via GitHub)" <gi...@apache.org> on 2026/04/21 03:57:23 UTC
[PR] [enhance](load) adaptive per-BE bucket assignment for random distribution load [doris]
sollhui opened a new pull request, #62661:
URL: https://github.com/apache/doris/pull/62661
## Problem
For tables with random distribution, the current import logic has two modes
controlled by `load_to_single_tablet`:
- `false` (default): each BE rotates through **all** buckets every batch
→ every BE holds open channels to every tablet, high memory pressure
- `true`: every BE writes to the **same single** tablet
→ severe data skew, single-tablet bottleneck
There is no middle ground: distribute writes evenly while keeping each BE's
active bucket set small.
## Solution
Remove the `load_to_single_tablet` knob and replace it with a new
`FIND_TABLET_RANDOM_BUCKET` mode. FE plans which bucket each BE starts on
and which buckets it is allowed to rotate into; BE switches to the next
assigned bucket only when a per-tablet write threshold (200 MB) is reached.
### Key design points
**1. CalculateBucketNum — limit active buckets by concurrency**
activeBuckets = min(numBackendsWithSink, totalBucketNum, totalSinkInstances)
maxLocalBucketsPerBe = ceil(activeBuckets / numBackendsWithSink)
For stream load (`totalSinkInstances = 1`): `activeBuckets = 1`, each BE
gets at most 1 local bucket — no switching, no data scatter for small loads.
For batch load (`totalSinkInstances = numBackendsWithSink`): each BE gets 1
local bucket as its starting point, and can grow as data accumulates.
**2. Per-BE bucket ownership — handles single and multi-replica**
Each bucket must be assigned to **exactly one** writer BE to avoid multiple
BEs converging on the same tablet:
- *Single replica*: `node_ids` has one entry → that BE is the owner.
- *Multi-replica*: sort `node_ids` by BE ID; assign bucket `i` to the BE at
position `i % numReplicas` in the sorted list. Deterministic and
coordination-free.
**3. FE sends `local_bucket_seqs` to BE**
Instead of having the BE re-derive ownership at runtime (which would require
scanning `node_ids` and could still produce wrong results for multi-replica),
the FE computes the final list and sends it as `local_bucket_seqs` in
`TOlapTablePartition`. The BE only needs to walk this list when switching.
---
## Example 1 — single import, 3 BEs, 6 buckets, single replica
Tablet layout (one replica per tablet):
| Bucket | node_ids |
|--------|----------|
| 0 | [BE1] |
| 1 | [BE2] |
| 2 | [BE3] |
| 3 | [BE1] |
| 4 | [BE2] |
| 5 | [BE3] |
`TabletLoadIndexRecorderMgr` returns `tabletIndex = 2` for this load.
`totalSinkInstances = 3`, `numBackendsWithSink = 3` → `activeBuckets = 3`,
`maxLocalBucketsPerBe = 1`.
FE plans:
| BE | localBuckets (before trim) | after trim | load_tablet_idx | local_bucket_seqs |
|-----|---------------------------|------------|-----------------|-------------------|
| BE1 | [0, 3] | [0] | **0** | [0] |
| BE2 | [1, 4] | [1] | **1** | [1] |
| BE3 | [2, 5] | [2] | **2** | [2] |
Each BE writes to one dedicated bucket. `local_bucket_seqs` has a single
entry so no switching occurs (small loads stay on one bucket).
---
## Example 2 — multiple imports, bucket rotation across loads
Same cluster. `tabletIndex` increments by 1 per load per partition
(starting from a random offset; suppose offset = 0).
| Load | tabletIndex | BE1 starts on | BE2 starts on | BE3 starts on |
|------|-------------|---------------|---------------|---------------|
| 1 | 0 | bucket **0** | bucket **1** | bucket **2** |
| 2 | 1 | bucket **3** | bucket **4** | bucket **5** |
| 3 | 2 | bucket **0** | bucket **1** | bucket **2** |
| 4 | 3 | bucket **3** | bucket **4** | bucket **5** |
Consecutive loads alternate between the two local buckets of each BE,
spreading data evenly across all 6 buckets over time.
---
## Example 3 — stream load (single instance)
Same cluster, stream load: `totalSinkInstances = 1`, `numBackendsWithSink = 1`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4834206630
# BE UT Coverage Report
Increment line coverage `13.77% (95/690)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/6cf651de54175959a128b07b857fa64e939af881_6cf651de54175959a128b07b857fa64e939af881/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/6cf651de54175959a128b07b857fa64e939af881_6cf651de54175959a128b07b857fa64e939af881/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 54.93% (21626/39372) |
| Line Coverage | 38.41% (206753/538210) |
| Region Coverage | 34.47% (162780/472215) |
| Branch Coverage | 35.49% (71236/200748) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4829125291
<details>
<summary>ClickBench: <b>Total hot run time: 25.22 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1ad5cc3a0a4c7e67ea8bd62c1cf9e84c7f493552, data reload: false
query1 0.01 0.01 0.00
query2 0.10 0.05 0.05
query3 0.26 0.14 0.15
query4 1.61 0.14 0.13
query5 0.24 0.22 0.23
query6 1.27 1.12 1.09
query7 0.04 0.01 0.00
query8 0.05 0.04 0.03
query9 0.38 0.31 0.32
query10 0.56 0.54 0.53
query11 0.20 0.14 0.14
query12 0.19 0.14 0.14
query13 0.47 0.47 0.47
query14 1.02 1.01 1.00
query15 0.60 0.59 0.62
query16 0.31 0.32 0.33
query17 1.10 1.10 1.11
query18 0.23 0.22 0.20
query19 2.12 1.90 1.97
query20 0.02 0.01 0.02
query21 15.43 0.21 0.13
query22 4.91 0.05 0.06
query23 16.15 0.30 0.12
query24 2.95 0.41 0.32
query25 0.11 0.04 0.04
query26 0.72 0.22 0.16
query27 0.04 0.03 0.04
query28 3.49 0.89 0.53
query29 12.49 4.33 3.45
query30 0.29 0.15 0.15
query31 2.78 0.59 0.31
query32 3.22 0.59 0.49
query33 3.31 3.23 3.23
query34 15.41 4.19 3.53
query35 3.58 3.52 3.53
query36 0.54 0.44 0.43
query37 0.09 0.07 0.06
query38 0.05 0.04 0.03
query39 0.03 0.03 0.04
query40 0.18 0.16 0.15
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.05 0.03 0.04
Total cold run time: 96.73 s
Total hot run time: 25.22 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3491739303
##########
fe/fe-core/src/main/java/org/apache/doris/planner/OlapTableSink.java:
##########
@@ -574,6 +581,375 @@ private TOlapTablePartitionParam createDummyPartition(long dbId, OlapTable table
return partitionParam;
}
+ public static final class AdaptiveBucketAssignment {
+ private final int loadTabletIdx;
+ private final AdaptiveIndexBucketAssignment firstIndexAssignment;
+ private final Map<Long, AdaptiveIndexBucketAssignment> indexAssignments;
+
+ public AdaptiveBucketAssignment(int loadTabletIdx,
+ AdaptiveIndexBucketAssignment firstIndexAssignment,
+ Map<Long, AdaptiveIndexBucketAssignment> indexAssignments) {
+ this.loadTabletIdx = loadTabletIdx;
+ this.firstIndexAssignment = firstIndexAssignment;
+ this.indexAssignments = new HashMap<>(indexAssignments);
+ }
+
+ public long getBucketBeId() {
+ return firstIndexAssignment == null ? -1L : firstIndexAssignment.getBucketBeId();
+ }
+
+ public int getLoadTabletIdx() {
+ return loadTabletIdx;
+ }
+
+ public List<Integer> getLocalBucketSeqs() {
+ return firstIndexAssignment == null ? Collections.emptyList()
+ : firstIndexAssignment.getLocalBucketSeqs();
+ }
+
+ public Map<Long, AdaptiveIndexBucketAssignment> getIndexAssignments() {
+ return indexAssignments;
+ }
+ }
+
+ public static final class AdaptiveIndexBucketAssignment {
+ private final long indexId;
+ private final long bucketBeId;
+ private final List<Integer> localBucketSeqs;
+
+ public AdaptiveIndexBucketAssignment(long indexId, long bucketBeId, List<Integer> localBucketSeqs) {
+ this.indexId = indexId;
+ this.bucketBeId = bucketBeId;
+ this.localBucketSeqs = new ArrayList<>(localBucketSeqs);
+ }
+
+ public long getIndexId() {
+ return indexId;
+ }
+
+ public long getBucketBeId() {
+ return bucketBeId;
+ }
+
+ public List<Integer> getLocalBucketSeqs() {
+ return localBucketSeqs;
+ }
+
+ @Override
+ public String toString() {
+ return "indexId=" + indexId + ",bucketBeId=" + bucketBeId
+ + ",localBucketSeqs=" + localBucketSeqs;
+ }
+ }
+
+ public static boolean shouldAssignAdaptiveRandomBucket(TOlapTableSink sink) {
+ return sink != null
+ && sink.isSetEnableAdaptiveRandomBucket()
+ && sink.isEnableAdaptiveRandomBucket()
+ && (!sink.isSetLoadToSingleTablet() || !sink.isLoadToSingleTablet())
+ && sink.isSetPartition()
+ && sink.getPartition() != null
+ && (!sink.getPartition().isSetDistributedColumns()
+ || sink.getPartition().getDistributedColumns().isEmpty());
+ }
+
+ public boolean shouldAssignAdaptiveRandomBucket() {
+ return tDataSink != null && shouldAssignAdaptiveRandomBucket(tDataSink.getOlapTableSink());
+ }
+
+ public static Map<Long, Map<Long, AdaptiveBucketAssignment>> computeAdaptiveRandomBucketAssignments(
+ List<Long> sinkBackendIds, List<TOlapTablePartition> partitions,
+ List<TTabletLocation> tabletLocations, int sinkInstanceNum) {
+ Map<Long, Map<Long, AdaptiveBucketAssignment>> assignments = new HashMap<>();
+ List<Long> orderedSinkBackendIds = sinkBackendIds.stream()
+ .distinct()
+ .sorted()
+ .collect(Collectors.toList());
+ for (Long sinkBackendId : orderedSinkBackendIds) {
+ assignments.put(sinkBackendId, new HashMap<>());
+ }
+ if (orderedSinkBackendIds.isEmpty() || partitions == null || tabletLocations == null) {
+ return assignments;
+ }
+
+ Map<Long, TTabletLocation> tabletLocationMap = new HashMap<>(tabletLocations.size());
+ for (TTabletLocation tabletLocation : tabletLocations) {
+ tabletLocationMap.put(tabletLocation.getTabletId(), tabletLocation);
+ }
+
+ for (TOlapTablePartition partition : partitions) {
+ if (!partition.isSetLoadTabletIdx() || partition.getNumBuckets() <= 0
+ || partition.getIndexes().isEmpty()) {
+ continue;
+ }
+ TOlapTableIndexTablets baseIndex = partition.getIndexes().get(0);
+ Map<Long, List<Integer>> beToBucketSeqs = buildBeToBucketSeqs(baseIndex, tabletLocationMap);
+ long baseTabletIndex = partition.getLoadTabletIdx();
+ int fallbackBucketIdx = (int) Math.floorMod(baseTabletIndex, (long) partition.getNumBuckets());
+ int targetBucketNum = Math.min(
Review Comment:
The new `sinkInstanceNum` still cannot increase the number of adaptive buckets when several sink instances run on the same BE. `computeAdaptiveRandomBucketAssignments()` deduplicates `sinkBackendIds` and then caps `targetBucketNum` by `orderedSinkBackendIds.size()`, and the result is keyed only by backend id. For example, with two sink BEs, four local sink instances per BE, and eight random buckets, `sinkInstanceNum` is 8 but this line still plans only `min(2, 8, 8) = 2` buckets, so all four senders on a BE open the same `local_bucket_seqs` and start on the same receiver tablet until rotation. Please either make the assignment granular per local sender/instance, or remove `sinkInstanceNum` from this calculation and document that adaptive random bucket intentionally works at backend granularity rather than sink-instance granularity.
##########
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:
##########
@@ -5745,4 +5768,132 @@ private TStatus checkMaster() {
}
return status;
}
+
+ private static final class AdaptiveBucketSinkContext {
+ private final boolean enableAdaptiveRandomBucket;
+ private final List<Long> sinkBackendIds;
+ private final int sinkInstanceNum;
+
+ private AdaptiveBucketSinkContext(boolean enableAdaptiveRandomBucket, List<Long> sinkBackendIds,
+ int sinkInstanceNum) {
+ this.enableAdaptiveRandomBucket = enableAdaptiveRandomBucket;
+ this.sinkBackendIds = sinkBackendIds;
+ this.sinkInstanceNum = sinkInstanceNum;
+ }
+ }
+
+ private static AdaptiveBucketSinkContext disabledAdaptiveBucketSinkContext() {
+ return new AdaptiveBucketSinkContext(false, Lists.newArrayList(), 1);
+ }
+
+ private static AdaptiveBucketSinkContext collectAdaptiveBucketSinkContext(TUniqueId queryId, long tableId) {
+ if (queryId == null) {
+ return disabledAdaptiveBucketSinkContext();
+ }
+ Coordinator coordinator = QeProcessorImpl.INSTANCE.getCoordinator(queryId);
+ if (coordinator == null) {
+ return disabledAdaptiveBucketSinkContext();
+ }
+ if (!(coordinator instanceof NereidsCoordinator)) {
+ Optional<Coordinator.AdaptiveRandomBucketSinkContext> context =
+ coordinator.getAdaptiveRandomBucketSinkContext(tableId);
+ if (context.isPresent()) {
+ return new AdaptiveBucketSinkContext(
+ true, context.get().getSinkBackendIds(), context.get().getSinkInstanceNum());
+ }
+ return disabledAdaptiveBucketSinkContext();
+ }
+ Set<Long> sinkBackendIds = new TreeSet<>();
+ int sinkInstanceNum = 0;
+ for (PipelineDistributedPlan distributedPlan :
+ ((NereidsCoordinator) coordinator).getCoordinatorContext().distributedPlans) {
+ if (!(distributedPlan.getFragmentJob().getFragment().getSink() instanceof OlapTableSink)) {
+ continue;
+ }
+ OlapTableSink sink = (OlapTableSink) distributedPlan.getFragmentJob().getFragment().getSink();
+ if (sink.getDstTable().getId() != tableId) {
+ continue;
+ }
+ if (!sink.shouldAssignAdaptiveRandomBucket()) {
+ continue;
+ }
+ sinkInstanceNum += distributedPlan.getInstanceJobs().size();
+ for (AssignedJob assignedJob : distributedPlan.getInstanceJobs()) {
+ sinkBackendIds.add(assignedJob.getAssignedWorker().id());
+ }
+ }
+ if (sinkBackendIds.isEmpty()) {
+ return disabledAdaptiveBucketSinkContext();
+ }
+ return new AdaptiveBucketSinkContext(true, new ArrayList<>(sinkBackendIds), Math.max(sinkInstanceNum, 1));
+ }
+
+ private static void assignAdaptiveBucketToPartition(TOlapTablePartition partition,
+ List<TTabletLocation> partitionTablets, long currentBeId, long tableId, TUniqueId queryId,
+ boolean enableAdaptiveRandomBucket) {
+ if (!enableAdaptiveRandomBucket || !Config.enable_adaptive_random_bucket_load
Review Comment:
This method should treat `enableAdaptiveRandomBucket` from the create/replace-partition request as the per-load source of truth. That flag was fixed when the sink was planned/opened and is sent by BE from `_tablet_finder->is_adaptive_random_bucket()`, but this extra mutable global config check can turn the feature off only for runtime-created/replaced partitions of an in-flight adaptive load. If `ADMIN SET FRONTEND CONFIG ('enable_adaptive_random_bucket_load' = 'false')` runs after planning, FE returns partitions without `bucket_be_id` / `local_bucket_seqs`; the adaptive sender then falls back to its own BE id for channel selection and can later fail with `unknown partition channel` when this BE is not the bucket owner. Please gate runtime replanning on the request's per-load flag, not the current global default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4854600023
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4646296562
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3371964168
##########
be/src/exec/sink/vtablet_finder.cpp:
##########
@@ -22,16 +22,89 @@
#include <gen_cpp/FrontendService_types.h>
#include <glog/logging.h>
+#include <algorithm>
#include <string>
#include <utility>
#include "common/compiler_util.h" // IWYU pragma: keep
+#include "common/config.h"
#include "common/status.h"
#include "core/block/block.h"
#include "runtime/runtime_state.h"
#include "storage/tablet_info.h"
namespace doris {
+
+void AdaptiveRandomBucketState::init_partition(int64_t partition_id,
+ const std::vector<int64_t>& tablets,
+ const std::vector<int32_t>& bucket_seqs,
+ int32_t start_tablet_idx) {
+ if (partition_id < 0 || tablets.empty()) {
+ return;
+ }
+ std::lock_guard<std::mutex> lock(_mutex);
+ if (_partition_states.contains(partition_id)) {
+ return;
Review Comment:
`TabletsChannel` is keyed by load/index and shared by all senders, but each sender's open request can carry a different FE-selected ordered bucket list for the same partition on the same receiver BE. The first sender to initialize the partition wins here, and all later opens for that partition return without merging or validating their `random_bucket_partitions`. For example, if two sink backends are assigned different local start buckets that both route to the same bucket-owner BE, the second sender's rows will be routed using the first sender's tablet order/start, defeating the FE assignment and potentially concentrating all rows on the wrong bucket until rotation. The receiver state needs to either include sender-specific state or merge/validate subsequent ordered lists instead of silently ignoring them.
##########
be/src/load/channel/tablets_channel.cpp:
##########
@@ -639,6 +681,133 @@ Status BaseTabletsChannel::_write_block_data(
return Status::OK();
}
+std::shared_ptr<std::mutex> BaseTabletsChannel::_get_partition_route_lock(int64_t partition_id) {
+ std::lock_guard<std::mutex> l(_partition_route_locks_lock);
+ auto& lock = _partition_route_locks[partition_id];
+ if (lock == nullptr) {
+ lock = std::make_shared<std::mutex>();
+ }
+ return lock;
+}
+
+Status BaseTabletsChannel::_write_block_data_for_receiver_side_random_bucket(
+ const PTabletWriterAddBlockRequest& request, int64_t cur_seq,
+ std::unordered_map<int64_t, DorisVector<uint32_t>>& partition_to_rowidxs,
+ PTabletWriterAddBlockResult* response) {
+ Block send_data;
+ [[maybe_unused]] size_t uncompressed_size = 0;
+ [[maybe_unused]] int64_t uncompressed_time = 0;
+ RETURN_IF_ERROR(send_data.deserialize(request.block(), &uncompressed_size, &uncompressed_time));
+ CHECK(send_data.rows() == request.partition_ids_size())
+ << "block rows: " << send_data.rows()
+ << ", partition_ids_size: " << request.partition_ids_size();
+
+ {
+ std::lock_guard<std::mutex> l(_lock);
+ for (const auto& [partition_id, _] : partition_to_rowidxs) {
+ _partition_ids.emplace(partition_id);
+ }
+ }
+
+ g_tablets_channel_send_data_allocated_size << send_data.allocated_bytes();
+ Defer defer {
+ [&]() { g_tablets_channel_send_data_allocated_size << -send_data.allocated_bytes(); }};
+
+ auto* tablet_errors = response->mutable_tablet_errors();
+ auto* tablet_load_infos = response->mutable_tablet_load_rowset_num_infos();
+
+ auto write_partition_data = [&](int64_t partition_id,
+ const DorisVector<uint32_t>& row_idxs) -> Status {
+ auto partition_lock = _get_partition_route_lock(partition_id);
+ std::lock_guard<std::mutex> partition_guard(*partition_lock);
+
+ CHECK(_adaptive_random_bucket_state != nullptr);
+ int64_t tablet_id = _adaptive_random_bucket_state->current_tablet(partition_id);
+ CHECK(tablet_id >= 0) << "invalid current tablet, load_id=" << _load_id
+ << ", partition_id=" << partition_id;
+ LOG(INFO) << "FIND_TABLET_RANDOM_BUCKET: route+write begin"
+ << ", load_id=" << _load_id << ", index_id=" << _index_id
+ << ", sender_id=" << request.sender_id()
+ << ", packet_seq=" << request.packet_seq() << ", partition_id=" << partition_id
+ << ", tablet_id=" << tablet_id << ", row_count=" << row_idxs.size();
+
+ {
+ std::shared_lock<std::shared_mutex> broken_rlock(_broken_tablets_lock);
+ if (_is_broken_tablet(tablet_id)) {
+ LOG(INFO) << "FIND_TABLET_RANDOM_BUCKET: skip broken tablet"
+ << ", load_id=" << _load_id << ", index_id=" << _index_id
+ << ", sender_id=" << request.sender_id()
+ << ", packet_seq=" << request.packet_seq()
+ << ", partition_id=" << partition_id << ", tablet_id=" << tablet_id;
+ return Status::OK();
+ }
+ }
+
+ decltype(_tablet_writers.find(tablet_id)) tablet_writer_it;
+ {
+ std::lock_guard<std::mutex> l(_tablet_writers_lock);
+ tablet_writer_it = _tablet_writers.find(tablet_id);
+ if (tablet_writer_it == _tablet_writers.end()) {
+ return Status::InternalError("unknown tablet to append data, tablet={}", tablet_id);
+ }
+ }
+
+ bool memtable_flushed = false;
+ Status st = tablet_writer_it->second->write(&send_data, row_idxs, &memtable_flushed);
+ if (!st.ok()) {
Review Comment:
This new receiver-side path keeps an `unordered_map` iterator after releasing `_tablet_writers_lock`, then dereferences it for `write()`, `cancel_with_status()`, and `set_tablet_load_rowset_num_info()`. `add_batch()` is explicitly allowed to run concurrently with `incremental_open()`, and `incremental_open()` inserts into `_tablet_writers` under the same lock; that insert can rehash the map and invalidate this iterator while the partition route lock is held. A concrete auto-partition load can be writing partition P here while another sender incremental-opens a different partition on the same tablets channel, causing undefined behavior or a write through an invalid iterator. Copy a stable `BaseDeltaWriter*` while holding `_tablet_writers_lock` and use that pointer after releasing the lock, matching the fact that writers are inserted but not erased during the channel lifetime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3373983820
##########
be/src/exec/sink/writer/vtablet_writer.cpp:
##########
@@ -973,8 +1054,10 @@ void VNodeChannel::try_send_pending_block(RuntimeState* state) {
}
if (request->eos()) {
- for (auto pid : _parent->_tablet_finder->partition_ids()) {
- request->add_partition_ids(pid);
+ if (!request->is_receiver_side_random_bucket()) {
+ for (auto pid : _parent->_tablet_finder->partition_ids()) {
Review Comment:
Skipping `partition_ids` on EOS for receiver-side random bucket changes the close semantics for touched partitions. In the normal random path every node channel receives the touched partition ids, so `TabletsChannel::close()` closes all writers in those partitions and produces the empty rowsets needed for tablets that did not receive rows. With the new receiver-side path, only receivers that happened to receive row batches record `_partition_ids` during `add_batch()`; all other receivers see an empty partition set on close and cancel their writers. For a random partition whose selected buckets live on one BE while other buckets live on other BEs, those other tablets get no rowset for this committed transaction, leaving tablet versions behind the partition visible version. Please still propagate the touched partition ids on EOS (or otherwise explicitly close the required empty writers) for receiver-side random bucket loads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4660596554
# FE Regression Coverage Report
Increment line coverage ` 42.23% (193/457)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_f552fc1cfb68972e5ff591a0b8668934955c7734_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_f552fc1cfb68972e5ff591a0b8668934955c7734_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3378489608
##########
be/src/load/channel/tablets_channel.cpp:
##########
@@ -675,6 +731,158 @@ Status BaseTabletsChannel::_write_block_data(
return Status::OK();
}
+std::shared_ptr<std::mutex> BaseTabletsChannel::_get_partition_route_lock(int64_t partition_id) {
+ std::lock_guard<std::mutex> l(_partition_route_locks_lock);
+ auto& lock = _partition_route_locks[partition_id];
+ if (lock == nullptr) {
+ lock = std::make_shared<std::mutex>();
+ }
+ return lock;
+}
+
+Status BaseTabletsChannel::_write_block_data_for_receiver_side_random_bucket(
+ const PTabletWriterAddBlockRequest& request, int64_t cur_seq,
+ std::unordered_map<int64_t, DorisVector<uint32_t>>& partition_to_rowidxs,
+ PTabletWriterAddBlockResult* response) {
+ Block send_data;
+ [[maybe_unused]] size_t uncompressed_size = 0;
+ [[maybe_unused]] int64_t uncompressed_time = 0;
+ RETURN_IF_ERROR(send_data.deserialize(request.block(), &uncompressed_size, &uncompressed_time));
+ if (send_data.rows() != request.partition_ids_size()) {
+ return Status::InternalError(
+ "invalid receiver-side random bucket add block request row count, load_id={}, "
+ "index_id={}, packet_seq={}, block_rows={}, partition_ids_size={}",
+ print_id(_load_id), _index_id, request.packet_seq(), send_data.rows(),
+ request.partition_ids_size());
+ }
+
+ {
+ std::lock_guard<std::mutex> l(_lock);
+ for (const auto& [partition_id, _] : partition_to_rowidxs) {
+ _partition_ids.emplace(partition_id);
+ }
+ }
+
+ g_tablets_channel_send_data_allocated_size << send_data.allocated_bytes();
+ Defer defer {
+ [&]() { g_tablets_channel_send_data_allocated_size << -send_data.allocated_bytes(); }};
+
+ auto* tablet_errors = response->mutable_tablet_errors();
+ auto* tablet_load_infos = response->mutable_tablet_load_rowset_num_infos();
+
+ auto write_partition_data = [&](int64_t partition_id,
+ const DorisVector<uint32_t>& row_idxs) -> Status {
+ auto partition_lock = _get_partition_route_lock(partition_id);
+ std::lock_guard<std::mutex> partition_guard(*partition_lock);
+
+ if (_adaptive_random_bucket_state == nullptr) {
+ return Status::InternalError(
+ "receiver-side random bucket state is not initialized, load_id={}, "
+ "index_id={}, packet_seq={}, partition_id={}",
+ print_id(_load_id), _index_id, request.packet_seq(), partition_id);
+ }
+ int64_t tablet_id = _adaptive_random_bucket_state->current_tablet(partition_id);
Review Comment:
This receiver-side random bucket path logs an `INFO` record before and after every partition write in every add-batch RPC. A large stream/broker load can send thousands of batches across many partitions, so these two unconditional logs become a hot-path cost and can flood BE logs/disk even when the load is healthy. Please demote the begin/done records to `VLOG_DEBUG`/`VLOG_PROGRESS` or rate-limit them; keep `INFO` for state transitions/errors such as initialization, rotation, or broken tablets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4670012505
# FE Regression Coverage Report
Increment line coverage ` 43.76% (193/441)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_1997c78bdbc8252f993c10c39035ec5e91e6b06b_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_1997c78bdbc8252f993c10c39035ec5e91e6b06b_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4899812193
<details>
<summary>ClickBench: <b>Total hot run time: 25.87 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 30a431f2b08623dcddbc518296a9a8b5c5a06387, data reload: false
query1 0.01 0.00 0.01
query2 0.15 0.10 0.09
query3 0.38 0.25 0.24
query4 1.61 0.24 0.24
query5 0.33 0.33 0.32
query6 1.16 0.67 0.66
query7 0.04 0.00 0.00
query8 0.09 0.07 0.07
query9 0.53 0.38 0.39
query10 0.60 0.60 0.62
query11 0.33 0.18 0.18
query12 0.32 0.20 0.19
query13 0.55 0.54 0.55
query14 0.94 0.94 0.92
query15 0.69 0.60 0.60
query16 0.40 0.39 0.39
query17 1.06 1.04 1.04
query18 0.33 0.30 0.31
query19 2.02 1.82 1.79
query20 0.02 0.02 0.01
query21 15.45 0.37 0.31
query22 4.98 0.14 0.13
query23 15.82 0.49 0.30
query24 2.48 0.61 0.44
query25 0.16 0.11 0.09
query26 0.75 0.27 0.21
query27 0.10 0.11 0.10
query28 3.38 0.93 0.53
query29 12.47 4.21 3.27
query30 0.38 0.27 0.26
query31 2.76 0.61 0.34
query32 3.22 0.60 0.48
query33 3.02 3.09 3.04
query34 15.66 4.08 3.40
query35 3.27 3.30 3.30
query36 0.65 0.52 0.51
query37 0.12 0.10 0.10
query38 0.09 0.07 0.07
query39 0.07 0.06 0.07
query40 0.21 0.18 0.17
query41 0.13 0.09 0.08
query42 0.08 0.06 0.06
query43 0.08 0.06 0.06
Total cold run time: 96.89 s
Total hot run time: 25.87 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4833553392
<details>
<summary>ClickBench: <b>Total hot run time: 25.25 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eee8cc536909690d1a4bebfa38a1ad0179a59224, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.25 0.13 0.13
query4 1.61 0.14 0.13
query5 0.25 0.25 0.22
query6 1.27 1.07 1.09
query7 0.04 0.00 0.00
query8 0.06 0.04 0.04
query9 0.38 0.31 0.32
query10 0.56 0.55 0.56
query11 0.20 0.15 0.15
query12 0.18 0.14 0.14
query13 0.48 0.48 0.49
query14 1.01 1.01 1.03
query15 0.62 0.60 0.60
query16 0.32 0.32 0.32
query17 1.14 1.10 1.08
query18 0.24 0.23 0.22
query19 2.01 1.93 1.89
query20 0.02 0.02 0.01
query21 15.46 0.21 0.13
query22 4.90 0.06 0.05
query23 16.10 0.30 0.13
query24 3.10 0.40 0.32
query25 0.12 0.05 0.05
query26 0.75 0.21 0.15
query27 0.04 0.04 0.04
query28 3.53 0.94 0.54
query29 12.49 4.31 3.43
query30 0.27 0.15 0.15
query31 2.76 0.60 0.32
query32 3.23 0.61 0.49
query33 3.20 3.28 3.18
query34 15.71 4.26 3.56
query35 3.53 3.53 3.51
query36 0.55 0.44 0.42
query37 0.09 0.06 0.06
query38 0.05 0.04 0.03
query39 0.04 0.02 0.02
query40 0.18 0.16 0.15
query41 0.09 0.04 0.04
query42 0.04 0.03 0.03
query43 0.04 0.04 0.04
Total cold run time: 97.02 s
Total hot run time: 25.25 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4786600723
<details>
<summary>TPC-H: <b>Total hot run time: 29250 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 591feb974a7f4b0b2831b4166b130e3dc8628771, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17596 3978 3948 3948
q2 2010 314 190 190
q3 10379 1416 840 840
q4 4677 477 339 339
q5 7497 870 602 602
q6 183 172 143 143
q7 798 850 623 623
q8 9313 1566 1538 1538
q9 5627 4531 4507 4507
q10 6778 1851 1559 1559
q11 440 283 238 238
q12 631 417 303 303
q13 18102 3505 2773 2773
q14 277 262 246 246
q15 q16 791 784 718 718
q17 947 960 927 927
q18 7084 5796 5561 5561
q19 1180 1248 1070 1070
q20 470 404 265 265
q21 5818 2695 2565 2565
q22 442 359 295 295
Total cold run time: 101040 ms
Total hot run time: 29250 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4311 4234 4256 4234
q2 313 350 219 219
q3 4655 4960 4461 4461
q4 2078 2211 1386 1386
q5 4470 4322 4291 4291
q6 224 177 129 129
q7 1729 1650 1848 1650
q8 2669 2230 2194 2194
q9 8271 8444 8152 8152
q10 4820 4873 4365 4365
q11 566 431 409 409
q12 764 750 588 588
q13 3237 3705 3006 3006
q14 280 313 289 289
q15 q16 719 748 632 632
q17 1352 1324 1321 1321
q18 8080 7363 7362 7362
q19 1188 1142 1067 1067
q20 2215 2212 1998 1998
q21 5293 4611 4423 4423
q22 537 478 408 408
Total cold run time: 57771 ms
Total hot run time: 52584 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4807174342
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4785756408
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4856554702
# BE UT Coverage Report
Increment line coverage `16.78% (120/715)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/a69300c67ec3c6c7ffaa3aa8879fc780ea092edc_a69300c67ec3c6c7ffaa3aa8879fc780ea092edc/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/a69300c67ec3c6c7ffaa3aa8879fc780ea092edc_a69300c67ec3c6c7ffaa3aa8879fc780ea092edc/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 55.11% (21753/39469) |
| Line Coverage | 38.60% (208384/539872) |
| Region Coverage | 34.64% (164149/473862) |
| Branch Coverage | 35.66% (71946/201745) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4859863010
# BE Regression && UT Coverage Report
Increment line coverage `64.34% (460/715)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_a69300c67ec3c6c7ffaa3aa8879fc780ea092edc_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_a69300c67ec3c6c7ffaa3aa8879fc780ea092edc_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 74.39% (28657/38525) |
| Line Coverage | 58.32% (312968/536683) |
| Region Coverage | 54.86% (261117/475967) |
| Branch Coverage | 56.40% (113923/201990) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4832865743
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/eee8cc536909690d1a4bebfa38a1ad0179a59224_eee8cc536909690d1a4bebfa38a1ad0179a59224_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/eee8cc536909690d1a4bebfa38a1ad0179a59224_eee8cc536909690d1a4bebfa38a1ad0179a59224_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.38% (1895/2449) |
| Line Coverage | 64.42% (33983/52753) |
| Region Coverage | 64.82% (17485/26973) |
| Branch Coverage | 54.02% (9370/17344) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4829105134
<details>
<summary>TPC-DS: <b>Total hot run time: 171828 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1ad5cc3a0a4c7e67ea8bd62c1cf9e84c7f493552, data reload: false
query5 4318 623 496 496
query6 424 190 178 178
query7 4828 571 295 295
query8 322 175 167 167
query9 8737 4042 4118 4042
query10 453 314 259 259
query11 5945 2348 2191 2191
query12 155 101 101 101
query13 1296 604 423 423
query14 6368 5304 5022 5022
query14_1 4292 4294 4283 4283
query15 216 206 183 183
query16 1031 481 475 475
query17 1141 739 610 610
query18 2728 481 358 358
query19 211 187 152 152
query20 116 113 110 110
query21 221 142 121 121
query22 13740 13789 13384 13384
query23 17522 16622 16349 16349
query23_1 16355 16324 16299 16299
query24 7698 1789 1314 1314
query24_1 1326 1311 1323 1311
query25 577 462 405 405
query26 1349 338 175 175
query27 2565 574 363 363
query28 4446 2047 2036 2036
query29 1071 625 501 501
query30 313 236 198 198
query31 1116 1087 966 966
query32 97 62 63 62
query33 522 317 260 260
query34 1171 1159 634 634
query35 769 791 679 679
query36 1400 1399 1209 1209
query37 152 111 91 91
query38 1874 1775 1669 1669
query39 931 932 917 917
query39_1 870 866 877 866
query40 219 120 97 97
query41 63 64 61 61
query42 87 88 86 86
query43 319 322 280 280
query44 1412 798 768 768
query45 203 189 179 179
query46 1074 1200 700 700
query47 2384 2343 2169 2169
query48 389 416 293 293
query49 578 428 307 307
query50 999 358 257 257
query51 4342 4443 4327 4327
query52 81 82 68 68
query53 251 256 191 191
query54 271 222 205 205
query55 73 68 64 64
query56 247 223 223 223
query57 1472 1455 1310 1310
query58 248 216 211 211
query59 1600 1653 1429 1429
query60 279 248 232 232
query61 157 156 149 149
query62 708 649 587 587
query63 238 190 188 188
query64 2485 789 619 619
query65 4846 4793 4775 4775
query66 1730 456 339 339
query67 28936 28925 28739 28739
query68 3317 1623 984 984
query69 411 346 264 264
query70 1043 983 979 979
query71 288 241 211 211
query72 2952 2639 2468 2468
query73 830 768 453 453
query74 5127 4972 4783 4783
query75 2582 2560 2186 2186
query76 2308 1173 776 776
query77 348 389 290 290
query78 12574 12517 11886 11886
query79 1469 1124 753 753
query80 1308 474 386 386
query81 519 293 241 241
query82 613 161 124 124
query83 326 271 254 254
query84 264 147 110 110
query85 923 518 422 422
query86 436 308 287 287
query87 1859 1836 1762 1762
query88 3680 2791 2762 2762
query89 425 381 329 329
query90 1953 188 183 183
query91 172 169 132 132
query92 68 62 54 54
query93 1673 1548 837 837
query94 729 350 304 304
query95 665 486 344 344
query96 1041 874 344 344
query97 2734 2689 2550 2550
query98 238 205 200 200
query99 1195 1162 1038 1038
Total cold run time: 258782 ms
Total hot run time: 171828 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4808361827
<details>
<summary>ClickBench: <b>Total hot run time: 25.27 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4f1c936ff2ae8acc7275baf70a22f673b037c6ca, data reload: false
query1 0.01 0.01 0.01
query2 0.09 0.05 0.05
query3 0.26 0.14 0.14
query4 1.60 0.14 0.14
query5 0.24 0.23 0.21
query6 1.26 1.10 1.07
query7 0.03 0.01 0.01
query8 0.06 0.04 0.03
query9 0.38 0.31 0.32
query10 0.55 0.57 0.54
query11 0.20 0.15 0.14
query12 0.19 0.14 0.16
query13 0.48 0.49 0.47
query14 1.03 1.02 1.02
query15 0.63 0.59 0.61
query16 0.32 0.33 0.33
query17 1.10 1.10 1.12
query18 0.23 0.21 0.21
query19 2.04 1.96 1.92
query20 0.02 0.01 0.01
query21 15.44 0.19 0.14
query22 5.00 0.05 0.05
query23 16.11 0.31 0.12
query24 2.94 0.40 0.33
query25 0.11 0.05 0.04
query26 0.73 0.21 0.16
query27 0.04 0.05 0.03
query28 3.53 0.92 0.54
query29 12.51 4.31 3.44
query30 0.28 0.16 0.16
query31 2.77 0.59 0.32
query32 3.23 0.59 0.48
query33 3.21 3.25 3.17
query34 15.63 4.25 3.53
query35 3.58 3.53 3.57
query36 0.55 0.44 0.42
query37 0.10 0.06 0.06
query38 0.05 0.04 0.03
query39 0.04 0.03 0.04
query40 0.17 0.16 0.15
query41 0.09 0.04 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.04
Total cold run time: 96.91 s
Total hot run time: 25.27 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4894298847
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/999)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_30a431f2b08623dcddbc518296a9a8b5c5a06387_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_30a431f2b08623dcddbc518296a9a8b5c5a06387_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](load) adaptive per-BE bucket assignment for random distribution load [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4285821834
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4709072984
<details>
<summary>TPC-DS: <b>Total hot run time: 168352 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 313fe8135eee7e6938e71bddc4ce73eb6e1050de, data reload: false
query5 4308 620 476 476
query6 438 183 167 167
query7 4806 572 320 320
query8 371 204 190 190
query9 8773 4066 4086 4066
query10 442 322 254 254
query11 5887 2308 2132 2132
query12 159 103 97 97
query13 1265 609 418 418
query14 6376 5369 5025 5025
query14_1 4450 4349 4342 4342
query15 202 193 171 171
query16 983 438 431 431
query17 1109 676 556 556
query18 2419 472 339 339
query19 198 177 135 135
query20 112 107 103 103
query21 211 133 119 119
query22 13568 13614 13416 13416
query23 17463 16421 16143 16143
query23_1 16272 16233 16241 16233
query24 7576 1784 1318 1318
query24_1 1299 1302 1308 1302
query25 556 458 393 393
query26 1302 301 168 168
query27 2701 549 330 330
query28 4489 2048 2036 2036
query29 1093 619 503 503
query30 313 236 206 206
query31 1109 1107 945 945
query32 104 60 57 57
query33 512 303 240 240
query34 1165 1099 637 637
query35 741 797 680 680
query36 1387 1365 1171 1171
query37 149 104 90 90
query38 3195 3123 3037 3037
query39 924 919 888 888
query39_1 857 863 871 863
query40 218 120 97 97
query41 63 62 59 59
query42 93 90 91 90
query43 313 316 272 272
query44
query45 197 182 177 177
query46 1058 1216 750 750
query47 2356 2371 2273 2273
query48 403 407 293 293
query49 617 463 347 347
query50 1005 365 262 262
query51 4376 4405 4244 4244
query52 85 89 75 75
query53 249 251 188 188
query54 262 214 191 191
query55 78 74 70 70
query56 225 225 204 204
query57 1441 1390 1328 1328
query58 236 209 205 205
query59 1577 1641 1443 1443
query60 290 235 224 224
query61 156 149 152 149
query62 699 648 584 584
query63 227 186 185 185
query64 2492 771 641 641
query65
query66 1780 445 349 349
query67 29653 29573 29478 29478
query68
query69 422 296 266 266
query70 966 999 950 950
query71 304 224 200 200
query72 2995 2732 2367 2367
query73 807 749 452 452
query74 5136 4930 4805 4805
query75 2641 2565 2230 2230
query76 2341 1116 763 763
query77 345 409 281 281
query78 12409 12388 11924 11924
query79 1255 1094 739 739
query80 518 480 396 396
query81 442 285 236 236
query82 235 152 123 123
query83 262 275 242 242
query84
query85 829 499 400 400
query86 320 298 277 277
query87 3379 3325 3150 3150
query88 3574 2742 2700 2700
query89 419 381 322 322
query90 2147 183 176 176
query91 165 153 134 134
query92 61 59 57 57
query93 1598 1479 879 879
query94 533 346 289 289
query95 668 458 346 346
query96 1007 845 346 346
query97 2674 2687 2600 2600
query98 208 206 197 197
query99 1137 1147 1044 1044
Total cold run time: 249638 ms
Total hot run time: 168352 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4716015835
<details>
<summary>TPC-H: <b>Total hot run time: 28958 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 52e2238c649552e3fd0cad1201b33579cb90c5ce, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17625 4042 4025 4025
q2 2022 323 195 195
q3 10866 1460 843 843
q4 4810 483 340 340
q5 8486 858 578 578
q6 361 178 147 147
q7 876 855 616 616
q8 10621 1508 1490 1490
q9 5925 4544 4510 4510
q10 6858 1780 1525 1525
q11 448 269 239 239
q12 649 426 294 294
q13 18150 3435 2760 2760
q14 263 256 247 247
q15 q16 790 777 711 711
q17 979 924 949 924
q18 6980 5857 5490 5490
q19 1189 1324 1044 1044
q20 465 409 265 265
q21 5686 2674 2416 2416
q22 446 366 299 299
Total cold run time: 104495 ms
Total hot run time: 28958 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4385 4245 4263 4245
q2 323 358 225 225
q3 4551 4955 4405 4405
q4 2074 2178 1387 1387
q5 4457 4315 4345 4315
q6 236 171 127 127
q7 2156 1926 1603 1603
q8 2512 2136 2159 2136
q9 8090 8071 7982 7982
q10 4837 4750 4332 4332
q11 707 417 390 390
q12 758 770 530 530
q13 3325 3662 3060 3060
q14 296 318 296 296
q15 q16 721 730 642 642
q17 1352 1300 1322 1300
q18 8065 7695 6803 6803
q19 1097 1075 1101 1075
q20 2240 2232 1959 1959
q21 5303 4608 4462 4462
q22 530 459 411 411
Total cold run time: 58015 ms
Total hot run time: 51685 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4897219589
# BE Regression && UT Coverage Report
Increment line coverage `25.31% (181/715)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_30a431f2b08623dcddbc518296a9a8b5c5a06387_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_30a431f2b08623dcddbc518296a9a8b5c5a06387_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 58.94% (23589/40019) |
| Line Coverage | 42.27% (235772/557819) |
| Region Coverage | 38.50% (190547/494872) |
| Branch Coverage | 39.05% (83175/213014) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4646054256
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4647422865
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4667903810
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4658394885
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](load) adaptive random bucket load: per-BE bucket assignment with threshold-based rotation [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4287350581
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](load) adaptive random bucket load: per-BE bucket assignment with threshold-based rotation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3116595339
##########
fe/fe-core/src/main/java/org/apache/doris/qe/runtime/ThriftPlansBuilder.java:
##########
@@ -242,8 +271,121 @@ private static void setParamsForOlapTableSink(List<PipelineDistributedPlan> dist
fragmentParams.setNumLocalSink(fragmentParams.getLocalParams().size());
LOG.info("num local sink for backend {} is {}", fragmentParams.getBackendId(),
fragmentParams.getNumLocalSink());
+
+ // Assign per-BE starting bucket and local_bucket_seqs for random distribution
+ if (originalOlapSink != null) {
+ assignRandomBucketPerBe(fragmentParams, worker.id(), originalOlapSink);
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * For random-distribution partitions, override load_tablet_idx and set bucket_be_id /
+ * local_bucket_seqs so that each BE starts writing to one of its own local buckets and
+ * can rotate within them once per-tablet bytes exceed the threshold.
+ *
+ * <p>local_bucket_seqs always contains all local buckets for this BE. The starting bucket
+ * rotates across consecutive load tasks via tabletIndex; within a single load, the BE
+ * switches to the next entry in the list only when the 200 MB threshold is reached.
+ * Small loads (e.g. stream load) naturally stay on one bucket because the threshold is
+ * never triggered.
+ *
+ * <p>When {@code Config.enable_adaptive_random_bucket_load} is true, this is called for all
+ * random-distribution tables, including those with {@code load_to_single_tablet=true}.
+ *
+ * @param originalOlapSink the shared TOlapTableSink captured before any per-BE mutation;
+ * every BE deep-copies from this original to avoid one BE's copy
+ * becoming the base for the next BE's copy.
+ */
+ private static void assignRandomBucketPerBe(TPipelineFragmentParams fragmentParams,
+ long beId, TOlapTableSink originalOlapSink) {
+ if (originalOlapSink.getPartition() == null
+ || !originalOlapSink.getPartition().isSetPartitions()) {
+ return;
+ }
+
+ // Check whether any partition uses random distribution (identified by load_tablet_idx being set)
+ boolean hasRandomPartition = originalOlapSink.getPartition().getPartitions().stream()
+ .anyMatch(TOlapTablePartition::isSetLoadTabletIdx);
+ if (!hasRandomPartition) {
+ return;
+ }
+
+ // Build tablet_id -> BE IDs index from the original location (not the potentially-mutated copy)
+ TOlapTableLocationParam location = originalOlapSink.getLocation();
+ if (location == null || !location.isSetTablets()) {
+ return;
+ }
+ Map<Long, List<Long>> tabletToBeIds = new HashMap<>();
+ for (TTabletLocation tabletLoc : location.getTablets()) {
+ tabletToBeIds.put(tabletLoc.getTabletId(), tabletLoc.getNodeIds());
+ }
+
+ // Deep-copy from the original so that per-BE mutations are fully isolated.
+ // Using originalOlapSink (not outputSink.getOlapTableSink()) ensures that even if a
+ // previous BE's call already replaced the shared reference, we still start from clean data.
+ TDataSink outputSink = fragmentParams.getFragment().getOutputSink();
+ TOlapTableSink sinkCopy = originalOlapSink.deepCopy();
+ outputSink.setOlapTableSink(sinkCopy);
+
+ for (TOlapTablePartition tPartition : sinkCopy.getPartition().getPartitions()) {
+ if (!tPartition.isSetLoadTabletIdx()) {
+ continue; // hash distribution – leave as-is
+ }
+
+ int tabletIndex = (int) tPartition.getLoadTabletIdx();
+ List<org.apache.doris.thrift.TOlapTableIndexTablets> indexes = tPartition.getIndexes();
+ if (indexes == null || indexes.isEmpty()) {
+ continue;
+ }
+ List<Long> tablets = indexes.get(0).getTablets();
+ int numBuckets = tPartition.getNumBuckets();
+
+ // Collect bucket indices assigned to this BE.
+ //
+ // For single-replica tablets, each bucket has exactly one entry in node_ids,
+ // so the bucket naturally belongs to one BE.
+ //
+ // For multi-replica tablets, multiple BEs host the same bucket. To avoid all BEs
+ // writing to the same bucket(s), we assign each bucket to exactly one BE using a
+ // deterministic rule: sort node_ids, then give bucket[i] to the BE at position
+ // (i % numReplicas) in the sorted list. This spreads buckets evenly and requires
+ // no coordination.
+ List<Integer> localBuckets = new ArrayList<>();
+ for (int bucketIdx = 0; bucketIdx < tablets.size(); bucketIdx++) {
+ List<Long> beIds = tabletToBeIds.get(tablets.get(bucketIdx));
+ if (beIds == null || beIds.isEmpty()) {
+ continue;
+ }
+ if (beIds.size() == 1) {
+ if (beIds.get(0).equals(beId)) {
+ localBuckets.add(bucketIdx);
+ }
+ } else {
+ // Multi-replica: deterministic per-bucket owner assignment
Review Comment:
`fragment.toThrift()` still reuses the same `TDataSink` instance for every worker because `PlanFragment.toThrift()` stores `sink.toThrift()` and `OlapTableSink.toThrift()` just returns its cached `tDataSink`. Here we only deep-copy the nested `TOlapTableSink`, then replace it on that shared `outputSink`, so the last worker processed overwrites the bucket assignment seen by earlier workers. To make the mutation truly BE-local we need to deep-copy the enclosing `TDataSink` (or `TPlanFragment`) before calling `setOlapTableSink()`.
##########
be/src/storage/tablet_info.cpp:
##########
@@ -700,10 +700,15 @@ Status VOlapTablePartitionParam::generate_partition_from(const TOlapTablePartiti
part_result = _obj_pool.add(new VOlapTablePartition(&_partition_block));
part_result->id = t_part.id;
part_result->is_mutable = t_part.is_mutable;
- // only load_to_single_tablet = true will set load_tablet_idx
if (t_part.__isset.load_tablet_idx) {
part_result->load_tablet_idx = t_part.load_tablet_idx;
}
+ if (t_part.__isset.bucket_be_id) {
+ part_result->bucket_be_id = t_part.bucket_be_id;
+ }
+ if (t_part.__isset.local_bucket_seqs) {
+ part_result->local_bucket_seqs = t_part.local_bucket_seqs;
Review Comment:
This only fixes the initial `generate_partition_from()` path. The runtime refresh paths still omit the new adaptive fields: FE `FrontendServiceImpl.createPartition()/replacePartition()` build `TOlapTablePartition` without `load_tablet_idx` and `local_bucket_seqs`, and BE `add_partitions()/replace_partitions()` also do not copy them. Once `FIND_TABLET_RANDOM_BUCKET` sees one of those new partitions, `load_tablet_idx` stays `-1` and `vtablet_finder.cpp` indexes `indexes[0].tablets[load_tablet_idx % num_buckets]`, which becomes an out-of-bounds access. The same propagation has to be added on the runtime partition-update path, not just initial sink construction.
##########
be/src/exec/sink/vtablet_finder.cpp:
##########
@@ -82,8 +83,82 @@ Status OlapTabletFinder::find_tablets(RuntimeState* state, Block* block, int row
if (_find_tablet_mode == FindTabletMode::FIND_TABLET_EVERY_ROW) {
_vpartition->find_tablets(block, qualified_rows, partitions, tablet_index);
+ } else if (_find_tablet_mode == FindTabletMode::FIND_TABLET_RANDOM_BUCKET) {
+ // Use per-batch caching: all rows in the same partition go to the same tablet this batch.
+ _vpartition->find_tablets(block, qualified_rows, partitions, tablet_index,
+ &_partition_to_tablet_map);
+
+ // Count actual rows per partition so we don't credit each partition with the
+ // full block size when a block spans multiple partitions.
+ flat_hash_map<VOlapTablePartition*, int64_t> partition_row_counts;
+ for (uint32_t qi : qualified_rows) {
+ partition_row_counts[partitions[qi]]++;
+ }
+
+ int64_t bytes_per_row = rows > 0 ? block->bytes() / rows : 0;
+ for (auto& [partition, _] : _partition_to_tablet_map) {
+ if (partition->indexes.empty()) {
+ continue;
+ }
+ const auto& seqs = partition->local_bucket_seqs;
Review Comment:
`_tablet_written_bytes` is lifetime bytes per tablet, but the threshold is supposed to be bytes written since this bucket became active. After bucket 0 crosses 200 MB once, wrapping back to bucket 0 later will re-enter this branch on the very next batch because its counter is still at or above 200 MB. That turns large loads into "rotate every batch after the first cycle" instead of writing roughly 200 MB chunks. We need to reset or subtract the current bucket counter when we advance, or track bytes per active stint rather than lifetime bytes per tablet.
##########
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##########
@@ -3610,4 +3610,15 @@ public static int metaServiceRpcRetryTimes() {
+ "obtaining partition version information when calculating the delete bitmap. Enabled "
+ "by default."})
public static boolean calc_delete_bitmap_get_versions_waiting_for_pending_txns = true;
+
+ @ConfField(mutable = true, masterOnly = true, description = {
+ "Whether to enable adaptive random bucket load. When enabled, the FE assigns each BE only the "
+ + "buckets whose primary replica it hosts (local_bucket_seqs), and the BE rotates across "
+ + "those buckets once per-tablet write volume exceeds the threshold (default 200 MB). "
+ + "This reduces import memory pressure and improves throughput for random-distribution "
+ + "tables. When enabled, load_to_single_tablet is also subsumed by this strategy.",
Review Comment:
This default `true` config changes the meaning of the existing `load_to_single_tablet=true` option on Nereids-distributed broker loads. `NereidsLoadPlanInfoCollector` still forwards `taskInfo.isLoadToSingleTablet()`, `LoadLoadingTask` goes through `ThriftPlansBuilder`, and the BE now prefers `FIND_TABLET_RANDOM_BUCKET` whenever `local_bucket_seqs` is present. So an explicit "write one tablet per partition for this load" request can now fan out across multiple local buckets, and even rotate after 200 MB, with no opt-in. That is a user-visible compatibility break; the adaptive rewrite should skip `load_to_single_tablet=true` or be behind a separate property.
##########
be/src/exec/sink/writer/vtablet_writer.cpp:
##########
@@ -1538,12 +1538,25 @@ Status VTabletWriter::_init(RuntimeState* state, RuntimeProfile* profile) {
if (table_sink.__isset.send_batch_parallelism && table_sink.send_batch_parallelism > 1) {
_send_batch_parallelism = table_sink.send_batch_parallelism;
}
- // if distributed column list is empty, we can ensure that tablet is with random distribution info
- // and if load_to_single_tablet is set and set to true, we should find only one tablet in one partition
- // for the whole olap table sink
+ // If distributed column list is empty, the table uses random distribution.
+ // Mode priority (highest to lowest):
+ // 1. FIND_TABLET_RANDOM_BUCKET: FE set local_bucket_seqs on at least one partition,
+ // meaning enable_adaptive_random_bucket_load is ON. This overrides load_to_single_tablet
Review Comment:
This comment overstates what the v1 sink path does. `VTabletWriter::_init()` still walks every tablet in every partition at lines 1654-1675 and `IndexChannel::init()/VNodeChannel::open()` opens them all up front, so the adaptive routing only changes which tablets receive rows later. The import-memory reduction claimed here is therefore only true on the lazy `VTabletWriterV2` path, not on `VTabletWriter`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4707503246
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3371318449
##########
be/src/cloud/cloud_tablets_channel.cpp:
##########
@@ -60,6 +60,21 @@ Status CloudTabletsChannel::add_batch(const PTabletWriterAddBlockRequest& reques
return Status::OK();
}
+ if (request.is_receiver_side_random_bucket()) {
+ std::unordered_map<int64_t, DorisVector<uint32_t>> partition_to_rowidxs;
+ _build_partition_to_rowidxs_for_receiver_side_random_bucket(request, &partition_to_rowidxs);
+ if (!partition_to_rowidxs.empty()) {
+ std::unordered_set<int64_t> partition_ids;
+ partition_ids.reserve(partition_to_rowidxs.size());
+ for (const auto& [partition_id, _] : partition_to_rowidxs) {
+ partition_ids.insert(partition_id);
+ }
+ RETURN_IF_ERROR(_init_writers_by_partition_ids(partition_ids));
Review Comment:
This new receiver-side path calls `_init_writers_by_partition_ids()` without holding `_tablet_writers_lock`, but `add_batch()` can run concurrently with `incremental_open()` (the existing tablet-routed path takes the lock before walking `_tablet_writers` for exactly that reason). In an auto-partition/replace-partition load, one sender can be incremental-opening new tablet writers while another sender reaches this branch and `_init_writers_by_partition_ids()` iterates `_tablet_writers`; that is an unsynchronized `unordered_map` traversal concurrent with insertion and can invalidate the iteration or miss writers before `CloudDeltaWriter::batch_init()`. Please take `_tablet_writers_lock` around this initialization path, matching the non-random cloud path below, before iterating or initializing writers by partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4762426544
# BE Regression && UT Coverage Report
Increment line coverage `65.73% (466/709)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_52e2238c649552e3fd0cad1201b33579cb90c5ce_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_52e2238c649552e3fd0cad1201b33579cb90c5ce_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 74.13% (28433/38354) |
| Line Coverage | 58.02% (310070/534403) |
| Region Coverage | 54.89% (260068/473824) |
| Branch Coverage | 56.17% (112753/200752) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4762081542
# FE UT Coverage Report
Increment line coverage ` 37.03% (157/424)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_52e2238c649552e3fd0cad1201b33579cb90c5ce/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_52e2238c649552e3fd0cad1201b33579cb90c5ce/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4808801369
# BE Regression && UT Coverage Report
Increment line coverage `65.01% (459/706)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_4f1c936ff2ae8acc7275baf70a22f673b037c6ca_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_4f1c936ff2ae8acc7275baf70a22f673b037c6ca_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 74.26% (28543/38437) |
| Line Coverage | 58.10% (310969/535241) |
| Region Coverage | 54.72% (259623/474419) |
| Branch Coverage | 56.20% (113010/201071) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4787129656
# FE Regression Coverage Report
Increment line coverage ` 50.00% (217/434)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_591feb974a7f4b0b2831b4166b130e3dc8628771_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_591feb974a7f4b0b2831b4166b130e3dc8628771_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3475262228
##########
fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java:
##########
@@ -539,6 +543,98 @@ public Map<String, Integer> getBeToInstancesNum() {
return result;
}
+ public static final class AdaptiveRandomBucketSinkContext {
+ private final List<Long> sinkBackendIds;
+ private final int planFragmentNum;
+
+ private AdaptiveRandomBucketSinkContext(List<Long> sinkBackendIds, int planFragmentNum) {
+ this.sinkBackendIds = sinkBackendIds;
+ this.planFragmentNum = planFragmentNum;
+ }
+
+ public List<Long> getSinkBackendIds() {
+ return sinkBackendIds;
+ }
+
+ public int getPlanFragmentNum() {
+ return planFragmentNum;
+ }
+ }
+
+ public Optional<AdaptiveRandomBucketSinkContext> getAdaptiveRandomBucketSinkContext(long tableId) {
+ Set<Long> sinkBackendIds = new TreeSet<>();
+ int planFragmentNum = 0;
+ for (PipelineExecContext context : pipelineExecContexts.values()) {
+ TPipelineFragmentParams params = context.rpcParams;
+ if (params.getFragment().getOutputSink() == null
+ || params.getFragment().getOutputSink().getType() != TDataSinkType.OLAP_TABLE_SINK) {
+ continue;
+ }
+ TOlapTableSink sink = params.getFragment().getOutputSink().getOlapTableSink();
+ if (sink.getTableId() != tableId) {
+ continue;
+ }
+ if (!OlapTableSink.shouldAssignAdaptiveRandomBucket(sink)) {
+ continue;
+ }
+ sinkBackendIds.add(params.getBackendId());
+ planFragmentNum += params.getLocalParamsSize();
+ }
+ if (sinkBackendIds.isEmpty()) {
+ return Optional.empty();
+ }
+ return Optional.of(new AdaptiveRandomBucketSinkContext(
+ new ArrayList<>(sinkBackendIds), Math.max(planFragmentNum, 1)));
+ }
+
+ private static void assignAdaptiveRandomBucketForFragment(
+ Collection<TPipelineFragmentParams> fragmentParamsList) {
+ List<TPipelineFragmentParams> sinkParams = fragmentParamsList.stream()
+ .filter(param -> param.getFragment().getOutputSink() != null
+ && param.getFragment().getOutputSink().getType() == TDataSinkType.OLAP_TABLE_SINK)
+ .collect(Collectors.toList());
+ if (sinkParams.isEmpty()) {
+ return;
+ }
+ TOlapTableSink sink = sinkParams.get(0).getFragment().getOutputSink().getOlapTableSink();
+ if (!OlapTableSink.shouldAssignAdaptiveRandomBucket(sink)) {
+ return;
+ }
+ List<Long> sinkBackendIds = sinkParams.stream()
+ .map(TPipelineFragmentParams::getBackendId)
+ .distinct()
+ .sorted()
+ .collect(Collectors.toList());
+ int planFragmentNum = sinkParams.stream()
Review Comment:
planFragmentNum -> sinkInstanceNum
##########
be/src/exec/sink/writer/vtablet_writer.cpp:
##########
@@ -676,6 +716,77 @@ void VNodeChannel::_open_internal(bool is_incremental) {
request->set_txn_expiration(_parent->_txn_expiration);
request->set_write_file_cache(_parent->_write_file_cache);
+ if (_parent->_tablet_finder->is_adaptive_random_bucket()) {
Review Comment:
Consider extracting the larger is_adaptive_random_bucket() blocks into named methods.
##########
be/src/load/channel/tablets_channel.h:
##########
@@ -185,6 +197,10 @@ class BaseTabletsChannel {
std::unordered_set<int64_t> _reducing_tablets;
std::unordered_set<int64_t> _partition_ids;
+ std::shared_ptr<AdaptiveRandomBucketState> _adaptive_random_bucket_state;
+ std::mutex _partition_route_locks_lock;
+ std::unordered_map<int32_t, std::unordered_map<int64_t, std::shared_ptr<std::mutex>>>
Review Comment:
add comment.
##########
be/src/load/channel/tablets_channel.cpp:
##########
@@ -683,6 +728,169 @@ Status BaseTabletsChannel::_write_block_data(
return Status::OK();
}
+std::shared_ptr<std::mutex> BaseTabletsChannel::_get_sender_partition_route_lock(
Review Comment:
why need this lock?
##########
be/src/common/config.cpp:
##########
@@ -749,6 +749,8 @@ DEFINE_mInt32(memory_gc_sleep_time_ms, "500");
// max write buffer size before flush, default 200MB
DEFINE_mInt64(write_buffer_size, "209715200");
DEFINE_mBool(enable_adaptive_write_buffer_size, "true");
+// Whether random bucket load rotates to the next local bucket when memtable flushes.
+DEFINE_mBool(enable_adaptive_random_bucket_load_bucket_rotation, "true");
Review Comment:
Do we need this config.
##########
be/src/load/channel/tablets_channel.cpp:
##########
@@ -702,10 +910,18 @@ Status TabletsChannel::add_batch(const PTabletWriterAddBlockRequest& request,
return Status::OK();
}
+ if (request.is_receiver_side_random_bucket()) {
Review Comment:
unify adaptive_random_bucket vs receiver_side_random_bucket
The same logical switch is renamed several times across the stack, which makes it hard to tell whether these are the same concept or different ones:
Config.enable_adaptive_random_bucket_load (FE config)
→ TOlapTableSink.enable_adaptive_random_bucket (thrift)
→ FIND_TABLET_RANDOM_BUCKET / is_adaptive_random_bucket() (BE finder)
→ PTabletWriter*.is_receiver_side_random_bucket (proto / receiver)
##########
be/src/exec/sink/vtablet_finder.h:
##########
@@ -19,15 +19,45 @@
#include <cstdint>
#include <map>
+#include <memory>
+#include <mutex>
+#include <unordered_map>
+#include <vector>
#include "common/status.h"
#include "core/block/block.h"
#include "exec/common/hash_table/phmap_fwd_decl.h"
#include "storage/tablet_info.h"
#include "util/bitmap.h"
+#include "util/uid_util.h"
namespace doris {
+class AdaptiveRandomBucketState {
Review Comment:
AdaptiveRandomBucketState looks misplaced in vtablet_finder.h
AdaptiveRandomBucketState is purely receiver-side state — it's only used by BaseTabletsChannel . The sender-side OlapTabletFinder in this same header never touches it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4894132749
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_30a431f2b08623dcddbc518296a9a8b5c5a06387_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_30a431f2b08623dcddbc518296a9a8b5c5a06387_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 65.07% (25835/39705) |
| Line Coverage | 48.83% (270451/553819) |
| Region Coverage | 45.57% (224180/491939) |
| Branch Coverage | 46.73% (98890/211625) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](load) adaptive random bucket load: per-BE bucket assignment with threshold-based rotation [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4286884797
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4854577010
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4828480587
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4829728487
# BE Regression && UT Coverage Report
Increment line coverage `64.91% (457/704)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_1ad5cc3a0a4c7e67ea8bd62c1cf9e84c7f493552_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_1ad5cc3a0a4c7e67ea8bd62c1cf9e84c7f493552_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 74.12% (28489/38434) |
| Line Coverage | 57.99% (310247/534976) |
| Region Coverage | 54.66% (259256/474295) |
| Branch Coverage | 56.08% (112702/200967) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4854117342
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4656309969
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3378088001
##########
be/src/load/delta_writer/delta_writer.cpp:
##########
@@ -155,10 +161,22 @@ Status BaseDeltaWriter::init() {
return Status::OK();
}
-Status DeltaWriter::write(const Block* block, const DorisVector<uint32_t>& row_idxs) {
+Status DeltaWriter::write(const Block* block, const DorisVector<uint32_t>& row_idxs,
+ bool* memtable_flushed) {
+ if (memtable_flushed != nullptr) {
+ *memtable_flushed = false;
+ }
if (UNLIKELY(row_idxs.empty())) {
return Status::OK();
}
+ if (_req.enable_table_memtable_backpressure) {
Review Comment:
This new table-level backpressure also applies to high-priority load channels. `LoadChannelMgr::add_batch()` explicitly skips `handle_memtable_flush()` for high-priority channels because blocking here can make the RPC time out, but receiver-side random-bucket writers set `enable_table_memtable_backpressure` and then enter this wait unconditionally. A high-priority stream/load with a short execution timeout can therefore block behind `table_memtable_flush_pending_count_limit`, defeating the existing high-priority bypass. Please skip this table-level wait for `_req.is_high_priority` (or otherwise keep it bounded/non-blocking for high-priority loads); the same consideration applies to the cloud and v2 writer paths that use this flag.
##########
be/src/load/memtable/memtable_writer.cpp:
##########
@@ -139,6 +144,9 @@ Status MemTableWriter::write(const Block* block, const DorisVector<uint32_t>& ro
}
if (UNLIKELY(_mem_table->need_flush())) {
RETURN_IF_ERROR(_flush_memtable());
+ if (memtable_flushed != nullptr) {
Review Comment:
`memtable_flushed` is only set for the `need_flush()` branch, but `write()` can also flush successfully before insert when the raw-row count would overflow `int32_t`. Receiver-side random-bucket routing advances only when `BaseTabletsChannel` sees `memtable_flushed=true`, so this flush path leaves the partition pinned to the same tablet even though a memtable was just submitted. Please set the flag for every successful `_flush_memtable()` in this method, including the raw-row overflow branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](load) adaptive random bucket load: per-BE bucket assignment with threshold-based rotation [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4645241130
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](load) adaptive random bucket load: per-BE bucket assignment with threshold-based rotation [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4286882878
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](load) adaptive random bucket load: per-BE bucket assignment with threshold-based rotation [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4286011073
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](load) adaptive random bucket load: per-BE bucket assignment with threshold-based rotation [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4286886119
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4807184460
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4832855961
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4833511318
<details>
<summary>TPC-DS: <b>Total hot run time: 172285 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6cf651de54175959a128b07b857fa64e939af881, data reload: false
query5 4321 626 489 489
query6 443 189 172 172
query7 4804 578 312 312
query8 334 187 166 166
query9 8776 4102 4108 4102
query10 444 321 263 263
query11 5797 2349 2205 2205
query12 153 101 95 95
query13 1246 585 418 418
query14 6238 5295 4894 4894
query14_1 4228 4237 4251 4237
query15 220 201 178 178
query16 1040 453 369 369
query17 1107 706 558 558
query18 2459 465 339 339
query19 201 183 142 142
query20 107 106 105 105
query21 224 144 118 118
query22 13622 13513 13427 13427
query23 17387 16507 16096 16096
query23_1 16278 16249 16238 16238
query24 7621 1814 1294 1294
query24_1 1266 1298 1286 1286
query25 536 433 357 357
query26 1302 322 162 162
query27 2670 548 355 355
query28 4492 2055 2029 2029
query29 1073 604 483 483
query30 307 238 196 196
query31 1100 1069 942 942
query32 109 63 55 55
query33 535 311 247 247
query34 1153 1175 664 664
query35 752 767 662 662
query36 1319 1413 1212 1212
query37 152 109 90 90
query38 1873 1707 1657 1657
query39 933 906 890 890
query39_1 862 873 939 873
query40 222 126 106 106
query41 71 69 69 69
query42 91 89 90 89
query43 331 322 279 279
query44 1516 792 786 786
query45 205 192 189 189
query46 1120 1198 729 729
query47 2348 2318 2241 2241
query48 437 418 310 310
query49 592 432 326 326
query50 993 382 270 270
query51 4428 4364 4294 4294
query52 84 83 72 72
query53 257 274 198 198
query54 279 241 214 214
query55 76 73 69 69
query56 255 244 242 242
query57 1428 1419 1321 1321
query58 252 219 222 219
query59 1617 1646 1451 1451
query60 293 253 236 236
query61 175 173 174 173
query62 710 659 587 587
query63 239 198 196 196
query64 2583 822 679 679
query65 4846 4752 4778 4752
query66 1790 473 356 356
query67 29691 29580 29577 29577
query68 3250 1558 924 924
query69 420 312 271 271
query70 1075 1001 984 984
query71 305 233 218 218
query72 2962 2563 2326 2326
query73 845 816 450 450
query74 5099 4921 4743 4743
query75 2559 2538 2172 2172
query76 2337 1200 822 822
query77 360 379 288 288
query78 12359 12378 11842 11842
query79 1453 1211 778 778
query80 574 459 369 369
query81 454 282 236 236
query82 575 156 119 119
query83 356 278 244 244
query84 265 137 111 111
query85 849 512 436 436
query86 373 292 280 280
query87 1845 1821 1778 1778
query88 3692 2797 2789 2789
query89 430 380 328 328
query90 1968 187 188 187
query91 176 165 132 132
query92 64 62 58 58
query93 1516 1526 919 919
query94 552 356 301 301
query95 668 381 357 357
query96 1137 830 331 331
query97 2700 2716 2568 2568
query98 226 208 197 197
query99 1166 1144 1036 1036
Total cold run time: 257533 ms
Total hot run time: 172285 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4715418809
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/52e2238c649552e3fd0cad1201b33579cb90c5ce_52e2238c649552e3fd0cad1201b33579cb90c5ce_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/52e2238c649552e3fd0cad1201b33579cb90c5ce_52e2238c649552e3fd0cad1201b33579cb90c5ce_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.32% (1889/2443) |
| Line Coverage | 64.39% (33952/52725) |
| Region Coverage | 64.76% (17451/26948) |
| Branch Coverage | 53.94% (9340/17316) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4714212890
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4715727573
# FE UT Coverage Report
Increment line coverage ` 37.03% (157/424)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_52e2238c649552e3fd0cad1201b33579cb90c5ce/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_52e2238c649552e3fd0cad1201b33579cb90c5ce/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4656392757
<details>
<summary>TPC-H: <b>Total hot run time: 29540 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2054fc372677684e4f1aa9088f1011f91a4ce5d9, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17552 4048 4000 4000
q2 q3 10696 1360 812 812
q4 4689 498 365 365
q5 7769 894 589 589
q6 194 177 137 137
q7 780 829 657 657
q8 9403 1608 1717 1608
q9 5936 4545 4560 4545
q10 6754 1805 1570 1570
q11 433 280 263 263
q12 634 429 292 292
q13 18168 3451 2811 2811
q14 266 251 234 234
q15 q16 835 791 718 718
q17 996 870 1024 870
q18 7230 5866 5679 5679
q19 1378 1256 1068 1068
q20 503 417 265 265
q21 6400 2806 2741 2741
q22 477 379 316 316
Total cold run time: 101093 ms
Total hot run time: 29540 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5215 4771 4944 4771
q2 q3 4919 5308 4711 4711
q4 2124 2216 1393 1393
q5 4827 5029 4728 4728
q6 229 182 129 129
q7 1946 1797 1616 1616
q8 2417 2124 2098 2098
q9 8076 7590 7483 7483
q10 4752 4719 4216 4216
q11 540 389 354 354
q12 746 745 532 532
q13 3099 3427 2804 2804
q14 276 278 248 248
q15 q16 693 708 615 615
q17 1309 1297 1305 1297
q18 7321 6938 6747 6747
q19 1129 1051 1098 1051
q20 2242 2215 1960 1960
q21 5314 4712 4528 4528
q22 512 465 422 422
Total cold run time: 57686 ms
Total hot run time: 51703 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3378768057
##########
be/src/exec/sink/writer/vtablet_writer_v2.cpp:
##########
@@ -158,9 +158,10 @@ Status VTabletWriterV2::_init(RuntimeState* state, RuntimeProfile* profile) {
_location = _pool->add(new OlapTableLocationParam(table_sink.location));
_nodes_info = _pool->add(new DorisNodesInfo(table_sink.nodes_info));
- // if distributed column list is empty, we can ensure that tablet is with random distribution info
- // and if load_to_single_tablet is set and set to true, we should find only one tablet in one partition
- // for the whole olap table sink
+ // If distributed column list is empty, the table uses random distribution.
+ // Mode priority (highest to lowest):
+ // 1. FIND_TABLET_EVERY_SINK: load_to_single_tablet=true (legacy single-tablet mode).
+ // 2. FIND_TABLET_EVERY_BATCH: default round-robin per batch.
Review Comment:
This V2 sink path still forces every random-distribution load into the legacy `FIND_TABLET_EVERY_BATCH` mode even when FE has set `enable_adaptive_random_bucket` and already rewritten each backend's `load_tablet_idx`/`local_bucket_seqs`. In cloud load-stream execution that means V2 does not use receiver-side local-bucket routing or memtable-flush rotation, but it does inherit the per-BE rewritten starting bucket from `ThriftPlansBuilder`/`Coordinator`. For example, two sink BEs can start from FE-selected buckets 0 and 1, then V2 increments the global `load_tablet_idx` per batch and routes subsequent batches to buckets 1 and 2 instead of staying within the assigned local sequences. Please either make V2 honor the adaptive mode end-to-end, or keep FE from applying adaptive assignments when this sink implementation will be used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4657948695
<details>
<summary>TPC-DS: <b>Total hot run time: 169025 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4b7e6eb1cca9bfa6551d41a3b3d5feeceaba7afc, data reload: false
query5 4325 633 486 486
query6 442 197 179 179
query7 4864 578 301 301
query8 373 215 214 214
query9 8776 4018 4002 4002
query10 463 303 256 256
query11 5888 2318 2154 2154
query12 158 102 103 102
query13 1265 595 428 428
query14 6435 5388 5088 5088
query14_1 4410 4371 4393 4371
query15 207 202 174 174
query16 1029 484 475 475
query17 1139 736 621 621
query18 2689 479 334 334
query19 195 177 141 141
query20 111 106 103 103
query21 212 137 116 116
query22 13595 13543 13367 13367
query23 17320 16483 16161 16161
query23_1 16280 16413 16282 16282
query24 7412 1764 1294 1294
query24_1 1292 1297 1280 1280
query25 558 448 372 372
query26 1287 323 164 164
query27 2706 556 343 343
query28 4453 1999 2016 1999
query29 1065 613 473 473
query30 306 240 200 200
query31 1109 1069 952 952
query32 108 63 59 59
query33 522 313 249 249
query34 1177 1109 655 655
query35 753 779 676 676
query36 1371 1424 1204 1204
query37 155 104 100 100
query38 3206 3167 3061 3061
query39 930 940 911 911
query39_1 897 879 878 878
query40 236 122 98 98
query41 64 62 61 61
query42 95 97 89 89
query43 316 357 281 281
query44
query45 196 188 179 179
query46 1067 1197 698 698
query47 2379 2400 2323 2323
query48 393 392 306 306
query49 635 468 348 348
query50 1030 355 245 245
query51 4356 4276 4304 4276
query52 93 88 86 86
query53 243 265 190 190
query54 277 212 203 203
query55 77 77 68 68
query56 226 223 209 209
query57 1441 1416 1339 1339
query58 253 217 215 215
query59 1534 1645 1400 1400
query60 286 248 230 230
query61 161 164 162 162
query62 696 649 572 572
query63 246 183 187 183
query64 2560 827 694 694
query65
query66 1745 465 339 339
query67 29019 29590 29457 29457
query68
query69 414 304 264 264
query70 977 945 934 934
query71 306 216 211 211
query72 3000 2767 2370 2370
query73 861 742 432 432
query74 5095 4997 4752 4752
query75 2654 2606 2236 2236
query76 2311 1134 765 765
query77 359 393 298 298
query78 12441 12499 11981 11981
query79 1236 1044 772 772
query80 568 511 430 430
query81 450 288 247 247
query82 246 161 126 126
query83 285 285 258 258
query84
query85 940 620 529 529
query86 323 315 290 290
query87 3379 3303 3269 3269
query88 3605 2763 2756 2756
query89 404 386 338 338
query90 2137 183 176 176
query91 173 156 135 135
query92 61 62 59 59
query93 1449 1419 874 874
query94 532 351 334 334
query95 674 366 440 366
query96 1051 794 338 338
query97 2698 2729 2582 2582
query98 211 206 209 206
query99 1156 1167 1039 1039
Total cold run time: 249585 ms
Total hot run time: 169025 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4658340803
# FE UT Coverage Report
Increment line coverage ` 4.88% (19/389)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_4b7e6eb1cca9bfa6551d41a3b3d5feeceaba7afc/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_4b7e6eb1cca9bfa6551d41a3b3d5feeceaba7afc/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4668239293
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/1997c78bdbc8252f993c10c39035ec5e91e6b06b_1997c78bdbc8252f993c10c39035ec5e91e6b06b_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/1997c78bdbc8252f993c10c39035ec5e91e6b06b_1997c78bdbc8252f993c10c39035ec5e91e6b06b_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.39% (1915/2443) |
| Line Coverage | 64.90% (34217/52725) |
| Region Coverage | 65.32% (17603/26948) |
| Branch Coverage | 54.03% (9355/17316) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3510193669
##########
regression-test/suites/load_p0/insert/test_adaptive_random_bucket_load.groovy:
##########
@@ -0,0 +1,76 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_adaptive_random_bucket_load", "p0,nonConcurrent") {
Review Comment:
This test is meant to cover adaptive random bucket in cloud mode, but with the current tags it appears to be skipped by the cloud CI path. `cloud_p0` selects `p0` suites, but its runner passes `-runNonConcurrent false`, and the framework returns before running `nonConcurrent`/SINGLE suites. The separate nonConcurrent pipeline can select this group, but that path is not cloud mode, so the suite returns at lines 19-20 before exercising the insert/routing path. Please retag or move this case so a cloud-mode pipeline actually executes it; otherwise the new adaptive path still has no automated regression coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4856796838
<details>
<summary>TPC-DS: <b>Total hot run time: 173352 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a69300c67ec3c6c7ffaa3aa8879fc780ea092edc, data reload: false
query5 4329 632 501 501
query6 489 219 206 206
query7 4959 608 343 343
query8 342 191 168 168
query9 8742 3994 4005 3994
query10 461 353 296 296
query11 5979 2358 2143 2143
query12 154 102 103 102
query13 1266 641 435 435
query14 6290 5315 4991 4991
query14_1 4321 4265 4300 4265
query15 227 202 181 181
query16 1085 471 407 407
query17 1135 730 583 583
query18 2703 479 356 356
query19 210 206 153 153
query20 114 112 109 109
query21 239 160 135 135
query22 13681 13659 13499 13499
query23 17439 16559 16248 16248
query23_1 16131 16227 16231 16227
query24 7458 1736 1294 1294
query24_1 1318 1288 1323 1288
query25 565 467 403 403
query26 1335 348 209 209
query27 2517 575 384 384
query28 4401 2015 2044 2015
query29 1110 650 511 511
query30 332 270 231 231
query31 1122 1102 985 985
query32 117 66 64 64
query33 535 326 266 266
query34 1184 1118 661 661
query35 758 808 670 670
query36 1413 1439 1235 1235
query37 156 110 93 93
query38 1890 1707 1668 1668
query39 936 929 889 889
query39_1 917 879 870 870
query40 237 162 134 134
query41 64 61 60 60
query42 93 91 99 91
query43 328 315 278 278
query44 1420 767 789 767
query45 205 191 178 178
query46 1048 1215 728 728
query47 2418 2334 2241 2241
query48 390 394 292 292
query49 575 424 304 304
query50 1082 427 332 332
query51 4398 4429 4357 4357
query52 87 87 73 73
query53 263 273 213 213
query54 297 227 207 207
query55 73 71 70 70
query56 297 278 276 276
query57 1467 1427 1331 1331
query58 292 271 234 234
query59 1595 1648 1400 1400
query60 296 271 241 241
query61 143 150 151 150
query62 690 660 583 583
query63 238 201 219 201
query64 2475 790 576 576
query65 4846 4764 4757 4757
query66 1767 500 391 391
query67 29680 29604 29568 29568
query68 3086 1473 953 953
query69 411 312 273 273
query70 1075 967 956 956
query71 339 327 317 317
query72 2923 2666 2295 2295
query73 853 840 450 450
query74 5121 4962 4781 4781
query75 2622 2593 2226 2226
query76 2346 1177 789 789
query77 342 381 283 283
query78 12560 12352 11718 11718
query79 1422 1246 785 785
query80 645 551 465 465
query81 467 323 278 278
query82 564 159 122 122
query83 393 313 288 288
query84 277 159 129 129
query85 898 624 529 529
query86 380 290 280 280
query87 1819 1819 1747 1747
query88 3695 2798 2805 2798
query89 447 403 358 358
query90 1977 209 201 201
query91 202 194 157 157
query92 67 56 57 56
query93 1601 1555 928 928
query94 558 354 324 324
query95 821 498 463 463
query96 1051 786 371 371
query97 2721 2680 2593 2593
query98 215 202 200 200
query99 1165 1155 1029 1029
Total cold run time: 258529 ms
Total hot run time: 173352 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4856512780
# BE UT Coverage Report
Increment line coverage `16.78% (120/715)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/ecf8ac6e1fe8d47743f6194b774795d51885756d_ecf8ac6e1fe8d47743f6194b774795d51885756d/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/ecf8ac6e1fe8d47743f6194b774795d51885756d_ecf8ac6e1fe8d47743f6194b774795d51885756d/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 55.11% (21753/39469) |
| Line Coverage | 38.60% (208402/539872) |
| Region Coverage | 34.66% (164224/473862) |
| Branch Coverage | 35.67% (71960/201745) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4861727510
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4861894050
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/30a431f2b08623dcddbc518296a9a8b5c5a06387_30a431f2b08623dcddbc518296a9a8b5c5a06387_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/30a431f2b08623dcddbc518296a9a8b5c5a06387_30a431f2b08623dcddbc518296a9a8b5c5a06387_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.39% (1896/2450) |
| Line Coverage | 64.46% (34073/52862) |
| Region Coverage | 64.86% (17534/27032) |
| Branch Coverage | 54.08% (9405/17390) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4646635471
# FE UT Coverage Report
Increment line coverage ` 5.43% (19/350)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_49f07beecaeffc490273e3adfd9058e7222888ac/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_49f07beecaeffc490273e3adfd9058e7222888ac/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4650259680
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4646055218
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4762333962
# FE Regression Coverage Report
Increment line coverage ` 51.18% (217/424)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_52e2238c649552e3fd0cad1201b33579cb90c5ce_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_52e2238c649552e3fd0cad1201b33579cb90c5ce_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4763330709
<details>
<summary>TPC-H: <b>Total hot run time: 29329 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 52e2238c649552e3fd0cad1201b33579cb90c5ce, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17649 4010 3997 3997
q2 2014 320 193 193
q3 10407 1465 817 817
q4 4755 471 340 340
q5 8210 891 580 580
q6 296 167 137 137
q7 819 877 645 645
q8 10632 1652 1645 1645
q9 6115 4567 4484 4484
q10 6852 1823 1531 1531
q11 451 284 242 242
q12 691 416 297 297
q13 18196 3939 2796 2796
q14 269 257 248 248
q15 q16 789 779 705 705
q17 1000 950 1000 950
q18 6881 5738 5622 5622
q19 1242 1246 982 982
q20 506 408 265 265
q21 5791 2626 2547 2547
q22 426 361 306 306
Total cold run time: 103991 ms
Total hot run time: 29329 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4337 4264 4250 4250
q2 330 357 228 228
q3 4597 4953 4367 4367
q4 2055 2139 1373 1373
q5 4452 4288 4320 4288
q6 235 177 128 128
q7 2320 1974 1627 1627
q8 2560 2226 2148 2148
q9 8237 8038 8091 8038
q10 4798 4744 4227 4227
q11 569 422 391 391
q12 775 767 534 534
q13 3299 3662 2969 2969
q14 291 307 288 288
q15 q16 712 717 650 650
q17 1364 1322 1485 1322
q18 8063 7286 7295 7286
q19 1182 1071 1093 1071
q20 2202 2210 1941 1941
q21 5209 4506 4362 4362
q22 529 473 417 417
Total cold run time: 58116 ms
Total hot run time: 51905 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4655415573
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4656753670
# FE UT Coverage Report
Increment line coverage ` 5.00% (19/380)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_2054fc372677684e4f1aa9088f1011f91a4ce5d9/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_2054fc372677684e4f1aa9088f1011f91a4ce5d9/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4786788328
# BE UT Coverage Report
Increment line coverage `14.39% (102/709)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/591feb974a7f4b0b2831b4166b130e3dc8628771_591feb974a7f4b0b2831b4166b130e3dc8628771/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/591feb974a7f4b0b2831b4166b130e3dc8628771_591feb974a7f4b0b2831b4166b130e3dc8628771/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 54.48% (21411/39300) |
| Line Coverage | 37.99% (204374/537916) |
| Region Coverage | 33.97% (160381/472077) |
| Branch Coverage | 34.96% (70172/200718) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3479822870
##########
be/src/load/channel/tablets_channel.cpp:
##########
@@ -702,10 +910,18 @@ Status TabletsChannel::add_batch(const PTabletWriterAddBlockRequest& request,
return Status::OK();
}
+ if (request.is_receiver_side_random_bucket()) {
Review Comment:
Added comments at the sender and receiver RPC boundaries to clarify the naming: FE exposes the feature as adaptive random bucket load, while the add-block/open protocol names the same mode receiver-side random bucket because the receiver routes by partition id and owns the selected-tablet rotation state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4828480956
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4828627521
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/1ad5cc3a0a4c7e67ea8bd62c1cf9e84c7f493552_1ad5cc3a0a4c7e67ea8bd62c1cf9e84c7f493552_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/1ad5cc3a0a4c7e67ea8bd62c1cf9e84c7f493552_1ad5cc3a0a4c7e67ea8bd62c1cf9e84c7f493552_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.38% (1895/2449) |
| Line Coverage | 64.43% (33990/52753) |
| Region Coverage | 64.81% (17482/26973) |
| Branch Coverage | 54.04% (9372/17344) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4833401041
<details>
<summary>TPC-H: <b>Total hot run time: 29404 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eee8cc536909690d1a4bebfa38a1ad0179a59224, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17599 4170 4224 4170
q2 2019 324 198 198
q3 10337 1487 832 832
q4 4681 471 342 342
q5 7646 1003 578 578
q6 179 169 139 139
q7 788 859 618 618
q8 9336 1588 1556 1556
q9 5577 4557 4515 4515
q10 6786 1790 1542 1542
q11 447 272 242 242
q12 624 424 292 292
q13 18063 3520 2690 2690
q14 268 254 242 242
q15 q16 792 804 710 710
q17 997 1003 1054 1003
q18 7284 5781 5597 5597
q19 1173 1248 1108 1108
q20 487 396 265 265
q21 5740 2565 2463 2463
q22 432 370 302 302
Total cold run time: 101255 ms
Total hot run time: 29404 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4514 4409 4321 4321
q2 327 372 228 228
q3 4557 5006 4410 4410
q4 2084 2183 1369 1369
q5 4467 4380 4308 4308
q6 233 176 134 134
q7 1745 1918 1860 1860
q8 2585 2258 2193 2193
q9 8405 8341 8112 8112
q10 4864 4763 4344 4344
q11 603 406 371 371
q12 778 800 536 536
q13 3340 3620 2970 2970
q14 308 309 296 296
q15 q16 707 720 641 641
q17 1362 1348 1479 1348
q18 7842 7463 7212 7212
q19 1144 1090 1126 1090
q20 2211 2273 1943 1943
q21 5276 4597 4476 4476
q22 506 458 393 393
Total cold run time: 57858 ms
Total hot run time: 52555 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3505634098
##########
regression-test/suites/load_p0/insert/test_adaptive_random_bucket_load.groovy:
##########
@@ -0,0 +1,77 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_adaptive_random_bucket_load", "p0,nonConcurrent") {
+ if (!isCloudMode()) {
+ return
+ }
+
+ def tableName = "test_adaptive_random_bucket_load"
+ def enableAdaptiveRandomBucketConfig =
+ sql """ ADMIN SHOW FRONTEND CONFIG LIKE 'enable_adaptive_random_bucket_load'; """
+ String oldEnableAdaptiveRandomBucket = enableAdaptiveRandomBucketConfig[0][1]
+
+ try {
+ sql """ ADMIN SET FRONTEND CONFIG ('enable_adaptive_random_bucket_load' = 'true') """
+ sql """ DROP TABLE IF EXISTS ${tableName} """
+ sql """
+ CREATE TABLE ${tableName} (
+ dt date NOT NULL,
+ k int NOT NULL,
+ v string
+ )
+ DUPLICATE KEY(dt, k)
+ AUTO PARTITION BY RANGE (date_trunc(dt, 'day')) ()
+ DISTRIBUTED BY RANDOM BUCKETS 4
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ )
+ """
+
+ sql """
+ INSERT INTO ${tableName} VALUES
+ ('2026-07-01', 1, 'a'),
+ ('2026-07-01', 2, 'b'),
+ ('2026-07-02', 3, 'c'),
+ ('2026-07-02', 4, 'd')
+ """
+ sql "sync"
+
+ def totalCount = sql "SELECT count(*) FROM ${tableName}"
+ assertEquals(totalCount[0][0], 4)
+
+ def partitions = sql "SHOW PARTITIONS FROM ${tableName}"
+ assertEquals(partitions.size(), 2)
+
+ def tablets = sql "SHOW TABLETS FROM ${tableName}"
+ int tabletsWithRows = 0
+ int tabletRowCount = 0
+ tablets.each { tablet ->
+ def tabletCount = sql "SELECT count(*) FROM ${tableName} TABLET(${tablet[0]})"
+ int count = tabletCount[0][0] as int
+ if (count > 0) {
+ tabletsWithRows++
+ tabletRowCount += count
+ }
+ }
+ assertTrue(tabletsWithRows > 0)
+ assertEquals(tabletRowCount, 4)
+ } finally {
+ sql """ ADMIN SET FRONTEND CONFIG ('enable_adaptive_random_bucket_load' = '${oldEnableAdaptiveRandomBucket}') """
+ sql """ DROP TABLE IF EXISTS ${tableName} """
Review Comment:
This new regression already drops the table before creating it, so the final cleanup should not drop it again. Doris regression-test standards ask tests to drop tables before use, not after completion, so failed runs keep their table/data for debugging. Please keep the config restore in `finally`, but remove this final `DROP TABLE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4832915169
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4715224835
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3417519029
##########
be/src/exec/sink/writer/vtablet_writer.cpp:
##########
@@ -1322,14 +1461,25 @@ void VNodeChannel::mark_close(bool hang_wait) {
return;
}
- _cur_add_block_request->set_eos(true);
- _cur_add_block_request->set_hang_wait(hang_wait);
+ bool need_receiver_side_random_bucket_eos =
+ _cur_add_block_request->is_receiver_side_random_bucket();
{
std::lock_guard<std::mutex> l(_pending_batches_lock);
if (!_cur_mutable_block) [[unlikely]] {
// never had a block arrived. add a dummy block
_cur_mutable_block = MutableBlock::create_unique();
}
+ if (need_receiver_side_random_bucket_eos && _cur_mutable_block->rows() > 0) {
+ _cur_add_block_request->set_eos(false);
+ auto tmp_add_block_request =
+ std::make_shared<PTabletWriterAddBlockRequest>(*_cur_add_block_request);
+ _pending_blocks.emplace(std::move(_cur_mutable_block), tmp_add_block_request);
+ _pending_batches_num++;
+ _cur_add_block_request->clear_partition_ids();
+ _cur_mutable_block = MutableBlock::create_unique();
Review Comment:
When adaptive close splits a final non-empty block from the EOS request, this request still contains the compatibility `tablet_ids` added in `add_block()`. The split enqueues the data block, clears only `partition_ids`, and then reuses `_cur_add_block_request` for an empty EOS block. `try_send_pending_block()` validates EOS requests against `tablet_ids_size()`, so a normal small adaptive load whose final block did not reach `_batch_size` will see `block.rows() == 0` and `request_rows == N`, cancel the channel locally, and never send EOS. Clear the compatibility tablet ids here as well. This is distinct from the existing EOS `partition_ids` thread because it is caused by the newly retained rolling-upgrade `tablet_ids` payload.
```suggestion
_cur_add_block_request->clear_tablet_ids();
_cur_add_block_request->clear_partition_ids();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3417844764
##########
fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java:
##########
@@ -538,6 +543,98 @@ public Map<String, Integer> getBeToInstancesNum() {
return result;
}
+ public static final class AdaptiveRandomBucketSinkContext {
+ private final List<Long> sinkBackendIds;
+ private final int planFragmentNum;
+
+ private AdaptiveRandomBucketSinkContext(List<Long> sinkBackendIds, int planFragmentNum) {
+ this.sinkBackendIds = sinkBackendIds;
+ this.planFragmentNum = planFragmentNum;
+ }
+
+ public List<Long> getSinkBackendIds() {
+ return sinkBackendIds;
+ }
+
+ public int getPlanFragmentNum() {
+ return planFragmentNum;
+ }
+ }
+
+ public Optional<AdaptiveRandomBucketSinkContext> getAdaptiveRandomBucketSinkContext(long tableId) {
+ Set<Long> sinkBackendIds = new TreeSet<>();
+ int planFragmentNum = 0;
+ for (PipelineExecContext context : pipelineExecContexts.values()) {
+ TPipelineFragmentParams params = context.rpcParams;
+ if (params.getFragment().getOutputSink() == null
+ || params.getFragment().getOutputSink().getType() != TDataSinkType.OLAP_TABLE_SINK) {
+ continue;
+ }
+ TOlapTableSink sink = params.getFragment().getOutputSink().getOlapTableSink();
+ if (sink.getTableId() != tableId) {
+ continue;
+ }
+ if (!OlapTableSink.shouldAssignAdaptiveRandomBucket(sink)) {
+ continue;
+ }
+ sinkBackendIds.add(params.getBackendId());
+ planFragmentNum += params.getLocalParamsSize();
+ }
+ if (sinkBackendIds.isEmpty()) {
+ return Optional.empty();
+ }
+ return Optional.of(new AdaptiveRandomBucketSinkContext(
+ new ArrayList<>(sinkBackendIds), Math.max(planFragmentNum, 1)));
+ }
+
+ private static void assignAdaptiveRandomBucketForFragment(
+ Collection<TPipelineFragmentParams> fragmentParamsList) {
+ List<TPipelineFragmentParams> sinkParams = fragmentParamsList.stream()
+ .filter(param -> param.getFragment().getOutputSink() != null
+ && param.getFragment().getOutputSink().getType() == TDataSinkType.OLAP_TABLE_SINK)
+ .collect(Collectors.toList());
+ if (sinkParams.isEmpty()) {
+ return;
+ }
+ TOlapTableSink sink = sinkParams.get(0).getFragment().getOutputSink().getOlapTableSink();
+ if (!OlapTableSink.shouldAssignAdaptiveRandomBucket(sink)) {
+ return;
+ }
+ List<Long> sinkBackendIds = sinkParams.stream()
+ .map(TPipelineFragmentParams::getBackendId)
+ .distinct()
+ .sorted()
+ .collect(Collectors.toList());
+ int planFragmentNum = sinkParams.stream()
+ .mapToInt(TPipelineFragmentParams::getLocalParamsSize)
+ .sum();
+ if (LOG.isInfoEnabled()) {
+ LOG.info("Adaptive random bucket planning in legacy fragment={}, sinkBackendIds={}, "
+ + "planFragmentNum={}",
+ sinkParams.get(0).getFragmentId(), sinkBackendIds, planFragmentNum);
+ }
+ Map<Long, Map<Long, OlapTableSink.AdaptiveBucketAssignment>> assignments =
+ OlapTableSink.computeAdaptiveRandomBucketAssignments(sinkBackendIds,
+ sink.getPartition().getPartitions(), sink.getLocation().getTablets(), planFragmentNum);
+ for (TPipelineFragmentParams sinkParam : sinkParams) {
+ Map<Long, OlapTableSink.AdaptiveBucketAssignment> partitionAssignments =
+ assignments.get(sinkParam.getBackendId());
+ if (partitionAssignments == null) {
+ continue;
+ }
+ TOlapTableSink copiedSink = deepCopyOlapTableSinkForCurrentBackend(sinkParam);
+ OlapTableSink.applyAdaptiveRandomBucketAssignments(
+ copiedSink.getPartition().getPartitions(),
+ partitionAssignments);
+ }
Review Comment:
`deepCopyOlapTableSinkForCurrentBackend()` still is not BE-local on the classic Coordinator path. In `FragmentExecParams.toThrift()` the legacy planner creates one `TPlanFragment fragmentThrift = fragment.toThrift()` before the backend loop and every `TPipelineFragmentParams` gets that same object via `params.setFragment(fragmentThrift)`. So this `setOutputSink()` mutates the shared fragment; as `assignAdaptiveRandomBucketForFragment()` iterates backends, the last backend's `bucket_be_id` / `local_bucket_seqs` overwrite the assignments for all earlier backends. A classic multi-BE load can then have every sender route using the last BE's adaptive assignment. This is distinct from the existing Nereids deep-copy thread because `ThriftPlansBuilder` now creates a fragment per worker before copying the sink; the classic Coordinator path still shares the fragment. Please deep-copy the `TPlanFragment` (or create per-backend `fragmentThrift`) before attaching the copied sink.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4862916728
<details>
<summary>TPC-H: <b>Total hot run time: 30278 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 30a431f2b08623dcddbc518296a9a8b5c5a06387, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17728 4061 4049 4049
q2 2020 321 206 206
q3 10466 1453 847 847
q4 4762 482 345 345
q5 8314 881 578 578
q6 313 171 136 136
q7 842 823 616 616
q8 10711 1674 1707 1674
q9 5961 4465 4478 4465
q10 6828 1791 1556 1556
q11 530 341 322 322
q12 742 571 448 448
q13 18101 3941 2794 2794
q14 271 268 243 243
q15 q16 798 775 715 715
q17 1053 946 1009 946
q18 7072 5904 5572 5572
q19 1188 1263 1071 1071
q20 929 756 567 567
q21 5925 2970 2805 2805
q22 452 361 323 323
Total cold run time: 105006 ms
Total hot run time: 30278 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4911 4877 5107 4877
q2 298 343 221 221
q3 4994 5362 4650 4650
q4 2126 2175 1513 1513
q5 4824 4637 4913 4637
q6 230 191 126 126
q7 1871 1742 1544 1544
q8 2447 2277 2110 2110
q9 7460 7349 7346 7346
q10 4664 4620 4188 4188
q11 529 383 352 352
q12 738 747 533 533
q13 3004 3389 2770 2770
q14 271 281 254 254
q15 q16 680 701 615 615
q17 1298 1257 1259 1257
q18 7381 7095 7097 7095
q19 1135 1114 1117 1114
q20 2242 2188 1949 1949
q21 5377 4705 4638 4638
q22 541 447 415 415
Total cold run time: 57021 ms
Total hot run time: 52204 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](load) adaptive random bucket load: per-BE bucket assignment with threshold-based rotation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4290266597
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/0e8fec709e17b52e8b214b7e4860e8977228e98a_0e8fec709e17b52e8b214b7e4860e8977228e98a_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/0e8fec709e17b52e8b214b7e4860e8977228e98a_0e8fec709e17b52e8b214b7e4860e8977228e98a_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.06% (1843/2361) |
| Line Coverage | 64.74% (32982/50947) |
| Region Coverage | 65.26% (16370/25085) |
| Branch Coverage | 55.79% (8732/15652) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](load) adaptive random bucket load: per-BE bucket assignment with threshold-based rotation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4286156821
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/35540cbbfb3bf09b86b55db3e6f2bb77f6897059_35540cbbfb3bf09b86b55db3e6f2bb77f6897059_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/35540cbbfb3bf09b86b55db3e6f2bb77f6897059_35540cbbfb3bf09b86b55db3e6f2bb77f6897059_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.06% (1843/2361) |
| Line Coverage | 64.74% (32982/50947) |
| Region Coverage | 65.25% (16368/25085) |
| Branch Coverage | 55.78% (8731/15652) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3479827116
##########
be/src/load/channel/tablets_channel.cpp:
##########
@@ -683,6 +728,169 @@ Status BaseTabletsChannel::_write_block_data(
return Status::OK();
}
+std::shared_ptr<std::mutex> BaseTabletsChannel::_get_sender_partition_route_lock(
Review Comment:
This lock is needed for correctness within the same sender and partition: current-tablet selection, the write, and the flush-triggered rotation must stay ordered so concurrent packets do not all write using the same old current tablet and then rotate repeatedly. I added comments in the header to make that explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4807756277
# FE UT Coverage Report
Increment line coverage ` 37.03% (157/424)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_4f1c936ff2ae8acc7275baf70a22f673b037c6ca/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_4f1c936ff2ae8acc7275baf70a22f673b037c6ca/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4786667542
<details>
<summary>TPC-DS: <b>Total hot run time: 173289 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 591feb974a7f4b0b2831b4166b130e3dc8628771, data reload: false
query5 4325 624 478 478
query6 428 219 170 170
query7 4812 583 313 313
query8 367 219 190 190
query9 8790 4044 4069 4044
query10 445 322 260 260
query11 5911 2369 2135 2135
query12 152 102 96 96
query13 1247 615 420 420
query14 6357 5416 5055 5055
query14_1 4366 4375 4390 4375
query15 200 196 172 172
query16 1000 465 417 417
query17 1096 700 560 560
query18 2434 459 341 341
query19 197 184 148 148
query20 115 112 106 106
query21 219 142 121 121
query22 13678 13574 13505 13505
query23 17486 16499 16223 16223
query23_1 16250 16379 16302 16302
query24 7459 1811 1323 1323
query24_1 1330 1321 1313 1313
query25 567 457 405 405
query26 1298 312 178 178
query27 2671 586 333 333
query28 4500 2056 2026 2026
query29 1072 623 510 510
query30 323 236 206 206
query31 1122 1085 972 972
query32 103 62 64 62
query33 520 328 260 260
query34 1174 1203 692 692
query35 765 790 695 695
query36 1353 1374 1252 1252
query37 157 105 94 94
query38 1888 1704 1684 1684
query39 952 920 894 894
query39_1 880 885 891 885
query40 223 126 104 104
query41 71 67 68 67
query42 90 89 89 89
query43 327 330 293 293
query44 1441 798 784 784
query45 199 187 184 184
query46 1095 1174 754 754
query47 2339 2333 2257 2257
query48 403 429 291 291
query49 645 473 380 380
query50 1033 361 264 264
query51 4415 4477 4322 4322
query52 84 86 73 73
query53 254 265 189 189
query54 285 231 212 212
query55 75 77 66 66
query56 259 240 244 240
query57 1486 1401 1362 1362
query58 237 214 215 214
query59 1578 1630 1427 1427
query60 276 247 229 229
query61 146 150 150 150
query62 703 647 595 595
query63 227 194 193 193
query64 2523 788 591 591
query65 4862 4816 4783 4783
query66 1785 488 335 335
query67 29911 29853 29823 29823
query68 3357 1586 926 926
query69 405 292 271 271
query70 1081 1000 1004 1000
query71 301 233 214 214
query72 2994 2689 2282 2282
query73 820 793 443 443
query74 5137 4989 4767 4767
query75 2621 2606 2265 2265
query76 2315 1197 784 784
query77 352 383 287 287
query78 12521 12601 11852 11852
query79 1417 1136 775 775
query80 718 471 398 398
query81 474 289 240 240
query82 566 155 123 123
query83 336 269 244 244
query84 261 137 117 117
query85 921 521 426 426
query86 385 290 288 288
query87 1849 1832 1777 1777
query88 3736 2815 2790 2790
query89 430 382 331 331
query90 1905 195 181 181
query91 175 159 135 135
query92 64 60 54 54
query93 1405 1460 869 869
query94 609 350 307 307
query95 676 472 349 349
query96 1056 790 327 327
query97 2721 2728 2584 2584
query98 214 206 199 199
query99 1200 1156 1022 1022
Total cold run time: 258468 ms
Total hot run time: 173289 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4897361390
# FE Regression Coverage Report
Increment line coverage ` 33.78% (507/1501)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_30a431f2b08623dcddbc518296a9a8b5c5a06387_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_30a431f2b08623dcddbc518296a9a8b5c5a06387_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4834876285
# FE Regression Coverage Report
Increment line coverage ` 51.54% (218/423)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_6cf651de54175959a128b07b857fa64e939af881_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_6cf651de54175959a128b07b857fa64e939af881_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4863022019
<details>
<summary>ClickBench: <b>Total hot run time: 25.26 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 30a431f2b08623dcddbc518296a9a8b5c5a06387, data reload: false
query1 0.01 0.01 0.01
query2 0.11 0.05 0.05
query3 0.25 0.14 0.14
query4 1.60 0.14 0.15
query5 0.24 0.22 0.22
query6 1.28 1.04 1.16
query7 0.04 0.00 0.01
query8 0.06 0.04 0.04
query9 0.37 0.32 0.30
query10 0.56 0.56 0.55
query11 0.22 0.15 0.15
query12 0.20 0.15 0.14
query13 0.47 0.46 0.47
query14 1.03 0.98 1.00
query15 0.60 0.60 0.59
query16 0.31 0.32 0.34
query17 1.12 1.10 1.10
query18 0.22 0.21 0.21
query19 1.99 1.91 1.94
query20 0.02 0.01 0.01
query21 15.50 0.23 0.13
query22 4.80 0.05 0.05
query23 16.16 0.31 0.12
query24 3.05 0.41 0.31
query25 0.12 0.05 0.06
query26 0.73 0.21 0.16
query27 0.04 0.04 0.04
query28 3.45 0.97 0.57
query29 12.52 4.31 3.47
query30 0.28 0.15 0.15
query31 2.77 0.59 0.33
query32 3.22 0.60 0.49
query33 3.16 3.19 3.21
query34 15.69 4.22 3.54
query35 3.49 3.50 3.52
query36 0.55 0.45 0.43
query37 0.09 0.06 0.07
query38 0.06 0.03 0.03
query39 0.04 0.03 0.03
query40 0.17 0.16 0.15
query41 0.08 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.04
Total cold run time: 96.75 s
Total hot run time: 25.26 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4861728041
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4856859069
<details>
<summary>ClickBench: <b>Total hot run time: 25.07 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a69300c67ec3c6c7ffaa3aa8879fc780ea092edc, data reload: false
query1 0.00 0.00 0.01
query2 0.10 0.06 0.05
query3 0.25 0.14 0.13
query4 1.61 0.14 0.14
query5 0.24 0.21 0.24
query6 1.22 1.05 1.05
query7 0.04 0.01 0.00
query8 0.06 0.04 0.04
query9 0.37 0.31 0.32
query10 0.60 0.56 0.58
query11 0.20 0.14 0.14
query12 0.19 0.14 0.14
query13 0.46 0.46 0.49
query14 1.05 1.01 1.02
query15 0.62 0.59 0.59
query16 0.33 0.32 0.32
query17 1.10 1.13 1.05
query18 0.23 0.21 0.21
query19 2.03 1.86 1.90
query20 0.02 0.02 0.01
query21 15.44 0.22 0.13
query22 4.90 0.04 0.05
query23 16.15 0.31 0.12
query24 2.96 0.41 0.30
query25 0.09 0.05 0.05
query26 0.74 0.21 0.16
query27 0.04 0.04 0.04
query28 3.55 0.97 0.51
query29 12.53 4.35 3.46
query30 0.28 0.15 0.15
query31 2.78 0.62 0.31
query32 3.23 0.59 0.50
query33 3.24 3.22 3.18
query34 15.54 4.25 3.53
query35 3.49 3.52 3.56
query36 0.55 0.43 0.42
query37 0.09 0.07 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.17 0.15
query41 0.09 0.04 0.03
query42 0.03 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 96.74 s
Total hot run time: 25.07 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4656178523
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/fad685172f132dd675eec0574703f90e52d3b2e2_fad685172f132dd675eec0574703f90e52d3b2e2_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/fad685172f132dd675eec0574703f90e52d3b2e2_fad685172f132dd675eec0574703f90e52d3b2e2_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.39% (1915/2443) |
| Line Coverage | 64.85% (34178/52701) |
| Region Coverage | 65.27% (17572/26922) |
| Branch Coverage | 53.96% (9331/17294) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4656210701
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4656993224
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4660079130
# BE UT Coverage Report
Increment line coverage `5.44% (36/662)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/f552fc1cfb68972e5ff591a0b8668934955c7734_f552fc1cfb68972e5ff591a0b8668934955c7734/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/f552fc1cfb68972e5ff591a0b8668934955c7734_f552fc1cfb68972e5ff591a0b8668934955c7734/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.85% (21134/39249) |
| Line Coverage | 37.54% (201236/536054) |
| Region Coverage | 33.54% (157731/470321) |
| Branch Coverage | 34.62% (69108/199605) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3414747874
##########
be/src/exec/sink/writer/vtablet_writer.cpp:
##########
@@ -777,13 +889,17 @@ Status VNodeChannel::add_block(Block* block, const Payload* payload) {
}
SCOPED_RAW_TIMER(&_stat.append_node_channel_ns);
- st = block->append_to_block_by_selector(_cur_mutable_block.get(), *(payload->first));
+ st = block->append_to_block_by_selector(_cur_mutable_block.get(), *payload->row_ids);
if (!st.ok()) {
_cancel_with_msg(fmt::format("{}, err: {}", channel_info(), st.to_string()));
return st;
}
Review Comment:
This needs a receiver-version gate before adaptive mode starts sending partition-ids-only add-block RPCs. In a rolling BE upgrade, a new sender can route an adaptive random bucket to an old receiver, for example when a selected bucket or rollup-index bucket owner is still on the old BE. This branch then omits `tablet_ids` and relies on `is_receiver_side_random_bucket`, but old receivers ignore that new protobuf field and still execute the old `BaseTabletsChannel::_write_block_data()` `CHECK(block.rows() == request.tablet_ids_size())`. With a non-empty block and zero `tablet_ids`, the old BE aborts or the load fails. Please only enable this mode when every possible tablet-writer receiver supports the new protocol, or fall back to sending `tablet_ids` to older receivers. This is distinct from the existing old-FE/`local_bucket_seqs` compatibility thread because it is BE-to-BE add-block payload compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4713883894
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3417586417
##########
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:
##########
@@ -5745,4 +5768,129 @@ private TStatus checkMaster() {
}
return status;
}
+
+ private static final class AdaptiveBucketSinkContext {
+ private final boolean enableAdaptiveRandomBucket;
+ private final List<Long> sinkBackendIds;
+ private final int planFragmentNum;
+
+ private AdaptiveBucketSinkContext(boolean enableAdaptiveRandomBucket, List<Long> sinkBackendIds,
+ int planFragmentNum) {
+ this.enableAdaptiveRandomBucket = enableAdaptiveRandomBucket;
Review Comment:
This runtime adaptive context needs to be scoped to the table/sink that issued the create/replace-partition RPC. `TCreatePartitionRequest` and `TReplacePartitionRequest` already carry `table_id`, but this helper only takes `queryId`; the classic branch and the Nereids loop aggregate every adaptive OLAP sink under the coordinator. Multi-table stream/routine load can generate several table plans under the same load id/query id, so when one table auto-creates or replaces a random partition, `assignAdaptiveBucketToPartition()` may compute its selected buckets with sink backends and `planFragmentNum` from unrelated tables. That changes `targetBucketNum`, selected bucket order, and the returned `bucket_be_id`/`local_bucket_seqs` compared with that table's initial sink assignment. This is distinct from the existing classic-fallback thread: the helper now finds a context, but it can be the union of multiple sinks. Please key/lookup the collected context by the requested table or sink, for
example using `TOlapTableSink.table_id` from the RPC, instead of aggregating at query level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4710730827
<details>
<summary>TPC-DS: <b>Total hot run time: 167886 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d8fb460dd1f77e6dd47d3649a8f89b0f7cb9922b, data reload: false
query5 4320 613 479 479
query6 443 187 170 170
query7 4817 562 306 306
query8 362 215 206 206
query9 8720 3999 4040 3999
query10 430 335 258 258
query11 5876 2364 2137 2137
query12 153 97 98 97
query13 1362 598 386 386
query14 6344 5340 5016 5016
query14_1 4377 4379 4349 4349
query15 199 190 169 169
query16 988 471 454 454
query17 1103 698 537 537
query18 2419 466 327 327
query19 208 176 137 137
query20 111 109 100 100
query21 217 137 115 115
query22 13631 13632 13366 13366
query23 17457 16501 16108 16108
query23_1 16311 16334 16402 16334
query24 7489 1786 1316 1316
query24_1 1327 1300 1320 1300
query25 572 469 404 404
query26 1324 300 165 165
query27 2701 584 337 337
query28 4487 2055 2016 2016
query29 1065 630 495 495
query30 326 236 197 197
query31 1121 1079 979 979
query32 109 61 60 60
query33 531 342 263 263
query34 1175 1110 644 644
query35 751 778 678 678
query36 1372 1385 1257 1257
query37 159 106 90 90
query38 3177 3144 3058 3058
query39 930 933 890 890
query39_1 872 869 861 861
query40 227 122 105 105
query41 71 65 66 65
query42 98 96 94 94
query43 315 316 275 275
query44
query45 197 190 178 178
query46 1078 1213 737 737
query47 2330 2390 2229 2229
query48 402 407 292 292
query49 666 489 347 347
query50 1087 360 258 258
query51 4496 4365 4290 4290
query52 89 91 77 77
query53 256 275 186 186
query54 276 251 213 213
query55 81 78 77 77
query56 241 233 231 231
query57 1415 1378 1308 1308
query58 248 224 215 215
query59 1555 1651 1412 1412
query60 303 283 219 219
query61 150 146 148 146
query62 688 649 585 585
query63 239 179 183 179
query64 2527 770 583 583
query65
query66 1822 467 340 340
query67 29686 29547 29427 29427
query68
query69 427 290 264 264
query70 978 921 932 921
query71 290 229 211 211
query72 2875 2589 2313 2313
query73 841 779 434 434
query74 5086 4956 4741 4741
query75 2657 2567 2209 2209
query76 2306 1131 781 781
query77 350 362 272 272
query78 12472 12400 11833 11833
query79 1446 1060 737 737
query80 1211 452 379 379
query81 521 282 243 243
query82 614 153 122 122
query83 319 273 248 248
query84
query85 885 500 406 406
query86 452 294 273 273
query87 3408 3363 3172 3172
query88 3661 2783 2758 2758
query89 423 382 323 323
query90 1902 180 175 175
query91 184 158 135 135
query92 67 61 54 54
query93 1585 1394 860 860
query94 705 361 295 295
query95 655 380 346 346
query96 1045 758 313 313
query97 2679 2692 2585 2585
query98 211 221 207 207
query99 1150 1164 1023 1023
Total cold run time: 251739 ms
Total hot run time: 167886 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4711761015
# FE Regression Coverage Report
Increment line coverage ` 51.90% (218/420)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_d8fb460dd1f77e6dd47d3649a8f89b0f7cb9922b_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_d8fb460dd1f77e6dd47d3649a8f89b0f7cb9922b_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4716098353
<details>
<summary>TPC-DS: <b>Total hot run time: 176067 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 52e2238c649552e3fd0cad1201b33579cb90c5ce, data reload: false
query5 4350 619 471 471
query6 447 193 168 168
query7 4802 539 307 307
query8 373 216 205 205
query9 8743 4114 4103 4103
query10 478 317 270 270
query11 5909 2355 2174 2174
query12 177 100 94 94
query13 1249 558 483 483
query14 6353 5374 5040 5040
query14_1 4376 4359 4371 4359
query15 206 197 171 171
query16 1044 449 450 449
query17 1121 687 551 551
query18 2568 460 336 336
query19 192 176 149 149
query20 112 110 111 110
query21 222 140 119 119
query22 13683 13584 13386 13386
query23 17502 16650 16261 16261
query23_1 16282 16260 16240 16240
query24 7589 1783 1334 1334
query24_1 1322 1326 1324 1324
query25 587 470 404 404
query26 1308 316 170 170
query27 2669 565 345 345
query28 4451 2042 2043 2042
query29 1112 623 512 512
query30 317 236 202 202
query31 1126 1076 967 967
query32 104 64 65 64
query33 541 327 271 271
query34 1173 1147 687 687
query35 756 802 701 701
query36 1361 1376 1260 1260
query37 158 110 91 91
query38 3240 3161 3079 3079
query39 948 935 890 890
query39_1 882 887 881 881
query40 232 127 106 106
query41 70 69 68 68
query42 99 98 101 98
query43 320 327 288 288
query44 1484 800 790 790
query45 196 188 179 179
query46 1107 1248 761 761
query47 2387 2380 2240 2240
query48 425 432 322 322
query49 640 484 382 382
query50 1072 352 267 267
query51 4329 4410 4291 4291
query52 91 90 79 79
query53 249 260 194 194
query54 281 231 209 209
query55 83 78 73 73
query56 244 242 224 224
query57 1426 1420 1335 1335
query58 265 222 220 220
query59 1618 1687 1471 1471
query60 285 243 226 226
query61 161 161 152 152
query62 687 657 574 574
query63 230 191 186 186
query64 2550 778 624 624
query65 4894 4789 4796 4789
query66 1768 469 342 342
query67 30000 29701 29646 29646
query68 3222 1624 923 923
query69 406 303 275 275
query70 1071 969 963 963
query71 306 235 215 215
query72 3067 2769 2352 2352
query73 886 745 460 460
query74 5135 4962 4772 4772
query75 2633 2610 2225 2225
query76 2340 1187 790 790
query77 359 385 289 289
query78 12511 12471 12046 12046
query79 1453 1129 743 743
query80 1277 462 414 414
query81 535 284 240 240
query82 773 155 121 121
query83 335 275 245 245
query84 257 145 115 115
query85 970 511 422 422
query86 436 319 293 293
query87 3422 3372 3218 3218
query88 3754 2848 2806 2806
query89 432 376 343 343
query90 1918 189 177 177
query91 173 163 135 135
query92 61 61 57 57
query93 1585 1660 841 841
query94 731 365 316 316
query95 679 470 341 341
query96 1025 742 370 370
query97 2693 2669 2557 2557
query98 231 211 198 198
query99 1185 1180 1026 1026
Total cold run time: 263320 ms
Total hot run time: 176067 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4761827796
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4762341416
# FE Regression Coverage Report
Increment line coverage ` 51.18% (217/424)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_52e2238c649552e3fd0cad1201b33579cb90c5ce_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_52e2238c649552e3fd0cad1201b33579cb90c5ce_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4829062411
<details>
<summary>TPC-H: <b>Total hot run time: 29482 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1ad5cc3a0a4c7e67ea8bd62c1cf9e84c7f493552, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17737 4075 4046 4046
q2 1992 311 197 197
q3 10515 1455 827 827
q4 4764 475 340 340
q5 8298 915 572 572
q6 332 170 135 135
q7 874 860 650 650
q8 10693 1684 1679 1679
q9 5789 4596 4547 4547
q10 6822 1787 1535 1535
q11 441 273 247 247
q12 646 426 295 295
q13 18137 3501 2732 2732
q14 264 267 249 249
q15 q16 797 777 719 719
q17 954 1009 983 983
q18 6850 5826 5656 5656
q19 1164 1394 1109 1109
q20 478 406 281 281
q21 5645 2653 2381 2381
q22 436 363 302 302
Total cold run time: 103628 ms
Total hot run time: 29482 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4379 4245 4273 4245
q2 323 353 233 233
q3 4639 4974 4375 4375
q4 2087 2164 1381 1381
q5 4431 4271 4326 4271
q6 238 183 133 133
q7 2224 1932 1675 1675
q8 2553 2214 2247 2214
q9 8426 8126 8181 8126
q10 4796 4762 4327 4327
q11 560 422 396 396
q12 772 744 536 536
q13 3330 3632 2943 2943
q14 308 299 264 264
q15 q16 713 746 650 650
q17 1372 1439 1330 1330
q18 7926 7202 7298 7202
q19 1101 1110 1174 1110
q20 2260 2231 1960 1960
q21 5257 4567 4429 4429
q22 537 455 407 407
Total cold run time: 58232 ms
Total hot run time: 52207 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4829462607
# BE UT Coverage Report
Increment line coverage `14.20% (100/704)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/1ad5cc3a0a4c7e67ea8bd62c1cf9e84c7f493552_1ad5cc3a0a4c7e67ea8bd62c1cf9e84c7f493552/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/1ad5cc3a0a4c7e67ea8bd62c1cf9e84c7f493552_1ad5cc3a0a4c7e67ea8bd62c1cf9e84c7f493552/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 54.88% (21610/39375) |
| Line Coverage | 38.39% (206620/538164) |
| Region Coverage | 34.45% (162686/472189) |
| Branch Coverage | 35.48% (71212/200724) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4856596469
# FE UT Coverage Report
Increment line coverage ` 37.12% (157/423)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_a69300c67ec3c6c7ffaa3aa8879fc780ea092edc/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_a69300c67ec3c6c7ffaa3aa8879fc780ea092edc/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4856683289
<details>
<summary>TPC-H: <b>Total hot run time: 30069 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a69300c67ec3c6c7ffaa3aa8879fc780ea092edc, data reload: false
------ Round 1 ----------------------------------
============================================
q1 18003 4065 4025 4025
q2 2003 328 200 200
q3 10288 1464 835 835
q4 4687 465 339 339
q5 7505 865 587 587
q6 185 177 137 137
q7 814 857 628 628
q8 9363 1671 1745 1671
q9 5644 4400 4416 4400
q10 6761 1809 1560 1560
q11 505 343 309 309
q12 724 552 435 435
q13 18074 3388 2801 2801
q14 264 262 241 241
q15 q16 797 791 710 710
q17 1032 1067 1056 1056
q18 7309 5755 5709 5709
q19 1173 1339 1119 1119
q20 772 650 537 537
q21 5632 2652 2468 2468
q22 432 358 302 302
Total cold run time: 101967 ms
Total hot run time: 30069 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4352 4285 4285 4285
q2 288 318 204 204
q3 4574 4958 4411 4411
q4 2089 2145 1358 1358
q5 4469 4317 4345 4317
q6 236 175 124 124
q7 1813 2186 1654 1654
q8 2571 2231 2178 2178
q9 8134 8166 7834 7834
q10 4820 4750 4324 4324
q11 596 407 383 383
q12 767 768 546 546
q13 3172 3572 3029 3029
q14 307 299 276 276
q15 q16 702 720 633 633
q17 1380 1322 1333 1322
q18 7859 7297 7450 7297
q19 1212 1155 1126 1126
q20 2217 2212 1951 1951
q21 5272 4538 4407 4407
q22 504 470 401 401
Total cold run time: 57334 ms
Total hot run time: 52060 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4655416296
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3378570991
##########
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:
##########
@@ -5695,4 +5735,103 @@ private TStatus checkMaster() {
}
return status;
}
+
+ private static final class AdaptiveBucketSinkContext {
+ private final List<Long> sinkBackendIds;
+ private final int planFragmentNum;
+
+ private AdaptiveBucketSinkContext(List<Long> sinkBackendIds, int planFragmentNum) {
+ this.sinkBackendIds = sinkBackendIds;
+ this.planFragmentNum = planFragmentNum;
+ }
+ }
+
+ private static AdaptiveBucketSinkContext collectAdaptiveBucketSinkContext(TUniqueId queryId, long currentBeId) {
+ if (queryId == null) {
+ return new AdaptiveBucketSinkContext(Lists.newArrayList(currentBeId), 1);
+ }
+ Coordinator coordinator = QeProcessorImpl.INSTANCE.getCoordinator(queryId);
+ if (coordinator == null) {
+ return new AdaptiveBucketSinkContext(Lists.newArrayList(currentBeId), 1);
+ }
+ if (!(coordinator instanceof NereidsCoordinator)) {
+ Optional<Coordinator.AdaptiveRandomBucketSinkContext> context =
+ coordinator.getAdaptiveRandomBucketSinkContext();
+ if (context.isPresent()) {
+ return new AdaptiveBucketSinkContext(
+ context.get().getSinkBackendIds(), context.get().getPlanFragmentNum());
+ }
+ return new AdaptiveBucketSinkContext(Lists.newArrayList(currentBeId), 1);
+ }
+ Set<Long> sinkBackendIds = new TreeSet<>();
+ int planFragmentNum = 0;
+ for (PipelineDistributedPlan distributedPlan :
+ ((NereidsCoordinator) coordinator).getCoordinatorContext().distributedPlans) {
+ if (!(distributedPlan.getFragmentJob().getFragment().getSink() instanceof OlapTableSink)) {
+ continue;
+ }
+ planFragmentNum += distributedPlan.getInstanceJobs().size();
+ for (AssignedJob assignedJob : distributedPlan.getInstanceJobs()) {
+ sinkBackendIds.add(assignedJob.getAssignedWorker().id());
+ }
+ }
+ if (sinkBackendIds.isEmpty()) {
+ sinkBackendIds.add(currentBeId);
+ }
+ return new AdaptiveBucketSinkContext(new ArrayList<>(sinkBackendIds), Math.max(planFragmentNum, 1));
+ }
+
+ private static void assignAdaptiveBucketToPartition(TOlapTablePartition partition,
+ List<TTabletLocation> partitionTablets, long currentBeId, TUniqueId queryId) {
+ if (!Config.enable_adaptive_random_bucket_load
+ || !partition.isSetLoadTabletIdx() || currentBeId <= 0) {
Review Comment:
This runtime partition replan has no way to know that the original sink was `load_to_single_tablet=true`, so it can still rewrite `load_tablet_idx` for auto-created/replaced random partitions in an explicit single-tablet load. BE keeps those loads in `FIND_TABLET_EVERY_SINK` and ignores `local_bucket_seqs`, but it does consume `partition.load_tablet_idx`; two receiver BEs can therefore get different rewritten tablet indexes for runtime partitions even if initial planning preserves the single-tablet setting. This is distinct from the initial-planning comments because it affects `createPartition()`/`replacePartition()` after the load has started. Please gate this helper on the same load mode, or pass enough sink context to skip adaptive rewriting for `load_to_single_tablet`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4656680335
# BE UT Coverage Report
Increment line coverage `5.27% (31/588)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/2054fc372677684e4f1aa9088f1011f91a4ce5d9_2054fc372677684e4f1aa9088f1011f91a4ce5d9/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/2054fc372677684e4f1aa9088f1011f91a4ce5d9_2054fc372677684e4f1aa9088f1011f91a4ce5d9/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.88% (21135/39226) |
| Line Coverage | 37.58% (201239/535512) |
| Region Coverage | 33.57% (157828/470183) |
| Branch Coverage | 34.65% (69092/199403) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4660507752
# BE Regression && UT Coverage Report
Increment line coverage `59.52% (394/662)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_f552fc1cfb68972e5ff591a0b8668934955c7734_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_f552fc1cfb68972e5ff591a0b8668934955c7734_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.78% (28284/38337) |
| Line Coverage | 57.84% (308283/532949) |
| Region Coverage | 54.52% (257631/472505) |
| Branch Coverage | 56.00% (111929/199876) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4656211386
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4656446146
<details>
<summary>TPC-DS: <b>Total hot run time: 169956 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2054fc372677684e4f1aa9088f1011f91a4ce5d9, data reload: false
query5 4342 635 484 484
query6 444 205 181 181
query7 4823 579 290 290
query8 376 219 213 213
query9 8734 4034 4040 4034
query10 444 318 258 258
query11 5789 2375 2181 2181
query12 149 102 96 96
query13 1279 602 430 430
query14 6711 5407 5076 5076
query14_1 4425 4473 4363 4363
query15 200 193 172 172
query16 1002 453 441 441
query17 1094 685 565 565
query18 2682 457 345 345
query19 203 183 139 139
query20 110 107 103 103
query21 212 134 115 115
query22 13668 13511 13453 13453
query23 17348 16510 16093 16093
query23_1 16177 16314 16409 16314
query24 7448 1783 1300 1300
query24_1 1323 1316 1305 1305
query25 543 451 378 378
query26 1290 331 172 172
query27 2599 544 341 341
query28 4349 2016 2033 2016
query29 1061 611 476 476
query30 305 235 200 200
query31 1141 1072 948 948
query32 105 61 58 58
query33 511 320 244 244
query34 1171 1105 644 644
query35 753 788 684 684
query36 1381 1367 1253 1253
query37 149 103 90 90
query38 3231 3117 3079 3079
query39 937 918 885 885
query39_1 874 870 884 870
query40 213 128 99 99
query41 67 63 62 62
query42 96 96 96 96
query43 323 320 272 272
query44
query45 197 183 178 178
query46 1081 1197 761 761
query47 2348 2368 2215 2215
query48 397 434 294 294
query49 639 479 355 355
query50 962 354 252 252
query51 4405 4305 4254 4254
query52 88 89 77 77
query53 247 269 197 197
query54 276 215 194 194
query55 78 75 70 70
query56 238 218 221 218
query57 1443 1418 1345 1345
query58 245 208 214 208
query59 1541 1619 1405 1405
query60 294 249 235 235
query61 180 182 183 182
query62 697 657 577 577
query63 242 190 194 190
query64 2566 829 670 670
query65
query66 1777 508 343 343
query67 29814 29768 29876 29768
query68
query69 434 319 276 276
query70 1029 964 941 941
query71 302 227 220 220
query72 3235 2717 2439 2439
query73 871 773 417 417
query74 5151 4926 4755 4755
query75 2654 2592 2235 2235
query76 2920 1181 795 795
query77 354 379 293 293
query78 12314 12518 11983 11983
query79 1217 1011 754 754
query80 516 503 405 405
query81 448 280 243 243
query82 241 162 121 121
query83 274 285 254 254
query84
query85 848 542 433 433
query86 359 303 275 275
query87 3365 3437 3217 3217
query88 3633 2731 2705 2705
query89 405 392 330 330
query90 2097 192 182 182
query91 176 162 138 138
query92 65 66 55 55
query93 1602 1507 949 949
query94 536 333 310 310
query95 670 454 355 355
query96 988 800 348 348
query97 2716 2720 2580 2580
query98 212 214 205 205
query99 1134 1186 995 995
Total cold run time: 250897 ms
Total hot run time: 169956 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4656918610
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4657587873
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4657832303
<details>
<summary>TPC-H: <b>Total hot run time: 29387 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4b7e6eb1cca9bfa6551d41a3b3d5feeceaba7afc, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17639 4150 4094 4094
q2 q3 10764 1424 815 815
q4 4686 487 346 346
q5 7543 873 585 585
q6 190 179 137 137
q7 801 851 643 643
q8 9335 1439 1441 1439
q9 5890 4527 4493 4493
q10 6785 1851 1547 1547
q11 436 270 252 252
q12 641 438 294 294
q13 18119 3397 2732 2732
q14 274 261 242 242
q15 q16 821 782 712 712
q17 939 961 1045 961
q18 7223 5727 5590 5590
q19 1180 1285 1170 1170
q20 515 420 264 264
q21 6221 2822 2752 2752
q22 456 397 319 319
Total cold run time: 100458 ms
Total hot run time: 29387 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5256 5018 4948 4948
q2 q3 4940 5312 4698 4698
q4 2159 2221 1447 1447
q5 4896 4931 4803 4803
q6 254 187 129 129
q7 1888 1773 1566 1566
q8 2592 2194 2140 2140
q9 8125 7941 7556 7556
q10 4822 4696 4201 4201
q11 566 386 351 351
q12 768 755 529 529
q13 3030 3363 2806 2806
q14 272 276 248 248
q15 q16 681 702 622 622
q17 1297 1289 1279 1279
q18 7670 6730 6911 6730
q19 1104 1088 1082 1082
q20 2221 2252 1967 1967
q21 5387 4654 4446 4446
q22 524 469 398 398
Total cold run time: 58452 ms
Total hot run time: 51946 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](load) adaptive random bucket load: per-BE bucket assignment with threshold-based rotation [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4645241520
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3371541935
##########
be/src/exec/sink/writer/vtablet_writer.cpp:
##########
@@ -657,6 +670,37 @@ void VNodeChannel::_open_internal(bool is_incremental) {
request->set_txn_expiration(_parent->_txn_expiration);
request->set_write_file_cache(_parent->_write_file_cache);
+ if (_parent->_tablet_finder->is_adaptive_random_bucket()) {
+ std::unordered_map<int64_t, std::vector<int64_t>> partition_to_ordered_tablets;
+ for (const auto& tablet : _all_tablets) {
+ partition_to_ordered_tablets[tablet.partition_id].push_back(tablet.tablet_id);
+ }
+ std::unordered_map<int64_t, const std::vector<int32_t>*> partition_to_bucket_seqs;
+ for (const auto* part : _parent->_vpartition->get_partitions()) {
+ if (!part->local_bucket_seqs.empty()) {
+ partition_to_bucket_seqs.emplace(part->id, &part->local_bucket_seqs);
+ }
+ }
+ for (const auto& [partition_id, ordered_tablets] : partition_to_ordered_tablets) {
+ auto* random_bucket_partition = request->add_random_bucket_partitions();
+ random_bucket_partition->set_partition_id(partition_id);
+ auto bucket_seqs_it = partition_to_bucket_seqs.find(partition_id);
+ if (bucket_seqs_it != partition_to_bucket_seqs.end()) {
+ for (auto bucket_seq : *bucket_seqs_it->second) {
+ CHECK(bucket_seq >= 0 && bucket_seq < cast_set<int32_t>(ordered_tablets.size()))
+ << "invalid local bucket seq, load_id=" << _parent->_load_id
Review Comment:
This treats `local_bucket_seqs` as indexes into `ordered_tablets`, but `ordered_tablets` is built from this `VNodeChannel`'s `_all_tablets`, not from the partition's full bucket array. In single-replica or otherwise sparse placement, a BE-local channel only contains the buckets hosted on that BE, while `local_bucket_seqs` still contains global bucket numbers. For example, if this BE owns global buckets `[2, 5]`, `_all_tablets` has two entries but the first `bucket_seq` is `2`, so this `CHECK` fails; if the vector happened to be longer but sparse, it would pick the wrong tablet. The open request should map global bucket seqs against the full partition tablet list, or build a `bucket_seq -> tablet_id` map before filtering to the selected local sequence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4668939935
<details>
<summary>TPC-H: <b>Total hot run time: 29558 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1997c78bdbc8252f993c10c39035ec5e91e6b06b, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17838 4119 4031 4031
q2 q3 10866 1469 822 822
q4 4762 485 344 344
q5 8503 874 590 590
q6 330 174 135 135
q7 903 871 645 645
q8 10844 1592 1613 1592
q9 7083 4522 4452 4452
q10 6825 1843 1533 1533
q11 451 275 253 253
q12 641 430 287 287
q13 18118 3450 2817 2817
q14 269 257 234 234
q15 q16 824 772 709 709
q17 980 950 918 918
q18 7307 5915 5631 5631
q19 1194 1285 1284 1284
q20 564 477 294 294
q21 5853 2737 2667 2667
q22 457 363 320 320
Total cold run time: 104612 ms
Total hot run time: 29558 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5004 4817 4865 4817
q2 q3 5030 5224 4566 4566
q4 2117 2196 1381 1381
q5 4966 4664 4712 4664
q6 235 182 131 131
q7 1884 1736 1531 1531
q8 2531 2138 2093 2093
q9 7591 7477 7477 7477
q10 4744 4645 4236 4236
q11 544 378 355 355
q12 724 734 524 524
q13 3034 3400 2754 2754
q14 271 278 254 254
q15 q16 687 695 609 609
q17 1294 1287 1268 1268
q18 7290 6796 6919 6796
q19 1156 1123 1131 1123
q20 2241 2209 1953 1953
q21 5252 4517 4373 4373
q22 549 479 404 404
Total cold run time: 57144 ms
Total hot run time: 51309 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4669951650
# BE Regression && UT Coverage Report
Increment line coverage `63.86% (417/653)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_1997c78bdbc8252f993c10c39035ec5e91e6b06b_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_1997c78bdbc8252f993c10c39035ec5e91e6b06b_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.88% (28325/38339) |
| Line Coverage | 57.91% (308697/533056) |
| Region Coverage | 54.66% (258264/472525) |
| Branch Coverage | 56.14% (112244/199931) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3413227400
##########
fe/fe-core/src/main/java/org/apache/doris/planner/OlapTableSink.java:
##########
@@ -574,6 +581,302 @@ private TOlapTablePartitionParam createDummyPartition(long dbId, OlapTable table
return partitionParam;
}
+ public static final class AdaptiveBucketAssignment {
+ private final long bucketBeId;
+ private final int loadTabletIdx;
+ private final List<Integer> localBucketSeqs;
+
+ public AdaptiveBucketAssignment(long bucketBeId, int loadTabletIdx, List<Integer> localBucketSeqs) {
+ this.bucketBeId = bucketBeId;
+ this.loadTabletIdx = loadTabletIdx;
+ this.localBucketSeqs = new ArrayList<>(localBucketSeqs);
+ }
+
+ public long getBucketBeId() {
+ return bucketBeId;
+ }
+
+ public int getLoadTabletIdx() {
+ return loadTabletIdx;
+ }
+
+ public List<Integer> getLocalBucketSeqs() {
+ return localBucketSeqs;
+ }
+ }
+
+ public static boolean shouldAssignAdaptiveRandomBucket(TOlapTableSink sink) {
+ return sink != null
+ && sink.isSetEnableAdaptiveRandomBucket()
+ && sink.isEnableAdaptiveRandomBucket()
+ && (!sink.isSetLoadToSingleTablet() || !sink.isLoadToSingleTablet())
+ && sink.isSetPartition()
+ && sink.getPartition() != null
+ && (!sink.getPartition().isSetDistributedColumns()
+ || sink.getPartition().getDistributedColumns().isEmpty());
+ }
+
+ public boolean shouldAssignAdaptiveRandomBucket() {
+ return tDataSink != null && shouldAssignAdaptiveRandomBucket(tDataSink.getOlapTableSink());
+ }
+
+ public static Map<Long, Map<Long, AdaptiveBucketAssignment>> computeAdaptiveRandomBucketAssignments(
+ List<Long> sinkBackendIds, List<TOlapTablePartition> partitions,
+ List<TTabletLocation> tabletLocations, int planFragmentNum) {
+ Map<Long, Map<Long, AdaptiveBucketAssignment>> assignments = new HashMap<>();
+ List<Long> orderedSinkBackendIds = sinkBackendIds.stream()
+ .distinct()
+ .sorted()
+ .collect(Collectors.toList());
+ for (Long sinkBackendId : orderedSinkBackendIds) {
+ assignments.put(sinkBackendId, new HashMap<>());
+ }
+ if (orderedSinkBackendIds.isEmpty() || partitions == null || tabletLocations == null) {
+ return assignments;
+ }
+
+ Map<Long, TTabletLocation> tabletLocationMap = new HashMap<>(tabletLocations.size());
+ for (TTabletLocation tabletLocation : tabletLocations) {
+ tabletLocationMap.put(tabletLocation.getTabletId(), tabletLocation);
+ }
+
+ for (TOlapTablePartition partition : partitions) {
+ if (!partition.isSetLoadTabletIdx() || partition.getNumBuckets() <= 0
+ || partition.getIndexes().isEmpty()) {
+ continue;
+ }
+ Map<Long, List<Integer>> beToBucketSeqs = buildBeToBucketSeqs(partition, tabletLocationMap);
+ long baseTabletIndex = partition.getLoadTabletIdx();
+ int fallbackBucketIdx = (int) Math.floorMod(baseTabletIndex, (long) partition.getNumBuckets());
+ int targetBucketNum = Math.min(
+ Math.min(orderedSinkBackendIds.size(), partition.getNumBuckets()),
+ Math.max(planFragmentNum, 1));
+ if (targetBucketNum <= 0) {
+ continue;
+ }
+
+ List<Long> rotatedSinkBackendIds = rotateSinkBackendIds(orderedSinkBackendIds, baseTabletIndex);
+ Map<Integer, Long> bucketToOwnerBe = buildBucketToOwnerBe(beToBucketSeqs);
+ List<Integer> selectedBucketSeqs = selectAdaptiveBucketSeqs(rotatedSinkBackendIds,
+ beToBucketSeqs, bucketToOwnerBe, baseTabletIndex, partition.getNumBuckets(), targetBucketNum);
+ if (selectedBucketSeqs.isEmpty()) {
+ selectedBucketSeqs = Collections.singletonList(fallbackBucketIdx);
+ }
+ if (selectedBucketSeqs.size() != targetBucketNum) {
+ LOG.warn("Adaptive random bucket selected {} buckets instead of target {} for partition {}, "
+ + "sinkBackendIds={}, beToBucketSeqs={}, selectedBucketSeqs={}, fallbackBucketIdx={}",
+ selectedBucketSeqs.size(), targetBucketNum, partition.getId(), orderedSinkBackendIds,
+ beToBucketSeqs, selectedBucketSeqs, fallbackBucketIdx);
+ }
+
+ Map<Long, List<Integer>> openedBeToBucketSeqs = buildBeToBucketSeqs(bucketToOwnerBe, selectedBucketSeqs);
+ Map<Integer, Integer> bucketUseCounts = new HashMap<>();
+ Map<Long, String> sinkAssignments = LOG.isInfoEnabled() ? new HashMap<>() : null;
+ for (Long sinkBackendId : rotatedSinkBackendIds) {
+ int bucketSeq = selectLeastUsedBucketSeq(
+ openedBeToBucketSeqs.get(sinkBackendId), bucketUseCounts, baseTabletIndex);
+ if (bucketSeq < 0) {
+ bucketSeq = selectLeastUsedBucketSeq(selectedBucketSeqs, bucketUseCounts, baseTabletIndex);
+ }
+ if (bucketSeq < 0) {
+ bucketSeq = fallbackBucketIdx;
+ }
+ long bucketBeId = bucketToOwnerBe.getOrDefault(bucketSeq, -1L);
+ if (bucketBeId <= 0) {
+ LOG.warn("Adaptive random bucket falls back to bucket {} without owner BE for partition {}, "
+ + "sinkBackendId={}, selectedBucketSeqs={}, beToBucketSeqs={}",
+ bucketSeq, partition.getId(), sinkBackendId, selectedBucketSeqs, beToBucketSeqs);
+ }
+ List<Integer> localBucketSeqs = rotateBucketSeqsForStartBucket(
+ beToBucketSeqs.get(bucketBeId), bucketSeq);
+ assignments.get(sinkBackendId).put(partition.getId(),
+ new AdaptiveBucketAssignment(bucketBeId, bucketSeq, localBucketSeqs));
+ bucketUseCounts.merge(bucketSeq, 1, Integer::sum);
+ if (sinkAssignments != null) {
+ sinkAssignments.put(sinkBackendId,
+ "bucket=" + bucketSeq + ",bucketBeId=" + bucketBeId
+ + ",localBucketSeqs=" + localBucketSeqs);
+ }
+ }
+ if (sinkAssignments != null) {
+ LOG.info("Adaptive random bucket plan partition={}, baseTabletIndex={}, targetBucketNum={}, "
+ + "sinkBackendIds={}, rotatedSinkBackendIds={}, beToBucketSeqs={}, "
+ + "selectedBucketSeqs={}, openedBeToBucketSeqs={}, sinkAssignments={}",
+ partition.getId(), baseTabletIndex, targetBucketNum, orderedSinkBackendIds,
+ rotatedSinkBackendIds, beToBucketSeqs, selectedBucketSeqs, openedBeToBucketSeqs,
+ sinkAssignments);
+ }
+ }
+ return assignments;
+ }
+
+ public static void applyAdaptiveRandomBucketAssignments(List<TOlapTablePartition> partitions,
+ Map<Long, AdaptiveBucketAssignment> partitionAssignments) {
+ if (partitions == null || partitionAssignments == null || partitionAssignments.isEmpty()) {
+ return;
+ }
+ for (TOlapTablePartition partition : partitions) {
+ AdaptiveBucketAssignment assignment = partitionAssignments.get(partition.getId());
+ if (assignment == null) {
+ continue;
+ }
+ partition.setLoadTabletIdx(assignment.getLoadTabletIdx());
+ if (assignment.getBucketBeId() > 0) {
+ partition.setBucketBeId(assignment.getBucketBeId());
+ } else if (partition.isSetBucketBeId()) {
+ partition.unsetBucketBeId();
+ }
+ if (!assignment.getLocalBucketSeqs().isEmpty()) {
+ partition.setLocalBucketSeqs(new ArrayList<>(assignment.getLocalBucketSeqs()));
+ } else if (partition.isSetLocalBucketSeqs()) {
+ partition.unsetLocalBucketSeqs();
+ }
+ if (LOG.isInfoEnabled()) {
+ LOG.info("Adaptive random bucket apply partition={}, bucketBeId={}, loadTabletIdx={}, "
+ + "localBucketSeqs={}",
+ partition.getId(), assignment.getBucketBeId(), assignment.getLoadTabletIdx(),
+ assignment.getLocalBucketSeqs());
+ }
+ }
+ }
+
+ private static Map<Long, List<Integer>> buildBeToBucketSeqs(TOlapTablePartition partition,
+ Map<Long, TTabletLocation> tabletLocationMap) {
Review Comment:
This derives the adaptive bucket owner from only the first materialized index, but the assignment is stored on `TOlapTablePartition` and `IndexChannel` reuses the same `bucket_be_id` / `local_bucket_seqs` for every index. In cloud a random table can have a rollup/materialized index whose tablet for the same bucket sequence is placed on a different BE than the base index. Then the secondary index channel either skips the selected seq as non-local during `_open_internal()` and fails later with no receiver state, or rotates to a different local seq than the base index. That can make a load into a random table with rollups fail or write different indexes to different bucket sequences. Please make the adaptive assignment per index, or gate it on verifying that every index has the same owner for each selected bucket sequence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4710197154
# FE UT Coverage Report
Increment line coverage ` 37.38% (157/420)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_d8fb460dd1f77e6dd47d3649a8f89b0f7cb9922b/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_d8fb460dd1f77e6dd47d3649a8f89b0f7cb9922b/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui merged PR #62661:
URL: https://github.com/apache/doris/pull/62661
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4896732257
# BE Regression && UT Coverage Report
Increment line coverage `26.01% (186/715)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_30a431f2b08623dcddbc518296a9a8b5c5a06387_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_30a431f2b08623dcddbc518296a9a8b5c5a06387_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 64.14% (25627/39954) |
| Line Coverage | 47.85% (265480/554767) |
| Region Coverage | 44.65% (219012/490490) |
| Branch Coverage | 45.58% (96236/211143) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](load) adaptive random bucket load: per-BE bucket assignment with threshold-based rotation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3116154079
##########
fe/fe-core/src/main/java/org/apache/doris/qe/runtime/ThriftPlansBuilder.java:
##########
@@ -242,11 +243,116 @@ private static void setParamsForOlapTableSink(List<PipelineDistributedPlan> dist
fragmentParams.setNumLocalSink(fragmentParams.getLocalParams().size());
LOG.info("num local sink for backend {} is {}", fragmentParams.getBackendId(),
fragmentParams.getNumLocalSink());
+
+ // Assign per-BE starting bucket and bucket_be_id for random distribution
+ assignRandomBucketPerBe(fragmentParams, worker.id());
Review Comment:
This runs for every `OLAP_TABLE_SINK`, including `load_to_single_tablet=true`. In that mode BE still chooses `FIND_TABLET_EVERY_SINK`, so it will honor the rewritten `load_tablet_idx` but ignore `local_bucket_seqs`. That changes the semantics from "one tablet for the whole load" to "one tablet per BE". For example, with two BEs and global `load_tablet_idx=0`, BE1 can be rewritten to bucket 0 while BE2 is rewritten to bucket 1. Please skip the per-BE rewrite when `load_to_single_tablet` is enabled.
##########
fe/fe-core/src/main/java/org/apache/doris/qe/runtime/ThriftPlansBuilder.java:
##########
@@ -242,11 +243,116 @@ private static void setParamsForOlapTableSink(List<PipelineDistributedPlan> dist
fragmentParams.setNumLocalSink(fragmentParams.getLocalParams().size());
LOG.info("num local sink for backend {} is {}", fragmentParams.getBackendId(),
fragmentParams.getNumLocalSink());
+
+ // Assign per-BE starting bucket and bucket_be_id for random distribution
+ assignRandomBucketPerBe(fragmentParams, worker.id());
}
}
}
}
+ /**
+ * For random-distribution partitions, override load_tablet_idx and set bucket_be_id /
+ * local_bucket_seqs so that each BE starts writing to one of its own local buckets and
+ * can rotate within them once per-tablet bytes exceed the threshold.
+ *
+ * <p>local_bucket_seqs always contains all local buckets for this BE. The starting bucket
+ * rotates across consecutive load tasks via tabletIndex; within a single load, the BE
+ * switches to the next entry in the list only when the 200 MB threshold is reached.
+ * Small loads (e.g. stream load) naturally stay on one bucket because the threshold is
+ * never triggered.
+ */
+ private static void assignRandomBucketPerBe(TPipelineFragmentParams fragmentParams, long beId) {
+ org.apache.doris.thrift.TDataSink outputSink = fragmentParams.getFragment().getOutputSink();
+ org.apache.doris.thrift.TOlapTableSink olapSink = outputSink.getOlapTableSink();
+ if (olapSink == null || olapSink.getPartition() == null
+ || !olapSink.getPartition().isSetPartitions()) {
+ return;
+ }
+
+ // Check whether any partition uses random distribution (identified by load_tablet_idx being set)
+ boolean hasRandomPartition = olapSink.getPartition().getPartitions().stream()
+ .anyMatch(org.apache.doris.thrift.TOlapTablePartition::isSetLoadTabletIdx);
+ if (!hasRandomPartition) {
+ return;
+ }
+
+ // Build tablet_id -> BE IDs index from location
+ org.apache.doris.thrift.TOlapTableLocationParam location = olapSink.getLocation();
+ if (location == null || !location.isSetTablets()) {
+ return;
+ }
+ Map<Long, List<Long>> tabletToBeIds = new java.util.HashMap<>();
+ for (org.apache.doris.thrift.TTabletLocation tabletLoc : location.getTablets()) {
+ tabletToBeIds.put(tabletLoc.getTabletId(), tabletLoc.getNodeIds());
+ }
+
+ // Deep-copy only the OlapTableSink so that per-BE modifications are isolated
+ org.apache.doris.thrift.TOlapTableSink sinkCopy = olapSink.deepCopy();
+ outputSink.setOlapTableSink(sinkCopy);
Review Comment:
`outputSink` is still shared across workers here. `PlanFragment.toThrift()` stores `sink.toThrift()`, and `OlapTableSink.toThrift()` just returns its cached `tDataSink`, so every worker's `TPlanFragment.outputSink` points to the same `TDataSink` instance. Replacing only the nested `TOlapTableSink` means the last worker that executes this code overwrites the previous workers' bucket assignments. To make the per-BE mutation real, the whole `TDataSink` (or enclosing `TPlanFragment`) needs to be deep-copied before mutating it.
##########
be/src/exec/sink/writer/vtablet_writer.cpp:
##########
@@ -1546,7 +1546,7 @@ Status VTabletWriter::_init(RuntimeState* state, RuntimeProfile* profile) {
if (table_sink.__isset.load_to_single_tablet && table_sink.load_to_single_tablet) {
find_tablet_mode = OlapTabletFinder::FindTabletMode::FIND_TABLET_EVERY_SINK;
} else {
- find_tablet_mode = OlapTabletFinder::FindTabletMode::FIND_TABLET_EVERY_BATCH;
+ find_tablet_mode = OlapTabletFinder::FindTabletMode::FIND_TABLET_RANDOM_BUCKET;
Review Comment:
Switching every random-distribution sink to `FIND_TABLET_RANDOM_BUCKET` is not backward-compatible yet. The new mode only works when FE has filled `local_bucket_seqs`, but that happens only in `ThriftPlansBuilder` today. Classic `Coordinator` and `NereidsStreamLoadPlanner` still serialize `fragment.toThrift()` directly, and mixed FE/BE versions also will not send the new fields. In all of those cases `seqs` stays empty, so we silently lose the old per-batch round-robin and pin the whole sink to a single bucket. This needs a compatibility fallback until every FE path populates the new fields.`
##########
be/src/exec/sink/vtablet_finder.cpp:
##########
@@ -82,8 +83,39 @@ Status OlapTabletFinder::find_tablets(RuntimeState* state, Block* block, int row
if (_find_tablet_mode == FindTabletMode::FIND_TABLET_EVERY_ROW) {
_vpartition->find_tablets(block, qualified_rows, partitions, tablet_index);
+ } else if (_find_tablet_mode == FindTabletMode::FIND_TABLET_RANDOM_BUCKET) {
+ // Use per-batch caching same as FIND_TABLET_EVERY_BATCH
+ _vpartition->find_tablets(block, qualified_rows, partitions, tablet_index,
+ &_partition_to_tablet_map);
+
+ int64_t bytes_per_row = rows > 0 ? block->bytes() / rows : 0;
+ for (auto& [partition, _] : _partition_to_tablet_map) {
+ if (partition->indexes.empty()) {
+ continue;
+ }
+ int64_t cur_bucket = partition->load_tablet_idx % partition->num_buckets;
+ int64_t cur_tablet_id = partition->indexes[0].tablets[cur_bucket];
+ _tablet_written_bytes[cur_tablet_id] += bytes_per_row * rows;
Review Comment:
This charges the full block size to every touched partition. If a batch contains rows for partitions A and B, both counters get `block->bytes()`, so the 200 MB threshold is reached 2x too fast; with N touched partitions it is N times too fast. The rotation threshold needs to use bytes (or at least row counts) for the rows that actually routed to this partition, not the whole batch.
##########
be/src/exec/sink/vtablet_finder.cpp:
##########
@@ -82,8 +83,39 @@ Status OlapTabletFinder::find_tablets(RuntimeState* state, Block* block, int row
if (_find_tablet_mode == FindTabletMode::FIND_TABLET_EVERY_ROW) {
_vpartition->find_tablets(block, qualified_rows, partitions, tablet_index);
+ } else if (_find_tablet_mode == FindTabletMode::FIND_TABLET_RANDOM_BUCKET) {
+ // Use per-batch caching same as FIND_TABLET_EVERY_BATCH
+ _vpartition->find_tablets(block, qualified_rows, partitions, tablet_index,
+ &_partition_to_tablet_map);
+
+ int64_t bytes_per_row = rows > 0 ? block->bytes() / rows : 0;
+ for (auto& [partition, _] : _partition_to_tablet_map) {
+ if (partition->indexes.empty()) {
+ continue;
+ }
+ int64_t cur_bucket = partition->load_tablet_idx % partition->num_buckets;
+ int64_t cur_tablet_id = partition->indexes[0].tablets[cur_bucket];
+ _tablet_written_bytes[cur_tablet_id] += bytes_per_row * rows;
+
+ // Switch to the next local bucket when the threshold is reached.
+ // local_bucket_seqs carries the exact set of buckets assigned to this BE by the FE,
+ // which handles both single-replica and multi-replica correctly.
+ const auto& seqs = partition->local_bucket_seqs;
+ if (_tablet_written_bytes[cur_tablet_id] >= BUCKET_SWITCH_THRESHOLD_BYTES &&
+ !seqs.empty()) {
+ auto it = std::find(seqs.begin(), seqs.end(), static_cast<int32_t>(cur_bucket));
+ if (it != seqs.end()) {
+ ++it;
+ if (it == seqs.end()) {
+ it = seqs.begin();
+ }
+ partition->load_tablet_idx = *it;
Review Comment:
When we advance to the next bucket, the counter for `cur_tablet_id` is left above the threshold forever. After the rotation wraps back to this tablet later, the next small batch immediately switches away again instead of allowing another ~200 MB on that bucket. For example, with local buckets `[0, 2]`, after writing 200 MB to 0 and 200 MB to 2, the first small batch back on 0 will rotate away again. The counter needs to be reset or have the threshold subtracted when the bucket advances.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3505034362
##########
regression-test/suites/load_p0/insert/test_insert_random_distribution_table.groovy:
##########
@@ -15,9 +15,15 @@
// specific language governing permissions and limitations
// under the License.
-suite("test_insert_random_distribution_table", "p0") {
+suite("test_insert_random_distribution_table", "p0,nonConcurrent") {
def tableName = "test_insert_random_distribution_table"
+ def enableAdaptiveRandomBucketConfig = sql """ ADMIN SHOW FRONTEND CONFIG LIKE 'enable_adaptive_random_bucket_load'; """
+ String oldEnableAdaptiveRandomBucket = enableAdaptiveRandomBucketConfig[0][1]
+ try {
+ // This case verifies the legacy batch-size based random tablet rotation.
+ // Keep the suite nonConcurrent because this FE config is global.
+ sql """ ADMIN SET FRONTEND CONFIG ('enable_adaptive_random_bucket_load' = 'false') """
Review Comment:
This PR enables adaptive random bucket load by default, but this regression explicitly disables the feature and therefore only preserves the legacy random-bucket behavior. The new path spans FE assignment, thrift/protobuf metadata, sender partition routing, receiver-side tablet selection/rotation, cloud lazy writer init, and close/commit semantics. Could we add an adaptive-enabled cloud load regression that keeps this config on and verifies rows are routed/loaded correctly, ideally including rotation after memtable flush or a runtime partition path?
##########
be/src/load/delta_writer/delta_writer.cpp:
##########
@@ -155,10 +161,22 @@ Status BaseDeltaWriter::init() {
return Status::OK();
}
-Status DeltaWriter::write(const Block* block, const DorisVector<uint32_t>& row_idxs) {
+Status DeltaWriter::write(const Block* block, const DorisVector<uint32_t>& row_idxs,
+ bool* memtable_flushed) {
+ if (memtable_flushed != nullptr) {
+ *memtable_flushed = false;
+ }
if (UNLIKELY(row_idxs.empty())) {
return Status::OK();
}
+ if (_req.enable_table_memtable_backpressure) {
Review Comment:
This table-level memtable backpressure does not appear to preserve the existing high-priority load bypass. LoadChannelMgr skips the global memtable limiter for high-priority loads because blocking here can cause RPC timeout, but adaptive writes can still enter this new table-level wait before taking the writer lock. Could we skip this wait, or make it bounded/non-blocking, when _req.is_high_priority is true? The same policy should probably be applied to CloudDeltaWriter and DeltaWriterV2 as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3505033025
##########
be/src/load/channel/adaptive_random_bucket_state.cpp:
##########
@@ -0,0 +1,81 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "load/channel/adaptive_random_bucket_state.h"
+
+#include <glog/logging.h>
+
+#include <utility>
+
+namespace doris {
+
+void AdaptiveRandomBucketState::init_partition(int64_t partition_id,
+ const std::vector<int64_t>& tablets,
+ const std::vector<int32_t>& bucket_seqs,
+ int32_t start_tablet_idx) {
+ if (partition_id < 0 || tablets.empty()) {
+ return;
+ }
+ std::lock_guard<std::mutex> lock(_mutex);
+ if (_partition_states.contains(partition_id)) {
Review Comment:
The assignment/state granularity still looks ambiguous here. FE computes adaptive assignments keyed by sink backend, but the receiver stores only one state per partition and ignores subsequent init for the same partition. If multiple senders route the same partition to the same receiver BE with different ordered bucket lists, the later sender will use the first sender's current tablet/sequence instead of its FE assignment. Could we either scope the receiver state by sender+partition, or explicitly prove/enforce that every open for the same receiver partition carries the same sequence? A regression with multiple sink instances/senders landing on the same receiver BE would also make the intended contract much clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](load) adaptive random bucket load: per-BE bucket assignment with threshold-based rotation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3116113334
##########
fe/fe-core/src/main/java/org/apache/doris/qe/runtime/ThriftPlansBuilder.java:
##########
@@ -242,11 +243,116 @@ private static void setParamsForOlapTableSink(List<PipelineDistributedPlan> dist
fragmentParams.setNumLocalSink(fragmentParams.getLocalParams().size());
LOG.info("num local sink for backend {} is {}", fragmentParams.getBackendId(),
fragmentParams.getNumLocalSink());
+
+ // Assign per-BE starting bucket and bucket_be_id for random distribution
Review Comment:
This only updates the Nereids `ThriftPlansBuilder` path, but BE now switches every random-distribution sink to `FIND_TABLET_RANDOM_BUCKET` in both `VTabletWriter` implementations. The classic `Coordinator` path (`Coordinator.FragmentExecParams.toThrift()`) and the one-shot stream-load / group-commit path (`NereidsStreamLoadPlanner.plan()`) still serialize the old sink shape and never populate `local_bucket_seqs`. On those paths `local_bucket_seqs` stays empty in `vtablet_finder.cpp`, so the writer no longer does the old per-batch round-robin and collapses back to a single bucket for the whole load. This needs the same FE propagation on every `OLAP_TABLE_SINK` planning path, not just this one.
##########
be/src/exec/sink/vtablet_finder.cpp:
##########
@@ -82,8 +83,39 @@ Status OlapTabletFinder::find_tablets(RuntimeState* state, Block* block, int row
if (_find_tablet_mode == FindTabletMode::FIND_TABLET_EVERY_ROW) {
_vpartition->find_tablets(block, qualified_rows, partitions, tablet_index);
+ } else if (_find_tablet_mode == FindTabletMode::FIND_TABLET_RANDOM_BUCKET) {
+ // Use per-batch caching same as FIND_TABLET_EVERY_BATCH
+ _vpartition->find_tablets(block, qualified_rows, partitions, tablet_index,
+ &_partition_to_tablet_map);
+
+ int64_t bytes_per_row = rows > 0 ? block->bytes() / rows : 0;
+ for (auto& [partition, _] : _partition_to_tablet_map) {
+ if (partition->indexes.empty()) {
+ continue;
+ }
+ int64_t cur_bucket = partition->load_tablet_idx % partition->num_buckets;
+ int64_t cur_tablet_id = partition->indexes[0].tablets[cur_bucket];
Review Comment:
This charges each touched partition with the full block size. `_partition_to_tablet_map` has one entry per partition, but `bytes_per_row * rows` is just `block->bytes()` for the whole batch, not the bytes belonging to that partition's rows. If one block fans out to N partitions, each partition is overcounted by roughly N and the 200 MB rotation fires much earlier than intended. The accounting needs to be partition-local (for example by counting rows per partition or actual bytes sent per tablet).
##########
be/src/exec/sink/vtablet_finder.cpp:
##########
@@ -82,8 +83,39 @@ Status OlapTabletFinder::find_tablets(RuntimeState* state, Block* block, int row
if (_find_tablet_mode == FindTabletMode::FIND_TABLET_EVERY_ROW) {
_vpartition->find_tablets(block, qualified_rows, partitions, tablet_index);
+ } else if (_find_tablet_mode == FindTabletMode::FIND_TABLET_RANDOM_BUCKET) {
+ // Use per-batch caching same as FIND_TABLET_EVERY_BATCH
+ _vpartition->find_tablets(block, qualified_rows, partitions, tablet_index,
+ &_partition_to_tablet_map);
+
+ int64_t bytes_per_row = rows > 0 ? block->bytes() / rows : 0;
+ for (auto& [partition, _] : _partition_to_tablet_map) {
+ if (partition->indexes.empty()) {
+ continue;
+ }
+ int64_t cur_bucket = partition->load_tablet_idx % partition->num_buckets;
+ int64_t cur_tablet_id = partition->indexes[0].tablets[cur_bucket];
+ _tablet_written_bytes[cur_tablet_id] += bytes_per_row * rows;
+
+ // Switch to the next local bucket when the threshold is reached.
+ // local_bucket_seqs carries the exact set of buckets assigned to this BE by the FE,
+ // which handles both single-replica and multi-replica correctly.
+ const auto& seqs = partition->local_bucket_seqs;
Review Comment:
The counter is never reset when we advance away from a bucket. After bucket 0 crosses 200 MB once, wrapping back to bucket 0 later will hit this condition again on the very next batch, so large loads degenerate into 'rotate every batch after the first full cycle' instead of writing ~200 MB chunks as described in the PR. We need to subtract/reset the current bucket's counter when switching, or track bytes since the current bucket became active rather than lifetime bytes per tablet.
##########
fe/fe-core/src/main/java/org/apache/doris/qe/runtime/ThriftPlansBuilder.java:
##########
@@ -242,11 +243,116 @@ private static void setParamsForOlapTableSink(List<PipelineDistributedPlan> dist
fragmentParams.setNumLocalSink(fragmentParams.getLocalParams().size());
LOG.info("num local sink for backend {} is {}", fragmentParams.getBackendId(),
fragmentParams.getNumLocalSink());
+
+ // Assign per-BE starting bucket and bucket_be_id for random distribution
+ assignRandomBucketPerBe(fragmentParams, worker.id());
}
}
}
}
+ /**
+ * For random-distribution partitions, override load_tablet_idx and set bucket_be_id /
+ * local_bucket_seqs so that each BE starts writing to one of its own local buckets and
+ * can rotate within them once per-tablet bytes exceed the threshold.
+ *
+ * <p>local_bucket_seqs always contains all local buckets for this BE. The starting bucket
+ * rotates across consecutive load tasks via tabletIndex; within a single load, the BE
+ * switches to the next entry in the list only when the 200 MB threshold is reached.
+ * Small loads (e.g. stream load) naturally stay on one bucket because the threshold is
+ * never triggered.
+ */
+ private static void assignRandomBucketPerBe(TPipelineFragmentParams fragmentParams, long beId) {
+ org.apache.doris.thrift.TDataSink outputSink = fragmentParams.getFragment().getOutputSink();
+ org.apache.doris.thrift.TOlapTableSink olapSink = outputSink.getOlapTableSink();
+ if (olapSink == null || olapSink.getPartition() == null
+ || !olapSink.getPartition().isSetPartitions()) {
+ return;
+ }
+
+ // Check whether any partition uses random distribution (identified by load_tablet_idx being set)
+ boolean hasRandomPartition = olapSink.getPartition().getPartitions().stream()
+ .anyMatch(org.apache.doris.thrift.TOlapTablePartition::isSetLoadTabletIdx);
+ if (!hasRandomPartition) {
+ return;
+ }
+
+ // Build tablet_id -> BE IDs index from location
+ org.apache.doris.thrift.TOlapTableLocationParam location = olapSink.getLocation();
+ if (location == null || !location.isSetTablets()) {
+ return;
+ }
+ Map<Long, List<Long>> tabletToBeIds = new java.util.HashMap<>();
+ for (org.apache.doris.thrift.TTabletLocation tabletLoc : location.getTablets()) {
+ tabletToBeIds.put(tabletLoc.getTabletId(), tabletLoc.getNodeIds());
+ }
+
+ // Deep-copy only the OlapTableSink so that per-BE modifications are isolated
+ org.apache.doris.thrift.TOlapTableSink sinkCopy = olapSink.deepCopy();
Review Comment:
`fragmentParams.getFragment().getOutputSink()` is not BE-local here. `PlanFragment.toThrift()` calls `sink.toThrift()`, and `OlapTableSink.toThrift()` returns the shared `tDataSink` object, so every `TPipelineFragmentParams` for this fragment still points at the same `TDataSink`. Replacing `olap_table_sink` on that shared object means the last worker processed overwrites the sink seen by all earlier workers. To make the per-BE mutation real, you need to deep-copy the enclosing `TDataSink` / `TPlanFragment` before mutating it, not only the nested `TOlapTableSink`.
##########
be/src/storage/tablet_info.cpp:
##########
@@ -700,10 +700,15 @@ Status VOlapTablePartitionParam::generate_partition_from(const TOlapTablePartiti
part_result = _obj_pool.add(new VOlapTablePartition(&_partition_block));
part_result->id = t_part.id;
part_result->is_mutable = t_part.is_mutable;
- // only load_to_single_tablet = true will set load_tablet_idx
if (t_part.__isset.load_tablet_idx) {
part_result->load_tablet_idx = t_part.load_tablet_idx;
}
+ if (t_part.__isset.bucket_be_id) {
+ part_result->bucket_be_id = t_part.bucket_be_id;
+ }
+ if (t_part.__isset.local_bucket_seqs) {
+ part_result->local_bucket_seqs = t_part.local_bucket_seqs;
Review Comment:
This only covers the initial `generate_partition_from()` path. The runtime partition refresh paths still build `VOlapTablePartition` manually in `add_partitions()` and `replace_partitions()` without copying `load_tablet_idx` / `local_bucket_seqs`. For random-distribution tables with auto partitioning or replace-partition during load, `FIND_TABLET_RANDOM_BUCKET` then sees `load_tablet_idx == -1` and line 96 in `vtablet_finder.cpp` indexes `indexes[0].tablets[cur_bucket]` with `cur_bucket = -1 % num_buckets`. Those paths need the same field propagation as the initial load path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](load) adaptive random bucket load: per-BE bucket assignment with threshold-based rotation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4287580742
# BE UT Coverage Report
Increment line coverage `9.09% (3/33)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/982c51652552ce932a3ab99979f4e6ba198d49dd_982c51652552ce932a3ab99979f4e6ba198d49dd/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/982c51652552ce932a3ab99979f4e6ba198d49dd_982c51652552ce932a3ab99979f4e6ba198d49dd/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.30% (20356/38192) |
| Line Coverage | 36.85% (191808/520462) |
| Region Coverage | 33.18% (149211/449735) |
| Branch Coverage | 34.28% (65248/190362) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4658264196
# BE UT Coverage Report
Increment line coverage `4.87% (31/637)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/4b7e6eb1cca9bfa6551d41a3b3d5feeceaba7afc_4b7e6eb1cca9bfa6551d41a3b3d5feeceaba7afc/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/4b7e6eb1cca9bfa6551d41a3b3d5feeceaba7afc_4b7e6eb1cca9bfa6551d41a3b3d5feeceaba7afc/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.85% (21134/39249) |
| Line Coverage | 37.54% (201252/536053) |
| Region Coverage | 33.56% (157829/470355) |
| Branch Coverage | 34.62% (69108/199597) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4658381191
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3380025028
##########
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:
##########
@@ -4561,12 +4570,26 @@ public TCreatePartitionResult createPartition(TCreatePartitionRequest request) t
tPartition.setNumBuckets(index.getTablets().size());
}
tPartition.setIsMutable(olapTable.getPartitionInfo().getIsMutable(partition.getId()));
+ if (partition.getDistributionInfo().getType() == DistributionInfo.DistributionInfoType.RANDOM) {
+ try {
+ int tabletIndex = Env.getCurrentEnv().getTabletLoadIndexRecorderMgr()
+ .getCurrentTabletLoadIndex(dbId, olapTable.getId(), partition);
+ tPartition.setLoadTabletIdx(tabletIndex);
Review Comment:
This runtime path allocates `load_tablet_idx` with `getCurrentTabletLoadIndex()`, which advances the shared per-partition counter every time a BE asks FE to create the same auto partition. In an adaptive multi-BE load, BE1 can create/cache partition P and replan from base bucket N, while BE2 later hits the cache for the same P and replan from base bucket N+1. Since `assignAdaptiveBucketToPartition()` computes the whole sink assignment from that base index and returns only the current BE's slice, the senders no longer share the same load-wide bucket plan for this runtime partition. The mirrored `replacePartition()` code has the same issue. Please make the runtime partition base index stable for the load/txn/partition, for example by caching the chosen index with the auto-partition cache/result, instead of consuming a new index per receiver RPC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3377508056
##########
be/src/exec/sink/writer/vtablet_writer.cpp:
##########
@@ -657,6 +666,70 @@ void VNodeChannel::_open_internal(bool is_incremental) {
request->set_txn_expiration(_parent->_txn_expiration);
request->set_write_file_cache(_parent->_write_file_cache);
+ if (_parent->_tablet_finder->is_adaptive_random_bucket()) {
+ std::unordered_map<int64_t, std::vector<int64_t>> partition_to_ordered_tablets;
+ std::unordered_map<int64_t, std::unordered_set<int64_t>> partition_to_local_tablets;
+ for (const auto& tablet : _all_tablets) {
+ partition_to_ordered_tablets[tablet.partition_id].push_back(tablet.tablet_id);
+ partition_to_local_tablets[tablet.partition_id].insert(tablet.tablet_id);
+ }
+ std::unordered_map<int64_t, const VOlapTablePartition*> id_to_partition;
Review Comment:
This adds a `random_bucket_partition` before proving that this `VNodeChannel` owns any of the FE-selected `local_bucket_seqs`. `IndexChannel` still opens channels for BEs that have replicas for the partition, so a non-selected BE can have entries in `_all_tablets` but none of the selected bucket tablets. The loop below then skips every selected tablet and sends an empty `ordered_tablet_ids`, which makes the receiver abort at `BaseTabletsChannel::_init_receiver_side_random_bucket_state()` with `CHECK(!params.ordered_tablet_ids.empty())`. This is distinct from the existing selected-order issue: even after mapping global bucket seqs correctly, channels with no selected local buckets need to be skipped instead of opening receiver-side random state with an empty partition.
##########
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:
##########
@@ -5695,4 +5735,94 @@ private TStatus checkMaster() {
}
return status;
}
+
+ private static final class AdaptiveBucketSinkContext {
+ private final List<Long> sinkBackendIds;
+ private final int planFragmentNum;
+
+ private AdaptiveBucketSinkContext(List<Long> sinkBackendIds, int planFragmentNum) {
+ this.sinkBackendIds = sinkBackendIds;
+ this.planFragmentNum = planFragmentNum;
Review Comment:
`createPartition()`/`replacePartition()` already look up a classic `Coordinator` to decide `needUseCache`, but this helper falls back to `[currentBeId], planFragmentNum=1` for every non-Nereids coordinator. In a classic multi-instance load that creates or replaces a random partition at runtime, each BE will independently replan that new partition as if it were the only sink backend, while the initial sink assignment was computed across all sink BEs in `Coordinator.assignAdaptiveRandomBucketForFragment()`. That can give different `bucket_be_id` / `local_bucket_seqs` for runtime partitions than for the rest of the load and duplicate bucket choices across senders. Please derive the sink backend set and fragment instance count for classic coordinators as well, or skip adaptive replanning for this path until it can use the same load-wide context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4657288542
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4646821948
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4710625683
<details>
<summary>TPC-H: <b>Total hot run time: 28904 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d8fb460dd1f77e6dd47d3649a8f89b0f7cb9922b, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17707 4016 4034 4016
q2 q3 10974 1452 823 823
q4 4804 476 338 338
q5 8515 876 585 585
q6 338 171 135 135
q7 912 862 613 613
q8 10874 1721 1622 1622
q9 7207 4566 4524 4524
q10 6795 1811 1551 1551
q11 430 267 251 251
q12 642 430 294 294
q13 18103 3492 2848 2848
q14 268 261 244 244
q15 q16 819 762 697 697
q17 981 1010 893 893
q18 6951 5830 5598 5598
q19 1181 1304 1018 1018
q20 520 409 255 255
q21 5504 2548 2291 2291
q22 432 358 308 308
Total cold run time: 103957 ms
Total hot run time: 28904 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4354 4303 4289 4289
q2 q3 4510 4964 4341 4341
q4 2121 2204 1371 1371
q5 4556 4337 4305 4305
q6 226 179 130 130
q7 2192 1885 1704 1704
q8 2633 2159 2091 2091
q9 8011 8116 7984 7984
q10 4866 4824 4517 4517
q11 585 406 386 386
q12 763 784 571 571
q13 3216 3762 2945 2945
q14 286 303 280 280
q15 q16 691 756 631 631
q17 1360 1327 1338 1327
q18 8015 7286 6893 6893
q19 1118 1162 1116 1116
q20 2224 2223 1946 1946
q21 5274 4527 4402 4402
q22 509 490 418 418
Total cold run time: 57510 ms
Total hot run time: 51647 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4713963266
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4708971686
<details>
<summary>TPC-H: <b>Total hot run time: 29103 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 313fe8135eee7e6938e71bddc4ce73eb6e1050de, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17653 4046 3984 3984
q2 q3 10917 1422 840 840
q4 4750 468 343 343
q5 8304 907 603 603
q6 311 173 138 138
q7 914 840 642 642
q8 10923 1781 1593 1593
q9 7056 4537 4508 4508
q10 6799 1814 1534 1534
q11 447 273 254 254
q12 648 435 297 297
q13 18169 3399 2847 2847
q14 270 257 238 238
q15 q16 822 784 713 713
q17 1012 1007 963 963
q18 7010 5833 5508 5508
q19 1153 1218 1147 1147
q20 521 398 260 260
q21 5554 2618 2392 2392
q22 432 360 299 299
Total cold run time: 103665 ms
Total hot run time: 29103 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4309 4228 4225 4225
q2 q3 4483 4919 4349 4349
q4 2084 2141 1379 1379
q5 4405 4286 4290 4286
q6 227 173 128 128
q7 2087 2031 1607 1607
q8 2545 2215 2160 2160
q9 8198 7976 7981 7976
q10 4834 4749 4314 4314
q11 572 416 378 378
q12 752 752 538 538
q13 3326 3645 2939 2939
q14 290 326 259 259
q15 q16 710 762 647 647
q17 1357 1331 1459 1331
q18 7802 7395 7211 7211
q19 1205 1073 1084 1073
q20 2231 2211 1939 1939
q21 5175 4557 4414 4414
q22 516 442 396 396
Total cold run time: 57108 ms
Total hot run time: 51549 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3414747878
##########
be/src/load/channel/tablets_channel.cpp:
##########
@@ -675,6 +719,160 @@ Status BaseTabletsChannel::_write_block_data(
return Status::OK();
}
+std::shared_ptr<std::mutex> BaseTabletsChannel::_get_partition_route_lock(int64_t partition_id) {
+ std::lock_guard<std::mutex> l(_partition_route_locks_lock);
+ auto& lock = _partition_route_locks[partition_id];
+ if (lock == nullptr) {
+ lock = std::make_shared<std::mutex>();
+ }
+ return lock;
+}
+
+Status BaseTabletsChannel::_write_block_data_for_receiver_side_random_bucket(
+ const PTabletWriterAddBlockRequest& request, int64_t cur_seq,
+ std::unordered_map<int64_t, DorisVector<uint32_t>>& partition_to_rowidxs,
+ PTabletWriterAddBlockResult* response) {
+ Block send_data;
+ [[maybe_unused]] size_t uncompressed_size = 0;
+ [[maybe_unused]] int64_t uncompressed_time = 0;
+ RETURN_IF_ERROR(send_data.deserialize(request.block(), &uncompressed_size, &uncompressed_time));
+ if (send_data.rows() != request.partition_ids_size()) {
+ return Status::InternalError(
+ "invalid receiver-side random bucket add block request row count, load_id={}, "
+ "index_id={}, packet_seq={}, block_rows={}, partition_ids_size={}",
+ print_id(_load_id), _index_id, request.packet_seq(), send_data.rows(),
+ request.partition_ids_size());
+ }
+
+ {
+ std::lock_guard<std::mutex> l(_lock);
+ for (const auto& [partition_id, _] : partition_to_rowidxs) {
+ _partition_ids.emplace(partition_id);
+ }
+ }
+
+ g_tablets_channel_send_data_allocated_size << send_data.allocated_bytes();
+ Defer defer {
+ [&]() { g_tablets_channel_send_data_allocated_size << -send_data.allocated_bytes(); }};
+
+ auto* tablet_errors = response->mutable_tablet_errors();
Review Comment:
This partition route lock is held while `tablet_writer->write()` runs, and that write can block on table-level memtable backpressure or submit a flush. The adaptive state is keyed by `(sender_id, partition_id)`, so different senders that FE assigns to different buckets on the same receiver do not need a partition-wide critical section. A concrete case is two sink BEs whose selected buckets are both owned by the same third BE: they should be able to write bucket 0 and bucket 1 concurrently, but this lock serializes all rows for the partition. Please narrow the lock to the state lookup/rotation, or key it by sender as well, and keep the potentially blocking write outside the partition-wide lock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4899788844
<details>
<summary>TPC-DS: <b>Total hot run time: 180151 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 30a431f2b08623dcddbc518296a9a8b5c5a06387, data reload: false
query5 4328 633 488 488
query6 457 236 209 209
query7 4840 601 334 334
query8 335 183 176 176
query9 8779 4106 4102 4102
query10 474 358 290 290
query11 5970 2409 2181 2181
query12 162 112 103 103
query13 1265 629 434 434
query14 6280 5301 5000 5000
query14_1 4313 4295 4318 4295
query15 230 204 180 180
query16 1022 478 459 459
query17 1147 754 615 615
query18 2425 463 338 338
query19 215 191 150 150
query20 115 104 104 104
query21 233 156 135 135
query22 13661 13606 13434 13434
query23 17369 16553 16095 16095
query23_1 16238 16228 16349 16228
query24 7617 1773 1291 1291
query24_1 1336 1298 1292 1292
query25 564 445 386 386
query26 1320 372 211 211
query27 2647 609 391 391
query28 4529 2027 1983 1983
query29 1065 621 473 473
query30 334 258 232 232
query31 1114 1096 988 988
query32 108 65 60 60
query33 520 305 275 275
query34 1203 1159 652 652
query35 784 775 668 668
query36 1419 1383 1204 1204
query37 155 103 100 100
query38 1898 1719 1656 1656
query39 917 927 888 888
query39_1 867 903 910 903
query40 252 163 139 139
query41 66 67 63 63
query42 96 93 89 89
query43 335 333 278 278
query44 1431 800 796 796
query45 210 185 185 185
query46 1071 1269 767 767
query47 2355 2339 2222 2222
query48 414 419 309 309
query49 600 428 334 334
query50 1016 419 343 343
query51 10922 10949 10580 10580
query52 87 89 84 84
query53 274 291 208 208
query54 295 263 233 233
query55 78 72 68 68
query56 314 321 308 308
query57 1440 1419 1304 1304
query58 293 267 252 252
query59 1621 1676 1481 1481
query60 317 288 271 271
query61 182 171 178 171
query62 707 651 593 593
query63 237 205 211 205
query64 2599 835 662 662
query65 5053 4843 4743 4743
query66 1858 538 411 411
query67 29615 29522 29341 29341
query68 3241 1517 973 973
query69 409 303 268 268
query70 1107 964 978 964
query71 342 330 295 295
query72 2934 2649 2309 2309
query73 870 819 457 457
query74 5141 4971 4759 4759
query75 2628 2604 2225 2225
query76 2351 1213 794 794
query77 348 385 290 290
query78 12258 12378 11881 11881
query79 1762 1213 793 793
query80 1279 547 481 481
query81 553 338 290 290
query82 780 160 121 121
query83 366 318 287 287
query84 287 164 129 129
query85 966 610 502 502
query86 437 298 288 288
query87 1819 1823 1771 1771
query88 3739 2815 2812 2812
query89 472 405 355 355
query90 1890 202 204 202
query91 199 193 158 158
query92 62 63 59 59
query93 1751 1475 955 955
query94 732 344 307 307
query95 785 564 473 473
query96 1125 810 353 353
query97 2679 2675 2560 2560
query98 212 207 203 203
query99 1198 1164 1025 1025
Total cold run time: 267076 ms
Total hot run time: 180151 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4893909875
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4894413262
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3479824213
##########
be/src/exec/sink/writer/vtablet_writer.cpp:
##########
@@ -676,6 +716,77 @@ void VNodeChannel::_open_internal(bool is_incremental) {
request->set_txn_expiration(_parent->_txn_expiration);
request->set_write_file_cache(_parent->_write_file_cache);
+ if (_parent->_tablet_finder->is_adaptive_random_bucket()) {
Review Comment:
Extracted the adaptive open-request setup into `VNodeChannel::_set_adaptive_random_bucket_open_request()` so `_open_internal()` keeps only the mode check and call site.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4808254565
<details>
<summary>TPC-H: <b>Total hot run time: 29756 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4f1c936ff2ae8acc7275baf70a22f673b037c6ca, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17577 4041 4004 4004
q2 2005 323 202 202
q3 10290 1410 826 826
q4 4680 488 349 349
q5 7514 853 575 575
q6 180 173 139 139
q7 771 856 634 634
q8 9337 1604 1728 1604
q9 5697 4569 4602 4569
q10 6769 1792 1573 1573
q11 442 275 246 246
q12 636 449 294 294
q13 18079 3457 2802 2802
q14 274 259 239 239
q15 q16 788 777 703 703
q17 1033 1028 888 888
q18 7065 5842 5602 5602
q19 1156 1230 1116 1116
q20 497 405 269 269
q21 6107 2808 2803 2803
q22 457 385 319 319
Total cold run time: 101354 ms
Total hot run time: 29756 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5126 4741 4648 4648
q2 326 381 230 230
q3 4924 5266 4688 4688
q4 2118 2163 1409 1409
q5 4934 4724 4678 4678
q6 229 175 125 125
q7 2121 1715 1540 1540
q8 2449 2152 2069 2069
q9 7975 7604 7604 7604
q10 4773 4702 4218 4218
q11 536 385 352 352
q12 740 756 531 531
q13 2956 3349 2777 2777
q14 269 288 245 245
q15 q16 686 713 601 601
q17 1291 1255 1256 1255
q18 7388 6932 6910 6910
q19 1122 1116 1095 1095
q20 2244 2222 1994 1994
q21 5285 4536 4416 4416
q22 519 472 405 405
Total cold run time: 58011 ms
Total hot run time: 51790 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3479729179
##########
be/src/exec/sink/writer/vtablet_writer.cpp:
##########
@@ -777,13 +889,17 @@ Status VNodeChannel::add_block(Block* block, const Payload* payload) {
}
SCOPED_RAW_TIMER(&_stat.append_node_channel_ns);
- st = block->append_to_block_by_selector(_cur_mutable_block.get(), *(payload->first));
+ st = block->append_to_block_by_selector(_cur_mutable_block.get(), *payload->row_ids);
if (!st.ok()) {
_cancel_with_msg(fmt::format("{}, err: {}", channel_info(), st.to_string()));
return st;
}
Review Comment:
Current head already keeps rolling-upgrade compatibility for this path. In adaptive add-block requests, the sender still appends a compatibility tablet id from `_adaptive_partition_compat_tablets`, so an old receiver will not see an empty `tablet_ids` list. The EOS split path also clears those compatibility tablet ids before sending the empty EOS request. I think no additional receiver-version gate is needed for this thread.
##########
be/src/load/channel/tablets_channel.h:
##########
@@ -185,6 +197,10 @@ class BaseTabletsChannel {
std::unordered_set<int64_t> _reducing_tablets;
std::unordered_set<int64_t> _partition_ids;
+ std::shared_ptr<AdaptiveRandomBucketState> _adaptive_random_bucket_state;
+ std::mutex _partition_route_locks_lock;
+ std::unordered_map<int32_t, std::unordered_map<int64_t, std::shared_ptr<std::mutex>>>
Review Comment:
Added comments for `_sender_partition_route_locks`: the map lock only protects the lock map, and each per-sender/partition lock serializes current-tablet selection, write, and rotation for one adaptive random-bucket stream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4787069423
# BE Regression && UT Coverage Report
Increment line coverage `64.60% (458/709)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_591feb974a7f4b0b2831b4166b130e3dc8628771_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_591feb974a7f4b0b2831b4166b130e3dc8628771_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 72.31% (27743/38367) |
| Line Coverage | 55.68% (297759/534753) |
| Region Coverage | 52.34% (248182/474205) |
| Branch Coverage | 53.49% (107491/200969) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4833560206
<details>
<summary>ClickBench: <b>Total hot run time: 25.23 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6cf651de54175959a128b07b857fa64e939af881, data reload: false
query1 0.01 0.01 0.01
query2 0.09 0.04 0.05
query3 0.25 0.13 0.13
query4 1.60 0.13 0.14
query5 0.23 0.23 0.23
query6 1.28 1.07 1.06
query7 0.04 0.01 0.01
query8 0.06 0.04 0.03
query9 0.37 0.32 0.31
query10 0.56 0.56 0.61
query11 0.19 0.14 0.15
query12 0.17 0.15 0.16
query13 0.46 0.47 0.49
query14 1.02 0.99 1.01
query15 0.62 0.59 0.60
query16 0.31 0.33 0.33
query17 1.12 1.10 1.18
query18 0.24 0.22 0.21
query19 1.98 1.97 1.97
query20 0.02 0.01 0.01
query21 15.44 0.21 0.14
query22 4.91 0.05 0.06
query23 16.14 0.31 0.12
query24 2.94 0.42 0.32
query25 0.10 0.06 0.05
query26 0.73 0.21 0.14
query27 0.05 0.04 0.03
query28 3.52 0.96 0.55
query29 12.47 4.33 3.47
query30 0.28 0.14 0.15
query31 2.77 0.59 0.30
query32 3.25 0.60 0.49
query33 3.16 3.15 3.27
query34 15.48 4.19 3.52
query35 3.60 3.48 3.54
query36 0.58 0.44 0.44
query37 0.09 0.06 0.06
query38 0.05 0.04 0.03
query39 0.04 0.03 0.03
query40 0.18 0.17 0.16
query41 0.09 0.04 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.03
Total cold run time: 96.57 s
Total hot run time: 25.23 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4832490257
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4832489305
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4862156755
# FE UT Coverage Report
Increment line coverage ` 39.29% (176/448)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_30a431f2b08623dcddbc518296a9a8b5c5a06387/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_30a431f2b08623dcddbc518296a9a8b5c5a06387/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4862989758
<details>
<summary>TPC-DS: <b>Total hot run time: 174481 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 30a431f2b08623dcddbc518296a9a8b5c5a06387, data reload: false
query5 4326 649 491 491
query6 480 251 212 212
query7 4930 632 348 348
query8 341 194 172 172
query9 8803 4076 4045 4045
query10 442 353 300 300
query11 5916 2362 2178 2178
query12 162 106 110 106
query13 1257 588 438 438
query14 6285 5337 4976 4976
query14_1 4287 4293 4302 4293
query15 220 206 180 180
query16 1061 472 455 455
query17 1158 718 584 584
query18 2564 474 363 363
query19 208 217 145 145
query20 111 106 104 104
query21 233 153 132 132
query22 13684 13625 13300 13300
query23 17319 16617 16270 16270
query23_1 16314 16177 16144 16144
query24 7563 1754 1301 1301
query24_1 1340 1297 1284 1284
query25 551 445 348 348
query26 1335 355 207 207
query27 2610 561 359 359
query28 4470 2017 2036 2017
query29 1070 593 480 480
query30 330 257 228 228
query31 1113 1086 998 998
query32 106 60 59 59
query33 525 315 243 243
query34 1171 1139 655 655
query35 758 771 687 687
query36 1450 1365 1259 1259
query37 160 115 93 93
query38 1883 1707 1638 1638
query39 932 900 899 899
query39_1 904 868 870 868
query40 241 160 140 140
query41 64 65 67 65
query42 103 94 91 91
query43 328 334 276 276
query44 1458 778 778 778
query45 201 186 183 183
query46 1084 1224 728 728
query47 2347 2324 2276 2276
query48 404 396 280 280
query49 578 443 312 312
query50 1014 415 334 334
query51 4489 4387 4389 4387
query52 88 85 72 72
query53 260 275 208 208
query54 283 278 237 237
query55 77 80 68 68
query56 323 313 292 292
query57 1485 1443 1327 1327
query58 306 275 294 275
query59 1615 1638 1456 1456
query60 338 283 265 265
query61 178 169 169 169
query62 702 647 596 596
query63 241 218 212 212
query64 2550 812 662 662
query65 4918 4765 4757 4757
query66 1821 539 402 402
query67 29641 29555 29389 29389
query68 3038 1612 964 964
query69 431 306 280 280
query70 1113 922 931 922
query71 356 329 343 329
query72 2936 2622 2368 2368
query73 834 864 452 452
query74 5100 4970 4763 4763
query75 2636 2605 2225 2225
query76 2314 1181 817 817
query77 359 382 287 287
query78 12393 12533 12374 12374
query79 1455 1179 769 769
query80 1290 556 465 465
query81 524 338 289 289
query82 640 158 122 122
query83 391 319 301 301
query84 281 166 136 136
query85 991 631 546 546
query86 429 293 296 293
query87 1815 1821 1767 1767
query88 3723 2814 2813 2813
query89 487 456 395 395
query90 1985 188 198 188
query91 199 193 165 165
query92 64 58 56 56
query93 1707 1631 946 946
query94 737 314 305 305
query95 784 576 467 467
query96 1109 814 359 359
query97 2839 2682 2564 2564
query98 213 203 206 203
query99 1178 1151 1039 1039
Total cold run time: 260352 ms
Total hot run time: 174481 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](load) adaptive random bucket load: per-BE bucket assignment with threshold-based rotation [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4286300603
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](load) adaptive random bucket load: per-BE bucket assignment with threshold-based rotation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4290690618
# BE UT Coverage Report
Increment line coverage `9.68% (9/93)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/0e8fec709e17b52e8b214b7e4860e8977228e98a_0e8fec709e17b52e8b214b7e4860e8977228e98a/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/0e8fec709e17b52e8b214b7e4860e8977228e98a_0e8fec709e17b52e8b214b7e4860e8977228e98a/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.29% (20354/38193) |
| Line Coverage | 36.84% (191778/520598) |
| Region Coverage | 33.19% (149344/450019) |
| Branch Coverage | 34.26% (65250/190440) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4714488903
<details>
<summary>TPC-H: <b>Total hot run time: 29025 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 635e8413bd63597f3c7976a1ed5e43d6c7f6cf8c, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17737 4083 4091 4083
q2 q3 10788 1426 820 820
q4 4679 475 331 331
q5 7582 872 563 563
q6 184 175 140 140
q7 766 830 619 619
q8 9338 1559 1580 1559
q9 6036 4436 4524 4436
q10 6770 1799 1513 1513
q11 432 269 240 240
q12 632 419 290 290
q13 18173 3422 2816 2816
q14 264 256 252 252
q15 q16 822 764 706 706
q17 969 974 970 970
q18 6691 5780 5648 5648
q19 1198 1333 1117 1117
q20 511 414 263 263
q21 5730 2545 2362 2362
q22 442 362 297 297
Total cold run time: 99744 ms
Total hot run time: 29025 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4466 4325 4363 4325
q2 q3 4554 4921 4308 4308
q4 2080 2209 1368 1368
q5 4512 4367 4349 4349
q6 238 176 129 129
q7 1729 1628 1700 1628
q8 2810 2346 2284 2284
q9 8228 8308 7946 7946
q10 4815 4875 4267 4267
q11 566 407 374 374
q12 754 758 542 542
q13 3466 3635 2930 2930
q14 295 307 282 282
q15 q16 705 722 642 642
q17 1376 1346 1345 1345
q18 8415 7519 7286 7286
q19 1198 1085 1099 1085
q20 2228 2228 1954 1954
q21 5288 4639 4406 4406
q22 539 447 389 389
Total cold run time: 58262 ms
Total hot run time: 51839 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3417688818
##########
be/src/cloud/cloud_tablets_channel.cpp:
##########
@@ -64,6 +64,25 @@ Status CloudTabletsChannel::add_batch(const PTabletWriterAddBlockRequest& reques
return Status::OK();
}
+ if (request.is_receiver_side_random_bucket()) {
+ std::unordered_map<int64_t, DorisVector<uint32_t>> partition_to_rowidxs;
+ RETURN_IF_ERROR(_build_partition_to_rowidxs_for_receiver_side_random_bucket(
+ request, &partition_to_rowidxs));
+ if (!partition_to_rowidxs.empty()) {
+ std::unordered_set<int64_t> partition_ids;
+ partition_ids.reserve(partition_to_rowidxs.size());
+ for (const auto& [partition_id, _] : partition_to_rowidxs) {
+ partition_ids.insert(partition_id);
+ }
+ {
+ std::lock_guard<std::mutex> l(_tablet_writers_lock);
+ RETURN_IF_ERROR(_init_writers_by_partition_ids(partition_ids));
Review Comment:
This adaptive cloud path initializes every writer in each touched partition before the receiver chooses the current adaptive tablet. In the normal cloud path, when `skip_writing_empty_rowset_metadata` is true (the default), only writers that actually receive rows are `batch_init`'d; untouched writers stay uninitialized so close goes through `_commit_empty_rowset()` and skips writing empty rowset metadata. Here a batch with one row for partition P calls `_init_writers_by_partition_ids(P)`, so all tablet writers for P become `is_init=true`. At close, writers that never received rows take the normal initialized `commit_rowset()` path instead of the skip-empty path, creating empty rowset metadata and rowset builders for every bucket on the receiver. That negates the cloud lazy/skip-empty optimization and can reintroduce the per-partition memory/metadata blow-up adaptive routing is trying to avoid. Please initialize only the current adaptive tablet writer(s) selected for this request,
leaving the other partition writers uninitialized until they actually receive rows or close uses the existing empty-rowset handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4714309327
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4714309837
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4715075432
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4714111243
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4709517584
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4716135204
<details>
<summary>ClickBench: <b>Total hot run time: 26.2 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 52e2238c649552e3fd0cad1201b33579cb90c5ce, data reload: false
query1 0.01 0.00 0.01
query2 0.15 0.09 0.08
query3 0.40 0.24 0.24
query4 1.61 0.25 0.25
query5 0.34 0.32 0.32
query6 1.16 0.68 0.68
query7 0.04 0.01 0.00
query8 0.10 0.07 0.07
query9 0.52 0.38 0.39
query10 0.60 0.63 0.62
query11 0.32 0.19 0.18
query12 0.33 0.19 0.19
query13 0.55 0.55 0.53
query14 0.94 0.93 0.92
query15 0.69 0.61 0.61
query16 0.39 0.39 0.39
query17 1.01 1.04 1.06
query18 0.31 0.30 0.31
query19 2.00 1.86 1.82
query20 0.02 0.01 0.01
query21 15.40 0.37 0.31
query22 4.92 0.14 0.13
query23 15.81 0.48 0.31
query24 2.59 0.61 0.44
query25 0.15 0.12 0.10
query26 0.75 0.26 0.22
query27 0.10 0.10 0.10
query28 3.47 0.93 0.56
query29 12.46 4.41 3.53
query30 0.40 0.24 0.25
query31 2.79 0.61 0.35
query32 3.24 0.62 0.51
query33 3.08 2.97 3.03
query34 15.87 4.07 3.41
query35 3.32 3.30 3.30
query36 0.67 0.52 0.50
query37 0.12 0.09 0.09
query38 0.08 0.06 0.06
query39 0.07 0.06 0.06
query40 0.21 0.19 0.17
query41 0.14 0.09 0.09
query42 0.09 0.06 0.06
query43 0.07 0.07 0.06
Total cold run time: 97.29 s
Total hot run time: 26.2 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3370738089
##########
be/src/exec/sink/writer/vtablet_writer.cpp:
##########
@@ -657,6 +670,20 @@ void VNodeChannel::_open_internal(bool is_incremental) {
request->set_txn_expiration(_parent->_txn_expiration);
request->set_write_file_cache(_parent->_write_file_cache);
+ if (_parent->_tablet_finder->is_adaptive_random_bucket()) {
+ std::unordered_map<int64_t, std::vector<int64_t>> partition_to_ordered_tablets;
+ for (const auto& tablet : _all_tablets) {
+ partition_to_ordered_tablets[tablet.partition_id].push_back(tablet.tablet_id);
+ }
Review Comment:
This receiver-side open path drops the FE-selected bucket sequence. `computeAdaptiveRandomBucketAssignments()` sets `load_tablet_idx` and `local_bucket_seqs`, and `IndexChannel::init()` uses only `bucket_be_id` to choose the target channel. But here the open request sends every tablet accumulated on that BE channel, and `BaseTabletsChannel::_init_receiver_side_random_bucket_state()` initializes the rotation from position `0` over the whole `ordered_tablet_ids` list. In a multi-replica partition where the chosen BE has replicas for buckets `[0,1,2,3,4,5]` but FE selected `local_bucket_seqs=[2,5]` with start bucket `2`, the receiver will start at bucket `0` and rotate through all six buckets after flushes. That routes rows to buckets that were not assigned to this sink and defeats the adaptive bucket selection. Please build `ordered_tablet_ids` from the partition's selected `local_bucket_seqs` in that order, and preserve the selected start bucket instead of implicitly starting at ze
ro.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3378876661
##########
be/src/exec/sink/writer/vtablet_writer_v2.cpp:
##########
@@ -158,9 +158,10 @@ Status VTabletWriterV2::_init(RuntimeState* state, RuntimeProfile* profile) {
_location = _pool->add(new OlapTableLocationParam(table_sink.location));
_nodes_info = _pool->add(new DorisNodesInfo(table_sink.nodes_info));
- // if distributed column list is empty, we can ensure that tablet is with random distribution info
- // and if load_to_single_tablet is set and set to true, we should find only one tablet in one partition
- // for the whole olap table sink
+ // If distributed column list is empty, the table uses random distribution.
+ // Mode priority (highest to lowest):
+ // 1. FIND_TABLET_EVERY_SINK: load_to_single_tablet=true (legacy single-tablet mode).
+ // 2. FIND_TABLET_EVERY_BATCH: default round-robin per batch.
Review Comment:
This path should not receive adaptive random bucket assignments in the current branch. FE only sets enable_adaptive_random_bucket in cloud mode, while BE only selects VTabletWriterV2 when !config::is_cloud_mode(). Therefore V2 does not run with receiver-side adaptive random bucket rewritten load_tablet_idx/local_bucket_seqs in the normal execution path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4667904468
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4660895198
<details>
<summary>TPC-H: <b>Total hot run time: 29829 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f552fc1cfb68972e5ff591a0b8668934955c7734, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17598 3993 4056 3993
q2 q3 10887 1487 803 803
q4 4765 487 345 345
q5 8356 875 594 594
q6 332 182 144 144
q7 922 870 654 654
q8 10937 1755 1688 1688
q9 7134 4629 4583 4583
q10 6838 1885 1530 1530
q11 449 282 260 260
q12 636 430 293 293
q13 18173 3581 2761 2761
q14 268 259 239 239
q15 q16 819 789 707 707
q17 979 993 991 991
q18 6957 5664 5565 5565
q19 1136 1350 1223 1223
q20 590 514 299 299
q21 5974 2940 2844 2844
q22 445 377 313 313
Total cold run time: 104195 ms
Total hot run time: 29829 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5049 4834 4962 4834
q2 q3 4859 5338 4720 4720
q4 2100 2216 1412 1412
q5 4909 4668 4807 4668
q6 259 200 136 136
q7 1883 1779 1560 1560
q8 2440 2261 2026 2026
q9 7525 7494 7503 7494
q10 4744 4660 4196 4196
q11 551 390 359 359
q12 742 734 526 526
q13 3096 3360 2831 2831
q14 271 277 254 254
q15 q16 685 699 620 620
q17 1320 1278 1265 1265
q18 7596 6877 6978 6877
q19 1144 1110 1158 1110
q20 2239 2210 1947 1947
q21 5328 4612 4517 4517
q22 519 476 397 397
Total cold run time: 57259 ms
Total hot run time: 51749 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4834773052
# BE Regression && UT Coverage Report
Increment line coverage `64.20% (443/690)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_6cf651de54175959a128b07b857fa64e939af881_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_6cf651de54175959a128b07b857fa64e939af881_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 74.23% (28529/38431) |
| Line Coverage | 58.12% (310949/535022) |
| Region Coverage | 54.79% (259904/474321) |
| Branch Coverage | 56.24% (113035/200991) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4834195621
# BE UT Coverage Report
Increment line coverage `14.47% (100/691)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/eee8cc536909690d1a4bebfa38a1ad0179a59224_eee8cc536909690d1a4bebfa38a1ad0179a59224/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/eee8cc536909690d1a4bebfa38a1ad0179a59224_eee8cc536909690d1a4bebfa38a1ad0179a59224/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 54.92% (21622/39371) |
| Line Coverage | 38.38% (206585/538211) |
| Region Coverage | 34.45% (162665/472232) |
| Branch Coverage | 35.47% (71210/200752) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4833504123
<details>
<summary>TPC-DS: <b>Total hot run time: 172281 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eee8cc536909690d1a4bebfa38a1ad0179a59224, data reload: false
query5 4344 622 477 477
query6 436 192 173 173
query7 4944 516 302 302
query8 333 182 174 174
query9 8747 4124 4070 4070
query10 478 320 257 257
query11 5958 2345 2148 2148
query12 155 103 101 101
query13 1261 594 448 448
query14 6348 5339 4994 4994
query14_1 4305 4335 4342 4335
query15 215 205 183 183
query16 1036 492 444 444
query17 1148 725 586 586
query18 2744 494 356 356
query19 212 197 145 145
query20 113 107 120 107
query21 217 146 114 114
query22 13681 13606 13443 13443
query23 17382 16529 16163 16163
query23_1 16366 16286 16285 16285
query24 7525 1786 1297 1297
query24_1 1339 1334 1319 1319
query25 590 469 406 406
query26 1327 302 174 174
query27 2595 586 350 350
query28 4467 2048 2042 2042
query29 1094 635 507 507
query30 300 239 195 195
query31 1140 1094 945 945
query32 110 66 61 61
query33 562 330 267 267
query34 1195 1149 669 669
query35 782 793 690 690
query36 1429 1396 1219 1219
query37 182 117 89 89
query38 1895 1711 1679 1679
query39 937 912 911 911
query39_1 863 857 866 857
query40 226 122 100 100
query41 65 62 63 62
query42 88 87 90 87
query43 320 322 274 274
query44 1425 794 775 775
query45 197 193 176 176
query46 1109 1220 736 736
query47 2335 2344 2198 2198
query48 394 408 308 308
query49 595 419 304 304
query50 990 360 264 264
query51 4369 4395 4308 4308
query52 81 82 69 69
query53 238 265 194 194
query54 258 215 194 194
query55 72 69 65 65
query56 243 218 214 214
query57 1437 1421 1348 1348
query58 238 210 203 203
query59 1566 1687 1450 1450
query60 280 240 219 219
query61 155 146 149 146
query62 688 650 593 593
query63 224 193 197 193
query64 2515 766 601 601
query65 4877 4786 4764 4764
query66 1763 461 366 366
query67 29206 29615 29593 29593
query68 3166 1559 856 856
query69 419 301 269 269
query70 1065 969 951 951
query71 292 235 215 215
query72 3003 2638 2329 2329
query73 865 820 447 447
query74 5133 4957 4760 4760
query75 2570 2544 2191 2191
query76 2314 1178 807 807
query77 358 375 285 285
query78 12552 12473 11917 11917
query79 1399 1149 764 764
query80 1300 475 379 379
query81 519 281 243 243
query82 578 154 123 123
query83 344 267 245 245
query84 262 142 113 113
query85 905 510 417 417
query86 430 307 292 292
query87 1850 1827 1766 1766
query88 3757 2816 2778 2778
query89 439 391 330 330
query90 1901 179 186 179
query91 168 160 136 136
query92 61 59 56 56
query93 1525 1498 850 850
query94 717 320 309 309
query95 677 468 347 347
query96 1102 849 351 351
query97 2729 2701 2588 2588
query98 220 213 200 200
query99 1170 1151 1034 1034
Total cold run time: 258892 ms
Total hot run time: 172281 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4828919677
# FE UT Coverage Report
Increment line coverage ` 37.03% (157/424)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_1ad5cc3a0a4c7e67ea8bd62c1cf9e84c7f493552/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_1ad5cc3a0a4c7e67ea8bd62c1cf9e84c7f493552/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4829784739
# FE Regression Coverage Report
Increment line coverage ` 51.18% (217/424)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_1ad5cc3a0a4c7e67ea8bd62c1cf9e84c7f493552_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_1ad5cc3a0a4c7e67ea8bd62c1cf9e84c7f493552_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4899852706
# FE Regression Coverage Report
Increment line coverage ` 38.58% (223/578)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_30a431f2b08623dcddbc518296a9a8b5c5a06387_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_30a431f2b08623dcddbc518296a9a8b5c5a06387_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4854116125
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4763356738
<details>
<summary>TPC-DS: <b>Total hot run time: 174761 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 52e2238c649552e3fd0cad1201b33579cb90c5ce, data reload: false
query5 4323 628 478 478
query6 450 206 187 187
query7 4878 572 295 295
query8 383 228 203 203
query9 8774 4028 4014 4014
query10 460 314 261 261
query11 5843 2341 2122 2122
query12 156 101 100 100
query13 1254 623 416 416
query14 6399 5374 5056 5056
query14_1 4402 4362 4360 4360
query15 217 203 175 175
query16 1054 473 449 449
query17 1125 704 576 576
query18 2588 485 346 346
query19 202 187 147 147
query20 112 107 104 104
query21 223 138 117 117
query22 13571 13635 13354 13354
query23 17401 16451 16086 16086
query23_1 16184 16276 16279 16276
query24 7524 1781 1327 1327
query24_1 1284 1372 1322 1322
query25 570 462 402 402
query26 1302 316 182 182
query27 2674 566 355 355
query28 4457 2050 2066 2050
query29 1065 603 465 465
query30 315 227 200 200
query31 1112 1066 947 947
query32 113 58 56 56
query33 519 308 239 239
query34 1176 1154 668 668
query35 762 771 682 682
query36 1392 1374 1220 1220
query37 155 103 88 88
query38 3225 3122 2956 2956
query39 918 913 888 888
query39_1 888 887 875 875
query40 231 123 98 98
query41 67 63 62 62
query42 97 94 93 93
query43 312 319 281 281
query44 1444 770 769 769
query45 192 179 177 177
query46 1095 1215 748 748
query47 2403 2374 2248 2248
query48 395 391 284 284
query49 626 445 340 340
query50 1025 346 242 242
query51 4364 4294 4201 4201
query52 87 87 77 77
query53 242 264 185 185
query54 259 221 192 192
query55 77 74 71 71
query56 237 212 206 206
query57 1454 1402 1333 1333
query58 245 221 206 206
query59 1597 1684 1457 1457
query60 264 228 227 227
query61 150 144 156 144
query62 693 646 583 583
query63 220 189 191 189
query64 2490 760 598 598
query65 4897 4819 4773 4773
query66 1771 453 334 334
query67 29164 29700 29498 29498
query68 3151 1555 891 891
query69 400 290 255 255
query70 1075 996 944 944
query71 294 233 208 208
query72 2868 2600 2316 2316
query73 864 760 447 447
query74 5145 4920 4729 4729
query75 2634 2590 2230 2230
query76 2349 1196 786 786
query77 345 379 281 281
query78 12613 12570 11927 11927
query79 1394 1189 773 773
query80 1283 512 405 405
query81 525 278 240 240
query82 603 159 125 125
query83 335 276 243 243
query84 268 144 112 112
query85 898 517 405 405
query86 432 309 285 285
query87 3373 3368 3193 3193
query88 3701 2772 2787 2772
query89 424 386 337 337
query90 2004 184 179 179
query91 171 161 135 135
query92 64 62 55 55
query93 1645 1570 849 849
query94 723 345 266 266
query95 678 372 344 344
query96 1046 804 343 343
query97 2730 2680 2556 2556
query98 215 201 196 196
query99 1160 1136 1029 1029
Total cold run time: 261424 ms
Total hot run time: 174761 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4763368092
<details>
<summary>ClickBench: <b>Total hot run time: 25.16 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 52e2238c649552e3fd0cad1201b33579cb90c5ce, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.25 0.14 0.14
query4 1.61 0.13 0.14
query5 0.23 0.23 0.22
query6 1.28 1.07 1.08
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.37 0.32 0.30
query10 0.53 0.54 0.54
query11 0.19 0.15 0.14
query12 0.18 0.14 0.14
query13 0.48 0.49 0.48
query14 1.03 0.99 0.99
query15 0.61 0.60 0.59
query16 0.31 0.31 0.31
query17 1.08 1.13 1.13
query18 0.23 0.22 0.21
query19 2.08 2.03 1.92
query20 0.01 0.01 0.02
query21 15.44 0.19 0.13
query22 5.02 0.05 0.05
query23 16.16 0.31 0.12
query24 2.92 0.42 0.33
query25 0.11 0.05 0.04
query26 0.74 0.20 0.14
query27 0.03 0.04 0.04
query28 3.53 0.97 0.53
query29 12.46 4.30 3.46
query30 0.27 0.14 0.15
query31 2.78 0.59 0.31
query32 3.22 0.59 0.48
query33 3.22 3.25 3.20
query34 15.58 4.15 3.49
query35 3.50 3.53 3.48
query36 0.56 0.44 0.42
query37 0.08 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.03 0.03
query40 0.18 0.16 0.15
query41 0.09 0.03 0.03
query42 0.03 0.03 0.02
query43 0.04 0.04 0.04
Total cold run time: 96.71 s
Total hot run time: 25.16 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4785757454
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4785898623
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/591feb974a7f4b0b2831b4166b130e3dc8628771_591feb974a7f4b0b2831b4166b130e3dc8628771_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/591feb974a7f4b0b2831b4166b130e3dc8628771_591feb974a7f4b0b2831b4166b130e3dc8628771_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.36% (1893/2447) |
| Line Coverage | 64.38% (33954/52739) |
| Region Coverage | 64.79% (17469/26962) |
| Branch Coverage | 53.95% (9349/17328) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4894905958
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/30a431f2b08623dcddbc518296a9a8b5c5a06387_30a431f2b08623dcddbc518296a9a8b5c5a06387_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/30a431f2b08623dcddbc518296a9a8b5c5a06387_30a431f2b08623dcddbc518296a9a8b5c5a06387_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.39% (1896/2450) |
| Line Coverage | 64.44% (34063/52862) |
| Region Coverage | 64.86% (17534/27032) |
| Branch Coverage | 54.01% (9393/17390) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](load) adaptive random bucket load: per-BE bucket assignment with threshold-based rotation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4286740181
# FE UT Coverage Report
Increment line coverage ` 55.77% (29/52)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_982c51652552ce932a3ab99979f4e6ba198d49dd/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_982c51652552ce932a3ab99979f4e6ba198d49dd/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](load) adaptive random bucket load: per-BE bucket assignment with threshold-based rotation [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4290072565
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4833683105
# FE UT Coverage Report
Increment line coverage ` 37.12% (157/423)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_6cf651de54175959a128b07b857fa64e939af881/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_6cf651de54175959a128b07b857fa64e939af881/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4833407086
<details>
<summary>TPC-H: <b>Total hot run time: 28894 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6cf651de54175959a128b07b857fa64e939af881, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17629 3978 3914 3914
q2 2003 312 184 184
q3 10322 1385 797 797
q4 4699 472 342 342
q5 7578 891 598 598
q6 218 169 141 141
q7 752 835 620 620
q8 10168 1550 1620 1550
q9 6029 4485 4442 4442
q10 6819 1809 1524 1524
q11 440 272 244 244
q12 640 420 300 300
q13 18119 3709 2764 2764
q14 273 257 245 245
q15 q16 785 765 704 704
q17 1031 993 925 925
q18 6806 5746 5439 5439
q19 1440 1262 1150 1150
q20 489 417 269 269
q21 5669 2583 2444 2444
q22 440 351 298 298
Total cold run time: 102349 ms
Total hot run time: 28894 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4326 4229 4228 4228
q2 308 337 236 236
q3 4579 4922 4423 4423
q4 2054 2157 1352 1352
q5 4430 4295 4312 4295
q6 232 178 127 127
q7 1740 1616 2038 1616
q8 2551 2162 2163 2162
q9 8183 8299 8067 8067
q10 4795 4755 4309 4309
q11 581 415 383 383
q12 720 781 526 526
q13 3251 3594 2994 2994
q14 287 294 282 282
q15 q16 739 742 612 612
q17 1361 1330 1320 1320
q18 7954 7306 7304 7304
q19 1162 1125 1113 1113
q20 2224 2237 1964 1964
q21 5238 4547 4394 4394
q22 519 461 408 408
Total cold run time: 57234 ms
Total hot run time: 52115 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4656431895
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/2054fc372677684e4f1aa9088f1011f91a4ce5d9_2054fc372677684e4f1aa9088f1011f91a4ce5d9_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/2054fc372677684e4f1aa9088f1011f91a4ce5d9_2054fc372677684e4f1aa9088f1011f91a4ce5d9_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.39% (1915/2443) |
| Line Coverage | 64.87% (34185/52701) |
| Region Coverage | 65.29% (17577/26922) |
| Branch Coverage | 54.01% (9340/17294) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3378000569
##########
be/src/load/delta_writer/delta_writer.cpp:
##########
@@ -155,10 +161,18 @@ Status BaseDeltaWriter::init() {
return Status::OK();
}
-Status DeltaWriter::write(const Block* block, const DorisVector<uint32_t>& row_idxs) {
+Status DeltaWriter::write(const Block* block, const DorisVector<uint32_t>& row_idxs,
+ bool* memtable_flushed) {
+ if (memtable_flushed != nullptr) {
+ *memtable_flushed = false;
+ }
if (UNLIKELY(row_idxs.empty())) {
return Status::OK();
}
+ if (_req.enable_table_memtable_backpressure) {
+ ExecEnv::GetInstance()->memtable_memory_limiter()->handle_table_memtable_backpressure(
+ nullptr, table_id());
+ }
Review Comment:
This new table-level backpressure wait is still uninterruptible on the non-cloud `DeltaWriter` path. `enable_table_memtable_backpressure` is set for receiver-side random-bucket opens in `TabletsChannel::_open_all_writers()`/`incremental_open()`, so a shared-nothing load can enter this branch before taking `_lock`; if the load is cancelled while the table already has `table_memtable_flush_pending_count_limit` pending memtables, `cancel_with_status()` cannot make this wait return because the predicate is `nullptr`. If flushes are stalled, the add-batch RPC remains stuck here and close/cancel cannot make progress. Please pass a cancellation predicate that observes `_is_cancelled` (matching the fixed `CloudDeltaWriter` path) or otherwise make this wait interruptible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4645637998
# FE UT Coverage Report
Increment line coverage ` 5.43% (19/350)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_3219248876af4abd01175df24c505875623b5888/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_3219248876af4abd01175df24c505875623b5888/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4649111029
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4656870786
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3373011366
##########
be/src/cloud/cloud_delta_writer.cpp:
##########
@@ -64,10 +66,18 @@ Status CloudDeltaWriter::batch_init(std::vector<CloudDeltaWriter*> writers) {
return cloud::bthread_fork_join(tasks, 10);
}
-Status CloudDeltaWriter::write(const Block* block, const DorisVector<uint32_t>& row_idxs) {
+Status CloudDeltaWriter::write(const Block* block, const DorisVector<uint32_t>& row_idxs,
+ bool* memtable_flushed) {
+ if (memtable_flushed != nullptr) {
+ *memtable_flushed = false;
+ }
if (row_idxs.empty()) [[unlikely]] {
return Status::OK();
}
+ if (_req.enable_table_memtable_backpressure) {
Review Comment:
This new table-level backpressure wait cannot observe load cancellation because it is called with a null `cancel_check`. In the adaptive cloud path this runs before taking the writer mutex, so if the table already has `table_memtable_flush_pending_count_limit` pending memtables, a cancelled load can remain stuck in `handle_table_memtable_backpressure()` until enough unrelated flushes complete; if flush is stalled, the add-batch RPC never returns and close/cancel cannot make progress. Please pass a cancellation predicate wired to the writer/channel state, or otherwise make this wait interruptible, matching the existing cancellation-aware memory-limit waits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4660998908
<details>
<summary>TPC-DS: <b>Total hot run time: 168839 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f552fc1cfb68972e5ff591a0b8668934955c7734, data reload: false
query5 4309 617 477 477
query6 455 201 181 181
query7 4913 592 303 303
query8 377 216 199 199
query9 8736 4120 4058 4058
query10 450 317 276 276
query11 5957 2317 2152 2152
query12 162 112 112 112
query13 1298 644 422 422
query14 6525 5374 5059 5059
query14_1 4451 4395 4385 4385
query15 209 196 177 177
query16 1051 474 404 404
query17 1116 713 574 574
query18 2730 493 336 336
query19 200 181 140 140
query20 113 104 113 104
query21 222 138 116 116
query22 13691 13580 13431 13431
query23 17415 16613 16199 16199
query23_1 16388 16296 16304 16296
query24 7542 1807 1303 1303
query24_1 1309 1322 1319 1319
query25 569 479 424 424
query26 1321 312 165 165
query27 2676 573 368 368
query28 4466 2044 2028 2028
query29 1117 657 512 512
query30 312 240 203 203
query31 1136 1080 980 980
query32 111 67 65 65
query33 535 346 279 279
query34 1185 1201 641 641
query35 785 813 686 686
query36 1391 1425 1280 1280
query37 155 114 98 98
query38 3204 3159 3039 3039
query39 945 922 875 875
query39_1 864 915 884 884
query40 227 130 111 111
query41 75 70 68 68
query42 104 100 97 97
query43 328 336 284 284
query44
query45 207 185 185 185
query46 1110 1215 719 719
query47 2369 2411 2264 2264
query48 407 416 314 314
query49 660 486 381 381
query50 1007 372 256 256
query51 4336 4362 4194 4194
query52 92 94 81 81
query53 259 277 199 199
query54 286 239 235 235
query55 82 79 74 74
query56 279 260 235 235
query57 1462 1401 1351 1351
query58 258 244 234 234
query59 1594 1649 1436 1436
query60 297 276 245 245
query61 207 157 162 157
query62 705 651 585 585
query63 228 193 193 193
query64 2535 790 645 645
query65
query66 1756 475 373 373
query67 29738 29741 29472 29472
query68
query69 413 305 270 270
query70 1037 980 987 980
query71 311 215 208 208
query72 3028 2691 2281 2281
query73 853 780 393 393
query74 5139 4967 4810 4810
query75 2635 2587 2235 2235
query76 2343 1180 771 771
query77 351 401 296 296
query78 12625 12458 11820 11820
query79 1461 1071 764 764
query80 1180 487 406 406
query81 505 274 242 242
query82 582 158 118 118
query83 362 285 248 248
query84
query85 935 519 438 438
query86 412 289 274 274
query87 3444 3366 3174 3174
query88 3680 2788 2736 2736
query89 431 406 332 332
query90 1853 186 187 186
query91 180 166 142 142
query92 64 66 58 58
query93 1554 1371 888 888
query94 658 342 304 304
query95 682 467 368 368
query96 1068 740 374 374
query97 2725 2687 2570 2570
query98 221 208 217 208
query99 1163 1164 1020 1020
Total cold run time: 253459 ms
Total hot run time: 168839 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4668671594
# FE UT Coverage Report
Increment line coverage ` 4.74% (19/401)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_1997c78bdbc8252f993c10c39035ec5e91e6b06b/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_1997c78bdbc8252f993c10c39035ec5e91e6b06b/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4669035646
<details>
<summary>TPC-DS: <b>Total hot run time: 169139 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1997c78bdbc8252f993c10c39035ec5e91e6b06b, data reload: false
query5 4325 630 492 492
query6 440 211 177 177
query7 4824 538 308 308
query8 368 222 245 222
query9 8772 4107 4112 4107
query10 447 308 260 260
query11 5921 2337 2138 2138
query12 158 101 98 98
query13 1279 621 446 446
query14 6428 5376 5071 5071
query14_1 4459 4422 4395 4395
query15 211 200 184 184
query16 1041 480 434 434
query17 1111 669 566 566
query18 2667 477 339 339
query19 198 184 139 139
query20 117 106 111 106
query21 213 137 114 114
query22 13668 13561 13339 13339
query23 17422 16530 16235 16235
query23_1 16249 16345 16341 16341
query24 7604 1795 1301 1301
query24_1 1338 1302 1309 1302
query25 541 449 383 383
query26 1344 320 171 171
query27 2652 557 331 331
query28 4476 2039 2055 2039
query29 1076 618 488 488
query30 319 241 197 197
query31 1127 1089 953 953
query32 116 61 89 61
query33 521 313 245 245
query34 1170 1149 656 656
query35 758 777 685 685
query36 1383 1353 1201 1201
query37 156 108 90 90
query38 3203 3155 3060 3060
query39 934 934 900 900
query39_1 867 876 887 876
query40 222 123 102 102
query41 64 62 62 62
query42 93 93 93 93
query43 321 322 275 275
query44
query45 196 191 174 174
query46 1098 1214 750 750
query47 2349 2350 2250 2250
query48 400 389 287 287
query49 634 472 355 355
query50 1039 337 266 266
query51 4320 4270 4245 4245
query52 87 87 79 79
query53 248 263 189 189
query54 269 216 196 196
query55 78 76 70 70
query56 238 231 215 215
query57 1437 1398 1336 1336
query58 243 216 221 216
query59 1630 1653 1454 1454
query60 295 260 239 239
query61 170 162 162 162
query62 710 652 598 598
query63 232 187 191 187
query64 2554 831 624 624
query65
query66 1763 454 349 349
query67 29727 29629 29403 29403
query68
query69 423 285 274 274
query70 991 983 955 955
query71 299 221 211 211
query72 2981 2670 2411 2411
query73 831 749 417 417
query74 5156 4952 4835 4835
query75 2691 2604 2256 2256
query76 2371 1139 772 772
query77 363 377 294 294
query78 12421 12318 11815 11815
query79 1251 1015 765 765
query80 569 512 425 425
query81 461 290 245 245
query82 241 161 125 125
query83 287 281 253 253
query84
query85 933 596 517 517
query86 325 312 282 282
query87 3386 3341 3172 3172
query88 3643 2766 2762 2762
query89 418 391 343 343
query90 2175 185 169 169
query91 195 175 198 175
query92 62 63 58 58
query93 1459 1467 868 868
query94 571 344 312 312
query95 692 476 343 343
query96 1075 798 343 343
query97 2711 2695 2573 2573
query98 211 209 205 205
query99 1154 1160 1032 1032
Total cold run time: 251143 ms
Total hot run time: 169139 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4709584620
# FE Regression Coverage Report
Increment line coverage ` 48.95% (187/382)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_313fe8135eee7e6938e71bddc4ce73eb6e1050de_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_313fe8135eee7e6938e71bddc4ce73eb6e1050de_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4709516622
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4707504029
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4714558529
<details>
<summary>TPC-DS: <b>Total hot run time: 167884 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 635e8413bd63597f3c7976a1ed5e43d6c7f6cf8c, data reload: false
query5 4343 631 481 481
query6 440 185 169 169
query7 4931 560 293 293
query8 370 212 212 212
query9 8777 4087 4048 4048
query10 439 301 257 257
query11 5913 2344 2159 2159
query12 152 95 92 92
query13 1260 599 402 402
query14 6314 5340 4995 4995
query14_1 4306 4339 4347 4339
query15 202 197 181 181
query16 1003 451 421 421
query17 1092 675 553 553
query18 2510 458 333 333
query19 200 185 133 133
query20 110 113 102 102
query21 214 135 114 114
query22 13570 13692 13459 13459
query23 17285 16486 16045 16045
query23_1 16249 16305 16244 16244
query24 7492 1777 1291 1291
query24_1 1315 1301 1302 1301
query25 574 490 412 412
query26 1294 312 186 186
query27 2680 598 372 372
query28 4491 2010 2010 2010
query29 1105 635 502 502
query30 315 239 198 198
query31 1114 1070 961 961
query32 109 63 62 62
query33 541 322 253 253
query34 1188 1104 649 649
query35 763 779 683 683
query36 1395 1414 1247 1247
query37 159 109 88 88
query38 3208 3143 2970 2970
query39 927 908 904 904
query39_1 887 870 879 870
query40 223 126 103 103
query41 70 67 66 66
query42 96 96 93 93
query43 317 321 275 275
query44
query45 199 186 185 185
query46 1083 1186 695 695
query47 2403 2400 2252 2252
query48 407 392 303 303
query49 631 478 366 366
query50 999 355 268 268
query51 4331 4304 4265 4265
query52 91 90 79 79
query53 247 265 195 195
query54 277 244 208 208
query55 79 75 72 72
query56 258 231 228 228
query57 1461 1404 1317 1317
query58 255 223 225 223
query59 1617 1653 1463 1463
query60 279 239 219 219
query61 150 141 146 141
query62 695 648 565 565
query63 228 187 188 187
query64 2525 763 587 587
query65
query66 1794 450 340 340
query67 29057 29666 29351 29351
query68
query69 416 296 268 268
query70 942 961 931 931
query71 299 221 209 209
query72 2913 2604 2316 2316
query73 880 769 430 430
query74 5121 4929 4748 4748
query75 2666 2544 2242 2242
query76 2305 1151 797 797
query77 347 363 273 273
query78 12468 12471 11813 11813
query79 1428 1008 760 760
query80 567 466 379 379
query81 449 275 237 237
query82 576 158 120 120
query83 351 271 242 242
query84
query85 886 516 419 419
query86 354 295 295 295
query87 3429 3341 3194 3194
query88 3570 2719 2693 2693
query89 426 375 328 328
query90 1987 181 185 181
query91 170 163 132 132
query92 63 60 56 56
query93 1541 1559 919 919
query94 530 366 291 291
query95 669 469 334 334
query96 999 817 357 357
query97 2697 2693 2521 2521
query98 213 212 217 212
query99 1161 1185 1030 1030
Total cold run time: 249789 ms
Total hot run time: 167884 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62661:
URL: https://github.com/apache/doris/pull/62661#discussion_r3464510484
##########
regression-test/suites/load_p0/insert/test_insert_random_distribution_table.groovy:
##########
@@ -15,9 +15,15 @@
// specific language governing permissions and limitations
// under the License.
-suite("test_insert_random_distribution_table", "p0") {
+suite("test_insert_random_distribution_table", "p0,nonConcurrent") {
def tableName = "test_insert_random_distribution_table"
+ def enableAdaptiveRandomBucketConfig = sql """ ADMIN SHOW FRONTEND CONFIG LIKE 'enable_adaptive_random_bucket_load'; """
+ String oldEnableAdaptiveRandomBucket = enableAdaptiveRandomBucketConfig[0][1]
+ try {
+ // This case verifies the legacy batch-size based random tablet rotation.
+ // Keep the suite nonConcurrent because this FE config is global.
+ sql """ ADMIN SET FRONTEND CONFIG ('enable_adaptive_random_bucket_load' = 'false') """
Review Comment:
This suite now disables `enable_adaptive_random_bucket_load`, but the PR turns that feature on by default and changes the FE planning, thrift/protobuf fields, BE sender routing, and receiver-side rotation path. The added unit tests only exercise helper assignment/state logic, and I do not see any regression/integration test that runs a real random-distribution load with adaptive mode enabled. Please add an end-to-end test that leaves the feature enabled (cloud mode if required) and verifies the planned adaptive assignments are propagated and rows are routed/loaded correctly, or add the closest executable integration coverage for that path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4786697727
<details>
<summary>ClickBench: <b>Total hot run time: 25.25 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 591feb974a7f4b0b2831b4166b130e3dc8628771, data reload: false
query1 0.00 0.01 0.00
query2 0.10 0.06 0.05
query3 0.26 0.14 0.14
query4 1.61 0.14 0.13
query5 0.24 0.23 0.22
query6 1.23 1.06 1.06
query7 0.03 0.02 0.00
query8 0.09 0.04 0.04
query9 0.38 0.32 0.31
query10 0.58 0.58 0.57
query11 0.19 0.14 0.13
query12 0.18 0.14 0.15
query13 0.47 0.48 0.49
query14 1.01 1.02 1.02
query15 0.63 0.59 0.61
query16 0.32 0.33 0.30
query17 1.18 1.09 1.09
query18 0.23 0.22 0.21
query19 2.06 2.01 1.96
query20 0.02 0.01 0.02
query21 15.44 0.20 0.13
query22 4.91 0.05 0.06
query23 16.12 0.31 0.12
query24 2.97 0.43 0.32
query25 0.11 0.06 0.04
query26 0.73 0.21 0.16
query27 0.03 0.04 0.03
query28 3.56 0.91 0.53
query29 12.50 4.32 3.45
query30 0.27 0.14 0.15
query31 2.78 0.60 0.32
query32 3.22 0.60 0.48
query33 3.13 3.25 3.18
query34 15.56 4.20 3.52
query35 3.57 3.54 3.55
query36 0.56 0.43 0.44
query37 0.09 0.07 0.06
query38 0.04 0.04 0.03
query39 0.03 0.03 0.03
query40 0.19 0.16 0.15
query41 0.08 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 96.78 s
Total hot run time: 25.25 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4808329372
<details>
<summary>TPC-DS: <b>Total hot run time: 171913 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4f1c936ff2ae8acc7275baf70a22f673b037c6ca, data reload: false
query5 4328 639 488 488
query6 439 207 190 190
query7 4806 559 316 316
query8 339 189 174 174
query9 8776 4053 4030 4030
query10 483 310 262 262
query11 5786 2370 2188 2188
query12 160 105 110 105
query13 1315 635 432 432
query14 6287 5359 5008 5008
query14_1 4307 4313 4314 4313
query15 210 208 192 192
query16 1059 504 447 447
query17 1158 731 598 598
query18 2504 497 363 363
query19 213 194 154 154
query20 113 110 110 110
query21 224 141 122 122
query22 13636 13591 13325 13325
query23 17506 16587 16137 16137
query23_1 16283 16233 16357 16233
query24 7640 1809 1309 1309
query24_1 1355 1334 1301 1301
query25 577 471 404 404
query26 1322 344 175 175
query27 2607 597 352 352
query28 4488 2006 2020 2006
query29 1139 632 510 510
query30 313 239 205 205
query31 1136 1084 973 973
query32 108 67 60 60
query33 525 342 304 304
query34 1168 1167 627 627
query35 750 789 678 678
query36 1402 1397 1223 1223
query37 156 110 90 90
query38 1891 1714 1624 1624
query39 923 915 897 897
query39_1 878 874 866 866
query40 219 123 111 111
query41 65 62 65 62
query42 89 98 92 92
query43 321 325 280 280
query44 1436 787 776 776
query45 198 188 175 175
query46 1064 1200 724 724
query47 2405 2383 2270 2270
query48 406 424 298 298
query49 592 429 316 316
query50 1027 351 253 253
query51 4446 4399 4316 4316
query52 82 82 71 71
query53 246 264 188 188
query54 253 229 194 194
query55 74 76 66 66
query56 220 228 219 219
query57 1407 1387 1343 1343
query58 247 210 207 207
query59 1528 1637 1389 1389
query60 271 253 229 229
query61 146 150 155 150
query62 692 647 582 582
query63 234 193 188 188
query64 2527 773 586 586
query65 4928 4814 4787 4787
query66 1764 461 335 335
query67 28969 29054 28843 28843
query68 3227 1617 1006 1006
query69 417 308 266 266
query70 1037 974 952 952
query71 284 237 218 218
query72 2948 2735 2313 2313
query73 861 763 458 458
query74 5144 5034 4793 4793
query75 2570 2539 2174 2174
query76 2338 1228 787 787
query77 360 381 285 285
query78 12505 12492 11926 11926
query79 1297 1158 781 781
query80 586 476 378 378
query81 450 278 244 244
query82 575 156 120 120
query83 354 277 243 243
query84 261 144 114 114
query85 870 527 413 413
query86 359 304 269 269
query87 1860 1836 1782 1782
query88 3748 2797 2787 2787
query89 422 398 325 325
query90 1960 185 185 185
query91 168 161 134 134
query92 64 63 57 57
query93 1575 1602 908 908
query94 534 338 316 316
query95 689 472 344 344
query96 1128 811 366 366
query97 2722 2710 2583 2583
query98 214 204 201 201
query99 1178 1154 1034 1034
Total cold run time: 257023 ms
Total hot run time: 171913 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4842038424
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4896822268
# FE Regression Coverage Report
Increment line coverage ` 46.94% (230/490)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_30a431f2b08623dcddbc518296a9a8b5c5a06387_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_30a431f2b08623dcddbc518296a9a8b5c5a06387_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4899725482
<details>
<summary>TPC-H: <b>Total hot run time: 29525 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 30a431f2b08623dcddbc518296a9a8b5c5a06387, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17682 4002 3963 3963
q2 2054 324 199 199
q3 10865 1399 841 841
q4 4739 468 346 346
q5 8371 846 570 570
q6 338 169 147 147
q7 843 832 625 625
q8 10589 1504 1512 1504
q9 5889 4403 4409 4403
q10 6809 1796 1495 1495
q11 517 350 310 310
q12 725 541 422 422
q13 18086 3338 2744 2744
q14 265 268 239 239
q15 q16 781 776 710 710
q17 1025 926 943 926
q18 7061 5700 5676 5676
q19 1194 1256 1118 1118
q20 768 656 530 530
q21 5587 2624 2453 2453
q22 419 363 304 304
Total cold run time: 104607 ms
Total hot run time: 29525 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4370 4315 4314 4314
q2 286 318 213 213
q3 4568 4972 4448 4448
q4 2052 2137 1354 1354
q5 4452 4287 4311 4287
q6 229 178 130 130
q7 2157 1910 1683 1683
q8 2455 2158 2128 2128
q9 7881 7903 7763 7763
q10 4741 4694 4229 4229
q11 590 454 375 375
q12 856 825 545 545
q13 3275 3648 2955 2955
q14 305 307 296 296
q15 q16 745 763 648 648
q17 1345 1323 1295 1295
q18 8229 7243 6882 6882
q19 1108 1056 1075 1056
q20 2221 2201 1947 1947
q21 5244 4579 4362 4362
q22 513 451 392 392
Total cold run time: 57622 ms
Total hot run time: 51302 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) introduce adaptive random bucket load routing [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62661:
URL: https://github.com/apache/doris/pull/62661#issuecomment-4899828519
# BE Regression && UT Coverage Report
Increment line coverage `25.31% (181/715)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62661_30a431f2b08623dcddbc518296a9a8b5c5a06387_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62661_30a431f2b08623dcddbc518296a9a8b5c5a06387_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 58.94% (23589/40019) |
| Line Coverage | 42.27% (235772/557819) |
| Region Coverage | 38.50% (190547/494872) |
| Branch Coverage | 39.05% (83175/213014) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org