You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@druid.apache.org by "abhishekrb19 (via GitHub)" <gi...@apache.org> on 2026/03/05 17:37:16 UTC

[PR] Add `strict` and `pooled` Broker tier selector strategies (druid)

abhishekrb19 opened a new pull request, #19094:
URL: https://github.com/apache/druid/pull/19094

### Summary

Adds two new Broker `TierSelectorStrategy` implementations to provide finer control over how Brokers select Historical and Realtime servers for query execution.

- **strict** – Only selects servers whose priorities match the configured list.
Unlike other existing strategies, there is no fallback to servers with other priorities if the configured priorities are unavailable. This also addresses a current limitation with watched tiers: when multiple tiers are configured, Brokers can still retain visibility into the state of the cluster, while enforcing query isolation at the time of server selection rather than filtering things at the time of server view.

- **pooled** – Pools servers across the configured priorities and selects among them, allowing queries to utilize multiple priority tiers for improved availability. This is particularly useful for querying realtime servers where the number of task replicas per tier may be limited for cost reasons.

Both strategies require the configured set of priorities to be non-empty. Similar to queries routed to tiers that are not part of the watched tiers, these strategies may result in queries returning no data if the configured priorities are unavailable.

### Release note

Operators can now configure two new Broker `TierSelectorStrategy` implementations:

- strict: Only selects servers whose priorities match the configured list. Example configuration: `druid.broker.select.tier=strict` and `druid.broker.select.tier.strict.priorities=[1]`.
- pooled: Pools servers across the configured priorities and selects among them, allowing queries to utilize multiple priority tiers for improved availability. Example configuration: `druid.broker.select.tier=pooled` and `druid.broker.select.tier.pooled.priorities=[2,1]`.
These strategies can also be configured for Realtime servers using `druid.broker.realtime.select.tier`.

This PR has:

- [x] been self-reviewed.
- [ ] added documentation for new or modified features or behaviors. (Will add docs in a follow up along with previous realtime tier configuration)
- [x] a release note entry in the PR description.
- [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [x] been tested in a test Druid cluster.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] Add `strict` and `pooled` Broker tier selector strategies (druid)

Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.

abhishekrb19 commented on code in PR #19094:
URL: https://github.com/apache/druid/pull/19094#discussion_r2909166474


##########
server/src/main/java/org/apache/druid/client/selector/StrictTierSelectorStrategy.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.client.selector;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import it.unimi.dsi.fastutil.ints.Int2ObjectMap;
+import it.unimi.dsi.fastutil.ints.Int2ObjectRBTreeMap;
+import org.apache.druid.client.QueryableDruidServer;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.Query;
+import org.apache.druid.query.metadata.metadata.SegmentMetadataQuery;
+import org.apache.druid.timeline.DataSegment;
+
+import javax.annotation.Nullable;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * A {@link TierSelectorStrategy} that only considers servers whose priorities
+ * are explicitly listed in {@link StrictTierSelectorStrategyConfig#getPriorities()}.
+ * <p>
+ * Unlike other strategies like {@link CustomTierSelectorStrategy} that falls back to servers with different priorities,
+ * this strategy strictly filters the available servers to the configured priorities.
+ * If no servers match the configured priorities, an empty server list is returned from {@link #pick(Query, Int2ObjectRBTreeMap, DataSegment, int)},
+ * which may cause queries to return partial or no data.
+ * <p>
+ * When multiple priorities are configured, they are evaluated in the order specified, with earlier priorities
+ * preferred over later ones.
+ * <p>
+ * Example configuration:
+ * <li> <code> druid.broker.select.tier=strict </code> </li>
+ * <li> <code> druid.broker.select.tier.strict.priorities=[2,1] </code> </li>
+ * <p>
+ * With this configuration, servers with priority 2 are preferred over servers with priority 1.
+ * Servers with any other tier priority are not considered.
+ * <p>
+ * This strategy is useful when query isolation is required between different server priority tiers. Brokers may still
+ * be configured to watch all server tiers, allowing them to retain visibility into the overall cluster state
+ * while enforcing isolation at query time.
+ */
+public class StrictTierSelectorStrategy extends AbstractTierSelectorStrategy
+{
+  private static final EmittingLogger log = new EmittingLogger(StrictTierSelectorStrategy.class);
+  public static final String TYPE = "strict";
+
+  private final StrictTierSelectorStrategyConfig config;
+  private final ServiceEmitter emitter;
+  private final Map<Integer, Integer> configuredPriorities;
+  private final Comparator<Integer> comparator;
+
+  @JsonCreator
+  public StrictTierSelectorStrategy(
+      @JacksonInject final ServerSelectorStrategy serverSelectorStrategy,
+      @JacksonInject final StrictTierSelectorStrategyConfig config,
+      @JacksonInject final ServiceEmitter emitter
+  )
+  {
+    super(serverSelectorStrategy);
+    this.config = config;
+    this.emitter = emitter;
+
+    configuredPriorities = new HashMap<>();
+    for (int i = 0; i < config.getPriorities().size(); i++) {
+      configuredPriorities.put(config.getPriorities().get(i), i);
+    }
+
+    this.comparator = (p1, p2) -> {
+      final Integer rank1 = configuredPriorities.get(p1);
+      final Integer rank2 = configuredPriorities.get(p2);
+
+      if (rank1 != null && rank2 != null) {
+        return Integer.compare(rank1, rank2);
+      }
+      if (rank1 != null) {
+        return -1;
+      }
+      if (rank2 != null) {
+        return 1;
+      }
+
+      // Priorities outside configuredPriorities don't matter and won't be selected in pick() for this strategy.
+      // This fallback ordering can be anything and is needed because the comparator may be used by ServerSelector/ServerView
+      // which maintains views of all servers, including those with unconfigured priorities.
+      return Integer.compare(p2, p1);
+    };
+  }
+
+  @Override
+  public <T> List<QueryableDruidServer> pick(
+      @Nullable final Query<T> query,
+      final Int2ObjectRBTreeMap<Set<QueryableDruidServer>> prioritizedServers,
+      final DataSegment segment,
+      final int numServersToPick
+  )
+  {
+    final Int2ObjectRBTreeMap<Set<QueryableDruidServer>> candidatePrioritizedServers = new Int2ObjectRBTreeMap<>(getComparator());
+
+    for (Int2ObjectMap.Entry<Set<QueryableDruidServer>> entry : prioritizedServers.int2ObjectEntrySet()) {
+      final int priority = entry.getIntKey();
+      final Set<QueryableDruidServer> servers = entry.getValue();
+
+      if (configuredPriorities.containsKey(priority)) {
+        candidatePrioritizedServers.put(priority, servers);
+      } else {
+        log.debug(
+            "Server priority[%d] not in the configured list of priorities[%s] so ignore servers[%s] for query[%s]",
+            priority, config.getPriorities(), servers, query
+        );
+      }
+    }
+
+    if (candidatePrioritizedServers.isEmpty()) {
+      if (query == null || query instanceof SegmentMetadataQuery) {

Review Comment:
   I think the legacy `dataSourceMetadata` queries can be treated the same as other query types. Unlike system-generated queries such as `segmentMetadata`, these would be user-initiated and won't likely cause alert/log spam



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org