You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "findingrish (via GitHub)" <gi...@apache.org> on 2023/09/14 08:11:48 UTC
[PR] Centralize table schema building in Coordinator by relocating SegmentMetadataCache (druid)
findingrish opened a new pull request, #14985:
URL: https://github.com/apache/druid/pull/14985
## Description
Original proposal https://github.com/abhishekagarwal87/druid/blob/metadata_design_proposal/design-proposal.md#3-proposed-solution-storing-segment-schema-in-metadata-store
In the current design, brokers query both data nodes and tasks to fetch the schema of the segments they serve. The table schema is then constructed by combining the schemas of all segments within a dataSource. However, this approach leads to a high number of segment metadata queries during broker startup, resulting in slow startup times and various issues outlined in the design proposal.
To address these challenges, we propose centralizing the table schema management process within the coordinator. This change is the first step in that direction. In the new arrangement, the coordinator will take on the responsibility of querying both data nodes and tasks to fetch segment schema and subsequently building the table schema. Brokers will now simply query the Coordinator to fetch table schema. Importantly, brokers will still retain the capability to build table schemas if the need arises, ensuring both flexibility and resilience.
## Design
### Coordinator changes
These changes enhance the coordinator's capabilities, enabling it to manage table schema and provide schema related information through its APIs.
**SegmentMetadataCache refactor**: We introduce a new class, `CoordinatorSegmentMetadataCache`, extending `AbstractSegmentMetadataCache`, which manages the cache and forms the core of the schema management process.
**Changes to Coordinator timeline**: A new interface, `CoordinatorTimeline`, is introduced, and the existing coordinator timeline, `CoordinatorServerView`, now implements it. Additionally, we add a new implementation, `QueryableCoordinatorServerView`, which extends `BrokerServerView`. This facilitates running segment metadata queries on the coordinator using `CachingClusteredClient`.
**API Changes**:
- We introduce a new container class, `DataSourceInformation`, for handling `RowSignature` information. An API is exposed to retrieve dataSource information.
- The existing api, responsible for returning used segments is modified to accept a new query parameter called `includeRealtimeSegments`. When set, this parameter will include realtime segments in the result. Additionally, we enhance the return object `SegmentStatusInCluster`, to provide additional information regarding `realtime` and `numRows`.
**Binding for Query Modules**: We add essential bindings for required query modules in `CliCoordinator`.
### Broker changes
Broker now defaults to querying the Coordinator for fetching table schema, eliminating the need for querying data nodes and tasks for segment schema. Additionally, there are changes in the logic for building sys segments table.
**Broker-side SegmentMetadataCache**: We introduce a new implementation of `AbstractSegmentMetadataCache` called `BrokerSegmentMetadataCache`. This implementation queries the coordinator for table schema and falls back to running a segment metadata query when needed. It also assumes the responsibility for building `PhysicalDataSourceMetadata`, for which a new class is introduced.
**System schema changes**: In `MetadataSegmentView`, requests for realtime segments are now included when polling the coordinator. The segment table building logic is updated accordingly.
### Upgrade considerations
The general upgrade order should be followed. The new code is behind a feature flag, so it is compatible with existing setups. Brokers with the new changes can communicate with old Coordinators without issues.
### Release note
This feature is experimental and addresses multiple challenges outlined in the description. To enable it, set `druid.coordinator.segmentMetadataCache.enabled` in Coordinator configurations.
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1334961021
##########
server/src/main/java/org/apache/druid/client/coordinator/CoordinatorClientImpl.java:
##########
@@ -107,6 +110,23 @@ public ListenableFuture<List<DataSegment>> fetchUsedSegments(String dataSource,
);
}
+ @Override
+ public ListenableFuture<List<DataSourceInformation>> fetchDataSourceInformation(Set<String> dataSources)
+ {
+ final String path = "/druid/coordinator/v1/metadata/dataSourceInformation";
+ if (null == dataSources) {
+ dataSources = new HashSet<>();
Review Comment:
I don't think it is needed. Removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1334961918
##########
extensions-core/druid-bloom-filter/src/test/java/org/apache/druid/query/aggregation/bloom/sql/BloomFilterSqlAggregatorTest.java:
##########
@@ -47,10 +47,10 @@
import org.apache.druid.segment.join.JoinableFactoryWrapper;
import org.apache.druid.segment.virtual.ExpressionVirtualColumn;
import org.apache.druid.segment.writeout.OffHeapMemorySegmentWriteOutMediumFactory;
+import org.apache.druid.server.SpecificSegmentsQuerySegmentWalker;
import org.apache.druid.sql.calcite.BaseCalciteQueryTest;
import org.apache.druid.sql.calcite.filtration.Filtration;
import org.apache.druid.sql.calcite.util.CalciteTests;
-import org.apache.druid.sql.calcite.util.SpecificSegmentsQuerySegmentWalker;
Review Comment:
This class was needed for testing `SegmentMetadataCache`. Since, the class `SegmentMetadataCache` and its test are moved to `server` module, it was required to move this class as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1334963360
##########
server/src/main/java/org/apache/druid/segment/metadata/DataSourceInformation.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.metadata;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.segment.column.RowSignature;
+
+import java.util.Objects;
+
+/**
+ * Encapsulates information about a datasource, such as its schema.
+ */
+public class DataSourceInformation
Review Comment:
Yes. The idea is to have a container object with schema in it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1334962538
##########
server/src/main/java/org/apache/druid/segment/metadata/CoordinatorSegmentMetadataCache.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.metadata;
+
+import com.google.common.collect.Sets;
+import com.google.inject.Inject;
+import org.apache.druid.client.InternalQueryConfig;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.server.QueryLifecycleFactory;
+import org.apache.druid.server.security.Escalator;
+import org.apache.druid.timeline.SegmentId;
+
+import java.io.IOException;
+import java.util.Set;
+
+/**
+ * Coordinator-side cache of segment metadata that combines segments to identify
Review Comment:
Updated to use the term `build` instead of `identify`. However, these are old docs from `SegmentMetadataCache` class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1334962011
##########
server/src/main/java/org/apache/druid/client/QueryableCoordinatorServerView.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.client;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Iterables;
+import com.google.inject.Inject;
+import org.apache.druid.client.selector.ServerSelector;
+import org.apache.druid.client.selector.TierSelectorStrategy;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.guice.annotations.EscalatedClient;
+import org.apache.druid.guice.annotations.Smile;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.http.client.HttpClient;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.QueryToolChestWarehouse;
+import org.apache.druid.query.QueryWatcher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+import org.apache.druid.timeline.VersionedIntervalTimeline;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Map;
+
+/**
+ * ServerView of coordinator for the state of segments being loaded in the cluster.
+ *
+ * <p>This class extends {@link BrokerServerView} and implements {@link CoordinatorTimeline}.
+ * The main distinction between this class and {@link CoordinatorServerView} is the maintenance of a timeline
+ * of {@link ServerSelector} objects, while the other class stores {@link SegmentLoadInfo} object in its timeline.</p>
+ *
+ * <p>A new timeline class (implementing {@link TimelineServerView}) is required for
+ * {@link org.apache.druid.segment.metadata.CoordinatorSegmentMetadataCache}, which will run on the Coordinator.</p>
+ */
+@ManageLifecycle
+public class QueryableCoordinatorServerView extends BrokerServerView implements CoordinatorTimeline
+{
+ private final FilteredServerInventoryView baseView;
+
+ @Inject
+ public QueryableCoordinatorServerView(
+ final QueryToolChestWarehouse warehouse,
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1372565864
##########
sql/src/main/java/org/apache/druid/sql/calcite/schema/BrokerSegmentMetadataCacheConfig.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.segment.metadata.SegmentMetadataCacheConfig;
+import org.joda.time.Period;
+
+/**
+ * Broker-side configuration class for managing segment polling from the Coordinator and
+ * customizing properties related to the SegmentMetadata cache which is used to infer datasources for SQL.
+ *
+ * <p>See {@link BrokerSegmentMetadataCache}, {@link MetadataSegmentView}.</p>
+ *
+ * <p>This class shares the same config root as {@link org.apache.druid.sql.calcite.planner.PlannerConfig}
+ * to maintain backward compatibility for when the properties here resided in {@code PlannerConfig}.</p>
+ *
+ * <p> The property {@link #awaitInitializationOnStart} is overridden in this class with a default value
+ * of {@code true}, which differs from the parent class. This ensures that the SegmentMetadata cache is
+ * fully initialized before other startup processes proceed.</p>
+ */
+public class BrokerSegmentMetadataCacheConfig extends SegmentMetadataCacheConfig
+{
+ // A flag indicating whether to cache polled segments from the Coordinator.
+ @JsonProperty
+ private boolean metadataSegmentCacheEnable = false;
+
+ // Interval for polling segments from the coordinator.
+ @JsonProperty
+ private long metadataSegmentPollPeriod = 60000;
+
+ // A flag indicating whether to wait for cache initialization during startup.
+ @JsonProperty
+ private boolean awaitInitializationOnStart = true;
+
+ @JsonProperty
+ private boolean disableSegmentMetadataQueries = false;
Review Comment:
This config is only needed for the integration test, wherein I disable the fallback mechanism in BrokerSegmentMetadataCache and thus ensure that table schema is built only in the Coordinator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1372515182
##########
sql/src/main/java/org/apache/druid/sql/calcite/schema/BrokerSegmentMetadataCacheConfig.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.segment.metadata.SegmentMetadataCacheConfig;
+import org.joda.time.Period;
+
+/**
+ * Broker-side configuration class for managing segment polling from the Coordinator and
+ * customizing properties related to the SegmentMetadata cache which is used to infer datasources for SQL.
+ *
+ * <p>See {@link BrokerSegmentMetadataCache}, {@link MetadataSegmentView}.</p>
+ *
+ * <p>This class shares the same config root as {@link org.apache.druid.sql.calcite.planner.PlannerConfig}
+ * to maintain backward compatibility for when the properties here resided in {@code PlannerConfig}.</p>
+ *
+ * <p> The property {@link #awaitInitializationOnStart} is overridden in this class with a default value
+ * of {@code true}, which differs from the parent class. This ensures that the SegmentMetadata cache is
+ * fully initialized before other startup processes proceed.</p>
+ */
+public class BrokerSegmentMetadataCacheConfig extends SegmentMetadataCacheConfig
+{
+ // A flag indicating whether to cache polled segments from the Coordinator.
+ @JsonProperty
+ private boolean metadataSegmentCacheEnable = false;
+
+ // Interval for polling segments from the coordinator.
+ @JsonProperty
+ private long metadataSegmentPollPeriod = 60000;
+
+ // A flag indicating whether to wait for cache initialization during startup.
+ @JsonProperty
+ private boolean awaitInitializationOnStart = true;
+
+ @JsonProperty
+ private boolean disableSegmentMetadataQueries = false;
Review Comment:
afaict, this is not documented, should it be?
##########
sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java:
##########
@@ -302,13 +303,30 @@ public Enumerable<Object[]> scan(DataContext root)
final DataSegment segment = val.getDataSegment();
segmentsAlreadySeen.add(segment.getId());
final PartialSegmentData partialSegmentData = partialSegmentDataMap.get(segment.getId());
- long numReplicas = 0L, numRows = 0L, isRealtime = 0L, isAvailable = 0L;
+ long numReplicas = 0L, numRows = 0L, isRealtime, isAvailable = 0L;
+
if (partialSegmentData != null) {
numReplicas = partialSegmentData.getNumReplicas();
- numRows = partialSegmentData.getNumRows();
isAvailable = partialSegmentData.isAvailable();
- isRealtime = partialSegmentData.isRealtime();
+ numRows = partialSegmentData.getNumRows();
+ }
+
+ // If table schema building is enabled on the Coordinator, SegmentMetadataCache on the
+ // broker might have outdated or no information regarding numRows and rowSignature for a segment.
+ // We should use {@code numRows} from the segment polled from the coordinator.
+ if (null != val.getNumRows()) {
+ numRows = val.getNumRows();
}
+
+ isRealtime = Boolean.TRUE.equals(val.isRealtime()) ? 1 : 0;
+
+ // set of segments returned from Coordinator include published and realtime segments
+ // so realtime segments are not published and vice versa
+ boolean isPublished = !val.isRealtime();
Review Comment:
nit: why does `isRealtime` do this `Boolean.TRUE.equals` thing for `val.isRealtime()`, while `isPublished` use `!val.isRealtime()`, and similar `isActive` uses it directly
##########
server/src/main/java/org/apache/druid/client/CoordinatorServerView.java:
##########
@@ -162,7 +180,28 @@ private void serverAddedSegment(final DruidServerMetadata server, final DataSegm
);
segmentLoadInfos.put(segmentId, segmentLoadInfo);
}
+
+ if (null != druidClientFactory) {
+ QueryRunner queryRunner = serverQueryRunners.get(server.getName());
+ if (null == queryRunner) {
+ DruidServer inventoryValue = baseView.getInventoryValue(server.getName());
+ if (inventoryValue == null) {
Review Comment:
nit: can you please either put `null` consistently on the left or right of these checks? My preference would be on the right since my gut says that is what the majority of the codebase is doing, but at least we should be consistent in the same function...
##########
server/src/test/java/org/apache/druid/segment/metadata/SegmentMetadataQuerySegmentWalkerTest.java:
##########
@@ -0,0 +1,343 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.metadata;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import org.apache.druid.client.CachingClusteredClientTest.ServerExpectation;
+import org.apache.druid.client.CachingClusteredClientTest.ServerExpectations;
+import org.apache.druid.client.CoordinatorSegmentWatcherConfig;
+import org.apache.druid.client.CoordinatorServerView;
+import org.apache.druid.client.DirectDruidClientFactory;
+import org.apache.druid.client.DruidServer;
+import org.apache.druid.client.SegmentLoadInfo;
+import org.apache.druid.client.ServerInventoryView;
+import org.apache.druid.guice.http.DruidHttpClientConfig;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.Pair;
+import org.apache.druid.java.util.common.guava.Sequence;
+import org.apache.druid.java.util.common.guava.Sequences;
+import org.apache.druid.java.util.common.guava.Yielder;
+import org.apache.druid.java.util.common.guava.Yielders;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.MapQueryToolChestWarehouse;
+import org.apache.druid.query.Query;
+import org.apache.druid.query.QueryContexts;
+import org.apache.druid.query.QueryPlus;
+import org.apache.druid.query.QueryRunner;
+import org.apache.druid.query.QueryToolChest;
+import org.apache.druid.query.QueryToolChestWarehouse;
+import org.apache.druid.query.SegmentDescriptor;
+import org.apache.druid.query.TableDataSource;
+import org.apache.druid.query.context.ResponseContext;
+import org.apache.druid.query.metadata.SegmentMetadataQueryConfig;
+import org.apache.druid.query.metadata.SegmentMetadataQueryQueryToolChest;
+import org.apache.druid.query.metadata.metadata.AllColumnIncluderator;
+import org.apache.druid.query.metadata.metadata.SegmentAnalysis;
+import org.apache.druid.query.metadata.metadata.SegmentMetadataQuery;
+import org.apache.druid.query.spec.MultipleSpecificSegmentSpec;
+import org.apache.druid.server.coordination.ServerType;
+import org.apache.druid.server.initialization.ServerConfig;
+import org.apache.druid.server.metrics.NoopServiceEmitter;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+import org.apache.druid.timeline.VersionedIntervalTimeline;
+import org.apache.druid.timeline.partition.ShardSpec;
+import org.apache.druid.timeline.partition.SingleDimensionShardSpec;
+import org.easymock.EasyMock;
+import org.easymock.IAnswer;
+import org.joda.time.Interval;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+public class SegmentMetadataQuerySegmentWalkerTest
+{
+ private final String DATASOURCE = "testDatasource";
+ private QueryToolChestWarehouse warehouse;
+ private DruidHttpClientConfig httpClientConfig;
+ private DruidServer[] servers;
+ private Random random;
+
+ @Before
+ public void setUp()
+ {
+ warehouse = new MapQueryToolChestWarehouse(
+ ImmutableMap.<Class<? extends Query>, QueryToolChest>builder()
+ .put(
+ SegmentMetadataQuery.class,
+ new SegmentMetadataQueryQueryToolChest(
+ new SegmentMetadataQueryConfig("P1W")
+
+ )
+ ).build());
+
+ httpClientConfig = new DruidHttpClientConfig()
+ {
+ @Override
+ public long getMaxQueuedBytes()
+ {
+ return 0L;
+ }
+ };
+
+ servers =
+ new DruidServer[]{
+ new DruidServer("test1", "test1", null, 10, ServerType.HISTORICAL, "bye", 0),
+ new DruidServer("test2", "test2", null, 10, ServerType.HISTORICAL, "bye", 0),
+ new DruidServer("test3", "test2", null, 10, ServerType.REALTIME, "bye", 0)
Review Comment:
I know this is just a test, but afaik nothing uses `REALTIME` anymore, so probably should use `INDEXER_EXECUTOR`...
`REALTIME` refers to the old realtime node type that was removed quite a while ago https://github.com/apache/druid/pull/7915
We should probably remove this `ServerType` completely now since it seems unlikely anyone will be rolling upgrade from such an old version at this point 😅
##########
server/src/test/java/org/apache/druid/segment/metadata/SegmentMetadataQuerySegmentWalkerTest.java:
##########
@@ -0,0 +1,343 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.metadata;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import org.apache.druid.client.CachingClusteredClientTest.ServerExpectation;
+import org.apache.druid.client.CachingClusteredClientTest.ServerExpectations;
+import org.apache.druid.client.CoordinatorSegmentWatcherConfig;
+import org.apache.druid.client.CoordinatorServerView;
+import org.apache.druid.client.DirectDruidClientFactory;
+import org.apache.druid.client.DruidServer;
+import org.apache.druid.client.SegmentLoadInfo;
+import org.apache.druid.client.ServerInventoryView;
+import org.apache.druid.guice.http.DruidHttpClientConfig;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.Pair;
+import org.apache.druid.java.util.common.guava.Sequence;
+import org.apache.druid.java.util.common.guava.Sequences;
+import org.apache.druid.java.util.common.guava.Yielder;
+import org.apache.druid.java.util.common.guava.Yielders;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.MapQueryToolChestWarehouse;
+import org.apache.druid.query.Query;
+import org.apache.druid.query.QueryContexts;
+import org.apache.druid.query.QueryPlus;
+import org.apache.druid.query.QueryRunner;
+import org.apache.druid.query.QueryToolChest;
+import org.apache.druid.query.QueryToolChestWarehouse;
+import org.apache.druid.query.SegmentDescriptor;
+import org.apache.druid.query.TableDataSource;
+import org.apache.druid.query.context.ResponseContext;
+import org.apache.druid.query.metadata.SegmentMetadataQueryConfig;
+import org.apache.druid.query.metadata.SegmentMetadataQueryQueryToolChest;
+import org.apache.druid.query.metadata.metadata.AllColumnIncluderator;
+import org.apache.druid.query.metadata.metadata.SegmentAnalysis;
+import org.apache.druid.query.metadata.metadata.SegmentMetadataQuery;
+import org.apache.druid.query.spec.MultipleSpecificSegmentSpec;
+import org.apache.druid.server.coordination.ServerType;
+import org.apache.druid.server.initialization.ServerConfig;
+import org.apache.druid.server.metrics.NoopServiceEmitter;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+import org.apache.druid.timeline.VersionedIntervalTimeline;
+import org.apache.druid.timeline.partition.ShardSpec;
+import org.apache.druid.timeline.partition.SingleDimensionShardSpec;
+import org.easymock.EasyMock;
+import org.easymock.IAnswer;
+import org.joda.time.Interval;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+public class SegmentMetadataQuerySegmentWalkerTest
+{
+ private final String DATASOURCE = "testDatasource";
+ private QueryToolChestWarehouse warehouse;
+ private DruidHttpClientConfig httpClientConfig;
+ private DruidServer[] servers;
+ private Random random;
+
+ @Before
+ public void setUp()
+ {
+ warehouse = new MapQueryToolChestWarehouse(
+ ImmutableMap.<Class<? extends Query>, QueryToolChest>builder()
+ .put(
+ SegmentMetadataQuery.class,
+ new SegmentMetadataQueryQueryToolChest(
+ new SegmentMetadataQueryConfig("P1W")
+
+ )
+ ).build());
+
+ httpClientConfig = new DruidHttpClientConfig()
+ {
+ @Override
+ public long getMaxQueuedBytes()
+ {
+ return 0L;
+ }
+ };
+
+ servers =
+ new DruidServer[]{
+ new DruidServer("test1", "test1", null, 10, ServerType.HISTORICAL, "bye", 0),
+ new DruidServer("test2", "test2", null, 10, ServerType.HISTORICAL, "bye", 0),
+ new DruidServer("test3", "test2", null, 10, ServerType.REALTIME, "bye", 0)
Review Comment:
I know this is just a test, but afaik nothing uses `REALTIME` anymore, so probably should use `INDEXER_EXECUTOR`...
`REALTIME` refers to the old realtime node type that was removed quite a while ago https://github.com/apache/druid/pull/7915
We should probably remove this `ServerType` completely now since it seems unlikely anyone will be rolling upgrade from such an old version at this point 😅
##########
server/src/main/java/org/apache/druid/client/CachingClusteredClient.java:
##########
@@ -844,23 +845,26 @@ private byte[] computeQueryCacheKeyWithJoin()
}
}
- private static class TimelineConverter implements UnaryOperator<TimelineLookup<String, ServerSelector>>
+ /**
+ * Helper class to build a new timeline of filtered segments.
+ */
+ public static class TimelineConverter<ObjectType extends Overshadowable<ObjectType>> implements UnaryOperator<TimelineLookup<String, ObjectType>>
Review Comment:
nit: is there really nothing more descriptive than `ObjectType`? Might as well just use `T` if not 😅
##########
sql/src/main/java/org/apache/druid/sql/calcite/schema/BrokerSegmentMetadataCache.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.google.common.base.Stopwatch;
+import com.google.common.collect.Sets;
+import com.google.inject.Inject;
+import org.apache.druid.client.InternalQueryConfig;
+import org.apache.druid.client.ServerView;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.client.coordinator.CoordinatorClient;
+import org.apache.druid.common.guava.FutureUtils;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.metadata.AbstractSegmentMetadataCache;
+import org.apache.druid.segment.metadata.DataSourceInformation;
+import org.apache.druid.server.QueryLifecycleFactory;
+import org.apache.druid.server.coordination.DruidServerMetadata;
+import org.apache.druid.server.security.Escalator;
+import org.apache.druid.sql.calcite.table.DatasourceTable.PhysicalDatasourceMetadata;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Broker-side cache of segment metadata that combines segments to build
+ * datasources which become "tables" in Calcite. This cache provides the "physical"
+ * metadata about a dataSource which is blended with catalog "logical" metadata
+ * to provide the final user-view of each dataSource.
+ * <p>
+ * This class extends {@link AbstractSegmentMetadataCache} and introduces following changes,
+ * <ul>
+ * <li>The refresh mechanism includes polling the coordinator for datasource schema,
+ * and falling back to running {@link org.apache.druid.query.metadata.metadata.SegmentMetadataQuery}.</li>
+ * <li>It builds and caches {@link PhysicalDatasourceMetadata} object for the table schema</li>
+ * </ul>
+ */
+@ManageLifecycle
+public class BrokerSegmentMetadataCache extends AbstractSegmentMetadataCache<PhysicalDatasourceMetadata>
+{
+ private static final EmittingLogger log = new EmittingLogger(BrokerSegmentMetadataCache.class);
+
+ private final PhysicalDatasourceMetadataFactory dataSourceMetadataFactory;
+ private final CoordinatorClient coordinatorClient;
+
+ private final BrokerSegmentMetadataCacheConfig config;
+
+ @Inject
+ public BrokerSegmentMetadataCache(
+ final QueryLifecycleFactory queryLifecycleFactory,
+ final TimelineServerView serverView,
+ final BrokerSegmentMetadataCacheConfig config,
+ final Escalator escalator,
+ final InternalQueryConfig internalQueryConfig,
+ final ServiceEmitter emitter,
+ final PhysicalDatasourceMetadataFactory dataSourceMetadataFactory,
+ final CoordinatorClient coordinatorClient
+ )
+ {
+ super(
+ queryLifecycleFactory,
+ config,
+ escalator,
+ internalQueryConfig,
+ emitter
+ );
+ this.dataSourceMetadataFactory = dataSourceMetadataFactory;
+ this.coordinatorClient = coordinatorClient;
+ this.config = config;
+ initServerViewTimelineCallback(serverView);
+ }
+
+ private void initServerViewTimelineCallback(final TimelineServerView serverView)
+ {
+ serverView.registerTimelineCallback(
+ callbackExec,
+ new TimelineServerView.TimelineCallback()
+ {
+ @Override
+ public ServerView.CallbackAction timelineInitialized()
+ {
+ synchronized (lock) {
+ isServerViewInitialized = true;
+ lock.notifyAll();
+ }
+
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction segmentAdded(final DruidServerMetadata server, final DataSegment segment)
+ {
+ addSegment(server, segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction segmentRemoved(final DataSegment segment)
+ {
+ removeSegment(segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction serverSegmentRemoved(
+ final DruidServerMetadata server,
+ final DataSegment segment
+ )
+ {
+ removeServerSegment(server, segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+ }
+ );
+ }
+
+ /**
+ * Refreshes the set of segments in two steps:
+ * <ul>
+ * <li>Polls the coordinator for the datasource schema.</li>
+ * <li>Refreshes the remaining set of segments by executing a SegmentMetadataQuery and
+ * builds datasource schema by combining segment schema.</li>
+ * </ul>
+ *
+ * @param segmentsToRefresh segments for which the schema might have changed
+ * @param dataSourcesToRebuild datasources for which the schema might have changed
+ * @throws IOException when querying segment schema from data nodes and tasks
+ */
+ @Override
+ public void refresh(final Set<SegmentId> segmentsToRefresh, final Set<String> dataSourcesToRebuild) throws IOException
+ {
+ // query schema for all datasources, which includes,
+ // datasources explicitly marked for rebuilding
+ // datasources for the segments to be refreshed
+ // prebuilt datasources
+ final Set<String> dataSourcesToQuery = new HashSet<>(dataSourcesToRebuild);
+
+ segmentsToRefresh.forEach(segment -> dataSourcesToQuery.add(segment.getDataSource()));
+
+ dataSourcesToQuery.addAll(tables.keySet());
+
+ // Fetch datasource information from the Coordinator
+ Map<String, PhysicalDatasourceMetadata> polledDataSourceMetadata = queryDataSourceInformation(dataSourcesToQuery);
Review Comment:
since this is not on by default in the coordinator, shouldn't we have a broker side config to indicate if we query the coordinator at all? Like it seems like there is no point in doing this if the coordinator cache is not running (the default behavior)
##########
server/src/test/java/org/apache/druid/segment/metadata/SegmentMetadataCacheCommon.java:
##########
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.metadata;
+
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.data.input.InputRow;
+import org.apache.druid.data.input.InputRowSchema;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.data.input.impl.MapInputRowParser;
+import org.apache.druid.data.input.impl.TimestampSpec;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.DefaultGenericQueryMetricsFactory;
+import org.apache.druid.query.DefaultQueryConfig;
+import org.apache.druid.query.Query;
+import org.apache.druid.query.QueryRunnerFactoryConglomerate;
+import org.apache.druid.query.QuerySegmentWalker;
+import org.apache.druid.query.QueryToolChest;
+import org.apache.druid.query.QueryToolChestWarehouse;
+import org.apache.druid.query.aggregation.CountAggregatorFactory;
+import org.apache.druid.query.aggregation.DoubleSumAggregatorFactory;
+import org.apache.druid.query.aggregation.LongSumAggregatorFactory;
+import org.apache.druid.query.aggregation.hyperloglog.HyperUniquesAggregatorFactory;
+import org.apache.druid.segment.IndexBuilder;
+import org.apache.druid.segment.QueryableIndex;
+import org.apache.druid.segment.incremental.IncrementalIndexSchema;
+import org.apache.druid.segment.writeout.OffHeapMemorySegmentWriteOutMediumFactory;
+import org.apache.druid.server.QueryLifecycleFactory;
+import org.apache.druid.server.QueryStackTests;
+import org.apache.druid.server.log.TestRequestLogger;
+import org.apache.druid.server.metrics.NoopServiceEmitter;
+import org.apache.druid.server.security.AuthConfig;
+import org.apache.druid.server.security.AuthTestUtils;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.LinearShardSpec;
+import org.apache.druid.timeline.partition.NumberedShardSpec;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.util.List;
+import java.util.Map;
+
+public abstract class SegmentMetadataCacheCommon
+{
+ public static final String DATASOURCE1 = "foo";
+ public static final String DATASOURCE2 = "foo2";
+ public static final String DATASOURCE3 = "foo3";
+ public static final String SOME_DATASOURCE = "some_datasource";
+ public static final String TIMESTAMP_COLUMN = "t";
+ private static final InputRowSchema FOO_SCHEMA = new InputRowSchema(
+ new TimestampSpec(TIMESTAMP_COLUMN, "iso", null),
+ new DimensionsSpec(
+ DimensionsSpec.getDefaultSchemas(ImmutableList.of("dim1", "dim2", "dim3"))
+ ),
+ null
+ );
+
+ public final List<InputRow> ROWS1 = ImmutableList.of(
+ createRow(ImmutableMap.of("t", "2000-01-01", "m1", "1.0", "dim1", "")),
+ createRow(ImmutableMap.of("t", "2000-01-02", "m1", "2.0", "dim1", "10.1")),
+ createRow(ImmutableMap.of("t", "2000-01-03", "m1", "3.0", "dim1", "2"))
+ );
+
+ public final List<InputRow> ROWS2 = ImmutableList.of(
+ createRow(ImmutableMap.of("t", "2001-01-01", "m1", "4.0", "dim2", ImmutableList.of("a"))),
+ createRow(ImmutableMap.of("t", "2001-01-02", "m1", "5.0", "dim2", ImmutableList.of("abc"))),
+ createRow(ImmutableMap.of("t", "2001-01-03", "m1", "6.0"))
+ );
+
+ public QueryRunnerFactoryConglomerate conglomerate;
+ public Closer resourceCloser;
+ public QueryToolChestWarehouse queryToolChestWarehouse;
+
+ @Rule
+ public TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ public QueryableIndex index1;
+ public QueryableIndex index2;
+
+ public QueryableIndex indexAuto1;
+ public QueryableIndex indexAuto2;
+
+ public DataSegment realtimeSegment1;
+ public DataSegment segment1;
+ public DataSegment segment2;
+ public DataSegment segment3;
+ public DataSegment segment4;
+ public DataSegment segment5;
+
+ static {
+ NullHandling.initializeForTests();
+ }
Review Comment:
nit: recommend extending `InitializedNullHandlingTest` since it also initializes `ExpressionProcessing` (the other static config), though since this probably isn't running any actual queries it probably isn't necessary..
##########
server/src/main/java/org/apache/druid/client/CachingClusteredClient.java:
##########
@@ -844,23 +845,26 @@ private byte[] computeQueryCacheKeyWithJoin()
}
}
- private static class TimelineConverter implements UnaryOperator<TimelineLookup<String, ServerSelector>>
+ /**
+ * Helper class to build a new timeline of filtered segments.
+ */
+ public static class TimelineConverter<ObjectType extends Overshadowable<ObjectType>> implements UnaryOperator<TimelineLookup<String, ObjectType>>
Review Comment:
nit: is there really nothing more descriptive than `ObjectType`? Might as well just use `T` if not 😅
##########
server/src/main/java/org/apache/druid/client/SegmentLoadInfo.java:
##########
@@ -59,6 +61,15 @@ public ImmutableSegmentLoadInfo toImmutableSegmentLoadInfo()
return new ImmutableSegmentLoadInfo(segment, servers);
}
+ /**
+ * Randomly return one server from the sets of {@code servers}
+ */
+ public DruidServerMetadata pickOne()
+ {
+ synchronized (this) {
Review Comment:
if all of the methods are synchronized, then we don't need a concurrent set backing the servers list, suggest either dropping the concurrent backing set or making random pick not require locks like this, i don't have a strong opinion either way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1334961550
##########
server/src/main/java/org/apache/druid/segment/metadata/AbstractSegmentMetadataCache.java:
##########
@@ -385,55 +378,26 @@ private void startCacheExec()
);
}
+ private void setInitialized(Stopwatch stopwatch)
Review Comment:
Renamed to `setInitializedAndReportInitTime`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1372480185
##########
benchmarks/src/test/java/org/apache/druid/benchmark/DruidSchemaInternRowSignatureBenchmark.java:
##########
@@ -71,26 +68,21 @@ public class DruidSchemaInternRowSignatureBenchmark
{
private SegmentMetadataCacheForBenchmark cache;
- private static class SegmentMetadataCacheForBenchmark extends SegmentMetadataCache
+ private static class SegmentMetadataCacheForBenchmark extends CoordinatorSegmentMetadataCache
Review Comment:
The code being tested is same in the two classes, however I have changed the cache implementation used to BrokerSegmentMetadataCache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on PR #14985:
URL: https://github.com/apache/druid/pull/14985#issuecomment-1764029268
The initial approach of extending BrokerServerView to run as Coordinator timeline and using CachingClusteredClient to execute segment metadata queries was causing extra memory usage on the Coordinator nodes, as observed in our performance testing.
To address these issues, we have implemented a new solution, which involves a new implementation for QuerySegmentWalker to execute segment metadata queries and minimal modifications to the existing Coordinator timeline.
Please note that some review comments on deleted files from the initial approach are not addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe commented on PR #14985:
URL: https://github.com/apache/druid/pull/14985#issuecomment-1793452476
Thanks you for the contribution @findingrish.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1375248637
##########
server/src/main/java/org/apache/druid/client/SegmentLoadInfo.java:
##########
@@ -59,6 +61,15 @@ public ImmutableSegmentLoadInfo toImmutableSegmentLoadInfo()
return new ImmutableSegmentLoadInfo(segment, servers);
}
+ /**
+ * Randomly return one server from the sets of {@code servers}
+ */
+ public DruidServerMetadata pickOne()
+ {
+ synchronized (this) {
Review Comment:
> hmm, this still isn't quite right i think,
yes, I must synchronise the other methods as well.
Or, should we continue to use the concurrent set and synchronise only `add`, `remove` and `pickOne`?
> So I think we can just iterate to pick one...
`Iterators.get(servers.iterator(), ThreadLocalRandom.current().nextInt(servers.size()));`
Why do you suggest that we don't need synchronisation for this method.
The `servers` size can change while iteration, if the server is removed then it can potentially result in `IndexOutOfBoundsException`.
> but if we do, then something should probably be pushing a computation into this class to do the thing so that we can do the whole operation under lock...
can you please clarify this statement?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1374120552
##########
server/src/main/java/org/apache/druid/client/CoordinatorServerView.java:
##########
@@ -162,7 +180,28 @@ private void serverAddedSegment(final DruidServerMetadata server, final DataSegm
);
segmentLoadInfos.put(segmentId, segmentLoadInfo);
}
+
+ if (null != druidClientFactory) {
+ QueryRunner queryRunner = serverQueryRunners.get(server.getName());
+ if (null == queryRunner) {
+ DruidServer inventoryValue = baseView.getInventoryValue(server.getName());
+ if (inventoryValue == null) {
Review Comment:
I have tried to adhere with the general convention and moved null to the right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1328533621
##########
server/src/main/java/org/apache/druid/client/CoordinatorTimeline.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.client;
+
+import org.apache.druid.query.DataSource;
+import org.apache.druid.timeline.SegmentId;
+import org.apache.druid.timeline.VersionedIntervalTimeline;
+
+import java.util.Map;
+
+/**
+ * Segment timeline maintained in the coordinator.
Review Comment:
Could you add more details in this java doc.
What does this class do?
How is this class used.
##########
server/src/main/java/org/apache/druid/client/BrokerServerView.java:
##########
@@ -85,9 +81,14 @@ public class BrokerServerView implements TimelineServerView
private final ServiceEmitter emitter;
private final BrokerSegmentWatcherConfig segmentWatcherConfig;
private final Predicate<Pair<DruidServerMetadata, DataSegment>> segmentFilter;
-
private final CountDownLatch initialized = new CountDownLatch(1);
+ protected final Object lock = new Object();
+
+ protected final Map<SegmentId, ServerSelector> selectors = new HashMap<>();
Review Comment:
Why are there 2 hashmap's here?
Are they both used in one flow ? If not lets document that
##########
server/src/main/java/org/apache/druid/client/QueryableCoordinatorServerView.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.client;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Iterables;
+import com.google.inject.Inject;
+import org.apache.druid.client.selector.ServerSelector;
+import org.apache.druid.client.selector.TierSelectorStrategy;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.guice.annotations.EscalatedClient;
+import org.apache.druid.guice.annotations.Smile;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.http.client.HttpClient;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.QueryToolChestWarehouse;
+import org.apache.druid.query.QueryWatcher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+import org.apache.druid.timeline.VersionedIntervalTimeline;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Map;
+
+/**
+ * ServerView of coordinator for the state of segments being loaded in the cluster.
+ *
+ * <p>This class extends {@link BrokerServerView} and implements {@link CoordinatorTimeline}.
+ * The main distinction between this class and {@link CoordinatorServerView} is the maintenance of a timeline
+ * of {@link ServerSelector} objects, while the other class stores {@link SegmentLoadInfo} object in its timeline.</p>
+ *
+ * <p>A new timeline class (implementing {@link TimelineServerView}) is required for
+ * {@link org.apache.druid.segment.metadata.CoordinatorSegmentMetadataCache}, which will run on the Coordinator.</p>
+ */
+@ManageLifecycle
+public class QueryableCoordinatorServerView extends BrokerServerView implements CoordinatorTimeline
+{
+ private final FilteredServerInventoryView baseView;
+
+ @Inject
+ public QueryableCoordinatorServerView(
+ final QueryToolChestWarehouse warehouse,
+ final QueryWatcher queryWatcher,
+ final @Smile ObjectMapper smileMapper,
+ final @EscalatedClient HttpClient httpClient,
+ FilteredServerInventoryView baseView,
+ TierSelectorStrategy tierSelectorStrategy,
+ ServiceEmitter emitter,
+ CoordinatorSegmentWatcherConfig segmentWatcherConfig
+ )
+ {
+ super(warehouse, queryWatcher, smileMapper, httpClient, baseView, tierSelectorStrategy, emitter, new BrokerSegmentWatcherConfig() {
+ @Override
+ public boolean isAwaitInitializationOnStart()
+ {
+ return segmentWatcherConfig.isAwaitInitializationOnStart();
+ }
+ });
+ this.baseView = baseView;
+ }
+
+ /**
+ * Since this class maintains a timeline of {@link ServerSelector} objects,
+ * this method converts and returns a new timeline of the object {@link SegmentLoadInfo}.
+ *
+ * @param dataSource dataSoruce
+ * @return timeline for the given dataSource
+ */
+ @Override
+ public VersionedIntervalTimeline<String, SegmentLoadInfo> getTimeline(DataSource dataSource)
+ {
+ String table = Iterables.getOnlyElement(dataSource.getTableNames());
+ VersionedIntervalTimeline<String, ServerSelector> timeline;
Review Comment:
nit: Lets call this baseTimeline
##########
server/src/main/java/org/apache/druid/client/QueryableCoordinatorServerView.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.client;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Iterables;
+import com.google.inject.Inject;
+import org.apache.druid.client.selector.ServerSelector;
+import org.apache.druid.client.selector.TierSelectorStrategy;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.guice.annotations.EscalatedClient;
+import org.apache.druid.guice.annotations.Smile;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.http.client.HttpClient;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.QueryToolChestWarehouse;
+import org.apache.druid.query.QueryWatcher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+import org.apache.druid.timeline.VersionedIntervalTimeline;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Map;
+
+/**
+ * ServerView of coordinator for the state of segments being loaded in the cluster.
+ *
+ * <p>This class extends {@link BrokerServerView} and implements {@link CoordinatorTimeline}.
+ * The main distinction between this class and {@link CoordinatorServerView} is the maintenance of a timeline
+ * of {@link ServerSelector} objects, while the other class stores {@link SegmentLoadInfo} object in its timeline.</p>
+ *
+ * <p>A new timeline class (implementing {@link TimelineServerView}) is required for
+ * {@link org.apache.druid.segment.metadata.CoordinatorSegmentMetadataCache}, which will run on the Coordinator.</p>
+ */
+@ManageLifecycle
+public class QueryableCoordinatorServerView extends BrokerServerView implements CoordinatorTimeline
+{
+ private final FilteredServerInventoryView baseView;
+
+ @Inject
+ public QueryableCoordinatorServerView(
+ final QueryToolChestWarehouse warehouse,
+ final QueryWatcher queryWatcher,
+ final @Smile ObjectMapper smileMapper,
+ final @EscalatedClient HttpClient httpClient,
+ FilteredServerInventoryView baseView,
+ TierSelectorStrategy tierSelectorStrategy,
+ ServiceEmitter emitter,
+ CoordinatorSegmentWatcherConfig segmentWatcherConfig
+ )
+ {
+ super(warehouse, queryWatcher, smileMapper, httpClient, baseView, tierSelectorStrategy, emitter, new BrokerSegmentWatcherConfig() {
+ @Override
+ public boolean isAwaitInitializationOnStart()
+ {
+ return segmentWatcherConfig.isAwaitInitializationOnStart();
+ }
+ });
+ this.baseView = baseView;
+ }
+
+ /**
+ * Since this class maintains a timeline of {@link ServerSelector} objects,
+ * this method converts and returns a new timeline of the object {@link SegmentLoadInfo}.
+ *
+ * @param dataSource dataSoruce
+ * @return timeline for the given dataSource
+ */
+ @Override
+ public VersionedIntervalTimeline<String, SegmentLoadInfo> getTimeline(DataSource dataSource)
+ {
+ String table = Iterables.getOnlyElement(dataSource.getTableNames());
Review Comment:
NIt: Lets use `CollectionUtils.getOnlyElement()` so that a better error message is thrown to the user.
##########
server/src/main/java/org/apache/druid/client/CoordinatorTimeline.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.client;
+
+import org.apache.druid.query.DataSource;
+import org.apache.druid.timeline.SegmentId;
+import org.apache.druid.timeline.VersionedIntervalTimeline;
+
+import java.util.Map;
+
+/**
+ * Segment timeline maintained in the coordinator.
+ */
+public interface CoordinatorTimeline extends InventoryView
+{
+ /**
+ * Retrieve timeline for a dataSource.
+ */
+ VersionedIntervalTimeline<String, SegmentLoadInfo> getTimeline(DataSource dataSource);
+
+ /**
+ * Server information for all segments in the timeline.
+ */
+ Map<SegmentId, SegmentLoadInfo> getSegmentLoadInfos();
Review Comment:
When should user use one API over the other. Example usages might help here.
Also lets rename this method to `getAllSegmentLoadInformation()`
##########
sql/src/test/java/org/apache/druid/sql/calcite/util/QueryFrameworkUtils.java:
##########
@@ -84,6 +86,7 @@
public class QueryFrameworkUtils
{
+
Review Comment:
Nit: space can be removed
##########
server/src/main/java/org/apache/druid/segment/metadata/AbstractSegmentMetadataCache.java:
##########
@@ -96,41 +89,37 @@
import java.util.stream.StreamSupport;
/**
- * Broker-side cache of segment metadata which combines segments to identify
- * datasources which become "tables" in Calcite. This cache provides the "physical"
- * metadata about a datasource which is blended with catalog "logical" metadata
- * to provide the final user-view of each datasource.
+ * An abstract class that listens for segment change events and caches segment metadata and periodically refreshes
Review Comment:
```suggestion
* An abstract class that listens for segment change events and caches segment metadata. It also periodically refreshes the segments and datasources by xxxx
```
Something like that will help the reader understand the details of this class.
##########
server/src/main/java/org/apache/druid/client/coordinator/CoordinatorClientImpl.java:
##########
@@ -107,6 +110,23 @@ public ListenableFuture<List<DataSegment>> fetchUsedSegments(String dataSource,
);
}
+ @Override
+ public ListenableFuture<List<DataSourceInformation>> fetchDataSourceInformation(Set<String> dataSources)
+ {
+ final String path = "/druid/coordinator/v1/metadata/dataSourceInformation";
+ if (null == dataSources) {
+ dataSources = new HashSet<>();
Review Comment:
Why is this null check here?
Should we throw an exception here instead of passing the empty payload to the client?
##########
server/src/main/java/org/apache/druid/client/QueryableCoordinatorServerView.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.client;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Iterables;
+import com.google.inject.Inject;
+import org.apache.druid.client.selector.ServerSelector;
+import org.apache.druid.client.selector.TierSelectorStrategy;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.guice.annotations.EscalatedClient;
+import org.apache.druid.guice.annotations.Smile;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.http.client.HttpClient;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.QueryToolChestWarehouse;
+import org.apache.druid.query.QueryWatcher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+import org.apache.druid.timeline.VersionedIntervalTimeline;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Map;
+
+/**
+ * ServerView of coordinator for the state of segments being loaded in the cluster.
+ *
+ * <p>This class extends {@link BrokerServerView} and implements {@link CoordinatorTimeline}.
+ * The main distinction between this class and {@link CoordinatorServerView} is the maintenance of a timeline
+ * of {@link ServerSelector} objects, while the other class stores {@link SegmentLoadInfo} object in its timeline.</p>
+ *
+ * <p>A new timeline class (implementing {@link TimelineServerView}) is required for
+ * {@link org.apache.druid.segment.metadata.CoordinatorSegmentMetadataCache}, which will run on the Coordinator.</p>
+ */
+@ManageLifecycle
+public class QueryableCoordinatorServerView extends BrokerServerView implements CoordinatorTimeline
+{
+ private final FilteredServerInventoryView baseView;
+
+ @Inject
+ public QueryableCoordinatorServerView(
+ final QueryToolChestWarehouse warehouse,
+ final QueryWatcher queryWatcher,
+ final @Smile ObjectMapper smileMapper,
+ final @EscalatedClient HttpClient httpClient,
+ FilteredServerInventoryView baseView,
+ TierSelectorStrategy tierSelectorStrategy,
+ ServiceEmitter emitter,
+ CoordinatorSegmentWatcherConfig segmentWatcherConfig
+ )
+ {
+ super(warehouse, queryWatcher, smileMapper, httpClient, baseView, tierSelectorStrategy, emitter, new BrokerSegmentWatcherConfig() {
+ @Override
+ public boolean isAwaitInitializationOnStart()
+ {
+ return segmentWatcherConfig.isAwaitInitializationOnStart();
+ }
+ });
+ this.baseView = baseView;
+ }
+
+ /**
+ * Since this class maintains a timeline of {@link ServerSelector} objects,
+ * this method converts and returns a new timeline of the object {@link SegmentLoadInfo}.
+ *
+ * @param dataSource dataSoruce
+ * @return timeline for the given dataSource
+ */
+ @Override
+ public VersionedIntervalTimeline<String, SegmentLoadInfo> getTimeline(DataSource dataSource)
+ {
+ String table = Iterables.getOnlyElement(dataSource.getTableNames());
+ VersionedIntervalTimeline<String, ServerSelector> timeline;
+
+ synchronized (lock) {
+ timeline = timelines.get(table);
+ }
+
+ VersionedIntervalTimeline<String, SegmentLoadInfo> newTimeline =
Review Comment:
Nit: rename this to segmentLoadInfoTimeline
##########
server/src/main/java/org/apache/druid/segment/metadata/AbstractSegmentMetadataCache.java:
##########
@@ -385,55 +378,26 @@ private void startCacheExec()
);
}
+ private void setInitialized(Stopwatch stopwatch)
Review Comment:
The method name is misleading here. One would assume that it would have something to do with stopWatch but it does entirely different things.
Could it be renamed to something else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1332609473
##########
server/src/main/java/org/apache/druid/client/QueryableCoordinatorServerView.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.client;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Iterables;
+import com.google.inject.Inject;
+import org.apache.druid.client.selector.ServerSelector;
+import org.apache.druid.client.selector.TierSelectorStrategy;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.guice.annotations.EscalatedClient;
+import org.apache.druid.guice.annotations.Smile;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.http.client.HttpClient;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.QueryToolChestWarehouse;
+import org.apache.druid.query.QueryWatcher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+import org.apache.druid.timeline.VersionedIntervalTimeline;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Map;
+
+/**
+ * ServerView of coordinator for the state of segments being loaded in the cluster.
+ *
+ * <p>This class extends {@link BrokerServerView} and implements {@link CoordinatorTimeline}.
+ * The main distinction between this class and {@link CoordinatorServerView} is the maintenance of a timeline
+ * of {@link ServerSelector} objects, while the other class stores {@link SegmentLoadInfo} object in its timeline.</p>
+ *
+ * <p>A new timeline class (implementing {@link TimelineServerView}) is required for
+ * {@link org.apache.druid.segment.metadata.CoordinatorSegmentMetadataCache}, which will run on the Coordinator.</p>
+ */
+@ManageLifecycle
+public class QueryableCoordinatorServerView extends BrokerServerView implements CoordinatorTimeline
+{
+ private final FilteredServerInventoryView baseView;
+
+ @Inject
+ public QueryableCoordinatorServerView(
+ final QueryToolChestWarehouse warehouse,
+ final QueryWatcher queryWatcher,
+ final @Smile ObjectMapper smileMapper,
+ final @EscalatedClient HttpClient httpClient,
+ FilteredServerInventoryView baseView,
+ TierSelectorStrategy tierSelectorStrategy,
+ ServiceEmitter emitter,
+ CoordinatorSegmentWatcherConfig segmentWatcherConfig
+ )
+ {
+ super(warehouse, queryWatcher, smileMapper, httpClient, baseView, tierSelectorStrategy, emitter, new BrokerSegmentWatcherConfig() {
+ @Override
+ public boolean isAwaitInitializationOnStart()
+ {
+ return segmentWatcherConfig.isAwaitInitializationOnStart();
+ }
+ });
+ this.baseView = baseView;
+ }
+
+ /**
+ * Since this class maintains a timeline of {@link ServerSelector} objects,
+ * this method converts and returns a new timeline of the object {@link SegmentLoadInfo}.
+ *
+ * @param dataSource dataSoruce
+ * @return timeline for the given dataSource
+ */
+ @Override
+ public VersionedIntervalTimeline<String, SegmentLoadInfo> getTimeline(DataSource dataSource)
+ {
+ String table = Iterables.getOnlyElement(dataSource.getTableNames());
+ VersionedIntervalTimeline<String, ServerSelector> timeline;
+
+ synchronized (lock) {
+ timeline = timelines.get(table);
+ }
+
+ VersionedIntervalTimeline<String, SegmentLoadInfo> newTimeline =
+ new VersionedIntervalTimeline<>(Comparator.naturalOrder());
+ newTimeline.addAll(
+ timeline.iterateAllObjects().stream()
+ .map(serverSelector -> new VersionedIntervalTimeline.PartitionChunkEntry<>(
+ serverSelector.getSegment().getInterval(),
+ serverSelector.getSegment().getVersion(),
+ serverSelector.getSegment().getShardSpec().createChunk(serverSelector.toSegmentLoadInfo())
+ )).iterator());
Review Comment:
How frequently is getTimeline called? Timeline can be big. Copying it is expensive and it takes extra memory too.
##########
server/src/main/java/org/apache/druid/client/CoordinatorTimeline.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.client;
+
+import org.apache.druid.query.DataSource;
+import org.apache.druid.timeline.SegmentId;
+import org.apache.druid.timeline.VersionedIntervalTimeline;
+
+import java.util.Map;
+
+/**
+ * Segment timeline maintained in the coordinator.
Review Comment:
can you add a note why one would write different implementations of this interface?
##########
extensions-core/druid-bloom-filter/src/test/java/org/apache/druid/query/aggregation/bloom/sql/BloomFilterSqlAggregatorTest.java:
##########
@@ -47,10 +47,10 @@
import org.apache.druid.segment.join.JoinableFactoryWrapper;
import org.apache.druid.segment.virtual.ExpressionVirtualColumn;
import org.apache.druid.segment.writeout.OffHeapMemorySegmentWriteOutMediumFactory;
+import org.apache.druid.server.SpecificSegmentsQuerySegmentWalker;
import org.apache.druid.sql.calcite.BaseCalciteQueryTest;
import org.apache.druid.sql.calcite.filtration.Filtration;
import org.apache.druid.sql.calcite.util.CalciteTests;
-import org.apache.druid.sql.calcite.util.SpecificSegmentsQuerySegmentWalker;
Review Comment:
what was the reason to move this class?
##########
server/src/main/java/org/apache/druid/client/QueryableCoordinatorServerView.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.client;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Iterables;
+import com.google.inject.Inject;
+import org.apache.druid.client.selector.ServerSelector;
+import org.apache.druid.client.selector.TierSelectorStrategy;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.guice.annotations.EscalatedClient;
+import org.apache.druid.guice.annotations.Smile;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.http.client.HttpClient;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.QueryToolChestWarehouse;
+import org.apache.druid.query.QueryWatcher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+import org.apache.druid.timeline.VersionedIntervalTimeline;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Map;
+
+/**
+ * ServerView of coordinator for the state of segments being loaded in the cluster.
+ *
+ * <p>This class extends {@link BrokerServerView} and implements {@link CoordinatorTimeline}.
+ * The main distinction between this class and {@link CoordinatorServerView} is the maintenance of a timeline
+ * of {@link ServerSelector} objects, while the other class stores {@link SegmentLoadInfo} object in its timeline.</p>
+ *
+ * <p>A new timeline class (implementing {@link TimelineServerView}) is required for
+ * {@link org.apache.druid.segment.metadata.CoordinatorSegmentMetadataCache}, which will run on the Coordinator.</p>
+ */
+@ManageLifecycle
+public class QueryableCoordinatorServerView extends BrokerServerView implements CoordinatorTimeline
+{
+ private final FilteredServerInventoryView baseView;
+
+ @Inject
+ public QueryableCoordinatorServerView(
+ final QueryToolChestWarehouse warehouse,
Review Comment:
why are some parameters final and some are not?
##########
server/src/main/java/org/apache/druid/segment/metadata/DataSourceInformation.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.metadata;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.segment.column.RowSignature;
+
+import java.util.Objects;
+
+/**
+ * Encapsulates information about a datasource, such as its schema.
+ */
+public class DataSourceInformation
Review Comment:
how about DataSourceSchema? Is that too specific?
##########
server/src/main/java/org/apache/druid/segment/metadata/CoordinatorSegmentMetadataCache.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.metadata;
+
+import com.google.common.collect.Sets;
+import com.google.inject.Inject;
+import org.apache.druid.client.InternalQueryConfig;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.server.QueryLifecycleFactory;
+import org.apache.druid.server.security.Escalator;
+import org.apache.druid.timeline.SegmentId;
+
+import java.io.IOException;
+import java.util.Set;
+
+/**
+ * Coordinator-side cache of segment metadata that combines segments to identify
Review Comment:
what does identifying datasource mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1375252618
##########
sql/src/main/java/org/apache/druid/sql/calcite/schema/BrokerSegmentMetadataCache.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.google.common.base.Stopwatch;
+import com.google.common.collect.Sets;
+import com.google.inject.Inject;
+import org.apache.druid.client.InternalQueryConfig;
+import org.apache.druid.client.ServerView;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.client.coordinator.CoordinatorClient;
+import org.apache.druid.common.guava.FutureUtils;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.metadata.AbstractSegmentMetadataCache;
+import org.apache.druid.segment.metadata.DataSourceInformation;
+import org.apache.druid.server.QueryLifecycleFactory;
+import org.apache.druid.server.coordination.DruidServerMetadata;
+import org.apache.druid.server.security.Escalator;
+import org.apache.druid.sql.calcite.table.DatasourceTable.PhysicalDatasourceMetadata;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Broker-side cache of segment metadata that combines segments to build
+ * datasources which become "tables" in Calcite. This cache provides the "physical"
+ * metadata about a dataSource which is blended with catalog "logical" metadata
+ * to provide the final user-view of each dataSource.
+ * <p>
+ * This class extends {@link AbstractSegmentMetadataCache} and introduces following changes,
+ * <ul>
+ * <li>The refresh mechanism includes polling the coordinator for datasource schema,
+ * and falling back to running {@link org.apache.druid.query.metadata.metadata.SegmentMetadataQuery}.</li>
+ * <li>It builds and caches {@link PhysicalDatasourceMetadata} object for the table schema</li>
+ * </ul>
+ */
+@ManageLifecycle
+public class BrokerSegmentMetadataCache extends AbstractSegmentMetadataCache<PhysicalDatasourceMetadata>
+{
+ private static final EmittingLogger log = new EmittingLogger(BrokerSegmentMetadataCache.class);
+
+ private final PhysicalDatasourceMetadataFactory dataSourceMetadataFactory;
+ private final CoordinatorClient coordinatorClient;
+
+ private final BrokerSegmentMetadataCacheConfig config;
+
+ @Inject
+ public BrokerSegmentMetadataCache(
+ final QueryLifecycleFactory queryLifecycleFactory,
+ final TimelineServerView serverView,
+ final BrokerSegmentMetadataCacheConfig config,
+ final Escalator escalator,
+ final InternalQueryConfig internalQueryConfig,
+ final ServiceEmitter emitter,
+ final PhysicalDatasourceMetadataFactory dataSourceMetadataFactory,
+ final CoordinatorClient coordinatorClient
+ )
+ {
+ super(
+ queryLifecycleFactory,
+ config,
+ escalator,
+ internalQueryConfig,
+ emitter
+ );
+ this.dataSourceMetadataFactory = dataSourceMetadataFactory;
+ this.coordinatorClient = coordinatorClient;
+ this.config = config;
+ initServerViewTimelineCallback(serverView);
+ }
+
+ private void initServerViewTimelineCallback(final TimelineServerView serverView)
+ {
+ serverView.registerTimelineCallback(
+ callbackExec,
+ new TimelineServerView.TimelineCallback()
+ {
+ @Override
+ public ServerView.CallbackAction timelineInitialized()
+ {
+ synchronized (lock) {
+ isServerViewInitialized = true;
+ lock.notifyAll();
+ }
+
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction segmentAdded(final DruidServerMetadata server, final DataSegment segment)
+ {
+ addSegment(server, segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction segmentRemoved(final DataSegment segment)
+ {
+ removeSegment(segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction serverSegmentRemoved(
+ final DruidServerMetadata server,
+ final DataSegment segment
+ )
+ {
+ removeServerSegment(server, segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+ }
+ );
+ }
+
+ /**
+ * Refreshes the set of segments in two steps:
+ * <ul>
+ * <li>Polls the coordinator for the datasource schema.</li>
+ * <li>Refreshes the remaining set of segments by executing a SegmentMetadataQuery and
+ * builds datasource schema by combining segment schema.</li>
+ * </ul>
+ *
+ * @param segmentsToRefresh segments for which the schema might have changed
+ * @param dataSourcesToRebuild datasources for which the schema might have changed
+ * @throws IOException when querying segment schema from data nodes and tasks
+ */
+ @Override
+ public void refresh(final Set<SegmentId> segmentsToRefresh, final Set<String> dataSourcesToRebuild) throws IOException
+ {
+ // query schema for all datasources, which includes,
+ // datasources explicitly marked for rebuilding
+ // datasources for the segments to be refreshed
+ // prebuilt datasources
+ final Set<String> dataSourcesToQuery = new HashSet<>(dataSourcesToRebuild);
+
+ segmentsToRefresh.forEach(segment -> dataSourcesToQuery.add(segment.getDataSource()));
+
+ dataSourcesToQuery.addAll(tables.keySet());
+
+ // Fetch datasource information from the Coordinator
+ Map<String, PhysicalDatasourceMetadata> polledDataSourceMetadata = queryDataSourceInformation(dataSourcesToQuery);
+
+ // update datasource metadata in the cache
+ polledDataSourceMetadata.forEach(this::updateDSMetadata);
+
+ // Remove segments of the datasource from refresh list for which we received schema from the Coordinator.
+ segmentsToRefresh.removeIf(segmentId -> polledDataSourceMetadata.containsKey(segmentId.getDataSource()));
+
+ Set<SegmentId> refreshed = new HashSet<>();
+
+ // Refresh the remaining segments.
+ if (!config.isDisableSegmentMetadataQueries()) {
+ refreshed = refreshSegments(segmentsToRefresh);
+ }
+
+ synchronized (lock) {
+ // Add missing segments back to the refresh list.
+ segmentsNeedingRefresh.addAll(Sets.difference(segmentsToRefresh, refreshed));
+
+ // Compute the list of datasources to rebuild tables for.
+ dataSourcesToRebuild.addAll(dataSourcesNeedingRebuild);
+ refreshed.forEach(segment -> dataSourcesToRebuild.add(segment.getDataSource()));
+
+ // Remove those datasource for which we received schema from the Coordinator.
+ dataSourcesToRebuild.removeAll(polledDataSourceMetadata.keySet());
+ dataSourcesNeedingRebuild.clear();
+ }
+
+ // Rebuild the datasources.
+ for (String dataSource : dataSourcesToRebuild) {
+ final RowSignature rowSignature = buildDataSourceRowSignature(dataSource);
+ if (rowSignature == null) {
+ log.info("datasource [%s] no longer exists, all metadata removed.", dataSource);
+ tables.remove(dataSource);
+ return;
+ }
+
+ final PhysicalDatasourceMetadata physicalDatasourceMetadata = dataSourceMetadataFactory.build(dataSource, rowSignature);
+ updateDSMetadata(dataSource, physicalDatasourceMetadata);
+ }
+ }
+
+ private Map<String, PhysicalDatasourceMetadata> queryDataSourceInformation(Set<String> dataSourcesToQuery)
+ {
+ Stopwatch stopwatch = Stopwatch.createStarted();
+
+ List<DataSourceInformation> dataSourceInformations = null;
+
+ emitter.emit(ServiceMetricEvent.builder().setMetric(
+ "metadatacache/schemaPoll/count", 1));
+ try {
+ dataSourceInformations = FutureUtils.getUnchecked(coordinatorClient.fetchDataSourceInformation(dataSourcesToQuery), true);
+ }
+ catch (Exception e) {
+ log.debug(e, "Failed to query datasource information from the Coordinator.");
+ emitter.emit(ServiceMetricEvent.builder().setMetric(
+ "metadatacache/schemaPoll/failed", 1));
+ }
+
+ emitter.emit(ServiceMetricEvent.builder().setMetric(
+ "metadatacache/schemaPoll/time",
+ stopwatch.elapsed(TimeUnit.MILLISECONDS)));
+
+ final Map<String, PhysicalDatasourceMetadata> polledDataSourceMetadata = new HashMap<>();
+
+ if (dataSourceInformations != null) {
+ dataSourceInformations.forEach(dataSourceInformation -> polledDataSourceMetadata.put(
+ dataSourceInformation.getDataSource(),
+ dataSourceMetadataFactory.build(
+ dataSourceInformation.getDataSource(),
+ dataSourceInformation.getRowSignature()
+ )
+ ));
+ }
+
+ return polledDataSourceMetadata;
+ }
+
+ private void updateDSMetadata(String dataSource, PhysicalDatasourceMetadata physicalDatasourceMetadata)
+ {
+ final PhysicalDatasourceMetadata oldTable = tables.put(dataSource, physicalDatasourceMetadata);
+ if (oldTable == null || !oldTable.getRowSignature().equals(physicalDatasourceMetadata.getRowSignature())) {
+ log.info("[%s] has new signature: %s.", dataSource, physicalDatasourceMetadata.getRowSignature());
Review Comment:
When the historical went offline, the table would have been removed from the cache, when it again came up, the datasource schema is built and the message is logged since this condition `oldTable == null` is met.
`RowSignature` being empty implies that SMQ queries succeeded but returned empty `rowSignature` for the segments? is that possible?
Not sure if this got introduced by the new changes, since we are just fetching the schema from Coordinator (which is having empty `rowSignature`) and the flow on Coordinator is same as what used to be on the Broker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1377449052
##########
docs/api-reference/legacy-metadata-api.md:
##########
@@ -116,10 +116,18 @@ Returns a list of all segments for one or more specific datasources enabled in t
Returns a list of all segments for each datasource with the full segment metadata and an extra field `overshadowed`.
+`GET /druid/coordinator/v1/metadata/segments?includeOvershadowedStatus&includeRealtimeSegments`
Review Comment:
Lets document that we need this feature to be enabled to use this param `includeRealtimeSegments`
##########
server/src/main/java/org/apache/druid/server/http/MetadataResource.java:
##########
@@ -180,26 +196,67 @@ private Response getAllUsedSegmentsWithAdditionalDetails(
.filter(dataSourceWithUsedSegments -> dataSources.contains(dataSourceWithUsedSegments.getName()))
.collect(Collectors.toList());
}
- final Stream<DataSegment> usedSegments = dataSourcesWithUsedSegments
- .stream()
- .flatMap(t -> t.getSegments().stream());
final Set<DataSegment> overshadowedSegments = dataSourcesSnapshot.getOvershadowedSegments();
+ final Set<SegmentId> segmentAlreadySeen = new HashSet<>();
+ final Stream<SegmentStatusInCluster> segmentStatus = dataSourcesWithUsedSegments
+ .stream()
+ .flatMap(t -> t.getSegments().stream())
+ .map(segment -> {
+ // The replication factor for unloaded segments is 0 as they will be unloaded soon
+ boolean isOvershadowed = overshadowedSegments.contains(segment);
+ Integer replicationFactor = isOvershadowed ? (Integer) 0
+ : coordinator.getReplicationFactor(segment.getId());
+
+ Long numRows = null;
+ if (null != coordinatorSegmentMetadataCache) {
+ AvailableSegmentMetadata availableSegmentMetadata = coordinatorSegmentMetadataCache.getAvailableSegmentMetadata(
+ segment.getDataSource(),
+ segment.getId()
+ );
+ if (null != availableSegmentMetadata) {
+ numRows = availableSegmentMetadata.getNumRows();
+ }
+ }
+ segmentAlreadySeen.add(segment.getId());
+ return new SegmentStatusInCluster(
+ segment,
+ isOvershadowed,
+ replicationFactor,
+ numRows,
+ // published segment can't be realtime
+ false
+ );
+ });
+
+ Stream<SegmentStatusInCluster> finalSegments = segmentStatus;
- final Stream<SegmentStatusInCluster> segmentStatus = usedSegments.map(segment -> {
- // The replication factor for unloaded segments is 0 as they will be unloaded soon
- boolean isOvershadowed = overshadowedSegments.contains(segment);
- Integer replicationFactor = isOvershadowed ? (Integer) 0
- : coordinator.getReplicationFactor(segment.getId());
+ // conditionally add realtime segments information
+ if (null != includeRealtimeSegments && null != coordinatorSegmentMetadataCache) {
Review Comment:
Since the realtime flag only works if the `centralizedSchemaManagement`is enabled lets document this in user facing docs and well as in the code.
In the code we should mention that we did not want to increase the payload of the broker to coordinator communication if this feature is disabled and we did not want to introduce a new feature flag on the broker so this was a trade off done. This api will be deprecated once we fully move over to coordinator to power sys.segments q's from the broker.
##########
server/src/main/java/org/apache/druid/client/SegmentLoadInfo.java:
##########
@@ -59,6 +61,15 @@ public ImmutableSegmentLoadInfo toImmutableSegmentLoadInfo()
return new ImmutableSegmentLoadInfo(segment, servers);
}
+ /**
+ * Randomly return one server from the sets of {@code servers}
+ */
+ public DruidServerMetadata pickOne()
+ {
+ synchronized (this) {
Review Comment:
Yes I kind of agree that we might hit 'IndexOutOfBoundsExceptions`
From concurrentHashMap java docs:
```
However, iterators are designed to be used by only one thread at a time.
* Bear in mind that the results of aggregate status methods including
* {@code size}, {@code isEmpty}, and {@code containsValue} are typically
* useful only when a map is not undergoing concurrent updates in other threads.
* Otherwise the results of these methods reflect transient states
* that may be adequate for monitoring or estimation purposes, but not
* for program control.
```
but since you can iterator the elements safely what you could do is
* get the position of the element that you want to compute using size checks.
* create iterator
* and then the following pseudo code
``` int index=0;
int prev_element;
for each (val:iterator){
if(index==pos){
return val;
}
prev_element=val;
index++;
}
return prev_element;
```
##########
server/src/main/java/org/apache/druid/client/SegmentLoadInfo.java:
##########
@@ -36,29 +38,46 @@ public SegmentLoadInfo(DataSegment segment)
{
Preconditions.checkNotNull(segment, "segment");
this.segment = segment;
- this.servers = Sets.newConcurrentHashSet();
Review Comment:
Let this remain a concurrent HashSet.
##########
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java:
##########
@@ -192,20 +193,21 @@ private JsonParserIterator<SegmentStatusInCluster> getMetadataSegments(
Set<String> watchedDataSources
)
{
- String query = "/druid/coordinator/v1/metadata/segments?includeOvershadowedStatus";
+ StringBuilder queryBuilder = new StringBuilder("/druid/coordinator/v1/metadata/segments?includeOvershadowedStatus&includeRealtimeSegments");
Review Comment:
lets comment here also about the nuances of includeRealtimeFlag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1372579988
##########
sql/src/main/java/org/apache/druid/sql/calcite/schema/BrokerSegmentMetadataCache.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.google.common.base.Stopwatch;
+import com.google.common.collect.Sets;
+import com.google.inject.Inject;
+import org.apache.druid.client.InternalQueryConfig;
+import org.apache.druid.client.ServerView;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.client.coordinator.CoordinatorClient;
+import org.apache.druid.common.guava.FutureUtils;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.metadata.AbstractSegmentMetadataCache;
+import org.apache.druid.segment.metadata.DataSourceInformation;
+import org.apache.druid.server.QueryLifecycleFactory;
+import org.apache.druid.server.coordination.DruidServerMetadata;
+import org.apache.druid.server.security.Escalator;
+import org.apache.druid.sql.calcite.table.DatasourceTable.PhysicalDatasourceMetadata;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Broker-side cache of segment metadata that combines segments to build
+ * datasources which become "tables" in Calcite. This cache provides the "physical"
+ * metadata about a dataSource which is blended with catalog "logical" metadata
+ * to provide the final user-view of each dataSource.
+ * <p>
+ * This class extends {@link AbstractSegmentMetadataCache} and introduces following changes,
+ * <ul>
+ * <li>The refresh mechanism includes polling the coordinator for datasource schema,
+ * and falling back to running {@link org.apache.druid.query.metadata.metadata.SegmentMetadataQuery}.</li>
+ * <li>It builds and caches {@link PhysicalDatasourceMetadata} object for the table schema</li>
+ * </ul>
+ */
+@ManageLifecycle
+public class BrokerSegmentMetadataCache extends AbstractSegmentMetadataCache<PhysicalDatasourceMetadata>
+{
+ private static final EmittingLogger log = new EmittingLogger(BrokerSegmentMetadataCache.class);
+
+ private final PhysicalDatasourceMetadataFactory dataSourceMetadataFactory;
+ private final CoordinatorClient coordinatorClient;
+
+ private final BrokerSegmentMetadataCacheConfig config;
+
+ @Inject
+ public BrokerSegmentMetadataCache(
+ final QueryLifecycleFactory queryLifecycleFactory,
+ final TimelineServerView serverView,
+ final BrokerSegmentMetadataCacheConfig config,
+ final Escalator escalator,
+ final InternalQueryConfig internalQueryConfig,
+ final ServiceEmitter emitter,
+ final PhysicalDatasourceMetadataFactory dataSourceMetadataFactory,
+ final CoordinatorClient coordinatorClient
+ )
+ {
+ super(
+ queryLifecycleFactory,
+ config,
+ escalator,
+ internalQueryConfig,
+ emitter
+ );
+ this.dataSourceMetadataFactory = dataSourceMetadataFactory;
+ this.coordinatorClient = coordinatorClient;
+ this.config = config;
+ initServerViewTimelineCallback(serverView);
+ }
+
+ private void initServerViewTimelineCallback(final TimelineServerView serverView)
+ {
+ serverView.registerTimelineCallback(
+ callbackExec,
+ new TimelineServerView.TimelineCallback()
+ {
+ @Override
+ public ServerView.CallbackAction timelineInitialized()
+ {
+ synchronized (lock) {
+ isServerViewInitialized = true;
+ lock.notifyAll();
+ }
+
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction segmentAdded(final DruidServerMetadata server, final DataSegment segment)
+ {
+ addSegment(server, segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction segmentRemoved(final DataSegment segment)
+ {
+ removeSegment(segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction serverSegmentRemoved(
+ final DruidServerMetadata server,
+ final DataSegment segment
+ )
+ {
+ removeServerSegment(server, segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+ }
+ );
+ }
+
+ /**
+ * Refreshes the set of segments in two steps:
+ * <ul>
+ * <li>Polls the coordinator for the datasource schema.</li>
+ * <li>Refreshes the remaining set of segments by executing a SegmentMetadataQuery and
+ * builds datasource schema by combining segment schema.</li>
+ * </ul>
+ *
+ * @param segmentsToRefresh segments for which the schema might have changed
+ * @param dataSourcesToRebuild datasources for which the schema might have changed
+ * @throws IOException when querying segment schema from data nodes and tasks
+ */
+ @Override
+ public void refresh(final Set<SegmentId> segmentsToRefresh, final Set<String> dataSourcesToRebuild) throws IOException
+ {
+ // query schema for all datasources, which includes,
+ // datasources explicitly marked for rebuilding
+ // datasources for the segments to be refreshed
+ // prebuilt datasources
+ final Set<String> dataSourcesToQuery = new HashSet<>(dataSourcesToRebuild);
+
+ segmentsToRefresh.forEach(segment -> dataSourcesToQuery.add(segment.getDataSource()));
+
+ dataSourcesToQuery.addAll(tables.keySet());
+
+ // Fetch datasource information from the Coordinator
+ Map<String, PhysicalDatasourceMetadata> polledDataSourceMetadata = queryDataSourceInformation(dataSourcesToQuery);
Review Comment:
That was the plan initially, but it seemed that having 2 different configs to enable the feature would be confusing.
On the other hand, the latency of querying the Coordinator is pretty low.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1334961176
##########
server/src/main/java/org/apache/druid/client/BrokerServerView.java:
##########
@@ -85,9 +81,14 @@ public class BrokerServerView implements TimelineServerView
private final ServiceEmitter emitter;
private final BrokerSegmentWatcherConfig segmentWatcherConfig;
private final Predicate<Pair<DruidServerMetadata, DataSegment>> segmentFilter;
-
private final CountDownLatch initialized = new CountDownLatch(1);
+ protected final Object lock = new Object();
+
+ protected final Map<SegmentId, ServerSelector> selectors = new HashMap<>();
Review Comment:
This is existing code. I have only changed their visibility.
Added comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1374976579
##########
server/src/main/java/org/apache/druid/segment/metadata/AbstractSegmentMetadataCache.java:
##########
@@ -385,55 +329,26 @@ private void startCacheExec()
);
}
+ private void setInitializedAndReportInitTime(Stopwatch stopwatch)
+ {
+ // report the cache init time
+ if (initialized.getCount() == 1) {
+ long elapsedTime = stopwatch.elapsed(TimeUnit.MILLISECONDS);
+ emitter.emit(ServiceMetricEvent.builder().setMetric("metadatacache/init/time", elapsedTime));
+ log.info("%s initialized in [%,d] ms.", getClass().getSimpleName(), elapsedTime);
+ stopwatch.stop();
+ }
+ initialized.countDown();
+ }
+
@LifecycleStart
public void start() throws InterruptedException
{
+ log.info("%s waiting for initialization.", getClass().getSimpleName());
startCacheExec();
if (config.isAwaitInitializationOnStart()) {
Review Comment:
why is this log message moved outside of the if?
##########
sql/src/main/java/org/apache/druid/sql/calcite/schema/BrokerSegmentMetadataCache.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.google.common.base.Stopwatch;
+import com.google.common.collect.Sets;
+import com.google.inject.Inject;
+import org.apache.druid.client.InternalQueryConfig;
+import org.apache.druid.client.ServerView;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.client.coordinator.CoordinatorClient;
+import org.apache.druid.common.guava.FutureUtils;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.metadata.AbstractSegmentMetadataCache;
+import org.apache.druid.segment.metadata.DataSourceInformation;
+import org.apache.druid.server.QueryLifecycleFactory;
+import org.apache.druid.server.coordination.DruidServerMetadata;
+import org.apache.druid.server.security.Escalator;
+import org.apache.druid.sql.calcite.table.DatasourceTable.PhysicalDatasourceMetadata;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Broker-side cache of segment metadata that combines segments to build
+ * datasources which become "tables" in Calcite. This cache provides the "physical"
+ * metadata about a dataSource which is blended with catalog "logical" metadata
+ * to provide the final user-view of each dataSource.
+ * <p>
+ * This class extends {@link AbstractSegmentMetadataCache} and introduces following changes,
+ * <ul>
+ * <li>The refresh mechanism includes polling the coordinator for datasource schema,
+ * and falling back to running {@link org.apache.druid.query.metadata.metadata.SegmentMetadataQuery}.</li>
+ * <li>It builds and caches {@link PhysicalDatasourceMetadata} object for the table schema</li>
+ * </ul>
+ */
+@ManageLifecycle
+public class BrokerSegmentMetadataCache extends AbstractSegmentMetadataCache<PhysicalDatasourceMetadata>
+{
+ private static final EmittingLogger log = new EmittingLogger(BrokerSegmentMetadataCache.class);
+
+ private final PhysicalDatasourceMetadataFactory dataSourceMetadataFactory;
+ private final CoordinatorClient coordinatorClient;
+
+ private final BrokerSegmentMetadataCacheConfig config;
+
+ @Inject
+ public BrokerSegmentMetadataCache(
+ final QueryLifecycleFactory queryLifecycleFactory,
+ final TimelineServerView serverView,
+ final BrokerSegmentMetadataCacheConfig config,
+ final Escalator escalator,
+ final InternalQueryConfig internalQueryConfig,
+ final ServiceEmitter emitter,
+ final PhysicalDatasourceMetadataFactory dataSourceMetadataFactory,
+ final CoordinatorClient coordinatorClient
+ )
+ {
+ super(
+ queryLifecycleFactory,
+ config,
+ escalator,
+ internalQueryConfig,
+ emitter
+ );
+ this.dataSourceMetadataFactory = dataSourceMetadataFactory;
+ this.coordinatorClient = coordinatorClient;
+ this.config = config;
+ initServerViewTimelineCallback(serverView);
+ }
+
+ private void initServerViewTimelineCallback(final TimelineServerView serverView)
+ {
+ serverView.registerTimelineCallback(
+ callbackExec,
+ new TimelineServerView.TimelineCallback()
+ {
+ @Override
+ public ServerView.CallbackAction timelineInitialized()
+ {
+ synchronized (lock) {
+ isServerViewInitialized = true;
+ lock.notifyAll();
+ }
+
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction segmentAdded(final DruidServerMetadata server, final DataSegment segment)
+ {
+ addSegment(server, segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction segmentRemoved(final DataSegment segment)
+ {
+ removeSegment(segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction serverSegmentRemoved(
+ final DruidServerMetadata server,
+ final DataSegment segment
+ )
+ {
+ removeServerSegment(server, segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+ }
+ );
+ }
+
+ /**
+ * Refreshes the set of segments in two steps:
+ * <ul>
+ * <li>Polls the coordinator for the datasource schema.</li>
+ * <li>Refreshes the remaining set of segments by executing a SegmentMetadataQuery and
+ * builds datasource schema by combining segment schema.</li>
+ * </ul>
+ *
+ * @param segmentsToRefresh segments for which the schema might have changed
+ * @param dataSourcesToRebuild datasources for which the schema might have changed
+ * @throws IOException when querying segment schema from data nodes and tasks
+ */
+ @Override
+ public void refresh(final Set<SegmentId> segmentsToRefresh, final Set<String> dataSourcesToRebuild) throws IOException
+ {
+ // query schema for all datasources, which includes,
+ // datasources explicitly marked for rebuilding
+ // datasources for the segments to be refreshed
+ // prebuilt datasources
+ final Set<String> dataSourcesToQuery = new HashSet<>(dataSourcesToRebuild);
+
+ segmentsToRefresh.forEach(segment -> dataSourcesToQuery.add(segment.getDataSource()));
+
+ dataSourcesToQuery.addAll(tables.keySet());
+
+ // Fetch datasource information from the Coordinator
+ Map<String, PhysicalDatasourceMetadata> polledDataSourceMetadata = queryDataSourceInformation(dataSourcesToQuery);
+
+ // update datasource metadata in the cache
+ polledDataSourceMetadata.forEach(this::updateDSMetadata);
+
+ // Remove segments of the datasource from refresh list for which we received schema from the Coordinator.
+ segmentsToRefresh.removeIf(segmentId -> polledDataSourceMetadata.containsKey(segmentId.getDataSource()));
+
+ Set<SegmentId> refreshed = new HashSet<>();
+
+ // Refresh the remaining segments.
+ if (!config.isDisableSegmentMetadataQueries()) {
+ refreshed = refreshSegments(segmentsToRefresh);
+ }
+
+ synchronized (lock) {
+ // Add missing segments back to the refresh list.
+ segmentsNeedingRefresh.addAll(Sets.difference(segmentsToRefresh, refreshed));
+
+ // Compute the list of datasources to rebuild tables for.
+ dataSourcesToRebuild.addAll(dataSourcesNeedingRebuild);
+ refreshed.forEach(segment -> dataSourcesToRebuild.add(segment.getDataSource()));
+
+ // Remove those datasource for which we received schema from the Coordinator.
+ dataSourcesToRebuild.removeAll(polledDataSourceMetadata.keySet());
+ dataSourcesNeedingRebuild.clear();
+ }
+
+ // Rebuild the datasources.
+ for (String dataSource : dataSourcesToRebuild) {
+ final RowSignature rowSignature = buildDataSourceRowSignature(dataSource);
+ if (rowSignature == null) {
+ log.info("datasource [%s] no longer exists, all metadata removed.", dataSource);
+ tables.remove(dataSource);
+ return;
+ }
+
+ final PhysicalDatasourceMetadata physicalDatasourceMetadata = dataSourceMetadataFactory.build(dataSource, rowSignature);
+ updateDSMetadata(dataSource, physicalDatasourceMetadata);
+ }
+ }
+
+ private Map<String, PhysicalDatasourceMetadata> queryDataSourceInformation(Set<String> dataSourcesToQuery)
+ {
+ Stopwatch stopwatch = Stopwatch.createStarted();
+
+ List<DataSourceInformation> dataSourceInformations = null;
+
+ emitter.emit(ServiceMetricEvent.builder().setMetric(
+ "metadatacache/schemaPoll/count", 1));
+ try {
+ dataSourceInformations = FutureUtils.getUnchecked(coordinatorClient.fetchDataSourceInformation(dataSourcesToQuery), true);
+ }
+ catch (Exception e) {
+ log.debug(e, "Failed to query datasource information from the Coordinator.");
+ emitter.emit(ServiceMetricEvent.builder().setMetric(
+ "metadatacache/schemaPoll/failed", 1));
+ }
+
+ emitter.emit(ServiceMetricEvent.builder().setMetric(
+ "metadatacache/schemaPoll/time",
+ stopwatch.elapsed(TimeUnit.MILLISECONDS)));
+
+ final Map<String, PhysicalDatasourceMetadata> polledDataSourceMetadata = new HashMap<>();
+
+ if (dataSourceInformations != null) {
+ dataSourceInformations.forEach(dataSourceInformation -> polledDataSourceMetadata.put(
+ dataSourceInformation.getDataSource(),
+ dataSourceMetadataFactory.build(
+ dataSourceInformation.getDataSource(),
+ dataSourceInformation.getRowSignature()
+ )
+ ));
+ }
+
+ return polledDataSourceMetadata;
+ }
+
+ private void updateDSMetadata(String dataSource, PhysicalDatasourceMetadata physicalDatasourceMetadata)
+ {
+ final PhysicalDatasourceMetadata oldTable = tables.put(dataSource, physicalDatasourceMetadata);
+ if (oldTable == null || !oldTable.getRowSignature().equals(physicalDatasourceMetadata.getRowSignature())) {
+ log.info("[%s] has new signature: %s.", dataSource, physicalDatasourceMetadata.getRowSignature());
Review Comment:
I haven't quite figured it out yet, or if it is related to the refactor in this PR or some pre-existing problem, but might be worth looking further into anyway...
while bouncing a historical repeatedly, so that datasources are going offline and online periodically, i keep finding log messages where it gets into a state that it is printing an empty row signature.
```
2023-10-28T03:33:46,084 INFO [DruidSchema-Cache-0] org.apache.druid.sql.calcite.schema.BrokerSegmentMetadataCache - [wikipedia-schemaless] has new signature: {}.
```
which I think must mean that it knows the table needs refreshed, but has not yet obtained any segment metadata. I'll try to see if i can determine how this happens
##########
server/src/main/java/org/apache/druid/client/SegmentLoadInfo.java:
##########
@@ -59,6 +61,15 @@ public ImmutableSegmentLoadInfo toImmutableSegmentLoadInfo()
return new ImmutableSegmentLoadInfo(segment, servers);
}
+ /**
+ * Randomly return one server from the sets of {@code servers}
+ */
+ public DruidServerMetadata pickOne()
+ {
+ synchronized (this) {
Review Comment:
hmm, this still isn't quite right i think, _everything_ needs to be synchronized if we are dropping the concurrent set. It seems simpler to add back the concurrent set, I don't think the synchronized was necessary. `Sets.newConcurrentHashSet` is backed by the keyset of a `ConcurrentHashMap`. Per javadocs,
```
Most concurrent Collection implementations (including most Queues) also differ from the usual java.util conventions in that their Iterators and Spliterators provide weakly consistent rather than fast-fail traversal:
* they may proceed concurrently with other operations
* they will never throw ConcurrentModificationException
* they are guaranteed to traverse elements as they existed upon construction exactly once, and may (but are not guaranteed to) reflect any modifications subsequent to construction.
```
So I think we can just iterate to pick one...
I don't think we need to do anything like guard from the `DruidServerMetadata` being removed from the `SegmentLoadInfo`, but if we do, then something should probably be pushing a computation into this class to do the thing so that we can do the whole operation under lock...
##########
server/src/main/java/org/apache/druid/client/coordinator/CoordinatorClientImpl.java:
##########
@@ -107,6 +110,23 @@ public ListenableFuture<List<DataSegment>> fetchUsedSegments(String dataSource,
);
}
+ @Override
+ public ListenableFuture<List<DataSourceInformation>> fetchDataSourceInformation(Set<String> dataSources)
+ {
+ final String path = "/druid/coordinator/v1/metadata/dataSourceInformation";
+ if (null == dataSources) {
+ dataSources = new HashSet<>();
Review Comment:
does this still need removed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1375241460
##########
server/src/main/java/org/apache/druid/segment/metadata/AbstractSegmentMetadataCache.java:
##########
@@ -385,55 +329,26 @@ private void startCacheExec()
);
}
+ private void setInitializedAndReportInitTime(Stopwatch stopwatch)
+ {
+ // report the cache init time
+ if (initialized.getCount() == 1) {
+ long elapsedTime = stopwatch.elapsed(TimeUnit.MILLISECONDS);
+ emitter.emit(ServiceMetricEvent.builder().setMetric("metadatacache/init/time", elapsedTime));
+ log.info("%s initialized in [%,d] ms.", getClass().getSimpleName(), elapsedTime);
+ stopwatch.stop();
+ }
+ initialized.countDown();
+ }
+
@LifecycleStart
public void start() throws InterruptedException
{
+ log.info("%s waiting for initialization.", getClass().getSimpleName());
startCacheExec();
if (config.isAwaitInitializationOnStart()) {
Review Comment:
I wanted this message to be logged irrespective of the `isAwaitInitializationOnStart` config, since this config is `false` by default on Coordinator.
I am changing the message to `%s starting cache initialization.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1334961550
##########
server/src/main/java/org/apache/druid/segment/metadata/AbstractSegmentMetadataCache.java:
##########
@@ -385,55 +378,26 @@ private void startCacheExec()
);
}
+ private void setInitialized(Stopwatch stopwatch)
Review Comment:
Renamed to `setInitializedAndReportInitTime`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1378764455
##########
server/src/main/java/org/apache/druid/client/SegmentLoadInfo.java:
##########
@@ -59,6 +61,15 @@ public ImmutableSegmentLoadInfo toImmutableSegmentLoadInfo()
return new ImmutableSegmentLoadInfo(segment, servers);
}
+ /**
+ * Randomly return one server from the sets of {@code servers}
+ */
+ public DruidServerMetadata pickOne()
+ {
+ synchronized (this) {
Review Comment:
yea, something similar to what @cryptoe suggests; `Iterators.get` is just taking the iterator returned by the collection and calling hasNext to navigate to a specific index, so we could just return the last element available if the index we picked is no longer present (though a while loop might be a bit cleaner)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1360499361
##########
.github/workflows/revised-its.yml:
##########
@@ -50,7 +50,7 @@ jobs:
matrix:
#jdk: [8, 11, 17]
jdk: [8]
- it: [HighAvailability, MultiStageQuery, Catalog, BatchIndex, MultiStageQueryWithMM, InputSource, InputFormat]
+ it: [MultiStageQuery, BatchIndex]
#indexer: [indexer, middleManager]
Review Comment:
lets revert this change before commit. I guess these are accidental.
##########
benchmarks/src/test/java/org/apache/druid/benchmark/DruidSchemaInternRowSignatureBenchmark.java:
##########
@@ -71,26 +68,21 @@ public class DruidSchemaInternRowSignatureBenchmark
{
private SegmentMetadataCacheForBenchmark cache;
- private static class SegmentMetadataCacheForBenchmark extends SegmentMetadataCache
+ private static class SegmentMetadataCacheForBenchmark extends CoordinatorSegmentMetadataCache
Review Comment:
Lets change the original one to the brokerSegmentMetadataCache and add a new one for CoordinatorSegmentMetadataCache and lets compare the results.
##########
docs/operations/metrics.md:
##########
@@ -358,6 +361,9 @@ These metrics are for the Druid Coordinator and are reset each time the Coordina
|`serverview/init/time`|Time taken to initialize the coordinator server view.||Depends on the number of segments.|
|`serverview/sync/healthy`|Sync status of the Coordinator with a segment-loading server such as a Historical or Peon. Emitted only when [HTTP-based server view](../configuration/index.md#segment-management) is enabled. You can use this metric in conjunction with `serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise|
|`serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator has been failing to sync with a segment-loading server. Emitted only when [HTTP-based server view](../configuration/index.md#segment-management) is enabled.|`server`, `tier`|Not emitted for synced servers.|
+|`metadatacache/init/time`|Time taken to initialize the coordinator segment metadata cache.|`dataSource`|Depends on the number of segments.|
+|`metadatacache/refresh/count`|Number of segments to refresh in coordinator segment metadata cache.|`dataSource`|
+|`metadatacache/refresh/time`|Time taken to refresh segments in coordinator segment metadata cache.|`dataSource`|
Review Comment:
Please also add the user facing docs for this feature.
##########
server/src/main/java/org/apache/druid/client/CoordinatorServerView.java:
##########
@@ -50,28 +57,33 @@ public class CoordinatorServerView implements InventoryView
private static final Logger log = new Logger(CoordinatorServerView.class);
private final Object lock = new Object();
-
private final Map<SegmentId, SegmentLoadInfo> segmentLoadInfos;
private final Map<String, VersionedIntervalTimeline<String, SegmentLoadInfo>> timelines;
-
+ private final ConcurrentMap<String, QueryRunner> serverQueryRunners;
Review Comment:
Can you please add java docs as to why this is a concurrent data structure.
##########
server/src/main/java/org/apache/druid/client/SegmentLoadInfo.java:
##########
@@ -59,6 +61,15 @@ public ImmutableSegmentLoadInfo toImmutableSegmentLoadInfo()
return new ImmutableSegmentLoadInfo(segment, servers);
}
+ /**
+ * Randomly return one server from the sets of {@code servers}
+ */
+ public DruidServerMetadata pickOne()
+ {
+ synchronized (this) {
Review Comment:
Why is this call made thread safe ?
##########
server/src/main/java/org/apache/druid/client/SegmentLoadInfo.java:
##########
@@ -59,6 +61,15 @@ public ImmutableSegmentLoadInfo toImmutableSegmentLoadInfo()
return new ImmutableSegmentLoadInfo(segment, servers);
}
+ /**
+ * Randomly return one server from the sets of {@code servers}
+ */
+ public DruidServerMetadata pickOne()
+ {
+ synchronized (this) {
Review Comment:
I guess since this object is used in `DataSourceResouce` we need to make it threadSafe.
##########
docs/operations/metrics.md:
##########
@@ -358,6 +361,9 @@ These metrics are for the Druid Coordinator and are reset each time the Coordina
|`serverview/init/time`|Time taken to initialize the coordinator server view.||Depends on the number of segments.|
|`serverview/sync/healthy`|Sync status of the Coordinator with a segment-loading server such as a Historical or Peon. Emitted only when [HTTP-based server view](../configuration/index.md#segment-management) is enabled. You can use this metric in conjunction with `serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise|
|`serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator has been failing to sync with a segment-loading server. Emitted only when [HTTP-based server view](../configuration/index.md#segment-management) is enabled.|`server`, `tier`|Not emitted for synced servers.|
+|`metadatacache/init/time`|Time taken to initialize the coordinator segment metadata cache.|`dataSource`|Depends on the number of segments.|
Review Comment:
I guess all of this should not contain data source as a dimension.
##########
server/src/main/java/org/apache/druid/segment/metadata/CoordinatorSegmentMetadataCache.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.metadata;
+
+import com.google.common.collect.Sets;
+import com.google.inject.Inject;
+import org.apache.druid.client.CoordinatorServerView;
+import org.apache.druid.client.InternalQueryConfig;
+import org.apache.druid.client.ServerView;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.server.QueryLifecycleFactory;
+import org.apache.druid.server.coordination.DruidServerMetadata;
+import org.apache.druid.server.security.Escalator;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+
+import java.io.IOException;
+import java.util.Set;
+
+/**
+ * Coordinator-side cache of segment metadata that combines segments to build
+ * datasources. The cache provides metadata about a datasource, see {@link DataSourceInformation}.
+ */
+@ManageLifecycle
+public class CoordinatorSegmentMetadataCache extends AbstractSegmentMetadataCache<DataSourceInformation>
+{
+ private static final EmittingLogger log = new EmittingLogger(CoordinatorSegmentMetadataCache.class);
+
+ @Inject
+ public CoordinatorSegmentMetadataCache(
+ QueryLifecycleFactory queryLifecycleFactory,
+ CoordinatorServerView serverView,
+ SegmentMetadataCacheConfig config,
+ Escalator escalator,
+ InternalQueryConfig internalQueryConfig,
+ ServiceEmitter emitter
+ )
+ {
+ super(queryLifecycleFactory, config, escalator, internalQueryConfig, emitter);
+
+ initServerViewTimelineCallback(serverView);
+ }
+
+ private void initServerViewTimelineCallback(final CoordinatorServerView serverView)
+ {
+ serverView.registerTimelineCallback(
+ callbackExec,
+ new TimelineServerView.TimelineCallback()
+ {
+ @Override
+ public ServerView.CallbackAction timelineInitialized()
+ {
+ synchronized (lock) {
+ isServerViewInitialized = true;
+ lock.notifyAll();
+ }
+
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction segmentAdded(final DruidServerMetadata server, final DataSegment segment)
+ {
+ addSegment(server, segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction segmentRemoved(final DataSegment segment)
+ {
+ removeSegment(segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction serverSegmentRemoved(
+ final DruidServerMetadata server,
+ final DataSegment segment
+ )
+ {
+ removeServerSegment(server, segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+ }
+ );
+ }
+
+ /**
+ * Fires SegmentMetadataQuery to fetch schema information for each segment in the refresh list.
+ * The schema information for individual segments is combined to construct a table schema, which is then cached.
+ *
+ * @param segmentsToRefresh segments for which the schema might have changed
+ * @param dataSourcesToRebuild datasources for which the schema might have changed
+ * @throws IOException when querying segment from data nodes and tasks
+ */
+ @Override
+ public void refresh(final Set<SegmentId> segmentsToRefresh, final Set<String> dataSourcesToRebuild) throws IOException
+ {
+ // Refresh the segments.
+ final Set<SegmentId> refreshed = refreshSegments(segmentsToRefresh);
+
+ synchronized (lock) {
+ // Add missing segments back to the refresh list.
+ segmentsNeedingRefresh.addAll(Sets.difference(segmentsToRefresh, refreshed));
+
+ // Compute the list of datasources to rebuild tables for.
+ dataSourcesToRebuild.addAll(dataSourcesNeedingRebuild);
+ refreshed.forEach(segment -> dataSourcesToRebuild.add(segment.getDataSource()));
+ dataSourcesNeedingRebuild.clear();
+ }
+
+ // Rebuild the datasources.
+ for (String dataSource : dataSourcesToRebuild) {
+ final RowSignature rowSignature = buildDruidTable(dataSource);
+ if (rowSignature == null) {
+ log.info("datasource [%s] no longer exists, all metadata removed.", dataSource);
Review Comment:
Lets change this info comment to rowSignature is empty for this data source hence removing the data source.
##########
docs/operations/metrics.md:
##########
@@ -72,6 +72,9 @@ Most metric values reset each emission period, as specified in `druid.monitoring
|`metadatacache/init/time`|Time taken to initialize the broker segment metadata cache. Useful to detect if brokers are taking too long to start||Depends on the number of segments.|
|`metadatacache/refresh/count`|Number of segments to refresh in broker segment metadata cache.|`dataSource`|
|`metadatacache/refresh/time`|Time taken to refresh segments in broker segment metadata cache.|`dataSource`|
+|`metadatacache/schemaPoll/count`|Number of coordinator polls to fetch datasource schema.|`dataSource`|
Review Comment:
Why is this at a data source level .
##########
server/src/main/java/org/apache/druid/segment/metadata/AbstractSegmentMetadataCache.java:
##########
@@ -808,7 +792,7 @@ private Set<SegmentId> refreshSegmentsForDataSource(final String dataSource, fin
@VisibleForTesting
@Nullable
- DatasourceTable.PhysicalDatasourceMetadata buildDruidTable(final String dataSource)
+ public RowSignature buildDruidTable(final String dataSource)
Review Comment:
lets rename this to buildRowSignatureForDataSource.
Please add javadocs.
##########
server/src/main/java/org/apache/druid/segment/metadata/AbstractSegmentMetadataCache.java:
##########
@@ -96,40 +87,37 @@
import java.util.stream.StreamSupport;
/**
- * Broker-side cache of segment metadata which combines segments to identify
- * datasources which become "tables" in Calcite. This cache provides the "physical"
- * metadata about a datasource which is blended with catalog "logical" metadata
- * to provide the final user-view of each datasource.
+ * An abstract class that listens for segment change events and caches segment metadata. It periodically refreshes
+ * the segments, by fetching their metadata which includes schema information from sources like
+ * data nodes, db (the logic is specificed in the child class) and builds table schema.
+ *
+ * <p>This class is generic and is parameterized by a type {@code T} that extends {@link DataSourceInformation}.</p>
+ *
+ * <p>This class has an abstract method {@link #refresh(Set, Set)} which the child class must override
+ * with the logic to build and cache table schema.</p>
+ *
+ * @param <T> The type of information associated with the data source, which must extend {@link DataSourceInformation}.
Review Comment:
```suggestion
* An abstract class that listens for segment change events and caches segment metadata in memory. It periodically refreshes
* the segments, by fetching their metadata from sources like
* data nodes, metadata db and builds table schema.
*
*
* <p>This class has an abstract method {@link #refresh(Set, Set)} which the child class must override
* with the logic to build and cache table schema.</p>
*
* @param <T> The type of information associated with the data source which must extend {@link DataSourceInformation}.
```
##########
processing/src/main/java/org/apache/druid/timeline/SegmentStatusInCluster.java:
##########
@@ -52,26 +54,40 @@ public class SegmentStatusInCluster implements Comparable<SegmentStatusInCluster
*/
@JsonUnwrapped
private final DataSegment dataSegment;
+ /**
+ * Number of rows in the segment.
+ */
+ private final Long numRows;
+ /**
+ * If the segment is realtime.
+ */
+ private final boolean realtime;
Review Comment:
@adarshsanjeev Since we added replication factor to this class for the `cold` segments, can we instead not pass a tristate enum with values `REALTIME,HISTORICAL,COLD` and remove this boolean. Names are not final though ?
##########
server/src/main/java/org/apache/druid/segment/metadata/AbstractSegmentMetadataCache.java:
##########
@@ -808,7 +792,7 @@ private Set<SegmentId> refreshSegmentsForDataSource(final String dataSource, fin
@VisibleForTesting
@Nullable
- DatasourceTable.PhysicalDatasourceMetadata buildDruidTable(final String dataSource)
+ public RowSignature buildDruidTable(final String dataSource)
Review Comment:
We should also think of getting the `aggFactories` here since it will be required for auto compaction improvements where in we can launch the job without downloading the segments if the index spec is not provided.
##########
server/src/main/java/org/apache/druid/client/SegmentLoadInfo.java:
##########
@@ -59,6 +61,15 @@ public ImmutableSegmentLoadInfo toImmutableSegmentLoadInfo()
return new ImmutableSegmentLoadInfo(segment, servers);
}
+ /**
+ * Randomly return one server from the sets of {@code servers}
+ */
+ public DruidServerMetadata pickOne()
+ {
+ synchronized (this) {
Review Comment:
We should take a lock in the adding/remove as well.
##########
sql/src/main/java/org/apache/druid/sql/calcite/schema/BrokerSegmentMetadataCache.java:
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.google.common.base.Stopwatch;
+import com.google.common.collect.Sets;
+import com.google.inject.Inject;
+import org.apache.druid.client.InternalQueryConfig;
+import org.apache.druid.client.ServerView;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.client.coordinator.CoordinatorClient;
+import org.apache.druid.common.guava.FutureUtils;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.metadata.AbstractSegmentMetadataCache;
+import org.apache.druid.server.QueryLifecycleFactory;
+import org.apache.druid.server.coordination.DruidServerMetadata;
+import org.apache.druid.server.security.Escalator;
+import org.apache.druid.sql.calcite.table.DatasourceTable.PhysicalDatasourceMetadata;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Broker-side cache of segment metadata that combines segments to build
+ * datasources which become "tables" in Calcite. This cache provides the "physical"
+ * metadata about a dataSource which is blended with catalog "logical" metadata
+ * to provide the final user-view of each dataSource.
+ * <p>
+ * This class extends {@link AbstractSegmentMetadataCache} and introduces following changes,
+ * <ul>
+ * <li>The refresh mechanism includes polling the coordinator for datasource schema,
+ * and falling back to running {@link org.apache.druid.query.metadata.metadata.SegmentMetadataQuery}.</li>
+ * <li>It builds and caches {@link PhysicalDatasourceMetadata} object for the table schema</li>
+ * </ul>
+ */
+@ManageLifecycle
+public class BrokerSegmentMetadataCache extends AbstractSegmentMetadataCache<PhysicalDatasourceMetadata>
+{
+ private static final EmittingLogger log = new EmittingLogger(BrokerSegmentMetadataCache.class);
+
+ private final PhysicalDatasourceMetadataFactory dataSourceMetadataFactory;
+ private final CoordinatorClient coordinatorClient;
+
+ private final BrokerSegmentMetadataCacheConfig config;
+
+ @Inject
+ public BrokerSegmentMetadataCache(
+ final QueryLifecycleFactory queryLifecycleFactory,
+ final TimelineServerView serverView,
+ final BrokerSegmentMetadataCacheConfig config,
+ final Escalator escalator,
+ final InternalQueryConfig internalQueryConfig,
+ final ServiceEmitter emitter,
+ final PhysicalDatasourceMetadataFactory dataSourceMetadataFactory,
+ final CoordinatorClient coordinatorClient
+ )
+ {
+ super(
+ queryLifecycleFactory,
+ config,
+ escalator,
+ internalQueryConfig,
+ emitter
+ );
+ this.dataSourceMetadataFactory = dataSourceMetadataFactory;
+ this.coordinatorClient = coordinatorClient;
+ this.config = config;
+ initServerViewTimelineCallback(serverView);
+ }
+
+ private void initServerViewTimelineCallback(final TimelineServerView serverView)
+ {
+ serverView.registerTimelineCallback(
+ callbackExec,
+ new TimelineServerView.TimelineCallback()
+ {
+ @Override
+ public ServerView.CallbackAction timelineInitialized()
+ {
+ synchronized (lock) {
+ isServerViewInitialized = true;
+ lock.notifyAll();
+ }
+
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction segmentAdded(final DruidServerMetadata server, final DataSegment segment)
+ {
+ addSegment(server, segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction segmentRemoved(final DataSegment segment)
+ {
+ removeSegment(segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction serverSegmentRemoved(
+ final DruidServerMetadata server,
+ final DataSegment segment
+ )
+ {
+ removeServerSegment(server, segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+ }
+ );
+ }
+
+ /**
+ * Refreshes the set of segments in two steps:
+ * <ul>
+ * <li>Polls the coordinator for the datasource schema.</li>
+ * <li>Refreshes the remaining set of segments by executing a SegmentMetadataQuery and
+ * builds datasource schema by combining segment schema.</li>
+ * </ul>
+ *
+ * @param segmentsToRefresh segments for which the schema might have changed
+ * @param dataSourcesToRebuild datasources for which the schema might have changed
+ * @throws IOException when querying segment schema from data nodes and tasks
+ */
+ @Override
+ public void refresh(final Set<SegmentId> segmentsToRefresh, final Set<String> dataSourcesToRebuild) throws IOException
+ {
+ // query all the datasource schema, which includes,
Review Comment:
```suggestion
// query schema for all the data sources which includes
```
##########
server/src/main/java/org/apache/druid/client/CachingClusteredClient.java:
##########
@@ -844,23 +845,23 @@ private byte[] computeQueryCacheKeyWithJoin()
}
}
- private static class TimelineConverter implements UnaryOperator<TimelineLookup<String, ServerSelector>>
+ public static class TimelineConverter<ObjectType extends Overshadowable<ObjectType>> implements UnaryOperator<TimelineLookup<String, ObjectType>>
Review Comment:
Can you please add java docs for this since this is a public class now.
##########
docs/configuration/index.md:
##########
@@ -2002,7 +2003,7 @@ The Druid SQL server is configured through the following properties on the Broke
|`druid.sql.planner.useApproximateTopN`|Whether to use approximate [TopN queries](../querying/topnquery.md) when a SQL query could be expressed as such. If false, exact [GroupBy queries](../querying/groupbyquery.md) will be used instead.|true|
|`druid.sql.planner.requireTimeCondition`|Whether to require SQL to have filter conditions on __time column so that all generated native queries will have user specified intervals. If true, all queries without filter condition on __time column will fail|false|
|`druid.sql.planner.sqlTimeZone`|Sets the default time zone for the server, which will affect how time functions and timestamp literals behave. Should be a time zone name like "America/Los_Angeles" or offset like "-08:00".|UTC|
-|`druid.sql.planner.metadataSegmentCacheEnable`|Whether to keep a cache of published segments in broker. If true, broker polls coordinator in background to get segments from metadata store and maintains a local cache. If false, coordinator's REST API will be invoked when broker needs published segments info.|false|
+|`druid.sql.planner.metadataSegmentCacheEnable`|Whether to keep a cache of published segments in broker. If true, broker polls coordinator in background to get segments from metadata store and maintains a local cache. If false, coordinator's REST API will be invoked when broker needs published segments info.|true|
Review Comment:
Lets move this as part of another PR.
##########
sql/src/main/java/org/apache/druid/sql/calcite/schema/BrokerSegmentMetadataCache.java:
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.google.common.base.Stopwatch;
+import com.google.common.collect.Sets;
+import com.google.inject.Inject;
+import org.apache.druid.client.InternalQueryConfig;
+import org.apache.druid.client.ServerView;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.client.coordinator.CoordinatorClient;
+import org.apache.druid.common.guava.FutureUtils;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.metadata.AbstractSegmentMetadataCache;
+import org.apache.druid.server.QueryLifecycleFactory;
+import org.apache.druid.server.coordination.DruidServerMetadata;
+import org.apache.druid.server.security.Escalator;
+import org.apache.druid.sql.calcite.table.DatasourceTable.PhysicalDatasourceMetadata;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Broker-side cache of segment metadata that combines segments to build
+ * datasources which become "tables" in Calcite. This cache provides the "physical"
+ * metadata about a dataSource which is blended with catalog "logical" metadata
+ * to provide the final user-view of each dataSource.
+ * <p>
+ * This class extends {@link AbstractSegmentMetadataCache} and introduces following changes,
+ * <ul>
+ * <li>The refresh mechanism includes polling the coordinator for datasource schema,
+ * and falling back to running {@link org.apache.druid.query.metadata.metadata.SegmentMetadataQuery}.</li>
+ * <li>It builds and caches {@link PhysicalDatasourceMetadata} object for the table schema</li>
+ * </ul>
+ */
+@ManageLifecycle
+public class BrokerSegmentMetadataCache extends AbstractSegmentMetadataCache<PhysicalDatasourceMetadata>
+{
+ private static final EmittingLogger log = new EmittingLogger(BrokerSegmentMetadataCache.class);
+
+ private final PhysicalDatasourceMetadataFactory dataSourceMetadataFactory;
+ private final CoordinatorClient coordinatorClient;
+
+ private final BrokerSegmentMetadataCacheConfig config;
+
+ @Inject
+ public BrokerSegmentMetadataCache(
+ final QueryLifecycleFactory queryLifecycleFactory,
+ final TimelineServerView serverView,
+ final BrokerSegmentMetadataCacheConfig config,
+ final Escalator escalator,
+ final InternalQueryConfig internalQueryConfig,
+ final ServiceEmitter emitter,
+ final PhysicalDatasourceMetadataFactory dataSourceMetadataFactory,
+ final CoordinatorClient coordinatorClient
+ )
+ {
+ super(
+ queryLifecycleFactory,
+ config,
+ escalator,
+ internalQueryConfig,
+ emitter
+ );
+ this.dataSourceMetadataFactory = dataSourceMetadataFactory;
+ this.coordinatorClient = coordinatorClient;
+ this.config = config;
+ initServerViewTimelineCallback(serverView);
+ }
+
+ private void initServerViewTimelineCallback(final TimelineServerView serverView)
+ {
+ serverView.registerTimelineCallback(
+ callbackExec,
+ new TimelineServerView.TimelineCallback()
+ {
+ @Override
+ public ServerView.CallbackAction timelineInitialized()
+ {
+ synchronized (lock) {
+ isServerViewInitialized = true;
+ lock.notifyAll();
+ }
+
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction segmentAdded(final DruidServerMetadata server, final DataSegment segment)
+ {
+ addSegment(server, segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction segmentRemoved(final DataSegment segment)
+ {
+ removeSegment(segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction serverSegmentRemoved(
+ final DruidServerMetadata server,
+ final DataSegment segment
+ )
+ {
+ removeServerSegment(server, segment);
+ return ServerView.CallbackAction.CONTINUE;
+ }
+ }
+ );
+ }
+
+ /**
+ * Refreshes the set of segments in two steps:
+ * <ul>
+ * <li>Polls the coordinator for the datasource schema.</li>
+ * <li>Refreshes the remaining set of segments by executing a SegmentMetadataQuery and
+ * builds datasource schema by combining segment schema.</li>
+ * </ul>
+ *
+ * @param segmentsToRefresh segments for which the schema might have changed
+ * @param dataSourcesToRebuild datasources for which the schema might have changed
+ * @throws IOException when querying segment schema from data nodes and tasks
+ */
+ @Override
+ public void refresh(final Set<SegmentId> segmentsToRefresh, final Set<String> dataSourcesToRebuild) throws IOException
+ {
+ // query all the datasource schema, which includes,
+ // datasources explicitly marked for rebuilding
+ // datasources for the segments to be refreshed
+ // prebuilt datasources
+ final Set<String> dataSourcesToQuery = new HashSet<>(dataSourcesToRebuild);
+
+ segmentsToRefresh.forEach(segment -> dataSourcesToQuery.add(segment.getDataSource()));
+
+ dataSourcesToQuery.addAll(tables.keySet());
+
+ // Fetch datasource information from the Coordinator
+ Map<String, PhysicalDatasourceMetadata> polledDataSourceMetadata = queryDataSourceInformation(dataSourcesToQuery);
+
+ // update datasource metadata in the cache
+ polledDataSourceMetadata.forEach(this::updateDSMetadata);
+
+ // Remove segments of the datasource from refresh list for which we received schema from the Coordinator.
+ segmentsToRefresh.removeIf(segmentId -> polledDataSourceMetadata.containsKey(segmentId.getDataSource()));
+
+ Set<SegmentId> refreshed = new HashSet<>();
+
+ // Refresh the remaining segments.
+ if (!config.isDisableSegmentMetadataQueries()) {
+ refreshed = refreshSegments(segmentsToRefresh);
+ }
+
+ synchronized (lock) {
+ // Add missing segments back to the refresh list.
+ segmentsNeedingRefresh.addAll(Sets.difference(segmentsToRefresh, refreshed));
+
+ // Compute the list of datasources to rebuild tables for.
+ dataSourcesToRebuild.addAll(dataSourcesNeedingRebuild);
+ refreshed.forEach(segment -> dataSourcesToRebuild.add(segment.getDataSource()));
+
+ // Remove those datasource for which we received schema from the Coordinator.
+ dataSourcesToRebuild.removeAll(polledDataSourceMetadata.keySet());
+ dataSourcesNeedingRebuild.clear();
+ }
+
+ // Rebuild the datasources.
+ for (String dataSource : dataSourcesToRebuild) {
+ final RowSignature rowSignature = buildDruidTable(dataSource);
+ if (rowSignature == null) {
+ log.info("datasource [%s] no longer exists, all metadata removed.", dataSource);
+ tables.remove(dataSource);
+ return;
+ }
+
+ final PhysicalDatasourceMetadata physicalDatasourceMetadata = dataSourceMetadataFactory.build(dataSource, rowSignature);
+ updateDSMetadata(dataSource, physicalDatasourceMetadata);
+ }
+ }
+
+ private Map<String, PhysicalDatasourceMetadata> queryDataSourceInformation(Set<String> dataSourcesToQuery)
+ {
+ final Map<String, PhysicalDatasourceMetadata> polledDataSourceMetadata = new HashMap<>();
+
+ Stopwatch stopwatch = Stopwatch.createStarted();
+
+ try {
+ emitter.emit(ServiceMetricEvent.builder().setMetric(
+ "metadatacache/schemaPoll/count", 1));
+ FutureUtils.getUnchecked(coordinatorClient.fetchDataSourceInformation(dataSourcesToQuery), true)
+ .forEach(dataSourceInformation -> polledDataSourceMetadata.put(
+ dataSourceInformation.getDataSource(),
+ dataSourceMetadataFactory.build(
+ dataSourceInformation.getDataSource(),
+ dataSourceInformation.getRowSignature()
+ )
+ ));
+ }
+ catch (Exception e) {
+ log.warn(e, "Failed to query datasource information from the Coordinator.");
+ emitter.emit(ServiceMetricEvent.builder().setMetric(
+ "metadatacache/schemaPoll/failed", 1));
+ }
+
+ emitter.emit(ServiceMetricEvent.builder().setMetric(
Review Comment:
poll time should be before we iterate over the results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1378769828
##########
server/src/main/java/org/apache/druid/client/SegmentLoadInfo.java:
##########
@@ -59,6 +61,15 @@ public ImmutableSegmentLoadInfo toImmutableSegmentLoadInfo()
return new ImmutableSegmentLoadInfo(segment, servers);
}
+ /**
+ * Randomly return one server from the sets of {@code servers}
+ */
+ public DruidServerMetadata pickOne()
+ {
+ synchronized (this) {
Review Comment:
>can you please clarify this statement?
I mean that since we just pick and return a server and release the lock, nothing technically prevents the segment from being removed from the server while the code that picked a server holds reference to it. I don't think that is very problematic, but if it was for some reason, we would need to do whatever we need to do with the server under the lock (load/drop would need to interact with the same locks if not already i think to truly prevent these sorts of issues)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe merged PR #14985:
URL: https://github.com/apache/druid/pull/14985
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1373120457
##########
sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java:
##########
@@ -302,13 +303,30 @@ public Enumerable<Object[]> scan(DataContext root)
final DataSegment segment = val.getDataSegment();
segmentsAlreadySeen.add(segment.getId());
final PartialSegmentData partialSegmentData = partialSegmentDataMap.get(segment.getId());
- long numReplicas = 0L, numRows = 0L, isRealtime = 0L, isAvailable = 0L;
+ long numReplicas = 0L, numRows = 0L, isRealtime, isAvailable = 0L;
+
if (partialSegmentData != null) {
numReplicas = partialSegmentData.getNumReplicas();
- numRows = partialSegmentData.getNumRows();
isAvailable = partialSegmentData.isAvailable();
- isRealtime = partialSegmentData.isRealtime();
+ numRows = partialSegmentData.getNumRows();
+ }
+
+ // If table schema building is enabled on the Coordinator, SegmentMetadataCache on the
+ // broker might have outdated or no information regarding numRows and rowSignature for a segment.
+ // We should use {@code numRows} from the segment polled from the coordinator.
+ if (null != val.getNumRows()) {
+ numRows = val.getNumRows();
}
+
+ isRealtime = Boolean.TRUE.equals(val.isRealtime()) ? 1 : 0;
+
+ // set of segments returned from Coordinator include published and realtime segments
+ // so realtime segments are not published and vice versa
+ boolean isPublished = !val.isRealtime();
Review Comment:
Thanks for pointing out, simplified it.
I guess earlier when I put up the draft changes this field was nullable, but that is no longer the case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1372566671
##########
sql/src/main/java/org/apache/druid/sql/calcite/schema/BrokerSegmentMetadataCacheConfig.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.segment.metadata.SegmentMetadataCacheConfig;
+import org.joda.time.Period;
+
+/**
+ * Broker-side configuration class for managing segment polling from the Coordinator and
+ * customizing properties related to the SegmentMetadata cache which is used to infer datasources for SQL.
+ *
+ * <p>See {@link BrokerSegmentMetadataCache}, {@link MetadataSegmentView}.</p>
+ *
+ * <p>This class shares the same config root as {@link org.apache.druid.sql.calcite.planner.PlannerConfig}
+ * to maintain backward compatibility for when the properties here resided in {@code PlannerConfig}.</p>
+ *
+ * <p> The property {@link #awaitInitializationOnStart} is overridden in this class with a default value
+ * of {@code true}, which differs from the parent class. This ensures that the SegmentMetadata cache is
+ * fully initialized before other startup processes proceed.</p>
+ */
+public class BrokerSegmentMetadataCacheConfig extends SegmentMetadataCacheConfig
+{
+ // A flag indicating whether to cache polled segments from the Coordinator.
+ @JsonProperty
+ private boolean metadataSegmentCacheEnable = false;
+
+ // Interval for polling segments from the coordinator.
+ @JsonProperty
+ private long metadataSegmentPollPeriod = 60000;
+
+ // A flag indicating whether to wait for cache initialization during startup.
+ @JsonProperty
+ private boolean awaitInitializationOnStart = true;
+
+ @JsonProperty
+ private boolean disableSegmentMetadataQueries = false;
Review Comment:
got it, could you leave a note about that in comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1334962011
##########
server/src/main/java/org/apache/druid/client/QueryableCoordinatorServerView.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.client;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Iterables;
+import com.google.inject.Inject;
+import org.apache.druid.client.selector.ServerSelector;
+import org.apache.druid.client.selector.TierSelectorStrategy;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.guice.annotations.EscalatedClient;
+import org.apache.druid.guice.annotations.Smile;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.http.client.HttpClient;
+import org.apache.druid.query.DataSource;
+import org.apache.druid.query.QueryToolChestWarehouse;
+import org.apache.druid.query.QueryWatcher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+import org.apache.druid.timeline.VersionedIntervalTimeline;
+import org.apache.druid.utils.CollectionUtils;
+
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Map;
+
+/**
+ * ServerView of coordinator for the state of segments being loaded in the cluster.
+ *
+ * <p>This class extends {@link BrokerServerView} and implements {@link CoordinatorTimeline}.
+ * The main distinction between this class and {@link CoordinatorServerView} is the maintenance of a timeline
+ * of {@link ServerSelector} objects, while the other class stores {@link SegmentLoadInfo} object in its timeline.</p>
+ *
+ * <p>A new timeline class (implementing {@link TimelineServerView}) is required for
+ * {@link org.apache.druid.segment.metadata.CoordinatorSegmentMetadataCache}, which will run on the Coordinator.</p>
+ */
+@ManageLifecycle
+public class QueryableCoordinatorServerView extends BrokerServerView implements CoordinatorTimeline
+{
+ private final FilteredServerInventoryView baseView;
+
+ @Inject
+ public QueryableCoordinatorServerView(
+ final QueryToolChestWarehouse warehouse,
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "github-code-scanning[bot] (via GitHub)" <gi...@apache.org>.
github-code-scanning[bot] commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1359403691
##########
server/src/test/java/org/apache/druid/segment/metadata/TestSegmentMetadataQueryWalker.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.metadata;
+
+import org.apache.druid.client.CoordinatorServerView;
+import org.apache.druid.guice.http.DruidHttpClientConfig;
+import org.apache.druid.java.util.common.Pair;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.guava.FunctionalIterable;
+import org.apache.druid.java.util.common.guava.Sequence;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.query.BySegmentQueryRunner;
+import org.apache.druid.query.FinalizeResultsQueryRunner;
+import org.apache.druid.query.QueryPlus;
+import org.apache.druid.query.QueryRunner;
+import org.apache.druid.query.QueryRunnerFactory;
+import org.apache.druid.query.QueryRunnerFactoryConglomerate;
+import org.apache.druid.query.QueryToolChest;
+import org.apache.druid.query.QueryToolChestWarehouse;
+import org.apache.druid.query.SegmentDescriptor;
+import org.apache.druid.query.context.ResponseContext;
+import org.apache.druid.segment.QueryableIndex;
+import org.apache.druid.segment.QueryableIndexSegment;
+import org.apache.druid.server.initialization.ServerConfig;
+import org.apache.druid.timeline.DataSegment;
+
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class TestSegmentMetadataQueryWalker extends SegmentMetadataQuerySegmentWalker
+{
+ private QueryRunnerFactoryConglomerate conglomerate;
+ private Map<SegmentDescriptor, Pair<QueryableIndex, DataSegment>> queryableIndexMap;
+
+ public TestSegmentMetadataQueryWalker(
+ CoordinatorServerView serverView,
+ DruidHttpClientConfig httpClientConfig,
+ QueryToolChestWarehouse warehouse,
+ ServerConfig serverConfig,
+ ServiceEmitter emitter,
+ QueryRunnerFactoryConglomerate conglomerate,
+ Map<SegmentDescriptor, Pair<QueryableIndex, DataSegment>> queryableIndexMap
+ )
+ {
+ super(
+ serverView,
+ httpClientConfig,
+ warehouse,
+ serverConfig,
+ emitter
+ );
+ this.conglomerate = conglomerate;
+ this.queryableIndexMap = queryableIndexMap;
+ }
+
+ public void add(DataSegment segment, QueryableIndex index)
+ {
+ queryableIndexMap.put(segment.toDescriptor(), Pair.of(index, segment));
+ }
+
+ @Override
+ <T> Sequence<T> getServerResults(
+ QueryRunner serverRunner,
+ QueryPlus<T> queryPlus,
+ ResponseContext responseContext,
+ long maxQueuedBytesPerServer,
+ List<SegmentDescriptor> segmentDescriptors
+ )
+ {
+ QueryRunnerFactory factory = conglomerate.findFactory(queryPlus.getQuery());
+ QueryToolChest toolChest = factory.getToolchest();
+
+ return new FinalizeResultsQueryRunner<>(
+ toolChest.mergeResults(
+ factory.mergeRunners(
+ Execs.directExecutor(),
+ FunctionalIterable
+ .create(segmentDescriptors)
+ .transform(
+ segment ->
+ new BySegmentQueryRunner<>(
+ queryableIndexMap.get(segment).rhs.getId(),
+ queryableIndexMap.get(segment).rhs.getInterval().getStart(),
+ factory.createRunner(
+ new QueryableIndexSegment(
+ queryableIndexMap.get(segment).lhs,
+ queryableIndexMap.get(segment).rhs.getId()))
+ )
+ )
+ )
Review Comment:
## Deprecated method or constructor invocation
Invoking [QueryRunnerFactory.mergeRunners](1) should be avoided because it has been deprecated.
[Show more details](https://github.com/apache/druid/security/code-scanning/5902)
##########
sql/src/test/java/org/apache/druid/sql/calcite/schema/BrokerSegmentMetadataCacheConcurrencyTest.java:
##########
@@ -0,0 +1,502 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Sets;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.BrokerServerView;
+import org.apache.druid.client.DirectDruidClient;
+import org.apache.druid.client.DirectDruidClientFactory;
+import org.apache.druid.client.DruidServer;
+import org.apache.druid.client.FilteredServerInventoryView;
+import org.apache.druid.client.FilteringSegmentCallback;
+import org.apache.druid.client.InternalQueryConfig;
+import org.apache.druid.client.ServerView;
+import org.apache.druid.client.TimelineServerView;
+import org.apache.druid.client.coordinator.NoopCoordinatorClient;
+import org.apache.druid.client.selector.HighestPriorityTierSelectorStrategy;
+import org.apache.druid.client.selector.RandomServerSelectorStrategy;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.NonnullPair;
+import org.apache.druid.java.util.common.Pair;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.query.TableDataSource;
+import org.apache.druid.query.aggregation.CountAggregatorFactory;
+import org.apache.druid.query.aggregation.DoubleSumAggregatorFactory;
+import org.apache.druid.segment.IndexBuilder;
+import org.apache.druid.segment.QueryableIndex;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.segment.incremental.IncrementalIndexSchema;
+import org.apache.druid.segment.join.MapJoinableFactory;
+import org.apache.druid.segment.metadata.AbstractSegmentMetadataCache;
+import org.apache.druid.segment.metadata.AvailableSegmentMetadata;
+import org.apache.druid.segment.writeout.OffHeapMemorySegmentWriteOutMediumFactory;
+import org.apache.druid.server.coordination.DruidServerMetadata;
+import org.apache.druid.server.coordination.ServerType;
+import org.apache.druid.server.metrics.NoopServiceEmitter;
+import org.apache.druid.server.security.NoopEscalator;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.SegmentId;
+import org.apache.druid.timeline.partition.NumberedShardSpec;
+import org.easymock.EasyMock;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+
+public class BrokerSegmentMetadataCacheConcurrencyTest extends BrokerSegmentMetadataCacheCommon
+{
+ private static final String DATASOURCE = "datasource";
+ static final BrokerSegmentMetadataCacheConfig SEGMENT_CACHE_CONFIG_DEFAULT = BrokerSegmentMetadataCacheConfig.create("PT1S");
+ private File tmpDir;
+ private TestServerInventoryView inventoryView;
+ private BrokerServerView serverView;
+ private AbstractSegmentMetadataCache schema;
+ private ExecutorService exec;
+
+ @Before
+ @Override
+ public void setUp() throws Exception
+ {
+ super.setUp();
+ tmpDir = temporaryFolder.newFolder();
+ // walker = new SpecificSegmentsQuerySegmentWalker(conglomerate);
+ inventoryView = new TestServerInventoryView();
+ serverView = newBrokerServerView(inventoryView);
+ inventoryView.init();
+ serverView.awaitInitialization();
+ exec = Execs.multiThreaded(4, "DruidSchemaConcurrencyTest-%d");
+ }
+
+ @After
+ @Override
+ public void tearDown() throws Exception
+ {
+ super.tearDown();
+ exec.shutdownNow();
+ }
+
+ /**
+ * This tests the contention between three components, {@link AbstractSegmentMetadataCache},
+ * {@code InventoryView}, and {@link BrokerServerView}. It first triggers
+ * refreshing {@code SegmentMetadataCache}. To mimic some heavy work done with
+ * {@link AbstractSegmentMetadataCache#lock}, {@link AbstractSegmentMetadataCache#buildDruidTable}
+ * is overridden to sleep before doing real work. While refreshing
+ * {@code SegmentMetadataCache}, more new segments are added to
+ * {@code InventoryView}, which triggers updates of {@code BrokerServerView}.
+ * Finally, while {@code BrokerServerView} is updated,
+ * {@link BrokerServerView#getTimeline} is continuously called to mimic user query
+ * processing. All these calls must return without heavy contention.
+ */
+ @Test(timeout = 30000L)
+ public void testSegmentMetadataRefreshAndInventoryViewAddSegmentAndBrokerServerViewGetTimeline()
+ throws InterruptedException, ExecutionException, TimeoutException
+ {
+ schema = new BrokerSegmentMetadataCache(
+ getQueryLifecycleFactory(walker),
+ serverView,
+ SEGMENT_CACHE_CONFIG_DEFAULT,
+ new NoopEscalator(),
+ new InternalQueryConfig(),
+ new NoopServiceEmitter(),
+ new PhysicalDatasourceMetadataFactory(new MapJoinableFactory(ImmutableSet.of(), ImmutableMap.of()), segmentManager),
+ new NoopCoordinatorClient()
+ )
+ {
+ @Override
+ public RowSignature buildDruidTable(final String dataSource)
+ {
+ doInLock(() -> {
+ try {
+ // Mimic some heavy work done in lock in DruidSchema
+ Thread.sleep(5000);
+ }
+ catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ return super.buildDruidTable(dataSource);
+ }
+ };
+
+ int numExistingSegments = 100;
+ int numServers = 19;
+ CountDownLatch segmentLoadLatch = new CountDownLatch(numExistingSegments);
+ serverView.registerTimelineCallback(
+ Execs.directExecutor(),
+ new TimelineServerView.TimelineCallback()
+ {
+ @Override
+ public ServerView.CallbackAction timelineInitialized()
+ {
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction segmentAdded(DruidServerMetadata server, DataSegment segment)
+ {
+ segmentLoadLatch.countDown();
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction segmentRemoved(DataSegment segment)
+ {
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction serverSegmentRemoved(DruidServerMetadata server, DataSegment segment)
+ {
+ return ServerView.CallbackAction.CONTINUE;
+ }
+ }
+ );
+ addSegmentsToCluster(0, numServers, numExistingSegments);
+ // Wait for all segments to be loaded in BrokerServerView
+ Assert.assertTrue(segmentLoadLatch.await(5, TimeUnit.SECONDS));
+
+ // Trigger refresh of DruidSchema. This will internally run the heavy work
+ // mimicked by the overridden buildDruidTable
+ Future<?> refreshFuture = exec.submit(() -> {
+ schema.refresh(
+ walker.getSegments().stream().map(DataSegment::getId).collect(Collectors.toSet()),
+ Sets.newHashSet(DATASOURCE)
+ );
+ return null;
+ });
+
+ // Trigger updates of BrokerServerView. This should be done asynchronously.
+ addSegmentsToCluster(numExistingSegments, numServers, 50); // add completely new segments
+ addReplicasToCluster(1, numServers, 30); // add replicas of the first 30 segments.
+ // for the first 30 segments, we will still have replicas.
+ // for the other 20 segments, they will be completely removed from the cluster.
+ removeSegmentsFromCluster(numServers, 50);
+ Assert.assertFalse(refreshFuture.isDone());
+
+ for (int i = 0; i < 1000; i++) {
+ boolean hasTimeline = exec.submit(
+ () -> serverView.getTimeline((new TableDataSource(DATASOURCE)).getAnalysis())
+ .isPresent()
+ ).get(100, TimeUnit.MILLISECONDS);
+ Assert.assertTrue(hasTimeline);
+ // We want to call getTimeline while BrokerServerView is being updated. Sleep might help with timing.
+ Thread.sleep(2);
+ }
+
+ refreshFuture.get(10, TimeUnit.SECONDS);
+ }
+
+ /**
+ * This tests the contention between two methods of {@link AbstractSegmentMetadataCache}:
+ * {@link AbstractSegmentMetadataCache#refresh} and
+ * {@link AbstractSegmentMetadataCache#getSegmentMetadataSnapshot()}. It first triggers
+ * refreshing {@code SegmentMetadataCache}. To mimic some heavy work done with
+ * {@link AbstractSegmentMetadataCache#lock}, {@link AbstractSegmentMetadataCache#buildDruidTable}
+ * is overridden to sleep before doing real work. While refreshing
+ * {@code SegmentMetadataCache}, {@code getSegmentMetadataSnapshot()} is continuously
+ * called to mimic reading the segments table of SystemSchema. All these calls
+ * must return without heavy contention.
+ */
+ @Test(timeout = 30000L)
+ public void testSegmentMetadataRefreshAndDruidSchemaGetSegmentMetadata()
+ throws InterruptedException, ExecutionException, TimeoutException
+ {
+ schema = new BrokerSegmentMetadataCache(
+ getQueryLifecycleFactory(walker),
+ serverView,
+ SEGMENT_CACHE_CONFIG_DEFAULT,
+ new NoopEscalator(),
+ new InternalQueryConfig(),
+ new NoopServiceEmitter(),
+ new PhysicalDatasourceMetadataFactory(new MapJoinableFactory(ImmutableSet.of(), ImmutableMap.of()), segmentManager),
+ new NoopCoordinatorClient()
+ )
+ {
+ @Override
+ public RowSignature buildDruidTable(final String dataSource)
+ {
+ doInLock(() -> {
+ try {
+ // Mimic some heavy work done in lock in SegmentMetadataCache
+ Thread.sleep(5000);
+ }
+ catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ return super.buildDruidTable(dataSource);
+ }
+ };
+
+ int numExistingSegments = 100;
+ int numServers = 19;
+ CountDownLatch segmentLoadLatch = new CountDownLatch(numExistingSegments);
+ serverView.registerTimelineCallback(
+ Execs.directExecutor(),
+ new TimelineServerView.TimelineCallback()
+ {
+ @Override
+ public ServerView.CallbackAction timelineInitialized()
+ {
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction segmentAdded(DruidServerMetadata server, DataSegment segment)
+ {
+ segmentLoadLatch.countDown();
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction segmentRemoved(DataSegment segment)
+ {
+ return ServerView.CallbackAction.CONTINUE;
+ }
+
+ @Override
+ public ServerView.CallbackAction serverSegmentRemoved(DruidServerMetadata server, DataSegment segment)
+ {
+ return ServerView.CallbackAction.CONTINUE;
+ }
+ }
+ );
+ addSegmentsToCluster(0, numServers, numExistingSegments);
+ // Wait for all segments to be loaded in BrokerServerView
+ Assert.assertTrue(segmentLoadLatch.await(5, TimeUnit.SECONDS));
+
+ // Trigger refresh of SegmentMetadataCache. This will internally run the heavy work mimicked
+ // by the overridden buildDruidTable
+ Future<?> refreshFuture = exec.submit(() -> {
+ schema.refresh(
+ walker.getSegments().stream().map(DataSegment::getId).collect(Collectors.toSet()),
+ Sets.newHashSet(DATASOURCE)
+ );
+ return null;
+ });
+ Assert.assertFalse(refreshFuture.isDone());
+
+ for (int i = 0; i < 1000; i++) {
+ Map<SegmentId, AvailableSegmentMetadata> segmentsMetadata = exec.submit(
+ () -> schema.getSegmentMetadataSnapshot()
+ ).get(100, TimeUnit.MILLISECONDS);
+ Assert.assertFalse(segmentsMetadata.isEmpty());
+ // We want to call getTimeline while refreshing. Sleep might help with timing.
+ Thread.sleep(2);
+ }
+
+ refreshFuture.get(10, TimeUnit.SECONDS);
+ }
+
+ private void addSegmentsToCluster(int partitionIdStart, int numServers, int numSegments)
+ {
+ for (int i = 0; i < numSegments; i++) {
+ DataSegment segment = newSegment(i + partitionIdStart);
+ QueryableIndex index = newQueryableIndex(i + partitionIdStart);
Review Comment:
## Unread local variable
Variable 'QueryableIndex index' is never read.
[Show more details](https://github.com/apache/druid/security/code-scanning/5901)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Relocating Table Schema Building: Shifting from Brokers to Coordinator for Improved Efficiency (druid)
Posted by "findingrish (via GitHub)" <gi...@apache.org>.
findingrish commented on code in PR #14985:
URL: https://github.com/apache/druid/pull/14985#discussion_r1334961176
##########
server/src/main/java/org/apache/druid/client/BrokerServerView.java:
##########
@@ -85,9 +81,14 @@ public class BrokerServerView implements TimelineServerView
private final ServiceEmitter emitter;
private final BrokerSegmentWatcherConfig segmentWatcherConfig;
private final Predicate<Pair<DruidServerMetadata, DataSegment>> segmentFilter;
-
private final CountDownLatch initialized = new CountDownLatch(1);
+ protected final Object lock = new Object();
+
+ protected final Map<SegmentId, ServerSelector> selectors = new HashMap<>();
Review Comment:
This is existing code. I have only changed their visibility.
Added comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org