You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2025/11/30 12:28:52 UTC
[PR] Add system tables support with registry and broker wiring [pinot]
xiangfu0 opened a new pull request, #17291:
URL: https://github.com/apache/pinot/pull/17291
## Summary
- Introduce a system table SPI (request, response, filter, provider, config utils) in pinot-spi.
- Add a registry and two providers (system.tables, system.instances) plus unit and integration tests.
- Wire the broker to expose system tables via a shared registry; update request handlers and admin client to support system queries.
- Include table size fetching and table metadata surface in the tables system table; add database utils support and tests.
## Testing
- mvn -pl pinot-broker -am -DskipTests compile
- Added unit tests: TablesSystemTableProviderTest, SystemTableBrokerRequestHandlerTest
- Added integration test: SystemTableIntegrationTest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2649183195
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,740 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.databind.JsonNode;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.datasource.InMemorySystemTableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableDataProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final String PINOT_ADMIN_CLIENT_CLASS_NAME = "org.apache.pinot.client.admin.PinotAdminClient";
Review Comment:
The string literal for the class name creates tight coupling and is error-prone. Consider using `PinotAdminClient.class.getName()` instead to benefit from compile-time validation and refactoring safety. If the class might not be on the classpath in some deployments, document this assumption clearly.
##########
pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/admin/PinotSegmentAdminClient.java:
##########
@@ -36,6 +38,10 @@ public class PinotSegmentAdminClient {
private final String _controllerAddress;
private final Map<String, String> _headers;
+ private static String encodePathSegment(String segment) {
+ return URLEncoder.encode(segment, StandardCharsets.UTF_8);
Review Comment:
The method name `encodePathSegment` suggests path segment encoding (RFC 3986), but `URLEncoder.encode` uses form URL encoding (application/x-www-form-urlencoded) which encodes spaces as `+` instead of `%20`. For proper path segment encoding, consider using `URLEncoder.encode(...).replace("+", "%20")` or a URI builder utility that handles path segments correctly.
```suggestion
return URLEncoder.encode(segment, StandardCharsets.UTF_8).replace("+", "%20");
```
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BrokerRequestHandlerDelegate.java:
##########
@@ -43,27 +44,32 @@
/**
* {@code BrokerRequestHandlerDelegate} delegates the inbound broker request to one of the enabled
- * {@link BrokerRequestHandler} based on the requested handle type.
- *
- * {@see: @CommonConstant
+ * {@link BrokerRequestHandler} implementations based on the request type.
*/
public class BrokerRequestHandlerDelegate implements BrokerRequestHandler {
+ private static final Pattern SYSTEM_TABLE_QUERY_PATTERN =
+ Pattern.compile("\\b(from|join)\\s+system\\.", Pattern.CASE_INSENSITIVE);
Review Comment:
The regex pattern for detecting system table queries is fragile and could produce false positives (e.g., in string literals or comments) or false negatives (e.g., with unusual whitespace or subqueries). Consider extracting the table name from the parsed PinotQuery/SqlNodeAndOptions instead of pattern matching on raw SQL text for more reliable detection.
```suggestion
Pattern.compile("(?i)(?<!['\"-])\\b(from|join)\\s+system\\s*\\.\\s*[a-zA-Z_][a-zA-Z0-9_]*\\b");
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,740 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.databind.JsonNode;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.datasource.InMemorySystemTableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableDataProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final String PINOT_ADMIN_CLIENT_CLASS_NAME = "org.apache.pinot.client.admin.PinotAdminClient";
+ private static final String SIZE_CACHE_TTL_MS_PROPERTY = "pinot.systemtable.tables.sizeCacheTtlMs";
+ private static final long DEFAULT_SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+ private static final long SIZE_CACHE_TTL_MS = getNonNegativeLongProperty(SIZE_CACHE_TTL_MS_PROPERTY,
+ DEFAULT_SIZE_CACHE_TTL_MS);
+
+ private static final String CONTROLLER_TIMEOUT_MS_PROPERTY = "pinot.systemtable.tables.controllerTimeoutMs";
+ private static final long DEFAULT_CONTROLLER_TIMEOUT_MS = Duration.ofSeconds(5).toMillis();
+ private static final long CONTROLLER_TIMEOUT_MS = getPositiveLongProperty(CONTROLLER_TIMEOUT_MS_PROPERTY,
+ DEFAULT_CONTROLLER_TIMEOUT_MS);
+
+ private static final long SIZE_FETCH_FAILURE_WARN_INTERVAL_MS = Duration.ofHours(1).toMillis();
+
+ private static final String ADMIN_TRANSPORT_REQUEST_TIMEOUT_MS = "pinot.admin.request.timeout.ms";
+ private static final String ADMIN_TRANSPORT_SCHEME = "pinot.admin.scheme";
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final @Nullable Class<?> PINOT_ADMIN_CLIENT_CLASS = loadAdminClientClass();
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+ private final Map<String, CachedSize> _sizeCache = new ConcurrentHashMap<>();
+ private final Map<String, AutoCloseable> _adminClientCache = new ConcurrentHashMap<>();
+ private final AtomicLong _lastSizeFetchFailureWarnLogMs = new AtomicLong();
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public TableConfig getTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ }
+
+ @Override
+ public void close()
+ throws Exception {
+ for (Map.Entry<String, AutoCloseable> entry : _adminClientCache.entrySet()) {
+ try {
+ entry.getValue().close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close admin client for {}: {}", entry.getKey(), e.toString());
+ }
+ }
+ _adminClientCache.clear();
+ }
+
+ @Override
+ public IndexSegment getDataSource() {
+ if (_tableCache == null) {
+ return new InMemorySystemTableSegment(TABLE_NAME, SCHEMA, 0, Collections.emptyMap());
+ }
+
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ List<String> controllerBaseUrls = getControllerBaseUrls();
+ Function<String, TableSize> sizeFetcher = getSizeFetcher();
+ class TableRow {
+ final String _tableNameWithType;
+ final TableType _tableType;
+ final String _rawTableName;
+ final @Nullable TableConfig _tableConfig;
+ private volatile @Nullable String _tableConfigJson;
+ private volatile @Nullable TableSize _tableSize;
+ private volatile boolean _tableSizeFetched;
+
+ private TableRow(String tableNameWithType, TableType tableType, String rawTableName,
+ @Nullable TableConfig tableConfig) {
+ _tableNameWithType = tableNameWithType;
+ _tableType = tableType;
+ _rawTableName = rawTableName;
+ _tableConfig = tableConfig;
+ }
+
+ @Nullable
+ private TableSize getTableSize() {
+ if (_tableSizeFetched) {
+ return _tableSize;
+ }
+ synchronized (this) {
+ if (_tableSizeFetched) {
+ return _tableSize;
+ }
+ _tableSize = fetchTableSize(_tableNameWithType, sizeFetcher, controllerBaseUrls);
+ _tableSizeFetched = true;
+ return _tableSize;
+ }
+ }
+
+ private String getStatus() {
+ if (_tableConfig != null) {
+ return "ONLINE";
+ }
+ TableSize sizeFromController = getTableSize();
+ int segments = sizeFromController != null ? getSegmentCount(sizeFromController, _tableType) : 0;
+ return segments > 0 ? "ONLINE" : "UNKNOWN";
+ }
+
+ private int getSegments() {
+ TableSize sizeFromController = getTableSize();
+ return sizeFromController != null ? getSegmentCount(sizeFromController, _tableType) : 0;
+ }
+
+ private long getTotalDocs() {
+ TableSize sizeFromController = getTableSize();
+ return sizeFromController != null ? TablesSystemTableProvider.this.getTotalDocs(sizeFromController, _tableType,
+ _tableNameWithType, controllerBaseUrls) : 0L;
+ }
+
+ private long getReportedSize() {
+ TableSize sizeFromController = getTableSize();
+ if (sizeFromController == null || sizeFromController._reportedSizeInBytes < 0) {
+ return 0L;
+ }
+ return sizeFromController._reportedSizeInBytes;
+ }
+
+ private long getEstimatedSize() {
+ TableSize sizeFromController = getTableSize();
+ if (sizeFromController == null || sizeFromController._estimatedSizeInBytes < 0) {
+ return 0L;
+ }
+ return sizeFromController._estimatedSizeInBytes;
+ }
+
+ private String getBrokerTenant() {
+ if (_tableConfig != null && _tableConfig.getTenantConfig() != null) {
+ String tenant = _tableConfig.getTenantConfig().getBroker();
+ return tenant != null ? tenant : "";
+ }
+ return "";
+ }
+
+ private String getServerTenant() {
+ if (_tableConfig != null && _tableConfig.getTenantConfig() != null) {
+ String tenant = _tableConfig.getTenantConfig().getServer();
+ return tenant != null ? tenant : "";
+ }
+ return "";
+ }
+
+ private int getReplicas() {
+ if (_tableConfig != null && _tableConfig.getValidationConfig() != null) {
+ Integer replicationNumber = _tableConfig.getValidationConfig().getReplicationNumber();
+ if (replicationNumber != null) {
+ return replicationNumber;
+ }
+ }
+ return 0;
+ }
+
+ private String getTableConfigJson() {
+ String cached = _tableConfigJson;
+ if (cached != null) {
+ return cached;
+ }
+ synchronized (this) {
+ cached = _tableConfigJson;
+ if (cached != null) {
+ return cached;
+ }
+ cached = "";
+ if (_tableConfig != null) {
+ try {
+ cached = JsonUtils.objectToString(_tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", _tableNameWithType, e.toString());
+ cached = _tableConfig.toString();
+ }
+ }
+ _tableConfigJson = cached;
+ return cached;
+ }
+ }
+ }
+
+ List<TableRow> tableRows = new ArrayList<>(sortedTableNames.size());
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ tableRows.add(new TableRow(tableNameWithType, tableType, rawTableName, tableConfig));
+ }
+
+ Map<String, IntFunction<Object>> valueProviders = new java.util.HashMap<>();
+ valueProviders.put("tableName", docId -> tableRows.get(docId)._rawTableName);
+ valueProviders.put("type", docId -> tableRows.get(docId)._tableType.name());
+ valueProviders.put("status", docId -> tableRows.get(docId).getStatus());
+ valueProviders.put("segments", docId -> tableRows.get(docId).getSegments());
+ valueProviders.put("totalDocs", docId -> tableRows.get(docId).getTotalDocs());
+ valueProviders.put("reportedSize", docId -> tableRows.get(docId).getReportedSize());
+ valueProviders.put("estimatedSize", docId -> tableRows.get(docId).getEstimatedSize());
+ valueProviders.put("brokerTenant", docId -> tableRows.get(docId).getBrokerTenant());
+ valueProviders.put("serverTenant", docId -> tableRows.get(docId).getServerTenant());
+ valueProviders.put("replicas", docId -> tableRows.get(docId).getReplicas());
+ valueProviders.put("tableConfig", docId -> tableRows.get(docId).getTableConfigJson());
+
+ return new InMemorySystemTableSegment(TABLE_NAME, SCHEMA, tableRows.size(), valueProviders);
+ }
+
+ @Nullable
+ private TableSize fetchTableSize(String tableNameWithType,
+ @Nullable Function<String, TableSize> fetcher, List<String> controllerBaseUrls) {
+ boolean cacheEnabled = SIZE_CACHE_TTL_MS > 0;
+ TableSize cached = cacheEnabled ? getCachedSize(tableNameWithType) : null;
+ if (cached != null) {
+ return cached;
+ }
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ if (cacheEnabled) {
+ cacheSize(tableNameWithType, fetched);
+ }
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(controllerBaseUrls, rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(controllerBaseUrls, tableNameWithType);
+ if (size == null) {
+ logSizeFetchFailure("{}: failed to fetch size for {} from controllers {} (tried '{}' and '{}')", TABLE_NAME,
+ tableNameWithType, controllerBaseUrls, rawTableName, tableNameWithType);
+ }
+ }
+ if (size != null && cacheEnabled) {
+ cacheSize(tableNameWithType, size);
+ }
+ return size;
+ }
+
+ @Nullable
+ private TableSize fetchTableSizeForName(List<String> controllerBaseUrls, String tableName) {
+ Class<?> clientClass = PINOT_ADMIN_CLIENT_CLASS;
+ if (clientClass == null) {
+ return null;
+ }
+ for (String baseUrl : controllerBaseUrls) {
+ try {
+ AutoCloseable adminClient = getOrCreateAdminClient(baseUrl);
+ if (adminClient == null) {
+ continue;
+ }
+
+ Object sizeNode;
+ try {
+ sizeNode = clientClass.getMethod("getTableSize", String.class, boolean.class, boolean.class)
+ .invoke(adminClient, tableName, true, false);
+ } catch (NoSuchMethodException e) {
+ Object tableClient = clientClass.getMethod("getTableClient").invoke(adminClient);
+ sizeNode = tableClient.getClass().getMethod("getTableSize", String.class, boolean.class, boolean.class)
+ .invoke(tableClient, tableName, true, false);
+ }
+
+ if (sizeNode == null) {
+ continue;
+ }
+
+ TableSize parsed = JsonUtils.stringToObject(sizeNode.toString(), TableSize.class);
+ LOGGER.debug("{}: controller size response for {} via {} -> segments offline={}, realtime={}, "
+ + "reportedSize={}, estimatedSize={}", TABLE_NAME, tableName, baseUrl,
+ parsed._offlineSegments != null && parsed._offlineSegments._segments != null
+ ? parsed._offlineSegments._segments.size() : 0,
+ parsed._realtimeSegments != null && parsed._realtimeSegments._segments != null
+ ? parsed._realtimeSegments._segments.size() : 0,
+ parsed._reportedSizeInBytes, parsed._estimatedSizeInBytes);
+ return parsed;
+ } catch (Exception e) {
+ logSizeFetchFailure("{}: error fetching table size for {} via {} using admin client", TABLE_NAME, tableName,
+ baseUrl, e);
+ }
+ }
+ return null;
+ }
+
+ private List<String> getControllerBaseUrls() {
+ Set<String> urls = new LinkedHashSet<>();
+ if (_helixAdmin != null) {
+ for (String controller : discoverControllersFromHelix()) {
+ String normalized = normalizeControllerUrl(controller);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ }
+ for (String url : _staticControllerUrls) {
+ String normalized = normalizeControllerUrl(url);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ return new ArrayList<>(urls);
+ }
+
+ private int getSegmentCount(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.size();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.size();
+ }
+ return 0;
+ }
+
+ private long getTotalDocsFromSize(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ return 0;
+ }
+
+ private long getTotalDocs(TableSize sizeFromController, TableType tableType, String tableNameWithType,
+ List<String> controllerBaseUrls) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ long cached = sizeFromController._offlineTotalDocs;
+ if (cached >= 0) {
+ return cached;
+ }
+ long totalDocsFromSize = getTotalDocsFromSize(sizeFromController, tableType);
+ if (totalDocsFromSize > 0) {
+ synchronized (sizeFromController) {
+ if (sizeFromController._offlineTotalDocs < 0) {
+ sizeFromController._offlineTotalDocs = totalDocsFromSize;
+ }
+ return sizeFromController._offlineTotalDocs;
+ }
+ }
+ long fetched = fetchTotalDocsFromSegmentMetadata(tableNameWithType, sizeFromController._offlineSegments._segments,
+ controllerBaseUrls);
+ synchronized (sizeFromController) {
+ if (sizeFromController._offlineTotalDocs < 0) {
+ sizeFromController._offlineTotalDocs = fetched;
+ }
+ return sizeFromController._offlineTotalDocs;
+ }
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ long cached = sizeFromController._realtimeTotalDocs;
+ if (cached >= 0) {
+ return cached;
+ }
+ long totalDocsFromSize = getTotalDocsFromSize(sizeFromController, tableType);
+ if (totalDocsFromSize > 0) {
+ synchronized (sizeFromController) {
+ if (sizeFromController._realtimeTotalDocs < 0) {
+ sizeFromController._realtimeTotalDocs = totalDocsFromSize;
+ }
+ return sizeFromController._realtimeTotalDocs;
+ }
+ }
+ long fetched = fetchTotalDocsFromSegmentMetadata(tableNameWithType,
+ sizeFromController._realtimeSegments._segments, controllerBaseUrls);
+ synchronized (sizeFromController) {
+ if (sizeFromController._realtimeTotalDocs < 0) {
+ sizeFromController._realtimeTotalDocs = fetched;
+ }
+ return sizeFromController._realtimeTotalDocs;
+ }
+ }
+ return 0;
+ }
+
+ private long fetchTotalDocsFromSegmentMetadata(String tableNameWithType, Map<String, SegmentSize> segments,
+ List<String> controllerBaseUrls) {
+ if (segments.isEmpty()) {
+ return 0;
+ }
+ Class<?> clientClass = PINOT_ADMIN_CLIENT_CLASS;
+ if (clientClass == null) {
+ return 0;
+ }
+ for (String baseUrl : controllerBaseUrls) {
+ try {
+ AutoCloseable adminClient = getOrCreateAdminClient(baseUrl);
+ if (adminClient == null) {
+ continue;
+ }
+ Object segmentMetadataFetcher;
+ java.lang.reflect.Method getSegmentMetadataMethod;
+ boolean returnsJsonNode;
+ try {
+ segmentMetadataFetcher = clientClass.getMethod("getSegmentApiClient").invoke(adminClient);
+ getSegmentMetadataMethod =
+ segmentMetadataFetcher.getClass().getMethod("getSegmentMetadata", String.class, String.class);
+ returnsJsonNode = true;
+ } catch (NoSuchMethodException e) {
+ segmentMetadataFetcher = clientClass.getMethod("getSegmentClient").invoke(adminClient);
+ getSegmentMetadataMethod = segmentMetadataFetcher.getClass()
+ .getMethod("getSegmentMetadata", String.class, String.class, List.class);
+ returnsJsonNode = false;
+ }
+
+ long totalDocs = 0;
+ if (returnsJsonNode) {
+ for (String segmentName : segments.keySet()) {
+ JsonNode segmentMetadata = (JsonNode) getSegmentMetadataMethod.invoke(segmentMetadataFetcher,
+ tableNameWithType, segmentName);
+ totalDocs += segmentMetadata.path(CommonConstants.Segment.TOTAL_DOCS).asLong(0);
+ }
+ } else {
+ for (String segmentName : segments.keySet()) {
+ @SuppressWarnings("unchecked")
+ Map<String, Object> segmentMetadata = (Map<String, Object>) getSegmentMetadataMethod
+ .invoke(segmentMetadataFetcher, tableNameWithType, segmentName, List.of());
+ Object totalDocsObj = segmentMetadata.get(CommonConstants.Segment.TOTAL_DOCS);
+ if (totalDocsObj != null) {
+ totalDocs += Long.parseLong(totalDocsObj.toString());
+ }
+ }
+ }
+ return totalDocs;
+ } catch (Exception e) {
+ logSizeFetchFailure("{}: error fetching segment metadata for {} via {}", TABLE_NAME, tableNameWithType, baseUrl,
+ e);
+ }
+ }
+ return 0;
+ }
+
+ @Nullable
+ private Function<String, TableSize> getSizeFetcher() {
+ if (_tableSizeFetcherOverride != null) {
+ return _tableSizeFetcherOverride;
+ }
+ return null;
+ }
+
+ private List<String> discoverControllersFromHelix() {
+ List<String> urls = new ArrayList<>();
+ try {
+ if (_clusterName == null) {
+ LOGGER.warn("Cannot discover controllers without cluster name");
+ return List.of();
+ }
+ for (String controllerId : _helixAdmin.getInstancesInCluster(_clusterName)) {
+ if (!InstanceTypeUtils.isController(controllerId)) {
+ continue;
+ }
+ int firstUnderscore = controllerId.indexOf('_');
+ int lastUnderscore = controllerId.lastIndexOf('_');
+ if (firstUnderscore > 0 && lastUnderscore > firstUnderscore && lastUnderscore < controllerId.length() - 1) {
+ String host = controllerId.substring(firstUnderscore + 1, lastUnderscore);
+ String port = controllerId.substring(lastUnderscore + 1);
Review Comment:
The controller instance ID parsing logic assumes a specific format (`Controller_<host>_<port>`) and will fail silently for variations. Consider using a regex pattern with capture groups for more robust parsing and clearer validation, or extracting this logic into a dedicated utility method with proper error handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2617593762
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/DatabaseUtils.java:
##########
@@ -67,6 +67,15 @@ public static String translateTableName(String tableName, @Nullable String datab
case 2:
Preconditions.checkArgument(!tableSplit[1].isEmpty(), "Invalid table name '%s'", tableName);
String databasePrefix = tableSplit[0];
+ /*
+ * Allow system tables to bypass database prefix validation so they can be queried regardless of the database
+ * header. System tables are intended to be globally accessible and are not subject to database-level access
+ * controls. Ensure system tables do not expose sensitive information because they can be queried without a
+ * matching database context.
Review Comment:
The comment warns about sensitive information exposure but doesn't specify what validation or access controls are in place. Consider documenting what security measures are implemented for system tables or adding a reference to where such controls are defined.
```suggestion
* matching database context.
*
* Security Note: System tables are intentionally not protected by database-level access controls. This is by design,
* as documented in the Pinot security model (see: https://docs.pinot.apache.org/basics/security or the relevant
* access control implementation in org.apache.pinot.spi.security.AccessControl). If you add new system tables,
* ensure they do not expose sensitive or restricted information.
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,598 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTableFilter;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.apache.pinot.spi.systemtable.SystemTableRequest;
+import org.apache.pinot.spi.systemtable.SystemTableResponse;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final boolean IS_ADMIN_CLIENT_AVAILABLE;
+ static {
+ boolean available;
+ try {
+ Class.forName("org.apache.pinot.client.admin.PinotAdminClient");
+ available = true;
+ } catch (ClassNotFoundException e) {
+ available = false;
+ }
+ IS_ADMIN_CLIENT_AVAILABLE = available;
+ }
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
Review Comment:
The field name `_staticControllerUrls` is ambiguous. Consider renaming to `_configuredControllerUrls` or `_providedControllerUrls` to clarify that these are explicitly configured URLs rather than URLs discovered dynamically from Helix.
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java:
##########
@@ -1033,6 +1046,244 @@ private void throwAccessDeniedError(long requestId, String query, RequestContext
throw new WebApplicationException("Permission denied." + failureMessage, Response.Status.FORBIDDEN);
}
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ private BrokerResponse handleSystemTableQuery(PinotQuery pinotQuery, String tableName,
+ RequestContext requestContext, @Nullable RequesterIdentity requesterIdentity, String query) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableProvider provider = _systemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return BrokerResponseNative.TABLE_DOES_NOT_EXIST;
+ }
+ try {
+ if (!isSupportedSystemTableQuery(pinotQuery)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION,
+ "System tables only support simple projection/filter/limit queries");
+ }
+ Schema systemSchema = provider.getSchema();
+ List<String> projectionColumns = extractProjectionColumns(pinotQuery, systemSchema);
+ int offset = Math.max(0, pinotQuery.getOffset());
+ int limit = Math.max(0, pinotQuery.getLimit());
+ SystemTableRequest systemTableRequest =
+ new SystemTableRequest(projectionColumns, toSystemTableFilter(pinotQuery.getFilterExpression()), offset,
+ limit);
+ SystemTableResponse systemTableResponse = provider.getRows(systemTableRequest);
+ BrokerResponseNative brokerResponse =
+ buildSystemTableBrokerResponse(tableName, systemSchema, projectionColumns, systemTableResponse,
+ requestContext);
+ brokerResponse.setTimeUsedMs(System.currentTimeMillis() - requestContext.getRequestArrivalTimeMillis());
+ _queryLogger.log(new QueryLogger.QueryLogParams(requestContext, tableName, brokerResponse,
+ QueryLogger.QueryLogParams.QueryEngine.SINGLE_STAGE, requesterIdentity, null));
+ return brokerResponse;
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table query {}: {}", tableName, e.getMessage(), e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+
+ private boolean isSupportedSystemTableQuery(PinotQuery pinotQuery) {
+ return (pinotQuery.getGroupByList() == null || pinotQuery.getGroupByList().isEmpty())
+ && pinotQuery.getHavingExpression() == null
+ && (pinotQuery.getOrderByList() == null || pinotQuery.getOrderByList().isEmpty());
Review Comment:
The method lacks documentation explaining why GROUP BY, HAVING, and ORDER BY are unsupported for system tables. Add a comment explaining that these operations are not supported because system tables currently only support simple projection/filter/limit queries without aggregations or sorting.
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java:
##########
@@ -1033,6 +1046,244 @@ private void throwAccessDeniedError(long requestId, String query, RequestContext
throw new WebApplicationException("Permission denied." + failureMessage, Response.Status.FORBIDDEN);
}
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ private BrokerResponse handleSystemTableQuery(PinotQuery pinotQuery, String tableName,
+ RequestContext requestContext, @Nullable RequesterIdentity requesterIdentity, String query) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableProvider provider = _systemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return BrokerResponseNative.TABLE_DOES_NOT_EXIST;
+ }
+ try {
+ if (!isSupportedSystemTableQuery(pinotQuery)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION,
+ "System tables only support simple projection/filter/limit queries");
+ }
+ Schema systemSchema = provider.getSchema();
+ List<String> projectionColumns = extractProjectionColumns(pinotQuery, systemSchema);
+ int offset = Math.max(0, pinotQuery.getOffset());
+ int limit = Math.max(0, pinotQuery.getLimit());
+ SystemTableRequest systemTableRequest =
+ new SystemTableRequest(projectionColumns, toSystemTableFilter(pinotQuery.getFilterExpression()), offset,
+ limit);
+ SystemTableResponse systemTableResponse = provider.getRows(systemTableRequest);
+ BrokerResponseNative brokerResponse =
+ buildSystemTableBrokerResponse(tableName, systemSchema, projectionColumns, systemTableResponse,
+ requestContext);
+ brokerResponse.setTimeUsedMs(System.currentTimeMillis() - requestContext.getRequestArrivalTimeMillis());
+ _queryLogger.log(new QueryLogger.QueryLogParams(requestContext, tableName, brokerResponse,
+ QueryLogger.QueryLogParams.QueryEngine.SINGLE_STAGE, requesterIdentity, null));
+ return brokerResponse;
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table query {}: {}", tableName, e.getMessage(), e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+
+ private boolean isSupportedSystemTableQuery(PinotQuery pinotQuery) {
+ return (pinotQuery.getGroupByList() == null || pinotQuery.getGroupByList().isEmpty())
+ && pinotQuery.getHavingExpression() == null
+ && (pinotQuery.getOrderByList() == null || pinotQuery.getOrderByList().isEmpty());
+ }
+
+ private List<String> extractProjectionColumns(PinotQuery pinotQuery, Schema schema)
+ throws BadQueryRequestException {
+ List<String> projections = new ArrayList<>();
+ boolean hasStar = false;
+ List<Expression> selectList = pinotQuery.getSelectList();
+ if (CollectionUtils.isEmpty(selectList)) {
+ throw new BadQueryRequestException("System tables require a projection list");
+ }
+ for (Expression expression : selectList) {
+ Identifier identifier = expression.getIdentifier();
+ if (identifier != null) {
+ if ("*".equals(identifier.getName())) {
+ hasStar = true;
+ } else {
+ projections.add(identifier.getName());
+ }
+ continue;
+ }
+ Function function = expression.getFunctionCall();
+ if (function != null && "AS".equalsIgnoreCase(function.getOperator()) && !function.getOperands().isEmpty()) {
+ Identifier aliased = function.getOperands().get(0).getIdentifier();
+ if (aliased != null) {
+ projections.add(aliased.getName());
+ continue;
+ }
+ }
+ throw new BadQueryRequestException("System tables only support column projections or '*'");
+ }
+ if (hasStar || projections.isEmpty()) {
+ projections = new ArrayList<>(schema.getColumnNames());
Review Comment:
The condition `projections.isEmpty()` should not expand to all columns automatically, as an empty projection list is semantically different from SELECT *. Either throw a BadQueryRequestException for empty projections, or document this fallback behavior explicitly.
```suggestion
if (hasStar) {
projections = new ArrayList<>(schema.getColumnNames());
} else if (projections.isEmpty()) {
throw new BadQueryRequestException("No valid projection columns found in SELECT list for system table");
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,598 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTableFilter;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.apache.pinot.spi.systemtable.SystemTableRequest;
+import org.apache.pinot.spi.systemtable.SystemTableResponse;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final boolean IS_ADMIN_CLIENT_AVAILABLE;
+ static {
+ boolean available;
+ try {
+ Class.forName("org.apache.pinot.client.admin.PinotAdminClient");
+ available = true;
+ } catch (ClassNotFoundException e) {
+ available = false;
+ }
+ IS_ADMIN_CLIENT_AVAILABLE = available;
+ }
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public SystemTableResponse getRows(SystemTableRequest request) {
+ if (_tableCache == null) {
+ return new SystemTableResponse(List.of(), System.currentTimeMillis(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ int offset = Math.max(0, request.getOffset());
+ int limit = request.getLimit();
+ boolean hasLimit = limit > 0;
+ int totalRows = 0;
+ int initialCapacity = hasLimit ? Math.min(sortedTableNames.size(), limit) : 0;
+ List<GenericRow> rows = new ArrayList<>(initialCapacity);
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType);
+ if (!matchesFilter(request.getFilter(), stats, rawTableName)) {
+ continue;
+ }
+ totalRows++;
+ if (offset > 0) {
+ offset--;
+ continue;
+ }
+ if (limit == 0) {
+ continue;
+ }
+ if (hasLimit && rows.size() >= limit) {
+ continue;
+ }
+ GenericRow row = new GenericRow();
+ row.putValue("tableName", rawTableName);
+ row.putValue("type", stats._type);
+ row.putValue("status", stats._status);
+ row.putValue("segments", stats._segments);
+ row.putValue("totalDocs", stats._totalDocs);
+ row.putValue("reportedSize", stats._reportedSizeInBytes);
+ row.putValue("estimatedSize", stats._estimatedSizeInBytes);
+ row.putValue("storageTier", stats._storageTier);
+ row.putValue("brokerTenant", stats._brokerTenant);
+ row.putValue("serverTenant", stats._serverTenant);
+ row.putValue("replicas", stats._replicas);
+ row.putValue("tableConfig", stats._tableConfig);
+ rows.add(row);
+ }
+ return new SystemTableResponse(rows, System.currentTimeMillis(), totalRows);
+ }
+
+ private TableStats buildStats(String tableNameWithType, TableType tableType) {
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ int segments = 0;
+ long totalDocs = 0;
+ long reportedSize = 0;
+ long estimatedSize = 0;
+ String tierValue = "";
+ String brokerTenant = "";
+ String serverTenant = "";
+ int replicas = 0;
+ if (tableConfig != null && tableConfig.getTenantConfig() != null) {
+ brokerTenant = tableConfig.getTenantConfig().getBroker();
+ serverTenant = tableConfig.getTenantConfig().getServer();
+ }
+ if (tableConfig != null && tableConfig.getValidationConfig() != null) {
+ Integer repl = tableConfig.getValidationConfig().getReplicationNumber();
+ replicas = repl != null ? repl : replicas;
+ }
+ // Use controller API only
+ TableSize sizeFromController = fetchTableSize(tableNameWithType);
+ if (sizeFromController != null) {
+ if (sizeFromController._reportedSizeInBytes >= 0) {
+ reportedSize = sizeFromController._reportedSizeInBytes;
+ }
+ if (sizeFromController._estimatedSizeInBytes >= 0) {
+ estimatedSize = sizeFromController._estimatedSizeInBytes;
+ }
+ segments = getSegmentCount(sizeFromController, tableType);
+ totalDocs = getTotalDocs(sizeFromController, tableType);
+ }
+ String status = tableConfig != null ? "ONLINE" : (segments > 0 ? "ONLINE" : "UNKNOWN");
+ String tableConfigJson = "";
+ if (tableConfig != null) {
+ try {
+ tableConfigJson = JsonUtils.objectToString(tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", tableNameWithType, e.toString());
+ tableConfigJson = tableConfig.toString();
+ }
+ }
+ return new TableStats(tableType.name(), status, segments, totalDocs, reportedSize, estimatedSize, tierValue,
+ tableConfigJson, brokerTenant, serverTenant, replicas);
+ }
+
+ private boolean matchesFilter(SystemTableFilter filter, TableStats stats, String rawTableName) {
+ if (filter == null) {
+ return true;
+ }
+ if (filter.getChildren().isEmpty()) {
+ return matchesLeafFilter(filter, stats, rawTableName);
+ }
+ switch (filter.getOperator()) {
+ case AND:
+ for (SystemTableFilter child : filter.getChildren()) {
+ if (!matchesFilter(child, stats, rawTableName)) {
+ return false;
+ }
+ }
+ return true;
+ case OR:
+ for (SystemTableFilter child : filter.getChildren()) {
+ if (matchesFilter(child, stats, rawTableName)) {
+ return true;
+ }
+ }
+ return false;
+ case NOT:
+ return filter.getChildren().isEmpty() || !matchesFilter(filter.getChildren().get(0), stats, rawTableName);
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesLeafFilter(SystemTableFilter filter, TableStats stats, String rawTableName) {
+ String column = filter.getColumn();
+ if (column == null) {
+ return true;
+ }
+ List<String> values = filter.getValues();
+ if (values == null || values.isEmpty()) {
+ return true;
+ }
+ switch (column.toLowerCase()) {
+ case "tablename":
+ return matchesString(values, rawTableName, filter.getOperator());
+ case "type":
+ return matchesString(values, stats._type, filter.getOperator());
+ case "status":
+ return matchesString(values, stats._status, filter.getOperator());
+ case "segments":
+ return matchesNumber(values, stats._segments, filter.getOperator());
+ case "reportedsize":
+ return matchesNumber(values, stats._reportedSizeInBytes, filter.getOperator());
+ case "estimatedsize":
+ return matchesNumber(values, stats._estimatedSizeInBytes, filter.getOperator());
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesString(List<String> candidates, String actual, SystemTableFilter.Operator operator) {
+ switch (operator) {
+ case EQ:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case IN:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case NEQ:
+ return candidates.stream().noneMatch(v -> v.equalsIgnoreCase(actual));
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesNumber(List<String> candidates, long actual, SystemTableFilter.Operator operator) {
+ try {
+ switch (operator) {
+ case EQ:
+ return candidates.stream().anyMatch(v -> Long.parseLong(v) == actual);
+ case NEQ:
+ return candidates.stream().noneMatch(v -> Long.parseLong(v) == actual);
+ case GT:
+ return candidates.stream().anyMatch(v -> actual > Long.parseLong(v));
+ case GTE:
+ return candidates.stream().anyMatch(v -> actual >= Long.parseLong(v));
+ case LT:
+ return candidates.stream().anyMatch(v -> actual < Long.parseLong(v));
+ case LTE:
+ return candidates.stream().anyMatch(v -> actual <= Long.parseLong(v));
+ case IN:
+ for (String candidate : candidates) {
+ if (actual == Long.parseLong(candidate)) {
+ return true;
+ }
+ }
+ return false;
+ default:
+ return true;
+ }
+ } catch (NumberFormatException e) {
+ LOGGER.debug("Failed to parse numeric filter value {}: {}", candidates, e.toString());
+ return true;
+ }
+ }
+
+ private @Nullable TableSize fetchTableSize(String tableNameWithType) {
+ Function<String, TableSize> fetcher = getSizeFetcher();
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(tableNameWithType);
+ if (size == null) {
+ LOGGER.warn("{}: failed to fetch size for {} (raw: {}), returning null", TABLE_NAME, tableNameWithType,
+ rawTableName);
+ }
+ }
+ return size;
+ }
+
+ private @Nullable TableSize fetchTableSizeForName(String tableName) {
+ for (String baseUrl : getControllerBaseUrls()) {
+ try {
+ String url = baseUrl + "/tables/" + tableName + "/size?verbose=true&includeReplacedSegments=false";
+ HttpRequest request = HttpRequest.newBuilder(URI.create(url))
+ .timeout(Duration.ofSeconds(5))
+ .GET()
+ .header("Accept", "application/json")
+ .build();
+ HttpResponse<String> response = _httpClient.send(request, HttpResponse.BodyHandlers.ofString());
+ if (response.statusCode() >= 200 && response.statusCode() < 300) {
+ TableSize parsed = JsonUtils.stringToObject(response.body(), TableSize.class);
+ LOGGER.debug("{}: controller size response for {} via {} -> segments offline={}, realtime={}, "
+ + "reportedSize={}, estimatedSize={}", TABLE_NAME, tableName, baseUrl,
+ parsed._offlineSegments != null && parsed._offlineSegments._segments != null
+ ? parsed._offlineSegments._segments.size() : 0,
+ parsed._realtimeSegments != null && parsed._realtimeSegments._segments != null
+ ? parsed._realtimeSegments._segments.size() : 0,
+ parsed._reportedSizeInBytes, parsed._estimatedSizeInBytes);
+ return parsed;
+ } else {
+ LOGGER.warn("{}: failed to fetch table size for {} via {}: status {}, body={}", TABLE_NAME, tableName,
+ baseUrl, response.statusCode(), response.body());
+ }
+ } catch (Exception e) {
+ LOGGER.warn("{}: error fetching table size for {} via {}", TABLE_NAME, tableName, baseUrl, e);
+ }
+ }
+ return null;
+ }
+
+ private List<String> getControllerBaseUrls() {
+ Set<String> urls = new LinkedHashSet<>();
+ if (_helixAdmin != null) {
+ for (String controller : discoverControllersFromHelix()) {
+ String normalized = normalizeControllerUrl(controller);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ }
+ for (String url : _staticControllerUrls) {
+ String normalized = normalizeControllerUrl(url);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ return new ArrayList<>(urls);
+ }
+
+ private int getSegmentCount(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.size();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.size();
+ }
+ return 0;
+ }
+
+ private long getTotalDocs(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ return 0;
+ }
+
+ private @Nullable Function<String, TableSize> getSizeFetcher() {
+ if (_tableSizeFetcherOverride != null) {
+ return _tableSizeFetcherOverride;
+ }
+ List<String> controllers = getControllerBaseUrls();
+ if (controllers.isEmpty() || !isAdminClientAvailable()) {
+ return null;
+ }
+ return tableNameWithType -> fetchWithAdminClient(controllers, tableNameWithType);
+ }
+
+ private List<String> discoverControllersFromHelix() {
+ List<String> urls = new ArrayList<>();
+ try {
+ if (_clusterName == null) {
+ LOGGER.warn("Cannot discover controllers without cluster name");
+ return List.of();
+ }
+ for (String controllerId : _helixAdmin.getInstancesInCluster(_clusterName)) {
+ if (!InstanceTypeUtils.isController(controllerId)) {
+ continue;
+ }
+ int idx = controllerId.lastIndexOf('_');
+ if (idx > 0 && idx < controllerId.length() - 1) {
+ String host = controllerId.substring(controllerId.indexOf('_') + 1, idx);
+ String port = controllerId.substring(idx + 1);
Review Comment:
The substring extraction uses `controllerId.indexOf('_')` which finds the first underscore, but `idx` is set to `lastIndexOf('_')`. This mismatch could extract an incorrect host string when multiple underscores are present. Both should use either `indexOf` or a different parsing strategy that accounts for the expected format.
```suggestion
String[] parts = controllerId.split("_");
if (parts.length >= 3) {
// Join all parts between the first and last as host (to handle underscores in host)
StringBuilder hostBuilder = new StringBuilder();
for (int i = 1; i < parts.length - 1; i++) {
if (i > 1) {
hostBuilder.append("_");
}
hostBuilder.append(parts[i]);
}
String host = hostBuilder.toString();
String port = parts[parts.length - 1];
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2745017784
##########
pinot-plugins/pinot-system-table/src/main/java/org/apache/pinot/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,662 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.databind.JsonNode;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.client.admin.PinotAdminClient;
+import org.apache.pinot.client.admin.PinotAdminTransport;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.systemtable.SystemTable;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.datasource.InMemorySystemTableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final String SIZE_CACHE_TTL_MS_PROPERTY = "pinot.systemtable.tables.sizeCacheTtlMs";
+ private static final long DEFAULT_SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+ private static final long SIZE_CACHE_TTL_MS = getNonNegativeLongProperty(SIZE_CACHE_TTL_MS_PROPERTY,
+ DEFAULT_SIZE_CACHE_TTL_MS);
+
+ private static final String CONTROLLER_TIMEOUT_MS_PROPERTY = "pinot.systemtable.tables.controllerTimeoutMs";
+ private static final long DEFAULT_CONTROLLER_TIMEOUT_MS = Duration.ofSeconds(5).toMillis();
+ private static final long CONTROLLER_TIMEOUT_MS = getPositiveLongProperty(CONTROLLER_TIMEOUT_MS_PROPERTY,
+ DEFAULT_CONTROLLER_TIMEOUT_MS);
+
+ private static final long SIZE_FETCH_FAILURE_WARN_INTERVAL_MS = Duration.ofHours(1).toMillis();
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _configuredControllerUrls;
+ private final Map<String, CachedSize> _sizeCache = new ConcurrentHashMap<>();
+ private final Map<String, PinotAdminClient> _adminClientCache = new ConcurrentHashMap<>();
+ private final AtomicLong _lastSizeFetchFailureWarnLogMs = new AtomicLong();
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _configuredControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public TableConfig getTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ }
+
+ @Override
+ public void close()
+ throws Exception {
+ for (Map.Entry<String, PinotAdminClient> entry : _adminClientCache.entrySet()) {
+ try {
+ entry.getValue().close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close admin client for {}: {}", entry.getKey(), e.toString());
+ }
+ }
+ _adminClientCache.clear();
+ }
+
+ @Override
+ public IndexSegment getDataSource() {
+ if (_tableCache == null) {
+ return new InMemorySystemTableSegment(TABLE_NAME, SCHEMA, 0, Collections.emptyMap());
+ }
+
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ List<String> controllerBaseUrls = getControllerBaseUrls();
+ Function<String, TableSize> sizeFetcher = getSizeFetcher();
+ class TableRow {
+ final String _tableNameWithType;
+ final TableType _tableType;
+ final String _rawTableName;
+ final @Nullable TableConfig _tableConfig;
+ private volatile @Nullable String _tableConfigJson;
+ private volatile @Nullable TableSize _tableSize;
Review Comment:
The volatile keyword combined with lazy initialization and synchronized blocks creates complex state management. Consider using AtomicReference or compute these values once during TableRow construction to simplify thread-safety and improve readability.
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/SystemTableRegistry.java:
##########
@@ -0,0 +1,202 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.IdentityHashMap;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.PinotReflectionUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Registry to hold and lifecycle-manage system table data providers.
+ */
+public final class SystemTableRegistry {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableRegistry.class);
+
+ // Providers are registered once at broker startup.
+ private static final Map<String, SystemTableProvider> PROVIDERS = new HashMap<>();
+
+ private SystemTableRegistry() {
+ throw new UnsupportedOperationException("This is a utility class and cannot be instantiated");
+ }
+
+ public static void register(SystemTableProvider provider) {
+ synchronized (PROVIDERS) {
+ PROVIDERS.put(normalize(provider.getTableName()), provider);
+ }
+ }
+
+ @Nullable
+ public static SystemTableProvider get(String tableName) {
+ synchronized (PROVIDERS) {
+ return PROVIDERS.get(normalize(tableName));
+ }
+ }
+
+ public static boolean isRegistered(String tableName) {
+ synchronized (PROVIDERS) {
+ return PROVIDERS.containsKey(normalize(tableName));
+ }
+ }
+
+ public static Collection<SystemTableProvider> getProviders() {
+ synchronized (PROVIDERS) {
+ return Collections.unmodifiableCollection(new java.util.ArrayList<>(PROVIDERS.values()));
+ }
+ }
+
+ /**
+ * Discover and register providers marked with {@link SystemTable} using the available dependencies.
+ * Follows the ScalarFunction pattern: any class annotated with @SystemTable under a "*.systemtable.*" package
+ * will be discovered via reflection and registered.
+ */
+ public static void registerAnnotatedProviders(TableCache tableCache, HelixAdmin helixAdmin, String clusterName,
+ @Nullable PinotConfiguration config) {
+ Set<Class<?>> classes =
+ PinotReflectionUtils.getClassesThroughReflection(".*\\.systemtable\\..*", SystemTable.class);
+ for (Class<?> clazz : classes) {
+ if (!SystemTableProvider.class.isAssignableFrom(clazz)) {
+ continue;
+ }
+ SystemTableProvider provider =
+ instantiateProvider(clazz.asSubclass(SystemTableProvider.class), tableCache, helixAdmin, clusterName, config);
+ if (provider == null) {
+ continue;
+ }
+ if (isRegistered(provider.getTableName())) {
+ continue;
+ }
+ LOGGER.info("Registering system table provider: {}", provider.getTableName());
+ register(provider);
+ }
+ }
+
+ public static void close()
+ throws Exception {
+ Exception firstException = null;
+ // Snapshot providers to avoid concurrent modifications and to close each provider at most once.
+ Map<SystemTableProvider, Boolean> providersToClose = new IdentityHashMap<>();
+ synchronized (PROVIDERS) {
+ for (SystemTableProvider provider : PROVIDERS.values()) {
+ providersToClose.put(provider, Boolean.TRUE);
+ }
+ }
+ try {
+ for (SystemTableProvider provider : providersToClose.keySet()) {
+ try {
+ provider.close();
+ } catch (Exception e) {
+ if (firstException == null) {
+ firstException = e;
+ } else {
+ firstException.addSuppressed(e);
+ }
+ }
+ }
+ } finally {
+ synchronized (PROVIDERS) {
+ PROVIDERS.clear();
+ }
+ }
+ if (firstException != null) {
+ throw firstException;
+ }
+ }
+
+ /**
+ * Initialize and register all annotated system table providers.
+ */
+ public static void init(TableCache tableCache, HelixAdmin helixAdmin, String clusterName) {
+ init(tableCache, helixAdmin, clusterName, null);
+ }
+
+ /**
+ * Initialize and register all annotated system table providers.
+ */
+ public static void init(TableCache tableCache, HelixAdmin helixAdmin, String clusterName,
+ @Nullable PinotConfiguration config) {
+ registerAnnotatedProviders(tableCache, helixAdmin, clusterName, config);
+ }
+
+ private static String normalize(String tableName) {
+ return tableName.toLowerCase(Locale.ROOT);
+ }
+
+ @Nullable
+ private static SystemTableProvider instantiateProvider(Class<? extends SystemTableProvider> clazz,
+ TableCache tableCache, HelixAdmin helixAdmin, String clusterName, @Nullable PinotConfiguration config) {
+ try {
+ // Prefer the most specific constructor available.
+ // System table providers may declare any of the following constructors, in decreasing order of specificity:
+ // (TableCache, HelixAdmin, String, PinotConfiguration), (TableCache, HelixAdmin, String),
+ // (TableCache, HelixAdmin), (TableCache), (PinotConfiguration), or a no-arg constructor.
+ // The registry will select the most specific available constructor so providers can opt in to the dependencies
+ // they need without forcing all implementations to depend on Helix or cluster metadata.
+ class ConstructorSpec {
+ final Class<?>[] _paramTypes;
+ final Object[] _args;
Review Comment:
The nested `ConstructorSpec` class is defined inside a method, making the method logic harder to follow. Consider extracting this as a private static class at the class level to improve readability and maintainability.
##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/SystemTableIntegrationTest.java:
##########
@@ -0,0 +1,130 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.integration.tests;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.io.File;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.pinot.client.admin.PinotAdminClient;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.util.TestUtils;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertTrue;
+
+
+/**
+ * Integration test to ensure system tables can be queried through the broker.
+ */
+public class SystemTableIntegrationTest extends BaseClusterIntegrationTestSet {
+
+ @Override
+ protected void overrideBrokerConf(PinotConfiguration brokerConf) {
+ // Provide controller URL so system.tables provider can call size API.
+ brokerConf.setProperty(CommonConstants.Broker.CONTROLLER_URL, getControllerBaseApiUrl());
+ }
+
+ @BeforeClass
+ public void setUp()
+ throws Exception {
+ TestUtils.ensureDirectoriesExistAndEmpty(_tempDir, _segmentDir, _tarDir);
+
+ startZk();
+ startController();
+ startBroker();
+ startServer();
+
+ Schema schema = createSchema();
+ addSchema(schema);
+ TableConfig tableConfig = createOfflineTableConfig();
+ addTableConfig(tableConfig);
+
+ List<File> avroFiles = unpackAvroData(_tempDir);
+ ClusterIntegrationTestUtils.buildSegmentsFromAvro(avroFiles, tableConfig, schema, 0, _segmentDir, _tarDir);
+ uploadSegments(getTableName(), _tarDir);
+ waitForAllDocsLoaded(600_000L);
+ }
+
+ @AfterClass
+ public void tearDown()
+ throws Exception {
+ stopServer();
+ stopBroker();
+ stopController();
+ stopZk();
+ }
+
+ @Test
+ public void testSystemTablesSize()
+ throws Exception {
+ JsonNode controllerSize = JsonUtils.stringToJsonNode(
+ sendGetRequest(getControllerBaseApiUrl() + "/tables/mytable/size?verbose=true&includeReplacedSegments=false"));
Review Comment:
URL construction using string concatenation is fragile. Consider using `UriBuilder` or similar utility to construct URLs with proper encoding and parameter handling.
```suggestion
URI controllerBaseUri = URI.create(getControllerBaseApiUrl());
URI sizeUri = controllerBaseUri.resolve("/tables/mytable/size?verbose=true&includeReplacedSegments=false");
JsonNode controllerSize = JsonUtils.stringToJsonNode(sendGetRequest(sizeUri.toString()));
```
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/SystemTableBrokerRequestHandler.java:
##########
@@ -0,0 +1,554 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.requesthandler;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.google.common.annotations.VisibleForTesting;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.Nullable;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.client5.http.classic.methods.HttpPost;
+import org.apache.hc.client5.http.config.RequestConfig;
+import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
+import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse;
+import org.apache.hc.client5.http.impl.classic.HttpClients;
+import org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager;
+import org.apache.hc.client5.http.io.HttpClientConnectionManager;
+import org.apache.hc.core5.http.io.entity.EntityUtils;
+import org.apache.hc.core5.http.io.entity.StringEntity;
+import org.apache.hc.core5.util.Timeout;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.HelixManager;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.pinot.broker.api.AccessControl;
+import org.apache.pinot.broker.broker.AccessControlFactory;
+import org.apache.pinot.broker.querylog.QueryLogger;
+import org.apache.pinot.broker.queryquota.QueryQuotaManager;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.common.datatable.DataTableFactory;
+import org.apache.pinot.common.datatable.DataTableImplV4;
+import org.apache.pinot.common.http.PoolingHttpClientConnectionManagerHelper;
+import org.apache.pinot.common.metrics.BrokerMeter;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.request.QuerySource;
+import org.apache.pinot.common.response.BrokerResponse;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.SystemTableRegistry;
+import org.apache.pinot.common.utils.NamedThreadFactory;
+import org.apache.pinot.common.utils.config.InstanceUtils;
+import org.apache.pinot.common.utils.config.QueryOptionsUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.core.operator.blocks.InstanceResponseBlock;
+import org.apache.pinot.core.plan.Plan;
+import org.apache.pinot.core.plan.maker.InstancePlanMakerImplV2;
+import org.apache.pinot.core.plan.maker.PlanMaker;
+import org.apache.pinot.core.query.reduce.BrokerReduceService;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils;
+import org.apache.pinot.core.routing.MultiClusterRoutingContext;
+import org.apache.pinot.core.routing.RoutingManager;
+import org.apache.pinot.core.transport.ServerRoutingInstance;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.SegmentContext;
+import org.apache.pinot.spi.accounting.ThreadAccountant;
+import org.apache.pinot.spi.auth.AuthorizationResult;
+import org.apache.pinot.spi.auth.broker.RequesterIdentity;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.query.QueryExecutionContext;
+import org.apache.pinot.spi.query.QueryThreadContext;
+import org.apache.pinot.spi.trace.RequestContext;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+import org.apache.pinot.sql.parsers.SqlNodeAndOptions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Broker request handler for system tables (handled entirely on the broker).
+ */
+public class SystemTableBrokerRequestHandler extends BaseBrokerRequestHandler {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableBrokerRequestHandler.class);
+ private static final String SYSTEM_TABLE_PSEUDO_HOST = "localhost";
+ private static final int SYSTEM_TABLE_PSEUDO_PORT = 0;
+ private static final String SYSTEM_TABLE_DATATABLE_API_PATH = "/query/systemTable/datatable";
+ private static final Set<String> HOP_BY_HOP_HEADERS_TO_SKIP =
+ Set.of("connection", "content-length", "transfer-encoding", "host");
+
+ private final BrokerReduceService _brokerReduceService;
+ private final PlanMaker _planMaker;
+ private final ExecutorService _executorService;
+ private final ExecutorService _scatterGatherExecutorService;
+ private final PoolingHttpClientConnectionManager _scatterGatherConnMgr;
+ private final CloseableHttpClient _scatterGatherHttpClient;
+ @Nullable
+ private final HelixManager _helixManager;
+
+ public SystemTableBrokerRequestHandler(PinotConfiguration config, String brokerId,
+ BrokerRequestIdGenerator requestIdGenerator, RoutingManager routingManager,
+ AccessControlFactory accessControlFactory, QueryQuotaManager queryQuotaManager, TableCache tableCache,
+ ThreadAccountant threadAccountant, @Nullable MultiClusterRoutingContext multiClusterRoutingContext,
+ @Nullable HelixManager helixManager) {
+ super(config, brokerId, requestIdGenerator, routingManager, accessControlFactory, queryQuotaManager, tableCache,
+ threadAccountant, multiClusterRoutingContext);
+ _brokerReduceService = new BrokerReduceService(_config);
+ _planMaker = new InstancePlanMakerImplV2();
+ _planMaker.init(_config);
+ _helixManager = helixManager;
+ int executorPoolSize = config.getProperty(CommonConstants.Broker.CONFIG_OF_SYSTEM_TABLE_EXECUTOR_POOL_SIZE,
+ CommonConstants.Broker.DEFAULT_SYSTEM_TABLE_EXECUTOR_POOL_SIZE);
+ executorPoolSize = Math.max(1, executorPoolSize);
+ _executorService = QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(executorPoolSize,
+ new NamedThreadFactory("system-table-query-executor")));
+ _scatterGatherExecutorService =
+ QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(executorPoolSize,
+ new NamedThreadFactory("system-table-scatter-gather-executor")));
+ _scatterGatherConnMgr = PoolingHttpClientConnectionManagerHelper.createWithSocketFactory();
+ Timeout timeout = Timeout.of(_brokerTimeoutMs, TimeUnit.MILLISECONDS);
+ RequestConfig defaultRequestConfig =
+ RequestConfig.custom().setConnectionRequestTimeout(timeout).setResponseTimeout(timeout).build();
+ _scatterGatherHttpClient =
+ HttpClients.custom().setConnectionManager(_scatterGatherConnMgr).setDefaultRequestConfig(defaultRequestConfig)
+ .build();
+ }
+
+ @Override
+ public void start() {
+ }
+
+ @Override
+ public void shutDown() {
+ _executorService.shutdownNow();
+ _scatterGatherExecutorService.shutdownNow();
+ try {
+ _scatterGatherHttpClient.close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close system table scatter-gather http client: {}", e.toString());
+ }
+ try {
+ _scatterGatherConnMgr.close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close system table scatter-gather connection manager: {}", e.toString());
+ }
+ _brokerReduceService.shutDown();
+ }
+
+ public boolean canHandle(String tableName) {
+ return isSystemTable(tableName) && SystemTableRegistry.isRegistered(tableName);
+ }
+
+ @Override
+ protected BrokerResponse handleRequest(long requestId, String query, SqlNodeAndOptions sqlNodeAndOptions,
+ JsonNode request, @Nullable RequesterIdentity requesterIdentity, RequestContext requestContext,
+ @Nullable HttpHeaders httpHeaders, AccessControl accessControl)
+ throws Exception {
+ long startTimeMs = requestContext.getRequestArrivalTimeMillis();
+ long deadlineMs = startTimeMs + _brokerTimeoutMs;
+ QueryExecutionContext executionContext =
+ new QueryExecutionContext(QueryExecutionContext.QueryType.STE, requestId, Long.toString(requestId),
+ QueryOptionsUtils.getWorkloadName(sqlNodeAndOptions.getOptions()), startTimeMs, deadlineMs, deadlineMs,
+ _brokerId, _brokerId, org.apache.pinot.spi.utils.CommonConstants.Broker.DEFAULT_QUERY_HASH);
+ try (QueryThreadContext ignore = QueryThreadContext.open(executionContext, _threadAccountant)) {
+ PinotQuery pinotQuery;
+ try {
+ pinotQuery = CalciteSqlParser.compileToPinotQuery(sqlNodeAndOptions);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return new BrokerResponseNative(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ Set<String> tableNames = RequestUtils.getTableNames(pinotQuery);
+ if (tableNames == null || tableNames.isEmpty()) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "Failed to extract table name");
+ }
+ if (tableNames.size() != 1) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "System tables do not support joins");
+ }
+ String tableName = tableNames.iterator().next();
+ if (!isSystemTable(tableName)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "Not a system table query");
+ }
+ AuthorizationResult authorizationResult =
+ hasTableAccess(requesterIdentity, Set.of(tableName), requestContext, httpHeaders);
+ if (!authorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return new BrokerResponseNative(QueryErrorCode.ACCESS_DENIED, authorizationResult.getFailureMessage());
+ }
+
+ boolean queryWasLogged = _queryLogger.logQueryReceived(requestId, query);
+ return handleSystemTableQuery(request, pinotQuery, tableName, requestContext, requesterIdentity, query,
+ httpHeaders, queryWasLogged);
+ }
+ }
+
+ @Override
+ protected boolean handleCancel(long queryId, int timeoutMs, Executor executor,
+ HttpClientConnectionManager connMgr, Map<String, Integer> serverResponses) {
+ return false;
+ }
+
+ @Override
+ public boolean cancelQueryByClientId(String clientQueryId, int timeoutMs, Executor executor,
+ HttpClientConnectionManager connMgr, Map<String, Integer> serverResponses)
+ throws Exception {
+ return false;
+ }
+
+ @Override
+ public Map<Long, String> getRunningQueries() {
+ return Collections.emptyMap();
+ }
+
+ @Override
+ public OptionalLong getRequestIdByClientId(String clientQueryId) {
+ return OptionalLong.empty();
+ }
+
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ /**
+ * Executes a system table query against the local broker and returns the raw {@link DataTable} results.
+ * <p>
+ * This method is used by the internal broker-to-broker scatter-gather endpoint and must never perform fanout.
+ */
+ public DataTable handleSystemTableDataTableRequest(JsonNode request, @Nullable RequesterIdentity requesterIdentity,
+ RequestContext requestContext, @Nullable HttpHeaders httpHeaders) {
+ long startTimeMs = requestContext.getRequestArrivalTimeMillis();
+ if (startTimeMs <= 0) {
+ startTimeMs = System.currentTimeMillis();
+ requestContext.setRequestArrivalTimeMillis(startTimeMs);
+ }
+ long requestId = _requestIdGenerator.get();
+ long deadlineMs = startTimeMs + _brokerTimeoutMs;
+
+ JsonNode sql = request.get(CommonConstants.Broker.Request.SQL);
+ if (sql == null || !sql.isTextual()) {
+ return exceptionDataTable(QueryErrorCode.JSON_PARSING, "Failed to find 'sql' in the request: " + request);
+ }
+ String query = sql.textValue();
+ requestContext.setQuery(query);
+
+ SqlNodeAndOptions sqlNodeAndOptions;
+ try {
+ sqlNodeAndOptions = RequestUtils.parseQuery(query, request);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return exceptionDataTable(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ QueryExecutionContext executionContext =
+ new QueryExecutionContext(QueryExecutionContext.QueryType.STE, requestId, Long.toString(requestId),
+ QueryOptionsUtils.getWorkloadName(sqlNodeAndOptions.getOptions()), startTimeMs, deadlineMs, deadlineMs,
+ _brokerId, _brokerId, org.apache.pinot.spi.utils.CommonConstants.Broker.DEFAULT_QUERY_HASH);
+ try (QueryThreadContext ignore = QueryThreadContext.open(executionContext, _threadAccountant)) {
+ AccessControl accessControl = _accessControlFactory.create();
+ AuthorizationResult authorizationResult = accessControl.authorize(requesterIdentity);
+ if (!authorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return exceptionDataTable(QueryErrorCode.ACCESS_DENIED, authorizationResult.getFailureMessage());
+ }
+
+ PinotQuery pinotQuery;
+ try {
+ pinotQuery = CalciteSqlParser.compileToPinotQuery(sqlNodeAndOptions);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return exceptionDataTable(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ Set<String> tableNames = RequestUtils.getTableNames(pinotQuery);
+ if (tableNames == null || tableNames.isEmpty()) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, "Failed to extract table name");
+ }
+ if (tableNames.size() != 1) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, "System tables do not support joins");
+ }
+ String tableName = tableNames.iterator().next();
+ if (!isSystemTable(tableName)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, "Not a system table query");
+ }
+
+ AuthorizationResult tableAuthorizationResult =
+ hasTableAccess(requesterIdentity, Set.of(tableName), requestContext, httpHeaders);
+ if (!tableAuthorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return exceptionDataTable(QueryErrorCode.ACCESS_DENIED, tableAuthorizationResult.getFailureMessage());
+ }
+
+ SystemTableProvider provider = SystemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return exceptionDataTable(QueryErrorCode.TABLE_DOES_NOT_EXIST, "System table does not exist: " + tableName);
+ }
+
+ try {
+ return executeLocalSystemTableQuery(pinotQuery, provider);
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table datatable query {}: {}", tableName, e.getMessage(),
+ e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return exceptionDataTable(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+ }
+
+ private BrokerResponse handleSystemTableQuery(JsonNode request, PinotQuery pinotQuery, String tableName,
+ RequestContext requestContext, @Nullable RequesterIdentity requesterIdentity, String query,
+ @Nullable HttpHeaders httpHeaders, boolean queryWasLogged) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableProvider provider = SystemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return BrokerResponseNative.TABLE_DOES_NOT_EXIST;
+ }
+ try {
+ Map<ServerRoutingInstance, DataTable> dataTableMap;
+ if (provider.getExecutionMode() == SystemTableProvider.ExecutionMode.BROKER_SCATTER_GATHER) {
+ dataTableMap = scatterGatherSystemTableDataTables(provider, pinotQuery, tableName, request, httpHeaders);
+ } else {
+ dataTableMap = new HashMap<>(1);
+ // Use a synthetic routing instance for broker-local execution of system table queries.
+ dataTableMap.put(new ServerRoutingInstance(SYSTEM_TABLE_PSEUDO_HOST, SYSTEM_TABLE_PSEUDO_PORT,
+ TableType.OFFLINE), executeLocalSystemTableQuery(pinotQuery, provider));
+ }
+
+ BrokerResponseNative brokerResponse;
+ BrokerRequest brokerRequest = new BrokerRequest();
+ QuerySource querySource = new QuerySource();
+ querySource.setTableName(tableName);
+ brokerRequest.setQuerySource(querySource);
+ brokerRequest.setPinotQuery(pinotQuery);
+ brokerResponse = _brokerReduceService.reduceOnDataTable(brokerRequest, brokerRequest, dataTableMap,
+ _brokerTimeoutMs, _brokerMetrics);
+ brokerResponse.setTablesQueried(Set.of(TableNameBuilder.extractRawTableName(tableName)));
+ brokerResponse.setTimeUsedMs(System.currentTimeMillis() - requestContext.getRequestArrivalTimeMillis());
+ _queryLogger.logQueryCompleted(
+ new QueryLogger.QueryLogParams(requestContext, tableName, brokerResponse,
+ QueryLogger.QueryLogParams.QueryEngine.SINGLE_STAGE, requesterIdentity, null),
+ queryWasLogged);
+ return brokerResponse;
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table query {}: {}", tableName, e.getMessage(), e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+
+ private DataTable executeLocalSystemTableQuery(PinotQuery pinotQuery, SystemTableProvider provider)
+ throws Exception {
+ IndexSegment dataSource = provider.getDataSource();
+ try {
+ QueryContext queryContext = QueryContextConverterUtils.getQueryContext(pinotQuery);
+ queryContext.setSchema(provider.getSchema());
+ queryContext.setEndTimeMs(System.currentTimeMillis() + _brokerTimeoutMs);
+
+ // Pass null for serverMetrics because system table queries run broker-local against an in-memory IndexSegment.
+ Plan plan = _planMaker.makeInstancePlan(List.of(new SegmentContext(dataSource)), queryContext, _executorService,
+ null);
+ InstanceResponseBlock instanceResponse = plan.execute();
+ return instanceResponse.toDataTable();
+ } finally {
+ dataSource.destroy();
+ }
+ }
+
+ static final class BrokerTarget {
Review Comment:
The nested class `BrokerTarget` is marked as package-private (`static final`) but appears to only be used within `SystemTableBrokerRequestHandler`. Consider making it `private` to better encapsulate its usage scope.
```suggestion
private static final class BrokerTarget {
```
##########
pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/admin/PinotSegmentApiClient.java:
##########
@@ -52,20 +54,25 @@ public static class QueryParameters {
private static final String SELECT_PATH = "/select";
private static final String METADATA_PATH = "/metadata";
+ private static String encodePathSegment(String pathSegment) {
+ return URLEncoder.encode(pathSegment, StandardCharsets.UTF_8).replace("+", "%20");
Review Comment:
The `encodePathSegment` method is duplicated in both `PinotSegmentApiClient` and `PinotSegmentAdminClient`. Consider extracting this to a shared utility class to avoid duplication.
##########
pinot-plugins/pinot-system-table/src/main/java/org/apache/pinot/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,662 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.databind.JsonNode;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.client.admin.PinotAdminClient;
+import org.apache.pinot.client.admin.PinotAdminTransport;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.systemtable.SystemTable;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.datasource.InMemorySystemTableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final String SIZE_CACHE_TTL_MS_PROPERTY = "pinot.systemtable.tables.sizeCacheTtlMs";
+ private static final long DEFAULT_SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+ private static final long SIZE_CACHE_TTL_MS = getNonNegativeLongProperty(SIZE_CACHE_TTL_MS_PROPERTY,
+ DEFAULT_SIZE_CACHE_TTL_MS);
Review Comment:
Corrected variable name from `SIZE_CACHE_TTL_MS` to match the property name pattern. The constant name should be `SIZE_CACHE_TTL_MS` but it's being assigned from `getNonNegativeLongProperty(SIZE_CACHE_TTL_MS_PROPERTY, DEFAULT_SIZE_CACHE_TTL_MS)`.
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/SystemTableBrokerRequestHandler.java:
##########
@@ -0,0 +1,554 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.requesthandler;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.google.common.annotations.VisibleForTesting;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.Nullable;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.client5.http.classic.methods.HttpPost;
+import org.apache.hc.client5.http.config.RequestConfig;
+import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
+import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse;
+import org.apache.hc.client5.http.impl.classic.HttpClients;
+import org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager;
+import org.apache.hc.client5.http.io.HttpClientConnectionManager;
+import org.apache.hc.core5.http.io.entity.EntityUtils;
+import org.apache.hc.core5.http.io.entity.StringEntity;
+import org.apache.hc.core5.util.Timeout;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.HelixManager;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.pinot.broker.api.AccessControl;
+import org.apache.pinot.broker.broker.AccessControlFactory;
+import org.apache.pinot.broker.querylog.QueryLogger;
+import org.apache.pinot.broker.queryquota.QueryQuotaManager;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.common.datatable.DataTableFactory;
+import org.apache.pinot.common.datatable.DataTableImplV4;
+import org.apache.pinot.common.http.PoolingHttpClientConnectionManagerHelper;
+import org.apache.pinot.common.metrics.BrokerMeter;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.request.QuerySource;
+import org.apache.pinot.common.response.BrokerResponse;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.SystemTableRegistry;
+import org.apache.pinot.common.utils.NamedThreadFactory;
+import org.apache.pinot.common.utils.config.InstanceUtils;
+import org.apache.pinot.common.utils.config.QueryOptionsUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.core.operator.blocks.InstanceResponseBlock;
+import org.apache.pinot.core.plan.Plan;
+import org.apache.pinot.core.plan.maker.InstancePlanMakerImplV2;
+import org.apache.pinot.core.plan.maker.PlanMaker;
+import org.apache.pinot.core.query.reduce.BrokerReduceService;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils;
+import org.apache.pinot.core.routing.MultiClusterRoutingContext;
+import org.apache.pinot.core.routing.RoutingManager;
+import org.apache.pinot.core.transport.ServerRoutingInstance;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.SegmentContext;
+import org.apache.pinot.spi.accounting.ThreadAccountant;
+import org.apache.pinot.spi.auth.AuthorizationResult;
+import org.apache.pinot.spi.auth.broker.RequesterIdentity;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.query.QueryExecutionContext;
+import org.apache.pinot.spi.query.QueryThreadContext;
+import org.apache.pinot.spi.trace.RequestContext;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+import org.apache.pinot.sql.parsers.SqlNodeAndOptions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Broker request handler for system tables (handled entirely on the broker).
+ */
+public class SystemTableBrokerRequestHandler extends BaseBrokerRequestHandler {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableBrokerRequestHandler.class);
+ private static final String SYSTEM_TABLE_PSEUDO_HOST = "localhost";
+ private static final int SYSTEM_TABLE_PSEUDO_PORT = 0;
+ private static final String SYSTEM_TABLE_DATATABLE_API_PATH = "/query/systemTable/datatable";
+ private static final Set<String> HOP_BY_HOP_HEADERS_TO_SKIP =
+ Set.of("connection", "content-length", "transfer-encoding", "host");
Review Comment:
The hop-by-hop headers list is incomplete. According to RFC 7230, headers like "keep-alive", "proxy-authenticate", "proxy-authorization", "te", "trailer", and "upgrade" should also be excluded. Consider adding these or documenting why they're intentionally omitted.
```suggestion
// Hop-by-hop headers as defined in RFC 7230, Section 6.1, plus Host which is specific to each hop.
private static final Set<String> HOP_BY_HOP_HEADERS_TO_SKIP = Set.of(
"connection",
"keep-alive",
"proxy-authenticate",
"proxy-authorization",
"te",
"trailer",
"transfer-encoding",
"upgrade",
"host");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2647110034
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BrokerRequestHandlerDelegate.java:
##########
@@ -106,7 +115,27 @@ public BrokerResponse handleRequest(JsonNode request, @Nullable SqlNodeAndOption
}
}
+ Set<String> tableNames = null;
+ if (_systemTableBrokerRequestHandler != null) {
+ try {
+ // NOTE: compileToPinotQuery(SqlNodeAndOptions) mutates the SqlNode (e.g. SqlOrderBy -> SqlSelect).
+ // Re-parse from raw SQL so the SqlNodeAndOptions passed to the handler is unchanged (important for
+ // Multi-Stage Engine (MSE)).
+ JsonNode sql = request.get(Request.SQL);
+ String sqlQuery =
+ (sql != null && sql.isTextual()) ? sql.asText() : sqlNodeAndOptions.getSqlNode().toString();
+ tableNames = RequestUtils.getTableNames(CalciteSqlParser.compileToPinotQuery(sqlQuery));
Review Comment:
This code re-parses the SQL query for every request to determine if it's a system table query. This adds unnecessary overhead to all queries. Consider extracting table names directly from the already-parsed `sqlNodeAndOptions.getSqlNode()` using a visitor pattern or checking if `RequestUtils.getTableNames` can accept a SqlNode directly, avoiding the redundant parse step.
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,720 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.databind.JsonNode;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.datasource.InMemorySystemTableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableDataProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final String PINOT_ADMIN_CLIENT_CLASS_NAME = "org.apache.pinot.client.admin.PinotAdminClient";
+ private static final String SIZE_CACHE_TTL_MS_PROPERTY = "pinot.systemtable.tables.sizeCacheTtlMs";
+ private static final long DEFAULT_SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+ private static final long SIZE_CACHE_TTL_MS = getNonNegativeLongProperty(SIZE_CACHE_TTL_MS_PROPERTY,
+ DEFAULT_SIZE_CACHE_TTL_MS);
+
+ private static final String CONTROLLER_TIMEOUT_MS_PROPERTY = "pinot.systemtable.tables.controllerTimeoutMs";
+ private static final long DEFAULT_CONTROLLER_TIMEOUT_MS = Duration.ofSeconds(5).toMillis();
+ private static final long CONTROLLER_TIMEOUT_MS = getPositiveLongProperty(CONTROLLER_TIMEOUT_MS_PROPERTY,
+ DEFAULT_CONTROLLER_TIMEOUT_MS);
+
+ private static final String ADMIN_TRANSPORT_REQUEST_TIMEOUT_MS = "pinot.admin.request.timeout.ms";
+ private static final String ADMIN_TRANSPORT_SCHEME = "pinot.admin.scheme";
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final @Nullable Class<?> PINOT_ADMIN_CLIENT_CLASS = loadAdminClientClass();
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+ private final Map<String, CachedSize> _sizeCache = new ConcurrentHashMap<>();
+ private final Map<String, AutoCloseable> _adminClientCache = new ConcurrentHashMap<>();
+ private final AtomicBoolean _loggedSizeFetchFailure = new AtomicBoolean();
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public TableConfig getTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ }
+
+ @Override
+ public void close()
+ throws Exception {
+ for (Map.Entry<String, AutoCloseable> entry : _adminClientCache.entrySet()) {
+ try {
+ entry.getValue().close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close admin client for {}: {}", entry.getKey(), e.toString());
+ }
+ }
+ _adminClientCache.clear();
+ }
+
+ @Override
+ public IndexSegment getDataSource() {
+ if (_tableCache == null) {
+ return new InMemorySystemTableSegment(TABLE_NAME, SCHEMA, 0, Collections.emptyMap());
+ }
+
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ List<String> controllerBaseUrls = getControllerBaseUrls();
+ Function<String, TableSize> sizeFetcher = getSizeFetcher();
+ class TableRow {
+ final String _tableNameWithType;
+ final TableType _tableType;
+ final String _rawTableName;
+ final @Nullable TableConfig _tableConfig;
+ private volatile @Nullable String _tableConfigJson;
+ private volatile @Nullable TableSize _tableSize;
+ private volatile boolean _tableSizeFetched;
+
+ private TableRow(String tableNameWithType, TableType tableType, String rawTableName,
+ @Nullable TableConfig tableConfig) {
+ _tableNameWithType = tableNameWithType;
+ _tableType = tableType;
+ _rawTableName = rawTableName;
+ _tableConfig = tableConfig;
+ }
+
+ @Nullable
+ private TableSize getTableSize() {
+ if (_tableSizeFetched) {
+ return _tableSize;
+ }
+ synchronized (this) {
+ if (_tableSizeFetched) {
+ return _tableSize;
+ }
+ _tableSize = fetchTableSize(_tableNameWithType, sizeFetcher, controllerBaseUrls);
+ _tableSizeFetched = true;
+ return _tableSize;
+ }
+ }
+
+ private String getStatus() {
+ if (_tableConfig != null) {
+ return "ONLINE";
+ }
+ TableSize sizeFromController = getTableSize();
+ int segments = sizeFromController != null ? getSegmentCount(sizeFromController, _tableType) : 0;
+ return segments > 0 ? "ONLINE" : "UNKNOWN";
+ }
+
+ private int getSegments() {
+ TableSize sizeFromController = getTableSize();
+ return sizeFromController != null ? getSegmentCount(sizeFromController, _tableType) : 0;
+ }
+
+ private long getTotalDocs() {
+ TableSize sizeFromController = getTableSize();
+ return sizeFromController != null ? TablesSystemTableProvider.this.getTotalDocs(sizeFromController, _tableType,
+ _tableNameWithType, controllerBaseUrls) : 0L;
+ }
+
+ private long getReportedSize() {
+ TableSize sizeFromController = getTableSize();
+ if (sizeFromController == null || sizeFromController._reportedSizeInBytes < 0) {
+ return 0L;
+ }
+ return sizeFromController._reportedSizeInBytes;
+ }
+
+ private long getEstimatedSize() {
+ TableSize sizeFromController = getTableSize();
+ if (sizeFromController == null || sizeFromController._estimatedSizeInBytes < 0) {
+ return 0L;
+ }
+ return sizeFromController._estimatedSizeInBytes;
+ }
+
+ private String getBrokerTenant() {
+ if (_tableConfig != null && _tableConfig.getTenantConfig() != null) {
+ String tenant = _tableConfig.getTenantConfig().getBroker();
+ return tenant != null ? tenant : "";
+ }
+ return "";
+ }
+
+ private String getServerTenant() {
+ if (_tableConfig != null && _tableConfig.getTenantConfig() != null) {
+ String tenant = _tableConfig.getTenantConfig().getServer();
+ return tenant != null ? tenant : "";
+ }
+ return "";
+ }
+
+ private int getReplicas() {
+ if (_tableConfig != null && _tableConfig.getValidationConfig() != null) {
+ Integer replicationNumber = _tableConfig.getValidationConfig().getReplicationNumber();
+ if (replicationNumber != null) {
+ return replicationNumber;
+ }
+ }
+ return 0;
+ }
+
+ private String getTableConfigJson() {
+ String cached = _tableConfigJson;
+ if (cached != null) {
+ return cached;
+ }
+ synchronized (this) {
+ cached = _tableConfigJson;
+ if (cached != null) {
+ return cached;
+ }
+ cached = "";
+ if (_tableConfig != null) {
+ try {
+ cached = JsonUtils.objectToString(_tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", _tableNameWithType, e.toString());
+ cached = _tableConfig.toString();
+ }
+ }
+ _tableConfigJson = cached;
+ return cached;
+ }
+ }
+ }
+
+ List<TableRow> tableRows = new ArrayList<>(sortedTableNames.size());
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ tableRows.add(new TableRow(tableNameWithType, tableType, rawTableName, tableConfig));
+ }
+
+ Map<String, IntFunction<Object>> valueProviders = new java.util.HashMap<>();
+ valueProviders.put("tableName", docId -> tableRows.get(docId)._rawTableName);
+ valueProviders.put("type", docId -> tableRows.get(docId)._tableType.name());
+ valueProviders.put("status", docId -> tableRows.get(docId).getStatus());
+ valueProviders.put("segments", docId -> tableRows.get(docId).getSegments());
+ valueProviders.put("totalDocs", docId -> tableRows.get(docId).getTotalDocs());
+ valueProviders.put("reportedSize", docId -> tableRows.get(docId).getReportedSize());
+ valueProviders.put("estimatedSize", docId -> tableRows.get(docId).getEstimatedSize());
+ valueProviders.put("brokerTenant", docId -> tableRows.get(docId).getBrokerTenant());
+ valueProviders.put("serverTenant", docId -> tableRows.get(docId).getServerTenant());
+ valueProviders.put("replicas", docId -> tableRows.get(docId).getReplicas());
+ valueProviders.put("tableConfig", docId -> tableRows.get(docId).getTableConfigJson());
+
+ return new InMemorySystemTableSegment(TABLE_NAME, SCHEMA, tableRows.size(), valueProviders);
+ }
+
+ @Nullable
+ private TableSize fetchTableSize(String tableNameWithType,
+ @Nullable Function<String, TableSize> fetcher, List<String> controllerBaseUrls) {
+ boolean cacheEnabled = SIZE_CACHE_TTL_MS > 0;
+ TableSize cached = cacheEnabled ? getCachedSize(tableNameWithType) : null;
+ if (cached != null) {
+ return cached;
+ }
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ if (cacheEnabled) {
+ cacheSize(tableNameWithType, fetched);
+ }
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(controllerBaseUrls, rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(controllerBaseUrls, tableNameWithType);
+ if (size == null) {
+ logSizeFetchFailure("{}: failed to fetch size for {} from controllers {} (tried '{}' and '{}')", TABLE_NAME,
+ tableNameWithType, controllerBaseUrls, rawTableName, tableNameWithType);
+ }
+ }
+ if (size != null && cacheEnabled) {
+ cacheSize(tableNameWithType, size);
+ }
+ return size;
+ }
+
+ @Nullable
+ private TableSize fetchTableSizeForName(List<String> controllerBaseUrls, String tableName) {
+ Class<?> clientClass = PINOT_ADMIN_CLIENT_CLASS;
+ if (clientClass == null) {
+ return null;
+ }
+ for (String baseUrl : controllerBaseUrls) {
+ try {
+ AutoCloseable adminClient = getOrCreateAdminClient(baseUrl);
+ if (adminClient == null) {
+ continue;
+ }
+
+ Object sizeNode;
+ try {
+ sizeNode = clientClass.getMethod("getTableSize", String.class, boolean.class, boolean.class)
+ .invoke(adminClient, tableName, true, false);
+ } catch (NoSuchMethodException e) {
+ Object tableClient = clientClass.getMethod("getTableClient").invoke(adminClient);
+ sizeNode = tableClient.getClass().getMethod("getTableSize", String.class, boolean.class, boolean.class)
+ .invoke(tableClient, tableName, true, false);
+ }
+
+ if (sizeNode == null) {
+ continue;
+ }
+
+ TableSize parsed = JsonUtils.stringToObject(sizeNode.toString(), TableSize.class);
+ LOGGER.debug("{}: controller size response for {} via {} -> segments offline={}, realtime={}, "
+ + "reportedSize={}, estimatedSize={}", TABLE_NAME, tableName, baseUrl,
+ parsed._offlineSegments != null && parsed._offlineSegments._segments != null
+ ? parsed._offlineSegments._segments.size() : 0,
+ parsed._realtimeSegments != null && parsed._realtimeSegments._segments != null
+ ? parsed._realtimeSegments._segments.size() : 0,
+ parsed._reportedSizeInBytes, parsed._estimatedSizeInBytes);
+ return parsed;
+ } catch (Exception e) {
+ logSizeFetchFailure("{}: error fetching table size for {} via {} using admin client", TABLE_NAME, tableName,
+ baseUrl, e);
+ }
+ }
+ return null;
+ }
+
+ private List<String> getControllerBaseUrls() {
+ Set<String> urls = new LinkedHashSet<>();
+ if (_helixAdmin != null) {
+ for (String controller : discoverControllersFromHelix()) {
+ String normalized = normalizeControllerUrl(controller);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ }
+ for (String url : _staticControllerUrls) {
+ String normalized = normalizeControllerUrl(url);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ return new ArrayList<>(urls);
+ }
+
+ private int getSegmentCount(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.size();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.size();
+ }
+ return 0;
+ }
+
+ private long getTotalDocsFromSize(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ return 0;
+ }
+
+ private long getTotalDocs(TableSize sizeFromController, TableType tableType, String tableNameWithType,
+ List<String> controllerBaseUrls) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ if (sizeFromController._offlineTotalDocs >= 0) {
+ return sizeFromController._offlineTotalDocs;
+ }
+ long totalDocsFromSize = getTotalDocsFromSize(sizeFromController, tableType);
+ if (totalDocsFromSize > 0) {
+ sizeFromController._offlineTotalDocs = totalDocsFromSize;
+ return totalDocsFromSize;
+ }
+ long fetched = fetchTotalDocsFromSegmentMetadata(tableNameWithType, sizeFromController._offlineSegments._segments,
+ controllerBaseUrls);
+ sizeFromController._offlineTotalDocs = fetched;
+ return fetched;
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ if (sizeFromController._realtimeTotalDocs >= 0) {
+ return sizeFromController._realtimeTotalDocs;
+ }
+ long totalDocsFromSize = getTotalDocsFromSize(sizeFromController, tableType);
+ if (totalDocsFromSize > 0) {
+ sizeFromController._realtimeTotalDocs = totalDocsFromSize;
+ return totalDocsFromSize;
+ }
+ long fetched = fetchTotalDocsFromSegmentMetadata(tableNameWithType,
+ sizeFromController._realtimeSegments._segments, controllerBaseUrls);
+ sizeFromController._realtimeTotalDocs = fetched;
+ return fetched;
+ }
+ return 0;
+ }
+
+ private long fetchTotalDocsFromSegmentMetadata(String tableNameWithType, Map<String, SegmentSize> segments,
+ List<String> controllerBaseUrls) {
+ if (segments.isEmpty()) {
+ return 0;
+ }
+ Class<?> clientClass = PINOT_ADMIN_CLIENT_CLASS;
+ if (clientClass == null) {
+ return 0;
+ }
+ for (String baseUrl : controllerBaseUrls) {
+ try {
+ AutoCloseable adminClient = getOrCreateAdminClient(baseUrl);
+ if (adminClient == null) {
+ continue;
+ }
+ Object segmentMetadataFetcher;
+ java.lang.reflect.Method getSegmentMetadataMethod;
+ boolean returnsJsonNode;
+ try {
+ segmentMetadataFetcher = clientClass.getMethod("getSegmentApiClient").invoke(adminClient);
+ getSegmentMetadataMethod =
+ segmentMetadataFetcher.getClass().getMethod("getSegmentMetadata", String.class, String.class);
+ returnsJsonNode = true;
+ } catch (NoSuchMethodException e) {
+ segmentMetadataFetcher = clientClass.getMethod("getSegmentClient").invoke(adminClient);
+ getSegmentMetadataMethod = segmentMetadataFetcher.getClass()
+ .getMethod("getSegmentMetadata", String.class, String.class, List.class);
+ returnsJsonNode = false;
+ }
+
+ long totalDocs = 0;
+ if (returnsJsonNode) {
+ for (String segmentName : segments.keySet()) {
+ JsonNode segmentMetadata = (JsonNode) getSegmentMetadataMethod.invoke(segmentMetadataFetcher,
+ tableNameWithType, segmentName);
+ totalDocs += segmentMetadata.path(CommonConstants.Segment.TOTAL_DOCS).asLong(0);
+ }
+ } else {
+ for (String segmentName : segments.keySet()) {
+ @SuppressWarnings("unchecked")
+ Map<String, Object> segmentMetadata = (Map<String, Object>) getSegmentMetadataMethod
+ .invoke(segmentMetadataFetcher, tableNameWithType, segmentName, List.of());
+ Object totalDocsObj = segmentMetadata.get(CommonConstants.Segment.TOTAL_DOCS);
+ if (totalDocsObj != null) {
+ totalDocs += Long.parseLong(totalDocsObj.toString());
+ }
+ }
+ }
+ return totalDocs;
+ } catch (Exception e) {
+ logSizeFetchFailure("{}: error fetching segment metadata for {} via {}", TABLE_NAME, tableNameWithType, baseUrl,
+ e);
+ }
+ }
+ return 0;
+ }
+
+ @Nullable
+ private Function<String, TableSize> getSizeFetcher() {
+ if (_tableSizeFetcherOverride != null) {
+ return _tableSizeFetcherOverride;
+ }
+ return null;
+ }
+
+ private List<String> discoverControllersFromHelix() {
+ List<String> urls = new ArrayList<>();
+ try {
+ if (_clusterName == null) {
+ LOGGER.warn("Cannot discover controllers without cluster name");
+ return List.of();
+ }
+ for (String controllerId : _helixAdmin.getInstancesInCluster(_clusterName)) {
+ if (!InstanceTypeUtils.isController(controllerId)) {
+ continue;
+ }
+ int firstUnderscore = controllerId.indexOf('_');
+ int lastUnderscore = controllerId.lastIndexOf('_');
+ if (firstUnderscore > 0 && lastUnderscore > firstUnderscore && lastUnderscore < controllerId.length() - 1) {
+ String host = controllerId.substring(firstUnderscore + 1, lastUnderscore);
+ String port = controllerId.substring(lastUnderscore + 1);
+ if (!host.isEmpty() && !port.isEmpty()) {
+ urls.add(host + ":" + port);
+ } else {
+ LOGGER.warn("Unable to parse controller address from instance id (empty host or port): {}", controllerId);
+ }
+ } else {
+ LOGGER.warn("Unable to parse controller address from instance id '{}' (expected 'Controller_<host>_<port>')",
+ controllerId);
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Failed to discover controllers from Helix", e);
+ }
+ return urls;
+ }
+
+ @Nullable
+ private AutoCloseable getOrCreateAdminClient(String controllerBaseUrl) {
+ Class<?> clientClass = PINOT_ADMIN_CLIENT_CLASS;
+ if (clientClass == null) {
+ return null;
+ }
+ String normalized = normalizeControllerUrl(controllerBaseUrl);
+ if (normalized == null) {
+ return null;
+ }
+ return _adminClientCache.computeIfAbsent(normalized, controller -> {
+ try {
+ String controllerAddress = stripScheme(controller);
+ Properties properties = new Properties();
+ properties.setProperty(ADMIN_TRANSPORT_REQUEST_TIMEOUT_MS, String.valueOf(CONTROLLER_TIMEOUT_MS));
+ properties.setProperty(ADMIN_TRANSPORT_SCHEME, controller.startsWith("https://") ? "https" : "http");
+ return (AutoCloseable) clientClass.getConstructor(String.class, Properties.class)
+ .newInstance(controllerAddress, properties);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to create admin client for {}: {}", controller, e.toString());
+ return null;
+ }
+ });
+ }
+
+ private static String normalizeControllerUrl(@Nullable String controllerUrl) {
+ if (controllerUrl == null || controllerUrl.isEmpty()) {
+ return null;
+ }
+ String normalized = controllerUrl;
+ if (!normalized.startsWith("http://") && !normalized.startsWith("https://")) {
+ normalized = "http://" + normalized;
+ }
+ if (normalized.endsWith("/")) {
+ normalized = normalized.substring(0, normalized.length() - 1);
+ }
+ return normalized;
+ }
+
+ private static String stripScheme(String controllerUrl) {
+ if (controllerUrl == null) {
+ return null;
+ }
+ if (controllerUrl.startsWith("http://")) {
+ return controllerUrl.substring("http://".length());
+ }
+ if (controllerUrl.startsWith("https://")) {
+ return controllerUrl.substring("https://".length());
+ }
+ return controllerUrl;
+ }
+
+ /**
+ * Minimal shape of controller table size response.
+ */
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ public static final class TableSize {
+ @JsonProperty("reportedSizeInBytes")
+ public long _reportedSizeInBytes = -1;
+
+ @JsonProperty("estimatedSizeInBytes")
+ public long _estimatedSizeInBytes = -1;
+
+ @JsonProperty("offlineSegments")
+ public TableSubType _offlineSegments;
+
+ @JsonProperty("realtimeSegments")
+ public TableSubType _realtimeSegments;
+
+ public long _offlineTotalDocs = -1;
+ public long _realtimeTotalDocs = -1;
+ }
+
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ public static final class TableSubType {
+ @JsonProperty("segments")
+ public Map<String, SegmentSize> _segments;
+ }
+
+ @JsonIgnoreProperties(ignoreUnknown = true)
+ public static final class SegmentSize {
+ @JsonProperty("totalDocs")
+ public long _totalDocs = 0;
+ }
+
+ private static @Nullable Class<?> loadAdminClientClass() {
+ try {
+ return Class.forName(PINOT_ADMIN_CLIENT_CLASS_NAME);
+ } catch (ClassNotFoundException e) {
+ return null;
+ }
+ }
+
+ private static long getPositiveLongProperty(String key, long defaultValue) {
+ long value = Long.getLong(key, defaultValue);
+ return value > 0 ? value : defaultValue;
+ }
+
+ private static long getNonNegativeLongProperty(String key, long defaultValue) {
+ long value = Long.getLong(key, defaultValue);
+ return value >= 0 ? value : defaultValue;
+ }
+
+ private void logSizeFetchFailure(String message, Object... args) {
+ if (_loggedSizeFetchFailure.compareAndSet(false, true)) {
+ LOGGER.warn(message, args);
+ } else {
+ LOGGER.debug(message, args);
Review Comment:
This method suppresses repeated warnings by only logging at WARN level once, then switching to DEBUG. However, it uses a single boolean flag that never resets, so if failures occur intermittently (e.g., controller down then up), only the first failure is visible at WARN level. Consider adding a time-based reset mechanism (e.g., log at WARN level once per hour) or tracking per-table failures separately to improve observability.
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/SystemTableRegistry.java:
##########
@@ -0,0 +1,169 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.IdentityHashMap;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.PinotReflectionUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Registry to hold and lifecycle-manage system table data providers.
+ */
+public final class SystemTableRegistry {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableRegistry.class);
+
+ // Providers are registered once at broker startup.
+ private static final Map<String, SystemTableDataProvider> PROVIDERS = new HashMap<>();
+
+ private SystemTableRegistry() {
+ throw new UnsupportedOperationException("This is a utility class and cannot be instantiated");
+ }
+
+ public static void register(SystemTableDataProvider provider) {
+ synchronized (PROVIDERS) {
+ PROVIDERS.put(normalize(provider.getTableName()), provider);
+ }
+ }
+
+ @Nullable
+ public static SystemTableDataProvider get(String tableName) {
+ synchronized (PROVIDERS) {
+ return PROVIDERS.get(normalize(tableName));
+ }
+ }
+
+ public static boolean isRegistered(String tableName) {
+ synchronized (PROVIDERS) {
+ return PROVIDERS.containsKey(normalize(tableName));
+ }
+ }
+
+ public static Collection<SystemTableDataProvider> getProviders() {
+ synchronized (PROVIDERS) {
+ return Collections.unmodifiableCollection(new java.util.ArrayList<>(PROVIDERS.values()));
+ }
+ }
+
+ /**
+ * Discover and register providers marked with {@link SystemTable} using the available dependencies.
+ * Follows the ScalarFunction pattern: any class annotated with @SystemTable under a "*.systemtable.*" package
+ * will be discovered via reflection and registered.
+ */
+ public static void registerAnnotatedProviders(TableCache tableCache, HelixAdmin helixAdmin, String clusterName) {
+ Set<Class<?>> classes =
+ PinotReflectionUtils.getClassesThroughReflection(".*\\.systemtable\\..*", SystemTable.class);
+ for (Class<?> clazz : classes) {
+ if (!SystemTableDataProvider.class.isAssignableFrom(clazz)) {
+ continue;
+ }
+ SystemTableDataProvider provider = instantiateProvider(
+ clazz.asSubclass(SystemTableDataProvider.class), tableCache, helixAdmin, clusterName);
+ if (provider == null) {
+ continue;
+ }
+ if (isRegistered(provider.getTableName())) {
+ continue;
+ }
+ LOGGER.info("Registering system table provider: {}", provider.getTableName());
+ register(provider);
+ }
+ }
+
+ public static void close()
+ throws Exception {
+ Exception firstException = null;
+ // Snapshot providers to avoid concurrent modifications and to close each provider at most once.
+ Map<SystemTableDataProvider, Boolean> providersToClose = new IdentityHashMap<>();
+ synchronized (PROVIDERS) {
+ for (SystemTableDataProvider provider : PROVIDERS.values()) {
+ providersToClose.put(provider, Boolean.TRUE);
+ }
+ }
+ try {
+ for (SystemTableDataProvider provider : providersToClose.keySet()) {
+ try {
+ provider.close();
+ } catch (Exception e) {
+ if (firstException == null) {
+ firstException = e;
+ } else {
+ firstException.addSuppressed(e);
+ }
+ }
+ }
+ } finally {
+ synchronized (PROVIDERS) {
+ PROVIDERS.clear();
+ }
+ }
+ if (firstException != null) {
+ throw firstException;
+ }
+ }
+
+ /**
+ * Initialize and register all annotated system table providers.
+ */
+ public static void init(TableCache tableCache, HelixAdmin helixAdmin, String clusterName) {
+ registerAnnotatedProviders(tableCache, helixAdmin, clusterName);
+ }
+
+ private static String normalize(String tableName) {
+ return tableName.toLowerCase(Locale.ROOT);
+ }
+
+ private static @Nullable SystemTableDataProvider instantiateProvider(Class<? extends SystemTableDataProvider> clazz,
+ TableCache tableCache, HelixAdmin helixAdmin, String clusterName) {
+ try {
+ // Prefer the most specific constructor available.
Review Comment:
The constructor fallback logic (lines 147-163) tries multiple constructor signatures but doesn't document which constructors are expected or why. Add a comment explaining that providers can implement any of these constructor signatures: `(TableCache, HelixAdmin, String)`, `(TableCache, HelixAdmin)`, `(TableCache)`, or no-arg constructor, and that the most specific available constructor is selected.
```suggestion
// System table providers may declare any of the following constructors, in decreasing order of specificity:
// (TableCache, HelixAdmin, String), (TableCache, HelixAdmin), (TableCache), or a no-arg constructor.
// The registry will select the most specific available constructor so that providers can opt in to the
// dependencies they require without forcing all implementations to depend on Helix or cluster metadata.
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,720 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.databind.JsonNode;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.datasource.InMemorySystemTableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableDataProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final String PINOT_ADMIN_CLIENT_CLASS_NAME = "org.apache.pinot.client.admin.PinotAdminClient";
+ private static final String SIZE_CACHE_TTL_MS_PROPERTY = "pinot.systemtable.tables.sizeCacheTtlMs";
+ private static final long DEFAULT_SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+ private static final long SIZE_CACHE_TTL_MS = getNonNegativeLongProperty(SIZE_CACHE_TTL_MS_PROPERTY,
+ DEFAULT_SIZE_CACHE_TTL_MS);
+
+ private static final String CONTROLLER_TIMEOUT_MS_PROPERTY = "pinot.systemtable.tables.controllerTimeoutMs";
+ private static final long DEFAULT_CONTROLLER_TIMEOUT_MS = Duration.ofSeconds(5).toMillis();
+ private static final long CONTROLLER_TIMEOUT_MS = getPositiveLongProperty(CONTROLLER_TIMEOUT_MS_PROPERTY,
+ DEFAULT_CONTROLLER_TIMEOUT_MS);
+
+ private static final String ADMIN_TRANSPORT_REQUEST_TIMEOUT_MS = "pinot.admin.request.timeout.ms";
+ private static final String ADMIN_TRANSPORT_SCHEME = "pinot.admin.scheme";
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final @Nullable Class<?> PINOT_ADMIN_CLIENT_CLASS = loadAdminClientClass();
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+ private final Map<String, CachedSize> _sizeCache = new ConcurrentHashMap<>();
+ private final Map<String, AutoCloseable> _adminClientCache = new ConcurrentHashMap<>();
+ private final AtomicBoolean _loggedSizeFetchFailure = new AtomicBoolean();
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public TableConfig getTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ }
+
+ @Override
+ public void close()
+ throws Exception {
+ for (Map.Entry<String, AutoCloseable> entry : _adminClientCache.entrySet()) {
+ try {
+ entry.getValue().close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close admin client for {}: {}", entry.getKey(), e.toString());
+ }
+ }
+ _adminClientCache.clear();
+ }
+
+ @Override
+ public IndexSegment getDataSource() {
+ if (_tableCache == null) {
+ return new InMemorySystemTableSegment(TABLE_NAME, SCHEMA, 0, Collections.emptyMap());
+ }
+
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ List<String> controllerBaseUrls = getControllerBaseUrls();
+ Function<String, TableSize> sizeFetcher = getSizeFetcher();
+ class TableRow {
+ final String _tableNameWithType;
+ final TableType _tableType;
+ final String _rawTableName;
+ final @Nullable TableConfig _tableConfig;
+ private volatile @Nullable String _tableConfigJson;
+ private volatile @Nullable TableSize _tableSize;
+ private volatile boolean _tableSizeFetched;
+
+ private TableRow(String tableNameWithType, TableType tableType, String rawTableName,
+ @Nullable TableConfig tableConfig) {
+ _tableNameWithType = tableNameWithType;
+ _tableType = tableType;
+ _rawTableName = rawTableName;
+ _tableConfig = tableConfig;
+ }
+
+ @Nullable
+ private TableSize getTableSize() {
+ if (_tableSizeFetched) {
+ return _tableSize;
+ }
+ synchronized (this) {
+ if (_tableSizeFetched) {
+ return _tableSize;
+ }
+ _tableSize = fetchTableSize(_tableNameWithType, sizeFetcher, controllerBaseUrls);
+ _tableSizeFetched = true;
+ return _tableSize;
+ }
+ }
+
+ private String getStatus() {
+ if (_tableConfig != null) {
+ return "ONLINE";
+ }
+ TableSize sizeFromController = getTableSize();
+ int segments = sizeFromController != null ? getSegmentCount(sizeFromController, _tableType) : 0;
+ return segments > 0 ? "ONLINE" : "UNKNOWN";
+ }
+
+ private int getSegments() {
+ TableSize sizeFromController = getTableSize();
+ return sizeFromController != null ? getSegmentCount(sizeFromController, _tableType) : 0;
+ }
+
+ private long getTotalDocs() {
+ TableSize sizeFromController = getTableSize();
+ return sizeFromController != null ? TablesSystemTableProvider.this.getTotalDocs(sizeFromController, _tableType,
+ _tableNameWithType, controllerBaseUrls) : 0L;
+ }
+
+ private long getReportedSize() {
+ TableSize sizeFromController = getTableSize();
+ if (sizeFromController == null || sizeFromController._reportedSizeInBytes < 0) {
+ return 0L;
+ }
+ return sizeFromController._reportedSizeInBytes;
+ }
+
+ private long getEstimatedSize() {
+ TableSize sizeFromController = getTableSize();
+ if (sizeFromController == null || sizeFromController._estimatedSizeInBytes < 0) {
+ return 0L;
+ }
+ return sizeFromController._estimatedSizeInBytes;
+ }
+
+ private String getBrokerTenant() {
+ if (_tableConfig != null && _tableConfig.getTenantConfig() != null) {
+ String tenant = _tableConfig.getTenantConfig().getBroker();
+ return tenant != null ? tenant : "";
+ }
+ return "";
+ }
+
+ private String getServerTenant() {
+ if (_tableConfig != null && _tableConfig.getTenantConfig() != null) {
+ String tenant = _tableConfig.getTenantConfig().getServer();
+ return tenant != null ? tenant : "";
+ }
+ return "";
+ }
+
+ private int getReplicas() {
+ if (_tableConfig != null && _tableConfig.getValidationConfig() != null) {
+ Integer replicationNumber = _tableConfig.getValidationConfig().getReplicationNumber();
+ if (replicationNumber != null) {
+ return replicationNumber;
+ }
+ }
+ return 0;
+ }
+
+ private String getTableConfigJson() {
+ String cached = _tableConfigJson;
+ if (cached != null) {
+ return cached;
+ }
+ synchronized (this) {
+ cached = _tableConfigJson;
+ if (cached != null) {
+ return cached;
+ }
+ cached = "";
+ if (_tableConfig != null) {
+ try {
+ cached = JsonUtils.objectToString(_tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", _tableNameWithType, e.toString());
+ cached = _tableConfig.toString();
+ }
+ }
+ _tableConfigJson = cached;
+ return cached;
+ }
+ }
+ }
+
+ List<TableRow> tableRows = new ArrayList<>(sortedTableNames.size());
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ tableRows.add(new TableRow(tableNameWithType, tableType, rawTableName, tableConfig));
+ }
+
+ Map<String, IntFunction<Object>> valueProviders = new java.util.HashMap<>();
+ valueProviders.put("tableName", docId -> tableRows.get(docId)._rawTableName);
+ valueProviders.put("type", docId -> tableRows.get(docId)._tableType.name());
+ valueProviders.put("status", docId -> tableRows.get(docId).getStatus());
+ valueProviders.put("segments", docId -> tableRows.get(docId).getSegments());
+ valueProviders.put("totalDocs", docId -> tableRows.get(docId).getTotalDocs());
+ valueProviders.put("reportedSize", docId -> tableRows.get(docId).getReportedSize());
+ valueProviders.put("estimatedSize", docId -> tableRows.get(docId).getEstimatedSize());
+ valueProviders.put("brokerTenant", docId -> tableRows.get(docId).getBrokerTenant());
+ valueProviders.put("serverTenant", docId -> tableRows.get(docId).getServerTenant());
+ valueProviders.put("replicas", docId -> tableRows.get(docId).getReplicas());
+ valueProviders.put("tableConfig", docId -> tableRows.get(docId).getTableConfigJson());
+
+ return new InMemorySystemTableSegment(TABLE_NAME, SCHEMA, tableRows.size(), valueProviders);
+ }
+
+ @Nullable
+ private TableSize fetchTableSize(String tableNameWithType,
+ @Nullable Function<String, TableSize> fetcher, List<String> controllerBaseUrls) {
+ boolean cacheEnabled = SIZE_CACHE_TTL_MS > 0;
+ TableSize cached = cacheEnabled ? getCachedSize(tableNameWithType) : null;
+ if (cached != null) {
+ return cached;
+ }
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ if (cacheEnabled) {
+ cacheSize(tableNameWithType, fetched);
+ }
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(controllerBaseUrls, rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(controllerBaseUrls, tableNameWithType);
+ if (size == null) {
+ logSizeFetchFailure("{}: failed to fetch size for {} from controllers {} (tried '{}' and '{}')", TABLE_NAME,
+ tableNameWithType, controllerBaseUrls, rawTableName, tableNameWithType);
+ }
+ }
+ if (size != null && cacheEnabled) {
+ cacheSize(tableNameWithType, size);
+ }
+ return size;
+ }
+
+ @Nullable
+ private TableSize fetchTableSizeForName(List<String> controllerBaseUrls, String tableName) {
+ Class<?> clientClass = PINOT_ADMIN_CLIENT_CLASS;
+ if (clientClass == null) {
+ return null;
+ }
+ for (String baseUrl : controllerBaseUrls) {
+ try {
+ AutoCloseable adminClient = getOrCreateAdminClient(baseUrl);
+ if (adminClient == null) {
+ continue;
+ }
+
+ Object sizeNode;
+ try {
+ sizeNode = clientClass.getMethod("getTableSize", String.class, boolean.class, boolean.class)
+ .invoke(adminClient, tableName, true, false);
+ } catch (NoSuchMethodException e) {
+ Object tableClient = clientClass.getMethod("getTableClient").invoke(adminClient);
+ sizeNode = tableClient.getClass().getMethod("getTableSize", String.class, boolean.class, boolean.class)
+ .invoke(tableClient, tableName, true, false);
+ }
+
+ if (sizeNode == null) {
+ continue;
+ }
+
+ TableSize parsed = JsonUtils.stringToObject(sizeNode.toString(), TableSize.class);
+ LOGGER.debug("{}: controller size response for {} via {} -> segments offline={}, realtime={}, "
+ + "reportedSize={}, estimatedSize={}", TABLE_NAME, tableName, baseUrl,
+ parsed._offlineSegments != null && parsed._offlineSegments._segments != null
+ ? parsed._offlineSegments._segments.size() : 0,
+ parsed._realtimeSegments != null && parsed._realtimeSegments._segments != null
+ ? parsed._realtimeSegments._segments.size() : 0,
+ parsed._reportedSizeInBytes, parsed._estimatedSizeInBytes);
+ return parsed;
+ } catch (Exception e) {
+ logSizeFetchFailure("{}: error fetching table size for {} via {} using admin client", TABLE_NAME, tableName,
+ baseUrl, e);
+ }
+ }
+ return null;
+ }
+
+ private List<String> getControllerBaseUrls() {
+ Set<String> urls = new LinkedHashSet<>();
+ if (_helixAdmin != null) {
+ for (String controller : discoverControllersFromHelix()) {
+ String normalized = normalizeControllerUrl(controller);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ }
+ for (String url : _staticControllerUrls) {
+ String normalized = normalizeControllerUrl(url);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ return new ArrayList<>(urls);
+ }
+
+ private int getSegmentCount(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.size();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.size();
+ }
+ return 0;
+ }
+
+ private long getTotalDocsFromSize(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ return 0;
+ }
+
+ private long getTotalDocs(TableSize sizeFromController, TableType tableType, String tableNameWithType,
+ List<String> controllerBaseUrls) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ if (sizeFromController._offlineTotalDocs >= 0) {
+ return sizeFromController._offlineTotalDocs;
+ }
+ long totalDocsFromSize = getTotalDocsFromSize(sizeFromController, tableType);
+ if (totalDocsFromSize > 0) {
+ sizeFromController._offlineTotalDocs = totalDocsFromSize;
+ return totalDocsFromSize;
+ }
+ long fetched = fetchTotalDocsFromSegmentMetadata(tableNameWithType, sizeFromController._offlineSegments._segments,
+ controllerBaseUrls);
+ sizeFromController._offlineTotalDocs = fetched;
+ return fetched;
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ if (sizeFromController._realtimeTotalDocs >= 0) {
+ return sizeFromController._realtimeTotalDocs;
+ }
+ long totalDocsFromSize = getTotalDocsFromSize(sizeFromController, tableType);
+ if (totalDocsFromSize > 0) {
+ sizeFromController._realtimeTotalDocs = totalDocsFromSize;
+ return totalDocsFromSize;
+ }
+ long fetched = fetchTotalDocsFromSegmentMetadata(tableNameWithType,
+ sizeFromController._realtimeSegments._segments, controllerBaseUrls);
+ sizeFromController._realtimeTotalDocs = fetched;
+ return fetched;
Review Comment:
The `TableSize` object fields `_offlineTotalDocs` and `_realtimeTotalDocs` are mutated across multiple method calls (lines 454, 459, 469, 474) without synchronization, but the `TableSize` object may be shared via the `_sizeCache` which is a `ConcurrentHashMap`. This can lead to race conditions where multiple threads reading/writing these cached values see inconsistent state. Consider using `AtomicLong` for these fields or ensuring synchronized access when mutating cached objects.
```suggestion
long totalDocsFromSize = getTotalDocsFromSize(sizeFromController, tableType);
if (totalDocsFromSize > 0) {
return totalDocsFromSize;
}
return fetchTotalDocsFromSegmentMetadata(tableNameWithType, sizeFromController._offlineSegments._segments,
controllerBaseUrls);
}
if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
&& sizeFromController._realtimeSegments._segments != null) {
long totalDocsFromSize = getTotalDocsFromSize(sizeFromController, tableType);
if (totalDocsFromSize > 0) {
return totalDocsFromSize;
}
return fetchTotalDocsFromSegmentMetadata(tableNameWithType,
sizeFromController._realtimeSegments._segments, controllerBaseUrls);
```
##########
pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/admin/PinotSegmentApiClient.java:
##########
@@ -52,20 +54,25 @@ public static class QueryParameters {
private static final String SELECT_PATH = "/select";
private static final String METADATA_PATH = "/metadata";
+ private static String encodePathSegment(String segment) {
+ return URLEncoder.encode(segment, StandardCharsets.UTF_8);
Review Comment:
The parameter name `segment` is ambiguous—it could refer to either a URL path segment or a Pinot segment name. Consider renaming to `pathSegment` or `value` to clarify that this method encodes arbitrary path components, not specifically Pinot segments.
```suggestion
private static String encodePathSegment(String pathSegment) {
return URLEncoder.encode(pathSegment, StandardCharsets.UTF_8);
```
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/SystemTableBrokerRequestHandler.java:
##########
@@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.requesthandler;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import javax.annotation.Nullable;
+import javax.ws.rs.core.HttpHeaders;
+import org.apache.hc.client5.http.io.HttpClientConnectionManager;
+import org.apache.pinot.broker.api.AccessControl;
+import org.apache.pinot.broker.broker.AccessControlFactory;
+import org.apache.pinot.broker.querylog.QueryLogger;
+import org.apache.pinot.broker.queryquota.QueryQuotaManager;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.common.metrics.BrokerMeter;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.request.QuerySource;
+import org.apache.pinot.common.response.BrokerResponse;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.SystemTableRegistry;
+import org.apache.pinot.common.utils.config.QueryOptionsUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.core.operator.blocks.InstanceResponseBlock;
+import org.apache.pinot.core.plan.Plan;
+import org.apache.pinot.core.plan.maker.InstancePlanMakerImplV2;
+import org.apache.pinot.core.plan.maker.PlanMaker;
+import org.apache.pinot.core.query.reduce.BrokerReduceService;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils;
+import org.apache.pinot.core.routing.RoutingManager;
+import org.apache.pinot.core.transport.ServerRoutingInstance;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.SegmentContext;
+import org.apache.pinot.spi.accounting.ThreadAccountant;
+import org.apache.pinot.spi.auth.AuthorizationResult;
+import org.apache.pinot.spi.auth.broker.RequesterIdentity;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.query.QueryExecutionContext;
+import org.apache.pinot.spi.query.QueryThreadContext;
+import org.apache.pinot.spi.trace.RequestContext;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+import org.apache.pinot.sql.parsers.SqlNodeAndOptions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Broker request handler for system tables (handled entirely on the broker).
+ */
+public class SystemTableBrokerRequestHandler extends BaseBrokerRequestHandler {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableBrokerRequestHandler.class);
+
+ private final BrokerReduceService _brokerReduceService;
+ private final PlanMaker _planMaker;
+ private final ExecutorService _executorService;
+
+ public SystemTableBrokerRequestHandler(PinotConfiguration config, String brokerId,
+ BrokerRequestIdGenerator requestIdGenerator, RoutingManager routingManager,
+ AccessControlFactory accessControlFactory, QueryQuotaManager queryQuotaManager, TableCache tableCache,
+ ThreadAccountant threadAccountant) {
+ super(config, brokerId, requestIdGenerator, routingManager, accessControlFactory, queryQuotaManager, tableCache,
+ threadAccountant);
+ _brokerReduceService = new BrokerReduceService(_config);
+ _planMaker = new InstancePlanMakerImplV2();
+ _planMaker.init(_config);
+ _executorService = QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(2));
Review Comment:
The thread pool size is hardcoded to 2. This may be insufficient under load or wasteful for low-traffic deployments. Consider making this configurable via `PinotConfiguration` with a reasonable default, similar to how other executor pools are configured in Pinot.
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,720 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.databind.JsonNode;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.datasource.InMemorySystemTableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableDataProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final String PINOT_ADMIN_CLIENT_CLASS_NAME = "org.apache.pinot.client.admin.PinotAdminClient";
+ private static final String SIZE_CACHE_TTL_MS_PROPERTY = "pinot.systemtable.tables.sizeCacheTtlMs";
+ private static final long DEFAULT_SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+ private static final long SIZE_CACHE_TTL_MS = getNonNegativeLongProperty(SIZE_CACHE_TTL_MS_PROPERTY,
+ DEFAULT_SIZE_CACHE_TTL_MS);
+
+ private static final String CONTROLLER_TIMEOUT_MS_PROPERTY = "pinot.systemtable.tables.controllerTimeoutMs";
+ private static final long DEFAULT_CONTROLLER_TIMEOUT_MS = Duration.ofSeconds(5).toMillis();
+ private static final long CONTROLLER_TIMEOUT_MS = getPositiveLongProperty(CONTROLLER_TIMEOUT_MS_PROPERTY,
+ DEFAULT_CONTROLLER_TIMEOUT_MS);
+
+ private static final String ADMIN_TRANSPORT_REQUEST_TIMEOUT_MS = "pinot.admin.request.timeout.ms";
+ private static final String ADMIN_TRANSPORT_SCHEME = "pinot.admin.scheme";
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final @Nullable Class<?> PINOT_ADMIN_CLIENT_CLASS = loadAdminClientClass();
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+ private final Map<String, CachedSize> _sizeCache = new ConcurrentHashMap<>();
+ private final Map<String, AutoCloseable> _adminClientCache = new ConcurrentHashMap<>();
+ private final AtomicBoolean _loggedSizeFetchFailure = new AtomicBoolean();
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public TableConfig getTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ }
+
+ @Override
+ public void close()
+ throws Exception {
+ for (Map.Entry<String, AutoCloseable> entry : _adminClientCache.entrySet()) {
+ try {
+ entry.getValue().close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close admin client for {}: {}", entry.getKey(), e.toString());
+ }
+ }
+ _adminClientCache.clear();
+ }
+
+ @Override
+ public IndexSegment getDataSource() {
+ if (_tableCache == null) {
+ return new InMemorySystemTableSegment(TABLE_NAME, SCHEMA, 0, Collections.emptyMap());
+ }
+
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ List<String> controllerBaseUrls = getControllerBaseUrls();
+ Function<String, TableSize> sizeFetcher = getSizeFetcher();
+ class TableRow {
+ final String _tableNameWithType;
+ final TableType _tableType;
+ final String _rawTableName;
+ final @Nullable TableConfig _tableConfig;
+ private volatile @Nullable String _tableConfigJson;
+ private volatile @Nullable TableSize _tableSize;
+ private volatile boolean _tableSizeFetched;
+
+ private TableRow(String tableNameWithType, TableType tableType, String rawTableName,
+ @Nullable TableConfig tableConfig) {
+ _tableNameWithType = tableNameWithType;
+ _tableType = tableType;
+ _rawTableName = rawTableName;
+ _tableConfig = tableConfig;
+ }
+
+ @Nullable
+ private TableSize getTableSize() {
+ if (_tableSizeFetched) {
+ return _tableSize;
+ }
+ synchronized (this) {
+ if (_tableSizeFetched) {
+ return _tableSize;
+ }
+ _tableSize = fetchTableSize(_tableNameWithType, sizeFetcher, controllerBaseUrls);
+ _tableSizeFetched = true;
+ return _tableSize;
+ }
+ }
+
+ private String getStatus() {
+ if (_tableConfig != null) {
+ return "ONLINE";
+ }
+ TableSize sizeFromController = getTableSize();
+ int segments = sizeFromController != null ? getSegmentCount(sizeFromController, _tableType) : 0;
+ return segments > 0 ? "ONLINE" : "UNKNOWN";
+ }
+
+ private int getSegments() {
+ TableSize sizeFromController = getTableSize();
+ return sizeFromController != null ? getSegmentCount(sizeFromController, _tableType) : 0;
+ }
+
+ private long getTotalDocs() {
+ TableSize sizeFromController = getTableSize();
+ return sizeFromController != null ? TablesSystemTableProvider.this.getTotalDocs(sizeFromController, _tableType,
+ _tableNameWithType, controllerBaseUrls) : 0L;
+ }
+
+ private long getReportedSize() {
+ TableSize sizeFromController = getTableSize();
+ if (sizeFromController == null || sizeFromController._reportedSizeInBytes < 0) {
+ return 0L;
+ }
+ return sizeFromController._reportedSizeInBytes;
+ }
+
+ private long getEstimatedSize() {
+ TableSize sizeFromController = getTableSize();
+ if (sizeFromController == null || sizeFromController._estimatedSizeInBytes < 0) {
+ return 0L;
+ }
+ return sizeFromController._estimatedSizeInBytes;
+ }
+
+ private String getBrokerTenant() {
+ if (_tableConfig != null && _tableConfig.getTenantConfig() != null) {
+ String tenant = _tableConfig.getTenantConfig().getBroker();
+ return tenant != null ? tenant : "";
+ }
+ return "";
+ }
+
+ private String getServerTenant() {
+ if (_tableConfig != null && _tableConfig.getTenantConfig() != null) {
+ String tenant = _tableConfig.getTenantConfig().getServer();
+ return tenant != null ? tenant : "";
+ }
+ return "";
+ }
+
+ private int getReplicas() {
+ if (_tableConfig != null && _tableConfig.getValidationConfig() != null) {
+ Integer replicationNumber = _tableConfig.getValidationConfig().getReplicationNumber();
+ if (replicationNumber != null) {
+ return replicationNumber;
+ }
+ }
+ return 0;
+ }
+
+ private String getTableConfigJson() {
+ String cached = _tableConfigJson;
+ if (cached != null) {
+ return cached;
+ }
+ synchronized (this) {
+ cached = _tableConfigJson;
+ if (cached != null) {
+ return cached;
+ }
+ cached = "";
+ if (_tableConfig != null) {
+ try {
+ cached = JsonUtils.objectToString(_tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", _tableNameWithType, e.toString());
+ cached = _tableConfig.toString();
+ }
+ }
+ _tableConfigJson = cached;
+ return cached;
+ }
+ }
+ }
+
+ List<TableRow> tableRows = new ArrayList<>(sortedTableNames.size());
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ tableRows.add(new TableRow(tableNameWithType, tableType, rawTableName, tableConfig));
+ }
+
+ Map<String, IntFunction<Object>> valueProviders = new java.util.HashMap<>();
+ valueProviders.put("tableName", docId -> tableRows.get(docId)._rawTableName);
+ valueProviders.put("type", docId -> tableRows.get(docId)._tableType.name());
+ valueProviders.put("status", docId -> tableRows.get(docId).getStatus());
+ valueProviders.put("segments", docId -> tableRows.get(docId).getSegments());
+ valueProviders.put("totalDocs", docId -> tableRows.get(docId).getTotalDocs());
+ valueProviders.put("reportedSize", docId -> tableRows.get(docId).getReportedSize());
+ valueProviders.put("estimatedSize", docId -> tableRows.get(docId).getEstimatedSize());
+ valueProviders.put("brokerTenant", docId -> tableRows.get(docId).getBrokerTenant());
+ valueProviders.put("serverTenant", docId -> tableRows.get(docId).getServerTenant());
+ valueProviders.put("replicas", docId -> tableRows.get(docId).getReplicas());
+ valueProviders.put("tableConfig", docId -> tableRows.get(docId).getTableConfigJson());
+
+ return new InMemorySystemTableSegment(TABLE_NAME, SCHEMA, tableRows.size(), valueProviders);
+ }
+
+ @Nullable
+ private TableSize fetchTableSize(String tableNameWithType,
+ @Nullable Function<String, TableSize> fetcher, List<String> controllerBaseUrls) {
+ boolean cacheEnabled = SIZE_CACHE_TTL_MS > 0;
+ TableSize cached = cacheEnabled ? getCachedSize(tableNameWithType) : null;
+ if (cached != null) {
+ return cached;
+ }
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ if (cacheEnabled) {
+ cacheSize(tableNameWithType, fetched);
+ }
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(controllerBaseUrls, rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(controllerBaseUrls, tableNameWithType);
+ if (size == null) {
+ logSizeFetchFailure("{}: failed to fetch size for {} from controllers {} (tried '{}' and '{}')", TABLE_NAME,
+ tableNameWithType, controllerBaseUrls, rawTableName, tableNameWithType);
+ }
+ }
+ if (size != null && cacheEnabled) {
+ cacheSize(tableNameWithType, size);
+ }
+ return size;
+ }
+
+ @Nullable
+ private TableSize fetchTableSizeForName(List<String> controllerBaseUrls, String tableName) {
+ Class<?> clientClass = PINOT_ADMIN_CLIENT_CLASS;
+ if (clientClass == null) {
+ return null;
+ }
+ for (String baseUrl : controllerBaseUrls) {
+ try {
+ AutoCloseable adminClient = getOrCreateAdminClient(baseUrl);
+ if (adminClient == null) {
+ continue;
+ }
+
+ Object sizeNode;
+ try {
+ sizeNode = clientClass.getMethod("getTableSize", String.class, boolean.class, boolean.class)
+ .invoke(adminClient, tableName, true, false);
+ } catch (NoSuchMethodException e) {
+ Object tableClient = clientClass.getMethod("getTableClient").invoke(adminClient);
+ sizeNode = tableClient.getClass().getMethod("getTableSize", String.class, boolean.class, boolean.class)
+ .invoke(tableClient, tableName, true, false);
+ }
+
+ if (sizeNode == null) {
+ continue;
+ }
+
+ TableSize parsed = JsonUtils.stringToObject(sizeNode.toString(), TableSize.class);
+ LOGGER.debug("{}: controller size response for {} via {} -> segments offline={}, realtime={}, "
+ + "reportedSize={}, estimatedSize={}", TABLE_NAME, tableName, baseUrl,
+ parsed._offlineSegments != null && parsed._offlineSegments._segments != null
+ ? parsed._offlineSegments._segments.size() : 0,
+ parsed._realtimeSegments != null && parsed._realtimeSegments._segments != null
+ ? parsed._realtimeSegments._segments.size() : 0,
+ parsed._reportedSizeInBytes, parsed._estimatedSizeInBytes);
+ return parsed;
+ } catch (Exception e) {
+ logSizeFetchFailure("{}: error fetching table size for {} via {} using admin client", TABLE_NAME, tableName,
+ baseUrl, e);
+ }
+ }
+ return null;
+ }
+
+ private List<String> getControllerBaseUrls() {
+ Set<String> urls = new LinkedHashSet<>();
+ if (_helixAdmin != null) {
+ for (String controller : discoverControllersFromHelix()) {
+ String normalized = normalizeControllerUrl(controller);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ }
+ for (String url : _staticControllerUrls) {
+ String normalized = normalizeControllerUrl(url);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ return new ArrayList<>(urls);
+ }
+
+ private int getSegmentCount(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.size();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.size();
+ }
+ return 0;
+ }
+
+ private long getTotalDocsFromSize(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ return 0;
+ }
+
+ private long getTotalDocs(TableSize sizeFromController, TableType tableType, String tableNameWithType,
+ List<String> controllerBaseUrls) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ if (sizeFromController._offlineTotalDocs >= 0) {
+ return sizeFromController._offlineTotalDocs;
+ }
+ long totalDocsFromSize = getTotalDocsFromSize(sizeFromController, tableType);
+ if (totalDocsFromSize > 0) {
+ sizeFromController._offlineTotalDocs = totalDocsFromSize;
+ return totalDocsFromSize;
+ }
+ long fetched = fetchTotalDocsFromSegmentMetadata(tableNameWithType, sizeFromController._offlineSegments._segments,
+ controllerBaseUrls);
+ sizeFromController._offlineTotalDocs = fetched;
+ return fetched;
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ if (sizeFromController._realtimeTotalDocs >= 0) {
+ return sizeFromController._realtimeTotalDocs;
+ }
+ long totalDocsFromSize = getTotalDocsFromSize(sizeFromController, tableType);
+ if (totalDocsFromSize > 0) {
+ sizeFromController._realtimeTotalDocs = totalDocsFromSize;
+ return totalDocsFromSize;
+ }
+ long fetched = fetchTotalDocsFromSegmentMetadata(tableNameWithType,
+ sizeFromController._realtimeSegments._segments, controllerBaseUrls);
+ sizeFromController._realtimeTotalDocs = fetched;
+ return fetched;
+ }
+ return 0;
+ }
+
+ private long fetchTotalDocsFromSegmentMetadata(String tableNameWithType, Map<String, SegmentSize> segments,
+ List<String> controllerBaseUrls) {
+ if (segments.isEmpty()) {
+ return 0;
+ }
+ Class<?> clientClass = PINOT_ADMIN_CLIENT_CLASS;
+ if (clientClass == null) {
+ return 0;
+ }
+ for (String baseUrl : controllerBaseUrls) {
+ try {
+ AutoCloseable adminClient = getOrCreateAdminClient(baseUrl);
+ if (adminClient == null) {
+ continue;
+ }
+ Object segmentMetadataFetcher;
+ java.lang.reflect.Method getSegmentMetadataMethod;
+ boolean returnsJsonNode;
+ try {
+ segmentMetadataFetcher = clientClass.getMethod("getSegmentApiClient").invoke(adminClient);
+ getSegmentMetadataMethod =
+ segmentMetadataFetcher.getClass().getMethod("getSegmentMetadata", String.class, String.class);
+ returnsJsonNode = true;
+ } catch (NoSuchMethodException e) {
+ segmentMetadataFetcher = clientClass.getMethod("getSegmentClient").invoke(adminClient);
+ getSegmentMetadataMethod = segmentMetadataFetcher.getClass()
+ .getMethod("getSegmentMetadata", String.class, String.class, List.class);
+ returnsJsonNode = false;
+ }
+
+ long totalDocs = 0;
+ if (returnsJsonNode) {
+ for (String segmentName : segments.keySet()) {
+ JsonNode segmentMetadata = (JsonNode) getSegmentMetadataMethod.invoke(segmentMetadataFetcher,
+ tableNameWithType, segmentName);
+ totalDocs += segmentMetadata.path(CommonConstants.Segment.TOTAL_DOCS).asLong(0);
+ }
+ } else {
+ for (String segmentName : segments.keySet()) {
+ @SuppressWarnings("unchecked")
+ Map<String, Object> segmentMetadata = (Map<String, Object>) getSegmentMetadataMethod
+ .invoke(segmentMetadataFetcher, tableNameWithType, segmentName, List.of());
+ Object totalDocsObj = segmentMetadata.get(CommonConstants.Segment.TOTAL_DOCS);
+ if (totalDocsObj != null) {
+ totalDocs += Long.parseLong(totalDocsObj.toString());
+ }
+ }
+ }
+ return totalDocs;
+ } catch (Exception e) {
+ logSizeFetchFailure("{}: error fetching segment metadata for {} via {}", TABLE_NAME, tableNameWithType, baseUrl,
+ e);
+ }
+ }
+ return 0;
+ }
+
+ @Nullable
+ private Function<String, TableSize> getSizeFetcher() {
+ if (_tableSizeFetcherOverride != null) {
+ return _tableSizeFetcherOverride;
+ }
+ return null;
+ }
+
+ private List<String> discoverControllersFromHelix() {
+ List<String> urls = new ArrayList<>();
+ try {
+ if (_clusterName == null) {
+ LOGGER.warn("Cannot discover controllers without cluster name");
+ return List.of();
+ }
+ for (String controllerId : _helixAdmin.getInstancesInCluster(_clusterName)) {
+ if (!InstanceTypeUtils.isController(controllerId)) {
+ continue;
+ }
+ int firstUnderscore = controllerId.indexOf('_');
+ int lastUnderscore = controllerId.lastIndexOf('_');
+ if (firstUnderscore > 0 && lastUnderscore > firstUnderscore && lastUnderscore < controllerId.length() - 1) {
+ String host = controllerId.substring(firstUnderscore + 1, lastUnderscore);
+ String port = controllerId.substring(lastUnderscore + 1);
+ if (!host.isEmpty() && !port.isEmpty()) {
+ urls.add(host + ":" + port);
+ } else {
+ LOGGER.warn("Unable to parse controller address from instance id (empty host or port): {}", controllerId);
+ }
+ } else {
+ LOGGER.warn("Unable to parse controller address from instance id '{}' (expected 'Controller_<host>_<port>')",
+ controllerId);
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Failed to discover controllers from Helix", e);
+ }
+ return urls;
+ }
+
+ @Nullable
+ private AutoCloseable getOrCreateAdminClient(String controllerBaseUrl) {
+ Class<?> clientClass = PINOT_ADMIN_CLIENT_CLASS;
+ if (clientClass == null) {
+ return null;
+ }
+ String normalized = normalizeControllerUrl(controllerBaseUrl);
+ if (normalized == null) {
+ return null;
+ }
+ return _adminClientCache.computeIfAbsent(normalized, controller -> {
+ try {
+ String controllerAddress = stripScheme(controller);
+ Properties properties = new Properties();
+ properties.setProperty(ADMIN_TRANSPORT_REQUEST_TIMEOUT_MS, String.valueOf(CONTROLLER_TIMEOUT_MS));
+ properties.setProperty(ADMIN_TRANSPORT_SCHEME, controller.startsWith("https://") ? "https" : "http");
+ return (AutoCloseable) clientClass.getConstructor(String.class, Properties.class)
+ .newInstance(controllerAddress, properties);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to create admin client for {}: {}", controller, e.toString());
+ return null;
+ }
+ });
+ }
+
+ private static String normalizeControllerUrl(@Nullable String controllerUrl) {
+ if (controllerUrl == null || controllerUrl.isEmpty()) {
+ return null;
+ }
+ String normalized = controllerUrl;
+ if (!normalized.startsWith("http://") && !normalized.startsWith("https://")) {
+ normalized = "http://" + normalized;
+ }
+ if (normalized.endsWith("/")) {
+ normalized = normalized.substring(0, normalized.length() - 1);
+ }
+ return normalized;
+ }
+
+ private static String stripScheme(String controllerUrl) {
+ if (controllerUrl == null) {
Review Comment:
This method can return null (line 618), but the caller at line 589 uses the result directly without null checking: `stripScheme(controller)`. If `controller` is somehow null (though unlikely given the computeIfAbsent logic), this would result in passing null to the PinotAdminClient constructor, potentially causing a NullPointerException.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2625181560
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/SystemTableRegistry.java:
##########
@@ -0,0 +1,129 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.ServiceLoader;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Registry to hold and lifecycle-manage system table providers.
+ */
+public final class SystemTableRegistry implements AutoCloseable {
+ public static final SystemTableRegistry INSTANCE = new SystemTableRegistry();
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableRegistry.class);
+
+ private final Map<String, SystemTableProvider> _providers = new ConcurrentHashMap<>();
+
+ public void register(SystemTableProvider provider) {
+ _providers.put(normalize(provider.getTableName()), provider);
+ }
+
+ public @Nullable SystemTableProvider get(String tableName) {
+ return _providers.get(normalize(tableName));
+ }
+
+ public boolean isRegistered(String tableName) {
+ return _providers.containsKey(normalize(tableName));
+ }
+
+ public Collection<SystemTableProvider> getProviders() {
+ return Collections.unmodifiableCollection(_providers.values());
+ }
+
+ /**
+ * Discover and register providers marked with {@link SystemTable} using the available dependencies.
+ */
+ public void registerAnnotatedProviders(@Nullable TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ for (SystemTableProvider discovered : ServiceLoader.load(SystemTableProvider.class)) {
+ Class<? extends SystemTableProvider> clazz = discovered.getClass();
+ if (!clazz.isAnnotationPresent(SystemTable.class)) {
+ continue;
+ }
+ Optional<SystemTableProvider> instantiated =
+ instantiateProvider(clazz, tableCache, helixAdmin, clusterName);
+ instantiated.ifPresent(provider -> {
+ if (isRegistered(provider.getTableName())) {
+ return;
+ }
+ LOGGER.info("Registering system table provider: {}", provider.getTableName());
+ register(provider);
+ });
+ }
+ }
+
+ @Override
+ public void close()
+ throws Exception {
+ for (SystemTableProvider provider : _providers.values()) {
+ provider.close();
+ }
+ }
+
+ private static String normalize(String tableName) {
+ return tableName.toLowerCase(Locale.ROOT);
+ }
+
+ private Optional<SystemTableProvider> instantiateProvider(Class<? extends SystemTableProvider> clazz,
+ @Nullable TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName) {
Review Comment:
Will they ever be `null`?
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/SystemTableRegistry.java:
##########
@@ -0,0 +1,129 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.ServiceLoader;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Registry to hold and lifecycle-manage system table providers.
+ */
+public final class SystemTableRegistry implements AutoCloseable {
+ public static final SystemTableRegistry INSTANCE = new SystemTableRegistry();
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableRegistry.class);
+
+ private final Map<String, SystemTableProvider> _providers = new ConcurrentHashMap<>();
Review Comment:
Regular map should be enough as we are not dynamically registering
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/SystemTableRegistry.java:
##########
@@ -0,0 +1,129 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.ServiceLoader;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Registry to hold and lifecycle-manage system table providers.
+ */
+public final class SystemTableRegistry implements AutoCloseable {
+ public static final SystemTableRegistry INSTANCE = new SystemTableRegistry();
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableRegistry.class);
+
+ private final Map<String, SystemTableProvider> _providers = new ConcurrentHashMap<>();
+
+ public void register(SystemTableProvider provider) {
+ _providers.put(normalize(provider.getTableName()), provider);
+ }
+
+ public @Nullable SystemTableProvider get(String tableName) {
+ return _providers.get(normalize(tableName));
+ }
+
+ public boolean isRegistered(String tableName) {
+ return _providers.containsKey(normalize(tableName));
+ }
+
+ public Collection<SystemTableProvider> getProviders() {
+ return Collections.unmodifiableCollection(_providers.values());
+ }
+
+ /**
+ * Discover and register providers marked with {@link SystemTable} using the available dependencies.
+ */
+ public void registerAnnotatedProviders(@Nullable TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ for (SystemTableProvider discovered : ServiceLoader.load(SystemTableProvider.class)) {
+ Class<? extends SystemTableProvider> clazz = discovered.getClass();
+ if (!clazz.isAnnotationPresent(SystemTable.class)) {
+ continue;
+ }
+ Optional<SystemTableProvider> instantiated =
+ instantiateProvider(clazz, tableCache, helixAdmin, clusterName);
+ instantiated.ifPresent(provider -> {
+ if (isRegistered(provider.getTableName())) {
+ return;
+ }
+ LOGGER.info("Registering system table provider: {}", provider.getTableName());
+ register(provider);
+ });
+ }
+ }
+
+ @Override
+ public void close()
+ throws Exception {
+ for (SystemTableProvider provider : _providers.values()) {
+ provider.close();
+ }
+ }
+
+ private static String normalize(String tableName) {
+ return tableName.toLowerCase(Locale.ROOT);
+ }
+
+ private Optional<SystemTableProvider> instantiateProvider(Class<? extends SystemTableProvider> clazz,
+ @Nullable TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName) {
+ try {
+ // Prefer the most specific constructor available.
+ if (tableCache != null && helixAdmin != null && clusterName != null) {
+ try {
+ return Optional.of(clazz.getDeclaredConstructor(TableCache.class, HelixAdmin.class, String.class)
+ .newInstance(tableCache, helixAdmin, clusterName));
Review Comment:
We usually use `init()` method to initialize the plug-ins
##########
pinot-spi/src/main/java/org/apache/pinot/spi/systemtable/SystemTableRequest.java:
##########
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.systemtable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+
+/**
+ * Request sent to a {@link SystemTableProvider} from the system query engine.
+ */
+public final class SystemTableRequest {
Review Comment:
Is this enough to cover all system table requests?
Why not using the existing request format?
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/SystemTableRegistry.java:
##########
@@ -0,0 +1,129 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.ServiceLoader;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Registry to hold and lifecycle-manage system table providers.
+ */
+public final class SystemTableRegistry implements AutoCloseable {
+ public static final SystemTableRegistry INSTANCE = new SystemTableRegistry();
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableRegistry.class);
+
+ private final Map<String, SystemTableProvider> _providers = new ConcurrentHashMap<>();
+
+ public void register(SystemTableProvider provider) {
+ _providers.put(normalize(provider.getTableName()), provider);
+ }
+
+ public @Nullable SystemTableProvider get(String tableName) {
Review Comment:
(nit) Put `@Nullable` in a separate line
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/SystemTableRegistry.java:
##########
@@ -0,0 +1,129 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.ServiceLoader;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Registry to hold and lifecycle-manage system table providers.
+ */
+public final class SystemTableRegistry implements AutoCloseable {
+ public static final SystemTableRegistry INSTANCE = new SystemTableRegistry();
Review Comment:
Directly exposing singleton instance might not be good practice. Given this is just a registry, making it a util class might be good enough (similar to `ConnectorRegistry`)
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/SystemTableRegistry.java:
##########
@@ -0,0 +1,129 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.ServiceLoader;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Registry to hold and lifecycle-manage system table providers.
+ */
+public final class SystemTableRegistry implements AutoCloseable {
+ public static final SystemTableRegistry INSTANCE = new SystemTableRegistry();
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableRegistry.class);
+
+ private final Map<String, SystemTableProvider> _providers = new ConcurrentHashMap<>();
+
+ public void register(SystemTableProvider provider) {
+ _providers.put(normalize(provider.getTableName()), provider);
+ }
+
+ public @Nullable SystemTableProvider get(String tableName) {
+ return _providers.get(normalize(tableName));
+ }
+
+ public boolean isRegistered(String tableName) {
+ return _providers.containsKey(normalize(tableName));
+ }
+
+ public Collection<SystemTableProvider> getProviders() {
+ return Collections.unmodifiableCollection(_providers.values());
+ }
+
+ /**
+ * Discover and register providers marked with {@link SystemTable} using the available dependencies.
+ */
+ public void registerAnnotatedProviders(@Nullable TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ for (SystemTableProvider discovered : ServiceLoader.load(SystemTableProvider.class)) {
+ Class<? extends SystemTableProvider> clazz = discovered.getClass();
+ if (!clazz.isAnnotationPresent(SystemTable.class)) {
+ continue;
+ }
+ Optional<SystemTableProvider> instantiated =
+ instantiateProvider(clazz, tableCache, helixAdmin, clusterName);
+ instantiated.ifPresent(provider -> {
+ if (isRegistered(provider.getTableName())) {
+ return;
+ }
+ LOGGER.info("Registering system table provider: {}", provider.getTableName());
+ register(provider);
+ });
+ }
+ }
+
+ @Override
+ public void close()
+ throws Exception {
+ for (SystemTableProvider provider : _providers.values()) {
+ provider.close();
+ }
+ }
+
+ private static String normalize(String tableName) {
+ return tableName.toLowerCase(Locale.ROOT);
+ }
+
+ private Optional<SystemTableProvider> instantiateProvider(Class<? extends SystemTableProvider> clazz,
Review Comment:
We are mixing usage of `null` and `Optional`. Suggest make it return `@Nullable` instead
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java:
##########
@@ -176,13 +184,15 @@ public abstract class BaseSingleStageBrokerRequestHandler extends BaseBrokerRequ
protected BlockingQueue<Pair<String, String>> _multistageCompileQueryQueue;
protected ImplicitHybridTableRouteProvider _implicitHybridTableRouteProvider;
protected LogicalTableRouteProvider _logicalTableRouteProvider;
+ protected final SystemTableRegistry _systemTableRegistry;
Review Comment:
Consider adding a broker request handler for system table. We shouldn't classify it under SSE
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/SystemTableRegistry.java:
##########
@@ -0,0 +1,129 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.ServiceLoader;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Registry to hold and lifecycle-manage system table providers.
+ */
+public final class SystemTableRegistry implements AutoCloseable {
+ public static final SystemTableRegistry INSTANCE = new SystemTableRegistry();
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableRegistry.class);
+
+ private final Map<String, SystemTableProvider> _providers = new ConcurrentHashMap<>();
+
+ public void register(SystemTableProvider provider) {
+ _providers.put(normalize(provider.getTableName()), provider);
+ }
+
+ public @Nullable SystemTableProvider get(String tableName) {
+ return _providers.get(normalize(tableName));
+ }
+
+ public boolean isRegistered(String tableName) {
+ return _providers.containsKey(normalize(tableName));
+ }
+
+ public Collection<SystemTableProvider> getProviders() {
Review Comment:
(minor) Not used
##########
pinot-spi/src/main/java/org/apache/pinot/spi/systemtable/SystemTableResponse.java:
##########
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.systemtable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import org.apache.pinot.spi.data.readers.GenericRow;
+
+
+/**
+ * Response returned by {@link SystemTableProvider} containing materialized rows.
+ */
+public final class SystemTableResponse {
Review Comment:
Any specific reason why not reuse the existing response format? Is there certain standard for system table response?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2874643671
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/SystemTableRegistry.java:
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.IdentityHashMap;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.PinotReflectionUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Registry to hold and lifecycle-manage system table data providers.
+ * <p>
+ * This class is thread-safe and manages the lifecycle of registered providers.
+ */
+public final class SystemTableRegistry implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableRegistry.class);
+ private static final String SYSTEM_TABLE_PREFIX = "system.";
+
+ // Providers are registered once at broker startup.
+ private final Map<String, SystemTableProvider> _providers = new HashMap<>();
+ private final SystemTableProviderContext _context;
+ private volatile boolean _closed = false;
+
+ /**
+ * Creates a new SystemTableRegistry and initializes it with annotated providers.
+ */
+ public SystemTableRegistry(TableCache tableCache, HelixAdmin helixAdmin, String clusterName,
+ @Nullable PinotConfiguration config) {
+ _context = new SystemTableProviderContext(tableCache, helixAdmin, clusterName, config);
+ registerAnnotatedProviders();
+ }
+
+ /**
+ * Registers a system table provider.
+ * <p>
+ * Note: This method does NOT call {@link SystemTableProvider#init(SystemTableProviderContext)} on the provider.
+ * It is intended for registering providers that have already been initialized (e.g. in tests).
+ *
+ * @throws IllegalArgumentException if the table name does not start with "system."
+ */
+ public void register(SystemTableProvider provider) {
+ String tableName = provider.getTableName();
+ if (!tableName.toLowerCase(Locale.ROOT).startsWith(SYSTEM_TABLE_PREFIX)) {
+ throw new IllegalArgumentException(
+ "System table name must start with '" + SYSTEM_TABLE_PREFIX + "', got: " + tableName);
+ }
+ synchronized (_providers) {
+ if (_closed) {
+ throw new IllegalStateException("Cannot register provider after registry is closed");
+ }
+ _providers.put(normalize(tableName), provider);
Review Comment:
`SystemTableRegistry.register()` unconditionally overwrites any existing provider for the same normalized table name. That can mask duplicate providers and can leak resources because the displaced provider is never closed. Consider rejecting duplicates (throw) or closing/replacing explicitly with logging.
```suggestion
String normalizedTableName = normalize(tableName);
SystemTableProvider existingProvider = _providers.get(normalizedTableName);
if (existingProvider != null && existingProvider != provider) {
throw new IllegalStateException(
"System table provider already registered for table '" + normalizedTableName + "'");
}
_providers.put(normalizedTableName, provider);
```
##########
pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/admin/PinotTableAdminClient.java:
##########
@@ -185,6 +185,23 @@ public List<String> cancelRebalance(String tableName)
return _transport.parseStringArray(response, "jobIds");
}
+ /**
+ * Fetches the table size details.
+ *
+ * @param tableName Table name (raw or with type)
+ * @param verbose Whether to include per-segment details
+ * @param includeReplacedSegments Whether to include replaced segments
+ * @return Table size response as JsonNode
+ * @throws PinotAdminException If the request fails
+ */
+ public JsonNode getTableSize(String tableName, boolean verbose, boolean includeReplacedSegments)
+ throws PinotAdminException {
+ Map<String, String> queryParams = new HashMap<>();
+ queryParams.put("verbose", String.valueOf(verbose));
+ queryParams.put("includeReplacedSegments", String.valueOf(includeReplacedSegments));
+ return _transport.executeGet(_controllerAddress, "/tables/" + tableName + "/size", queryParams, _headers);
+ }
Review Comment:
`getTableSize()` builds the path with the raw `tableName` ("/tables/" + tableName + "/size") but does not URL-encode it. Other endpoints in this PR encode path segments to handle special characters safely; this one should do the same to avoid malformed requests for table names containing characters like spaces or '/'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2649756663
##########
pinot-plugins/pinot-system-table/src/main/java/org/apache/pinot/common/systemtable/datasource/InMemorySystemTableSegment.java:
##########
@@ -0,0 +1,723 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.datasource;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.node.JsonNodeFactory;
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.IntFunction;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.Constants;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.SegmentVersion;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.datasource.DataSourceMetadata;
+import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexType;
+import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.column.ColumnIndexContainer;
+import org.apache.pinot.segment.spi.index.multicolumntext.MultiColumnTextMetadata;
+import org.apache.pinot.segment.spi.index.reader.BloomFilterReader;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReaderContext;
+import org.apache.pinot.segment.spi.index.reader.H3IndexReader;
+import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
+import org.apache.pinot.segment.spi.index.reader.JsonIndexReader;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+import org.apache.pinot.segment.spi.index.reader.RangeIndexReader;
+import org.apache.pinot.segment.spi.index.reader.TextIndexReader;
+import org.apache.pinot.segment.spi.index.reader.VectorIndexReader;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2Metadata;
+import org.apache.pinot.segment.spi.partition.PartitionFunction;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.joda.time.Duration;
+import org.joda.time.Interval;
+
+
+/**
+ * In-memory {@link IndexSegment} implementation intended for system table queries.
+ * <p>
+ * This segment is backed by per-column value functions (docId -> value) and exposes raw forward indexes (no
+ * dictionaries/inverted indexes) so that the standard v1 query engine can operate on it.
+ */
+public final class InMemorySystemTableSegment implements IndexSegment {
+ private final String _segmentName;
+ private final Schema _schema;
+ private final int _numDocs;
+ private final Map<String, IntFunction<Object>> _valueProvidersByColumn;
+ private final Map<String, DataSource> _dataSourcesByColumn;
+ private final SegmentMetadata _segmentMetadata;
+
+ public InMemorySystemTableSegment(String segmentName, Schema schema, int numDocs,
+ Map<String, IntFunction<Object>> valueProvidersByColumn) {
+ _segmentName = segmentName;
+ _schema = schema;
+ _numDocs = numDocs;
+ _valueProvidersByColumn = new HashMap<>(valueProvidersByColumn);
+ _dataSourcesByColumn = new HashMap<>(schema.getColumnNames().size());
+ for (String column : schema.getColumnNames()) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ if (fieldSpec == null) {
+ continue;
+ }
+ IntFunction<Object> provider = _valueProvidersByColumn.get(column);
+ if (provider == null) {
+ provider = docId -> defaultValue(fieldSpec.getDataType());
+ _valueProvidersByColumn.put(column, provider);
+ }
+ _dataSourcesByColumn.put(column, new FunctionBasedDataSource(fieldSpec, numDocs, provider));
+ }
+ _segmentMetadata = new InMemorySegmentMetadata(segmentName, schema, numDocs);
+ }
+
+ @Override
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ @Override
+ public SegmentMetadata getSegmentMetadata() {
+ return _segmentMetadata;
+ }
+
+ @Override
+ public Set<String> getColumnNames() {
+ return _schema.getColumnNames();
+ }
+
+ @Override
+ public Set<String> getPhysicalColumnNames() {
+ return _schema.getPhysicalColumnNames();
+ }
+
+ @Nullable
+ @Override
+ public DataSource getDataSourceNullable(String column) {
+ return _dataSourcesByColumn.get(column);
+ }
+
+ @Override
+ public DataSource getDataSource(String column, Schema schema) {
+ DataSource dataSource = getDataSourceNullable(column);
+ if (dataSource != null) {
+ return dataSource;
+ }
+ throw new IllegalStateException("Failed to find data source for column: " + column);
+ }
+
+ @Nullable
+ @Override
+ public List<StarTreeV2> getStarTrees() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public TextIndexReader getMultiColumnTextIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public org.apache.pinot.segment.spi.index.mutable.ThreadSafeMutableRoaringBitmap getValidDocIds() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public org.apache.pinot.segment.spi.index.mutable.ThreadSafeMutableRoaringBitmap getQueryableDocIds() {
Review Comment:
Use import statements for `ThreadSafeMutableRoaringBitmap` instead of fully qualified names on lines 152 and 158.
##########
pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/admin/PinotSegmentAdminClient.java:
##########
@@ -36,6 +38,10 @@ public class PinotSegmentAdminClient {
private final String _controllerAddress;
private final Map<String, String> _headers;
+ private static String encodePathSegment(String segment) {
+ return URLEncoder.encode(segment, StandardCharsets.UTF_8);
Review Comment:
URLEncoder.encode uses form encoding (application/x-www-form-urlencoded) which converts spaces to '+' rather than '%20'. For URL path segments, use `URLEncoder.encode().replace(\"+\", \"%20\")` or a dedicated URI encoding utility to ensure RFC 3986 compliance.
```suggestion
return URLEncoder.encode(segment, StandardCharsets.UTF_8).replace("+", "%20");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2617600154
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,603 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTableFilter;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.apache.pinot.spi.systemtable.SystemTableRequest;
+import org.apache.pinot.spi.systemtable.SystemTableResponse;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final boolean IS_ADMIN_CLIENT_AVAILABLE;
+ static {
+ boolean available;
+ try {
+ Class.forName("org.apache.pinot.client.admin.PinotAdminClient");
+ available = true;
+ } catch (ClassNotFoundException e) {
+ available = false;
+ }
+ IS_ADMIN_CLIENT_AVAILABLE = available;
+ }
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public SystemTableResponse getRows(SystemTableRequest request) {
+ if (_tableCache == null) {
+ return new SystemTableResponse(List.of(), System.currentTimeMillis(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ int offset = Math.max(0, request.getOffset());
+ int limit = request.getLimit();
+ boolean hasLimit = limit > 0;
+ int totalRows = 0;
+ int initialCapacity = hasLimit ? Math.min(sortedTableNames.size(), limit) : 0;
+ List<GenericRow> rows = new ArrayList<>(initialCapacity);
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType);
+ if (!matchesFilter(request.getFilter(), stats, rawTableName)) {
+ continue;
+ }
+ totalRows++;
+ if (offset > 0) {
+ offset--;
+ continue;
+ }
Review Comment:
When limit is 0, the provider should not collect any rows but should still count totalRows for the response. However, this condition is checked after totalRows++ but before offset handling completes. If offset > 0 and limit == 0, this logic can prematurely skip rows that should be counted in totalRows. The limit check should occur after offset handling is complete.
```suggestion
if (offset > 0) {
offset--;
continue;
}
totalRows++;
```
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java:
##########
@@ -1033,6 +1046,247 @@ private void throwAccessDeniedError(long requestId, String query, RequestContext
throw new WebApplicationException("Permission denied." + failureMessage, Response.Status.FORBIDDEN);
}
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ private BrokerResponse handleSystemTableQuery(PinotQuery pinotQuery, String tableName,
+ RequestContext requestContext, @Nullable RequesterIdentity requesterIdentity, String query) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableProvider provider = _systemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return BrokerResponseNative.TABLE_DOES_NOT_EXIST;
+ }
+ try {
+ if (!isSupportedSystemTableQuery(pinotQuery)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION,
+ "System tables only support simple projection/filter/limit queries");
+ }
+ Schema systemSchema = provider.getSchema();
+ List<String> projectionColumns = extractProjectionColumns(pinotQuery, systemSchema);
+ int offset = Math.max(0, pinotQuery.getOffset());
+ int limit = Math.max(0, pinotQuery.getLimit());
+ SystemTableRequest systemTableRequest =
+ new SystemTableRequest(projectionColumns, toSystemTableFilter(pinotQuery.getFilterExpression()), offset,
+ limit);
+ SystemTableResponse systemTableResponse = provider.getRows(systemTableRequest);
+ BrokerResponseNative brokerResponse =
+ buildSystemTableBrokerResponse(tableName, systemSchema, projectionColumns, systemTableResponse,
+ requestContext);
+ brokerResponse.setTimeUsedMs(System.currentTimeMillis() - requestContext.getRequestArrivalTimeMillis());
+ _queryLogger.log(new QueryLogger.QueryLogParams(requestContext, tableName, brokerResponse,
+ QueryLogger.QueryLogParams.QueryEngine.SINGLE_STAGE, requesterIdentity, null));
+ return brokerResponse;
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table query {}: {}", tableName, e.getMessage(), e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+
+ // System tables are broker-served virtual tables; they only support projection/filter/offset/limit today.
+ private boolean isSupportedSystemTableQuery(PinotQuery pinotQuery) {
+ return (pinotQuery.getGroupByList() == null || pinotQuery.getGroupByList().isEmpty())
+ && pinotQuery.getHavingExpression() == null
+ && (pinotQuery.getOrderByList() == null || pinotQuery.getOrderByList().isEmpty());
+ }
+
+ private List<String> extractProjectionColumns(PinotQuery pinotQuery, Schema schema)
+ throws BadQueryRequestException {
+ List<String> projections = new ArrayList<>();
+ boolean hasStar = false;
+ List<Expression> selectList = pinotQuery.getSelectList();
+ if (CollectionUtils.isEmpty(selectList)) {
+ throw new BadQueryRequestException("System tables require a projection list");
+ }
+ for (Expression expression : selectList) {
+ Identifier identifier = expression.getIdentifier();
+ if (identifier != null) {
+ if ("*".equals(identifier.getName())) {
+ hasStar = true;
+ } else {
+ projections.add(identifier.getName());
+ }
+ continue;
+ }
+ Function function = expression.getFunctionCall();
+ if (function != null && "AS".equalsIgnoreCase(function.getOperator()) && !function.getOperands().isEmpty()) {
+ Identifier aliased = function.getOperands().get(0).getIdentifier();
+ if (aliased != null) {
+ projections.add(aliased.getName());
+ continue;
+ }
+ }
+ throw new BadQueryRequestException("System tables only support column projections or '*'");
+ }
+ if (hasStar) {
+ projections = new ArrayList<>(schema.getColumnNames());
+ } else if (projections.isEmpty()) {
+ throw new BadQueryRequestException("No valid projection columns found in SELECT list for system table");
Review Comment:
This error condition appears unreachable based on the logic flow. If hasStar is true, projections gets reassigned to all schema columns. If hasStar is false and projections is empty, it means the select list had no identifiers and no AS functions with identifiers, which would have thrown an exception at line 1128. Consider removing this check or adding a code comment explaining when this can occur.
```suggestion
```
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/DatabaseUtils.java:
##########
@@ -67,6 +67,19 @@ public static String translateTableName(String tableName, @Nullable String datab
case 2:
Preconditions.checkArgument(!tableSplit[1].isEmpty(), "Invalid table name '%s'", tableName);
String databasePrefix = tableSplit[0];
+ /*
+ * Allow system tables to bypass database prefix validation so they can be queried regardless of the database
+ * header. System tables are intended to be globally accessible and are not subject to database-level access
+ * controls. Ensure system tables do not expose sensitive information because they can be queried without a
+ * matching database context.
+ *
+ * Security note: system tables are intentionally exempt from database-scoped access control; broader Pinot
+ * security is handled by AccessControl implementations (see org.apache.pinot.spi.security.AccessControl) and
+ * the Pinot security model documentation. New system tables should be vetted for sensitive content.
+ */
Review Comment:
While this security note is helpful, consider adding it to the SystemTableProvider interface documentation as well, since that's where new providers are implemented. This ensures developers creating new system tables see the security guidance at the primary extension point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2650760654
##########
pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/admin/PinotSegmentAdminClient.java:
##########
@@ -36,6 +38,10 @@ public class PinotSegmentAdminClient {
private final String _controllerAddress;
private final Map<String, String> _headers;
+ private static String encodePathSegment(String segment) {
+ return URLEncoder.encode(segment, StandardCharsets.UTF_8).replace("+", "%20");
Review Comment:
The space-encoding workaround (`.replace("+", "%20")`) is applied here but not in PinotSegmentApiClient's `encodePathSegment`. For consistency, both implementations should use the same encoding strategy. Consider centralizing this logic in a shared utility method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2659005435
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/SystemTableBrokerRequestHandler.java:
##########
@@ -0,0 +1,550 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.requesthandler;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.Nullable;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.client5.http.classic.methods.HttpPost;
+import org.apache.hc.client5.http.config.RequestConfig;
+import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
+import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse;
+import org.apache.hc.client5.http.impl.classic.HttpClients;
+import org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager;
+import org.apache.hc.client5.http.io.HttpClientConnectionManager;
+import org.apache.hc.core5.http.io.entity.EntityUtils;
+import org.apache.hc.core5.http.io.entity.StringEntity;
+import org.apache.hc.core5.util.Timeout;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.HelixManager;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.pinot.broker.api.AccessControl;
+import org.apache.pinot.broker.broker.AccessControlFactory;
+import org.apache.pinot.broker.querylog.QueryLogger;
+import org.apache.pinot.broker.queryquota.QueryQuotaManager;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.common.datatable.DataTableFactory;
+import org.apache.pinot.common.datatable.DataTableImplV4;
+import org.apache.pinot.common.http.PoolingHttpClientConnectionManagerHelper;
+import org.apache.pinot.common.metrics.BrokerMeter;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.request.QuerySource;
+import org.apache.pinot.common.response.BrokerResponse;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.SystemTableRegistry;
+import org.apache.pinot.common.utils.NamedThreadFactory;
+import org.apache.pinot.common.utils.config.InstanceUtils;
+import org.apache.pinot.common.utils.config.QueryOptionsUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.core.operator.blocks.InstanceResponseBlock;
+import org.apache.pinot.core.plan.Plan;
+import org.apache.pinot.core.plan.maker.InstancePlanMakerImplV2;
+import org.apache.pinot.core.plan.maker.PlanMaker;
+import org.apache.pinot.core.query.reduce.BrokerReduceService;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils;
+import org.apache.pinot.core.routing.MultiClusterRoutingContext;
+import org.apache.pinot.core.routing.RoutingManager;
+import org.apache.pinot.core.transport.ServerRoutingInstance;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.SegmentContext;
+import org.apache.pinot.spi.accounting.ThreadAccountant;
+import org.apache.pinot.spi.auth.AuthorizationResult;
+import org.apache.pinot.spi.auth.broker.RequesterIdentity;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.query.QueryExecutionContext;
+import org.apache.pinot.spi.query.QueryThreadContext;
+import org.apache.pinot.spi.trace.RequestContext;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+import org.apache.pinot.sql.parsers.SqlNodeAndOptions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Broker request handler for system tables (handled entirely on the broker).
+ */
+public class SystemTableBrokerRequestHandler extends BaseBrokerRequestHandler {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableBrokerRequestHandler.class);
+ private static final String SYSTEM_TABLE_PSEUDO_HOST = "localhost";
+ private static final int SYSTEM_TABLE_PSEUDO_PORT = 0;
+ private static final String SYSTEM_TABLE_DATATABLE_API_PATH = "/query/systemTable/datatable";
+ private static final Set<String> HOP_BY_HOP_HEADERS_TO_SKIP =
+ Set.of("connection", "content-length", "transfer-encoding", "host");
+
+ private final BrokerReduceService _brokerReduceService;
+ private final PlanMaker _planMaker;
+ private final ExecutorService _executorService;
+ private final ExecutorService _scatterGatherExecutorService;
+ private final PoolingHttpClientConnectionManager _scatterGatherConnMgr;
+ private final CloseableHttpClient _scatterGatherHttpClient;
+ @Nullable
+ private final HelixManager _helixManager;
+
+ public SystemTableBrokerRequestHandler(PinotConfiguration config, String brokerId,
+ BrokerRequestIdGenerator requestIdGenerator, RoutingManager routingManager,
+ AccessControlFactory accessControlFactory, QueryQuotaManager queryQuotaManager, TableCache tableCache,
+ ThreadAccountant threadAccountant, @Nullable MultiClusterRoutingContext multiClusterRoutingContext,
+ @Nullable HelixManager helixManager) {
+ super(config, brokerId, requestIdGenerator, routingManager, accessControlFactory, queryQuotaManager, tableCache,
+ threadAccountant, multiClusterRoutingContext);
+ _brokerReduceService = new BrokerReduceService(_config);
+ _planMaker = new InstancePlanMakerImplV2();
+ _planMaker.init(_config);
+ _helixManager = helixManager;
+ int executorPoolSize = config.getProperty(CommonConstants.Broker.CONFIG_OF_SYSTEM_TABLE_EXECUTOR_POOL_SIZE,
+ CommonConstants.Broker.DEFAULT_SYSTEM_TABLE_EXECUTOR_POOL_SIZE);
+ executorPoolSize = Math.max(1, executorPoolSize);
+ _executorService = QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(executorPoolSize,
+ new NamedThreadFactory("system-table-query-executor")));
+ _scatterGatherExecutorService =
+ QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(executorPoolSize,
+ new NamedThreadFactory("system-table-scatter-gather-executor")));
+ _scatterGatherConnMgr = PoolingHttpClientConnectionManagerHelper.createWithSocketFactory();
+ Timeout timeout = Timeout.of(_brokerTimeoutMs, TimeUnit.MILLISECONDS);
+ RequestConfig defaultRequestConfig =
+ RequestConfig.custom().setConnectionRequestTimeout(timeout).setResponseTimeout(timeout).build();
+ _scatterGatherHttpClient =
+ HttpClients.custom().setConnectionManager(_scatterGatherConnMgr).setDefaultRequestConfig(defaultRequestConfig)
+ .build();
+ }
+
+ @Override
+ public void start() {
+ }
+
+ @Override
+ public void shutDown() {
+ _executorService.shutdownNow();
+ _scatterGatherExecutorService.shutdownNow();
+ try {
+ _scatterGatherHttpClient.close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close system table scatter-gather http client: {}", e.toString());
+ }
+ try {
+ _scatterGatherConnMgr.close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close system table scatter-gather connection manager: {}", e.toString());
+ }
+ _brokerReduceService.shutDown();
+ }
+
+ public boolean canHandle(String tableName) {
+ return isSystemTable(tableName) && SystemTableRegistry.isRegistered(tableName);
+ }
+
+ @Override
+ protected BrokerResponse handleRequest(long requestId, String query, SqlNodeAndOptions sqlNodeAndOptions,
+ JsonNode request, @Nullable RequesterIdentity requesterIdentity, RequestContext requestContext,
+ @Nullable HttpHeaders httpHeaders, AccessControl accessControl)
+ throws Exception {
+ long startTimeMs = requestContext.getRequestArrivalTimeMillis();
+ long deadlineMs = startTimeMs + _brokerTimeoutMs;
+ QueryExecutionContext executionContext =
+ new QueryExecutionContext(QueryExecutionContext.QueryType.STE, requestId, Long.toString(requestId),
+ QueryOptionsUtils.getWorkloadName(sqlNodeAndOptions.getOptions()), startTimeMs, deadlineMs, deadlineMs,
+ _brokerId, _brokerId, org.apache.pinot.spi.utils.CommonConstants.Broker.DEFAULT_QUERY_HASH);
+ try (QueryThreadContext ignore = QueryThreadContext.open(executionContext, _threadAccountant)) {
+ PinotQuery pinotQuery;
+ try {
+ pinotQuery = CalciteSqlParser.compileToPinotQuery(sqlNodeAndOptions);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return new BrokerResponseNative(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ Set<String> tableNames = RequestUtils.getTableNames(pinotQuery);
+ if (tableNames == null || tableNames.isEmpty()) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "Failed to extract table name");
+ }
+ if (tableNames.size() != 1) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "System tables do not support joins");
+ }
+ String tableName = tableNames.iterator().next();
+ if (!isSystemTable(tableName)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "Not a system table query");
+ }
+ AuthorizationResult authorizationResult =
+ hasTableAccess(requesterIdentity, Set.of(tableName), requestContext, httpHeaders);
+ if (!authorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return new BrokerResponseNative(QueryErrorCode.ACCESS_DENIED, authorizationResult.getFailureMessage());
+ }
+
+ return handleSystemTableQuery(request, pinotQuery, tableName, requestContext, requesterIdentity, query,
+ httpHeaders);
+ }
+ }
+
+ @Override
+ protected boolean handleCancel(long queryId, int timeoutMs, Executor executor,
+ HttpClientConnectionManager connMgr, Map<String, Integer> serverResponses) {
+ return false;
+ }
+
+ @Override
+ public boolean cancelQueryByClientId(String clientQueryId, int timeoutMs, Executor executor,
+ HttpClientConnectionManager connMgr, Map<String, Integer> serverResponses)
+ throws Exception {
+ return false;
+ }
+
+ @Override
+ public Map<Long, String> getRunningQueries() {
+ return Collections.emptyMap();
+ }
+
+ @Override
+ public OptionalLong getRequestIdByClientId(String clientQueryId) {
+ return OptionalLong.empty();
+ }
+
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ /**
+ * Executes a system table query against the local broker and returns the raw {@link DataTable} results.
+ * <p>
+ * This method is used by the internal broker-to-broker scatter-gather endpoint and must never perform fanout.
+ */
+ public DataTable handleSystemTableDataTableRequest(JsonNode request, @Nullable RequesterIdentity requesterIdentity,
+ RequestContext requestContext, @Nullable HttpHeaders httpHeaders) {
+ long startTimeMs = requestContext.getRequestArrivalTimeMillis();
+ if (startTimeMs <= 0) {
+ startTimeMs = System.currentTimeMillis();
+ requestContext.setRequestArrivalTimeMillis(startTimeMs);
+ }
+ long requestId = _requestIdGenerator.get();
+ long deadlineMs = startTimeMs + _brokerTimeoutMs;
+
+ JsonNode sql = request.get(CommonConstants.Broker.Request.SQL);
+ if (sql == null || !sql.isTextual()) {
+ return exceptionDataTable(QueryErrorCode.JSON_PARSING, "Failed to find 'sql' in the request: " + request);
+ }
+ String query = sql.textValue();
+ requestContext.setQuery(query);
+
+ SqlNodeAndOptions sqlNodeAndOptions;
+ try {
+ sqlNodeAndOptions = RequestUtils.parseQuery(query, request);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return exceptionDataTable(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ QueryExecutionContext executionContext =
+ new QueryExecutionContext(QueryExecutionContext.QueryType.STE, requestId, Long.toString(requestId),
+ QueryOptionsUtils.getWorkloadName(sqlNodeAndOptions.getOptions()), startTimeMs, deadlineMs, deadlineMs,
+ _brokerId, _brokerId, org.apache.pinot.spi.utils.CommonConstants.Broker.DEFAULT_QUERY_HASH);
+ try (QueryThreadContext ignore = QueryThreadContext.open(executionContext, _threadAccountant)) {
+ AccessControl accessControl = _accessControlFactory.create();
+ AuthorizationResult authorizationResult = accessControl.authorize(requesterIdentity);
+ if (!authorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return exceptionDataTable(QueryErrorCode.ACCESS_DENIED, authorizationResult.getFailureMessage());
+ }
+
+ PinotQuery pinotQuery;
+ try {
+ pinotQuery = CalciteSqlParser.compileToPinotQuery(sqlNodeAndOptions);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return exceptionDataTable(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ Set<String> tableNames = RequestUtils.getTableNames(pinotQuery);
+ if (tableNames == null || tableNames.isEmpty()) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, "Failed to extract table name");
+ }
+ if (tableNames.size() != 1) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, "System tables do not support joins");
+ }
+ String tableName = tableNames.iterator().next();
+ if (!isSystemTable(tableName)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, "Not a system table query");
+ }
+
+ AuthorizationResult tableAuthorizationResult =
+ hasTableAccess(requesterIdentity, Set.of(tableName), requestContext, httpHeaders);
+ if (!tableAuthorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return exceptionDataTable(QueryErrorCode.ACCESS_DENIED, tableAuthorizationResult.getFailureMessage());
+ }
+
+ SystemTableProvider provider = SystemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return exceptionDataTable(QueryErrorCode.TABLE_DOES_NOT_EXIST, "System table does not exist: " + tableName);
+ }
+
+ try {
+ return executeLocalSystemTableQuery(pinotQuery, provider);
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table datatable query {}: {}", tableName, e.getMessage(),
+ e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return exceptionDataTable(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+ }
+
+ private BrokerResponse handleSystemTableQuery(JsonNode request, PinotQuery pinotQuery, String tableName,
+ RequestContext requestContext, @Nullable RequesterIdentity requesterIdentity, String query,
+ @Nullable HttpHeaders httpHeaders) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableProvider provider = SystemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return BrokerResponseNative.TABLE_DOES_NOT_EXIST;
+ }
+ try {
+ Map<ServerRoutingInstance, DataTable> dataTableMap;
+ if (provider.getExecutionMode() == SystemTableProvider.ExecutionMode.BROKER_SCATTER_GATHER) {
+ dataTableMap = scatterGatherSystemTableDataTables(provider, pinotQuery, tableName, request, httpHeaders);
+ } else {
+ dataTableMap = new HashMap<>(1);
+ // Use a synthetic routing instance for broker-local execution of system table queries.
+ dataTableMap.put(new ServerRoutingInstance(SYSTEM_TABLE_PSEUDO_HOST, SYSTEM_TABLE_PSEUDO_PORT,
+ TableType.OFFLINE), executeLocalSystemTableQuery(pinotQuery, provider));
+ }
+
+ BrokerResponseNative brokerResponse;
+ BrokerRequest brokerRequest = new BrokerRequest();
+ QuerySource querySource = new QuerySource();
+ querySource.setTableName(tableName);
+ brokerRequest.setQuerySource(querySource);
+ brokerRequest.setPinotQuery(pinotQuery);
+ brokerResponse = _brokerReduceService.reduceOnDataTable(brokerRequest, brokerRequest, dataTableMap,
+ _brokerTimeoutMs, _brokerMetrics);
+ brokerResponse.setTablesQueried(Set.of(TableNameBuilder.extractRawTableName(tableName)));
+ brokerResponse.setTimeUsedMs(System.currentTimeMillis() - requestContext.getRequestArrivalTimeMillis());
+ _queryLogger.log(new QueryLogger.QueryLogParams(requestContext, tableName, brokerResponse,
+ QueryLogger.QueryLogParams.QueryEngine.SINGLE_STAGE, requesterIdentity, null));
+ return brokerResponse;
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table query {}: {}", tableName, e.getMessage(), e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+
+ private DataTable executeLocalSystemTableQuery(PinotQuery pinotQuery, SystemTableProvider provider)
+ throws Exception {
+ IndexSegment dataSource = provider.getDataSource();
+ try {
+ QueryContext queryContext = QueryContextConverterUtils.getQueryContext(pinotQuery);
+ queryContext.setSchema(provider.getSchema());
+ queryContext.setEndTimeMs(System.currentTimeMillis() + _brokerTimeoutMs);
+
+ // Pass null for serverMetrics because system table queries run broker-local against an in-memory IndexSegment.
+ Plan plan = _planMaker.makeInstancePlan(List.of(new SegmentContext(dataSource)), queryContext, _executorService,
+ null);
+ InstanceResponseBlock instanceResponse = plan.execute();
+ return instanceResponse.toDataTable();
+ } finally {
+ dataSource.destroy();
+ }
+ }
+
+ static final class BrokerTarget {
+ final ServerRoutingInstance _routingInstance;
+ final String _dataTableUrl;
+
+ BrokerTarget(ServerRoutingInstance routingInstance, String dataTableUrl) {
+ _routingInstance = routingInstance;
+ _dataTableUrl = dataTableUrl;
+ }
+ }
+
+ @com.google.common.annotations.VisibleForTesting
+ protected Map<ServerRoutingInstance, DataTable> scatterGatherSystemTableDataTables(SystemTableProvider provider,
+ PinotQuery pinotQuery, String tableName, JsonNode request, @Nullable HttpHeaders httpHeaders) {
+ if (_helixManager == null) {
+ throw new IllegalStateException(
+ "HelixManager is required for scatter-gather execution of system table: " + tableName);
+ }
+
+ HelixDataAccessor dataAccessor = _helixManager.getHelixDataAccessor();
+ List<String> liveInstances = dataAccessor.getChildNames(dataAccessor.keyBuilder().liveInstances());
+ if (liveInstances == null || liveInstances.isEmpty()) {
+ throw new IllegalStateException("No live instances found for scatter-gather execution of system table: "
+ + tableName);
+ }
+
+ String localInstanceId = _brokerId;
+ List<BrokerTarget> remoteTargets = new ArrayList<>();
+ @Nullable ServerRoutingInstance localRoutingInstance = null;
+ for (String instanceId : liveInstances) {
+ if (!InstanceTypeUtils.isBroker(instanceId)) {
+ continue;
+ }
+ InstanceConfig instanceConfig = dataAccessor.getProperty(dataAccessor.keyBuilder().instanceConfig(instanceId));
+ if (instanceConfig == null) {
+ continue;
+ }
+ URI baseUri = URI.create(InstanceUtils.getInstanceBaseUri(instanceConfig));
+ ServerRoutingInstance routingInstance = new ServerRoutingInstance(baseUri.getHost(), baseUri.getPort(),
+ TableType.OFFLINE);
+ if (instanceId.equals(localInstanceId)) {
+ localRoutingInstance = routingInstance;
+ } else {
+ remoteTargets.add(new BrokerTarget(routingInstance, baseUri.toString() + SYSTEM_TABLE_DATATABLE_API_PATH));
+ }
+ }
+
+ Map<ServerRoutingInstance, DataTable> dataTableMap = new HashMap<>(remoteTargets.size() + 1);
+ ServerRoutingInstance routingInstance = localRoutingInstance != null ? localRoutingInstance
+ : new ServerRoutingInstance(SYSTEM_TABLE_PSEUDO_HOST, SYSTEM_TABLE_PSEUDO_PORT, TableType.OFFLINE);
Review Comment:
The ternary expression spans multiple lines without clear formatting. Consider breaking this into an explicit if-else statement or ensuring the continuation is more clearly indented for better readability.
```suggestion
ServerRoutingInstance routingInstance;
if (localRoutingInstance != null) {
routingInstance = localRoutingInstance;
} else {
routingInstance = new ServerRoutingInstance(SYSTEM_TABLE_PSEUDO_HOST, SYSTEM_TABLE_PSEUDO_PORT,
TableType.OFFLINE);
}
```
##########
pinot-plugins/pinot-system-table/src/main/java/org/apache/pinot/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,662 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.databind.JsonNode;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.client.admin.PinotAdminClient;
+import org.apache.pinot.client.admin.PinotAdminTransport;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.systemtable.SystemTable;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.datasource.InMemorySystemTableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final String SIZE_CACHE_TTL_MS_PROPERTY = "pinot.systemtable.tables.sizeCacheTtlMs";
+ private static final long DEFAULT_SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+ private static final long SIZE_CACHE_TTL_MS = getNonNegativeLongProperty(SIZE_CACHE_TTL_MS_PROPERTY,
+ DEFAULT_SIZE_CACHE_TTL_MS);
+
+ private static final String CONTROLLER_TIMEOUT_MS_PROPERTY = "pinot.systemtable.tables.controllerTimeoutMs";
+ private static final long DEFAULT_CONTROLLER_TIMEOUT_MS = Duration.ofSeconds(5).toMillis();
+ private static final long CONTROLLER_TIMEOUT_MS = getPositiveLongProperty(CONTROLLER_TIMEOUT_MS_PROPERTY,
+ DEFAULT_CONTROLLER_TIMEOUT_MS);
+
+ private static final long SIZE_FETCH_FAILURE_WARN_INTERVAL_MS = Duration.ofHours(1).toMillis();
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _configuredControllerUrls;
+ private final Map<String, CachedSize> _sizeCache = new ConcurrentHashMap<>();
+ private final Map<String, PinotAdminClient> _adminClientCache = new ConcurrentHashMap<>();
+ private final AtomicLong _lastSizeFetchFailureWarnLogMs = new AtomicLong();
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _configuredControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public TableConfig getTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ }
+
+ @Override
+ public void close()
+ throws Exception {
+ for (Map.Entry<String, PinotAdminClient> entry : _adminClientCache.entrySet()) {
+ try {
+ entry.getValue().close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close admin client for {}: {}", entry.getKey(), e.toString());
+ }
+ }
+ _adminClientCache.clear();
+ }
+
+ @Override
+ public IndexSegment getDataSource() {
+ if (_tableCache == null) {
+ return new InMemorySystemTableSegment(TABLE_NAME, SCHEMA, 0, Collections.emptyMap());
+ }
+
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ List<String> controllerBaseUrls = getControllerBaseUrls();
+ Function<String, TableSize> sizeFetcher = getSizeFetcher();
+ class TableRow {
+ final String _tableNameWithType;
+ final TableType _tableType;
+ final String _rawTableName;
+ final @Nullable TableConfig _tableConfig;
+ private volatile @Nullable String _tableConfigJson;
+ private volatile @Nullable TableSize _tableSize;
+ private volatile boolean _tableSizeFetched;
+
+ private TableRow(String tableNameWithType, TableType tableType, String rawTableName,
+ @Nullable TableConfig tableConfig) {
+ _tableNameWithType = tableNameWithType;
+ _tableType = tableType;
+ _rawTableName = rawTableName;
+ _tableConfig = tableConfig;
+ }
+
+ @Nullable
+ private TableSize getTableSize() {
+ if (_tableSizeFetched) {
+ return _tableSize;
+ }
+ synchronized (this) {
+ if (_tableSizeFetched) {
+ return _tableSize;
+ }
+ _tableSize = fetchTableSize(_tableNameWithType, sizeFetcher, controllerBaseUrls);
+ _tableSizeFetched = true;
+ return _tableSize;
+ }
+ }
+
+ private String getStatus() {
+ if (_tableConfig != null) {
+ return "ONLINE";
+ }
+ TableSize sizeFromController = getTableSize();
+ int segments = sizeFromController != null ? getSegmentCount(sizeFromController, _tableType) : 0;
+ return segments > 0 ? "ONLINE" : "UNKNOWN";
+ }
+
+ private int getSegments() {
+ TableSize sizeFromController = getTableSize();
+ return sizeFromController != null ? getSegmentCount(sizeFromController, _tableType) : 0;
+ }
+
+ private long getTotalDocs() {
+ TableSize sizeFromController = getTableSize();
+ return sizeFromController != null ? TablesSystemTableProvider.this.getTotalDocs(sizeFromController, _tableType,
+ _tableNameWithType, controllerBaseUrls) : 0L;
+ }
+
+ private long getReportedSize() {
+ TableSize sizeFromController = getTableSize();
+ if (sizeFromController == null || sizeFromController._reportedSizeInBytes < 0) {
+ return 0L;
+ }
+ return sizeFromController._reportedSizeInBytes;
+ }
+
+ private long getEstimatedSize() {
+ TableSize sizeFromController = getTableSize();
+ if (sizeFromController == null || sizeFromController._estimatedSizeInBytes < 0) {
+ return 0L;
+ }
+ return sizeFromController._estimatedSizeInBytes;
+ }
+
+ private String getBrokerTenant() {
+ if (_tableConfig != null && _tableConfig.getTenantConfig() != null) {
+ String tenant = _tableConfig.getTenantConfig().getBroker();
+ return tenant != null ? tenant : "";
+ }
+ return "";
+ }
+
+ private String getServerTenant() {
+ if (_tableConfig != null && _tableConfig.getTenantConfig() != null) {
+ String tenant = _tableConfig.getTenantConfig().getServer();
+ return tenant != null ? tenant : "";
+ }
+ return "";
+ }
+
+ private int getReplicas() {
+ if (_tableConfig != null && _tableConfig.getValidationConfig() != null) {
+ Integer replicationNumber = _tableConfig.getValidationConfig().getReplicationNumber();
+ if (replicationNumber != null) {
+ return replicationNumber;
+ }
+ }
+ return 0;
+ }
+
+ private String getTableConfigJson() {
+ String cached = _tableConfigJson;
+ if (cached != null) {
+ return cached;
+ }
+ synchronized (this) {
+ cached = _tableConfigJson;
+ if (cached != null) {
+ return cached;
+ }
+ cached = "";
+ if (_tableConfig != null) {
+ try {
+ cached = JsonUtils.objectToString(_tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", _tableNameWithType, e.toString());
+ cached = _tableConfig.toString();
+ }
+ }
+ _tableConfigJson = cached;
+ return cached;
+ }
+ }
+ }
+
+ List<TableRow> tableRows = new ArrayList<>(sortedTableNames.size());
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ tableRows.add(new TableRow(tableNameWithType, tableType, rawTableName, tableConfig));
+ }
+
+ Map<String, IntFunction<Object>> valueProviders = new java.util.HashMap<>();
+ valueProviders.put("tableName", docId -> tableRows.get(docId)._rawTableName);
+ valueProviders.put("type", docId -> tableRows.get(docId)._tableType.name());
+ valueProviders.put("status", docId -> tableRows.get(docId).getStatus());
+ valueProviders.put("segments", docId -> tableRows.get(docId).getSegments());
+ valueProviders.put("totalDocs", docId -> tableRows.get(docId).getTotalDocs());
+ valueProviders.put("reportedSize", docId -> tableRows.get(docId).getReportedSize());
+ valueProviders.put("estimatedSize", docId -> tableRows.get(docId).getEstimatedSize());
+ valueProviders.put("brokerTenant", docId -> tableRows.get(docId).getBrokerTenant());
+ valueProviders.put("serverTenant", docId -> tableRows.get(docId).getServerTenant());
+ valueProviders.put("replicas", docId -> tableRows.get(docId).getReplicas());
+ valueProviders.put("tableConfig", docId -> tableRows.get(docId).getTableConfigJson());
+
+ return new InMemorySystemTableSegment(TABLE_NAME, SCHEMA, tableRows.size(), valueProviders);
+ }
+
+ @Nullable
+ private TableSize fetchTableSize(String tableNameWithType,
+ @Nullable Function<String, TableSize> fetcher, List<String> controllerBaseUrls) {
+ boolean cacheEnabled = SIZE_CACHE_TTL_MS > 0;
+ TableSize cached = cacheEnabled ? getCachedSize(tableNameWithType) : null;
+ if (cached != null) {
+ return cached;
+ }
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ if (cacheEnabled) {
+ cacheSize(tableNameWithType, fetched);
+ }
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(controllerBaseUrls, rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(controllerBaseUrls, tableNameWithType);
+ if (size == null) {
+ logSizeFetchFailure("{}: failed to fetch size for {} from controllers {} "
+ + "(tried raw table name '{}' and table name with type '{}')",
+ TABLE_NAME, tableNameWithType, controllerBaseUrls, rawTableName, tableNameWithType);
+ }
+ }
+ if (size != null && cacheEnabled) {
+ cacheSize(tableNameWithType, size);
+ }
+ return size;
+ }
+
+ @Nullable
+ private TableSize fetchTableSizeForName(List<String> controllerBaseUrls, String tableName) {
+ for (String baseUrl : controllerBaseUrls) {
+ try {
+ PinotAdminClient adminClient = getOrCreateAdminClient(baseUrl);
+ if (adminClient == null) {
+ continue;
+ }
+
+ JsonNode sizeNode = adminClient.getTableSize(tableName, true, false);
+
+ if (sizeNode == null) {
+ continue;
+ }
+
+ TableSize parsed = JsonUtils.stringToObject(sizeNode.toString(), TableSize.class);
+ LOGGER.debug("{}: controller size response for {} via {} -> segments offline={}, realtime={}, "
+ + "reportedSize={}, estimatedSize={}", TABLE_NAME, tableName, baseUrl,
+ parsed._offlineSegments != null && parsed._offlineSegments._segments != null
+ ? parsed._offlineSegments._segments.size() : 0,
+ parsed._realtimeSegments != null && parsed._realtimeSegments._segments != null
+ ? parsed._realtimeSegments._segments.size() : 0,
+ parsed._reportedSizeInBytes, parsed._estimatedSizeInBytes);
+ return parsed;
+ } catch (Exception e) {
+ logSizeFetchFailure("{}: error fetching table size for {} via {} using admin client", TABLE_NAME, tableName,
+ baseUrl, e);
+ }
+ }
+ return null;
+ }
+
+ private List<String> getControllerBaseUrls() {
+ Set<String> urls = new LinkedHashSet<>();
+ if (_helixAdmin != null) {
+ for (String controller : discoverControllersFromHelix()) {
+ String normalized = normalizeControllerUrl(controller);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ }
+ for (String url : _configuredControllerUrls) {
+ String normalized = normalizeControllerUrl(url);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ return new ArrayList<>(urls);
+ }
+
+ private int getSegmentCount(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.size();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.size();
+ }
+ return 0;
+ }
+
+ private long getTotalDocsFromSize(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ return 0;
+ }
+
+ private long getTotalDocs(TableSize sizeFromController, TableType tableType, String tableNameWithType,
+ List<String> controllerBaseUrls) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ long cached = sizeFromController._offlineTotalDocs;
+ if (cached >= 0) {
+ return cached;
+ }
+ long totalDocsFromSize = getTotalDocsFromSize(sizeFromController, tableType);
+ if (totalDocsFromSize > 0) {
+ synchronized (sizeFromController) {
+ if (sizeFromController._offlineTotalDocs < 0) {
+ sizeFromController._offlineTotalDocs = totalDocsFromSize;
+ }
+ return sizeFromController._offlineTotalDocs;
+ }
+ }
+ long fetched = fetchTotalDocsFromSegmentMetadata(tableNameWithType, sizeFromController._offlineSegments._segments,
+ controllerBaseUrls);
+ synchronized (sizeFromController) {
+ if (sizeFromController._offlineTotalDocs < 0) {
+ sizeFromController._offlineTotalDocs = fetched;
+ }
+ return sizeFromController._offlineTotalDocs;
+ }
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ long cached = sizeFromController._realtimeTotalDocs;
+ if (cached >= 0) {
+ return cached;
+ }
+ long totalDocsFromSize = getTotalDocsFromSize(sizeFromController, tableType);
+ if (totalDocsFromSize > 0) {
+ synchronized (sizeFromController) {
+ if (sizeFromController._realtimeTotalDocs < 0) {
+ sizeFromController._realtimeTotalDocs = totalDocsFromSize;
+ }
+ return sizeFromController._realtimeTotalDocs;
+ }
+ }
+ long fetched = fetchTotalDocsFromSegmentMetadata(tableNameWithType,
+ sizeFromController._realtimeSegments._segments, controllerBaseUrls);
+ synchronized (sizeFromController) {
+ if (sizeFromController._realtimeTotalDocs < 0) {
+ sizeFromController._realtimeTotalDocs = fetched;
+ }
+ return sizeFromController._realtimeTotalDocs;
+ }
+ }
Review Comment:
The caching logic for `_offlineTotalDocs` and `_realtimeTotalDocs` is duplicated in lines 436-446 and 459-470. Extract this into a helper method that takes the cached field reference and segment type to reduce code duplication and improve maintainability.
```suggestion
private long getTotalDocsWithCaching(TableSize sizeFromController, TableType tableType, String tableNameWithType,
List<String> controllerBaseUrls) {
long cached =
(tableType == TableType.OFFLINE) ? sizeFromController._offlineTotalDocs : sizeFromController._realtimeTotalDocs;
if (cached >= 0) {
return cached;
}
long totalDocsFromSize = getTotalDocsFromSize(sizeFromController, tableType);
if (totalDocsFromSize > 0) {
synchronized (sizeFromController) {
if (tableType == TableType.OFFLINE) {
if (sizeFromController._offlineTotalDocs < 0) {
sizeFromController._offlineTotalDocs = totalDocsFromSize;
}
return sizeFromController._offlineTotalDocs;
} else {
if (sizeFromController._realtimeTotalDocs < 0) {
sizeFromController._realtimeTotalDocs = totalDocsFromSize;
}
return sizeFromController._realtimeTotalDocs;
}
}
}
long fetched;
if (tableType == TableType.OFFLINE) {
fetched = fetchTotalDocsFromSegmentMetadata(tableNameWithType, sizeFromController._offlineSegments._segments,
controllerBaseUrls);
} else {
fetched = fetchTotalDocsFromSegmentMetadata(tableNameWithType, sizeFromController._realtimeSegments._segments,
controllerBaseUrls);
}
synchronized (sizeFromController) {
if (tableType == TableType.OFFLINE) {
if (sizeFromController._offlineTotalDocs < 0) {
sizeFromController._offlineTotalDocs = fetched;
}
return sizeFromController._offlineTotalDocs;
} else {
if (sizeFromController._realtimeTotalDocs < 0) {
sizeFromController._realtimeTotalDocs = fetched;
}
return sizeFromController._realtimeTotalDocs;
}
}
}
private long getTotalDocs(TableSize sizeFromController, TableType tableType, String tableNameWithType,
List<String> controllerBaseUrls) {
if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
&& sizeFromController._offlineSegments._segments != null) {
return getTotalDocsWithCaching(sizeFromController, TableType.OFFLINE, tableNameWithType, controllerBaseUrls);
}
if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
&& sizeFromController._realtimeSegments._segments != null) {
return getTotalDocsWithCaching(sizeFromController, TableType.REALTIME, tableNameWithType, controllerBaseUrls);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2874517431
##########
pinot-plugins/pinot-system-table/src/main/java/org/apache/pinot/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,604 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.databind.JsonNode;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.client.admin.PinotAdminClient;
+import org.apache.pinot.client.admin.PinotAdminTransport;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.systemtable.SystemTable;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.SystemTableProviderContext;
+import org.apache.pinot.common.systemtable.datasource.InMemorySystemTableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final String SIZE_CACHE_TTL_MS_PROPERTY = "pinot.systemtable.tables.sizeCacheTtlMs";
+ private static final long DEFAULT_SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+
+ private static final String CONTROLLER_TIMEOUT_MS_PROPERTY = "pinot.systemtable.tables.controllerTimeoutMs";
+ private static final long DEFAULT_CONTROLLER_TIMEOUT_MS = Duration.ofSeconds(5).toMillis();
+
+ private static final long SIZE_FETCH_FAILURE_WARN_INTERVAL_MS = Duration.ofHours(1).toMillis();
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSizeBytes", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSizeBytes", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ @Nullable
+ private TableCache _tableCache;
+ @Nullable
+ private HelixAdmin _helixAdmin;
+ @Nullable
+ private String _clusterName;
+ @Nullable
+ private Function<String, TableSize> _tableSizeFetcherOverride;
+ private List<String> _configuredControllerUrls = List.of();
+ private long _sizeCacheTtlMs = DEFAULT_SIZE_CACHE_TTL_MS;
+ private long _controllerTimeoutMs = DEFAULT_CONTROLLER_TIMEOUT_MS;
+ private final Map<String, CachedSize> _sizeCache = new ConcurrentHashMap<>();
+ private final Map<String, PinotAdminClient> _adminClientCache = new ConcurrentHashMap<>();
+ private final AtomicLong _lastSizeFetchFailureWarnLogMs = new AtomicLong();
+
+ public TablesSystemTableProvider() {
+ }
+
+ @Override
+ public void init(SystemTableProviderContext context) {
+ _tableCache = context.getTableCache();
+ _helixAdmin = context.getHelixAdmin();
+ _clusterName = context.getClusterName();
+ if (context.getConfig() != null) {
+ _sizeCacheTtlMs = context.getConfig().getProperty(SIZE_CACHE_TTL_MS_PROPERTY, DEFAULT_SIZE_CACHE_TTL_MS);
+ _controllerTimeoutMs =
+ context.getConfig().getProperty(CONTROLLER_TIMEOUT_MS_PROPERTY, DEFAULT_CONTROLLER_TIMEOUT_MS);
+ }
Review Comment:
`_configuredControllerUrls` is never populated from `SystemTableProviderContext#getConfig()` (only set via the package-private test constructor), so `getControllerBaseUrls()` effectively depends solely on Helix discovery. If Helix InstanceConfig base URIs are missing/misconfigured, `system.tables` will always return 0 rows/sizes. Consider reading `CommonConstants.Broker.CONTROLLER_URL` (or a dedicated system-table controller URL config) in `init()` and splitting it into `_configuredControllerUrls` as a fallback.
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/datasource/InMemorySystemTableSegment.java:
##########
@@ -0,0 +1,734 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.datasource;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.node.JsonNodeFactory;
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.Constants;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.SegmentVersion;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.datasource.DataSourceMetadata;
+import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexType;
+import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.column.ColumnIndexContainer;
+import org.apache.pinot.segment.spi.index.multicolumntext.MultiColumnTextMetadata;
+import org.apache.pinot.segment.spi.index.mutable.ThreadSafeMutableRoaringBitmap;
+import org.apache.pinot.segment.spi.index.reader.BloomFilterReader;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReaderContext;
+import org.apache.pinot.segment.spi.index.reader.H3IndexReader;
+import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
+import org.apache.pinot.segment.spi.index.reader.JsonIndexReader;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+import org.apache.pinot.segment.spi.index.reader.RangeIndexReader;
+import org.apache.pinot.segment.spi.index.reader.TextIndexReader;
+import org.apache.pinot.segment.spi.index.reader.VectorIndexReader;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2Metadata;
+import org.apache.pinot.segment.spi.partition.PartitionFunction;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.joda.time.Duration;
+import org.joda.time.Interval;
+
+
+/**
+ * In-memory {@link IndexSegment} implementation intended for system table queries.
+ * <p>
+ * This segment is backed by per-column value functions (docId -> value) and exposes raw forward indexes (no
+ * dictionaries/inverted indexes) so that the standard v1 query engine can operate on it.
+ */
+public final class InMemorySystemTableSegment implements IndexSegment {
+ private final String _segmentName;
+ private final Schema _schema;
+ private final int _numDocs;
+ private final Map<String, IntFunction<Object>> _valueProvidersByColumn;
+ private final Map<String, DataSource> _dataSourcesByColumn;
+ private final SegmentMetadata _segmentMetadata;
+
+ public InMemorySystemTableSegment(String segmentName, Schema schema, int numDocs,
+ Map<String, IntFunction<Object>> valueProvidersByColumn) {
+ _segmentName = segmentName;
+ _schema = schema;
+ _numDocs = numDocs;
+ _valueProvidersByColumn = new HashMap<>(valueProvidersByColumn);
+ _dataSourcesByColumn = new HashMap<>(schema.getColumnNames().size());
+ for (String column : schema.getColumnNames()) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ if (fieldSpec == null) {
+ continue;
+ }
+ IntFunction<Object> provider = _valueProvidersByColumn.get(column);
+ if (provider == null) {
+ provider = docId -> defaultValue(fieldSpec.getDataType());
+ _valueProvidersByColumn.put(column, provider);
+ }
+ _dataSourcesByColumn.put(column, new FunctionBasedDataSource(fieldSpec, numDocs, provider));
+ }
+ _segmentMetadata = new InMemorySegmentMetadata(segmentName, schema, numDocs);
+ }
+
+ @Override
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ @Override
+ public SegmentMetadata getSegmentMetadata() {
+ return _segmentMetadata;
+ }
+
+ @Override
+ public Set<String> getColumnNames() {
+ return _schema.getColumnNames();
+ }
+
+ @Override
+ public Set<String> getPhysicalColumnNames() {
+ return _schema.getPhysicalColumnNames();
+ }
+
+ @Nullable
+ @Override
+ public DataSource getDataSourceNullable(String column) {
+ return _dataSourcesByColumn.get(column);
+ }
+
+ @Override
+ public DataSource getDataSource(String column, Schema schema) {
+ DataSource dataSource = getDataSourceNullable(column);
+ if (dataSource != null) {
+ return dataSource;
+ }
+ throw new IllegalStateException("Failed to find data source for column: " + column);
+ }
+
+ @Nullable
+ @Override
+ public List<StarTreeV2> getStarTrees() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public TextIndexReader getMultiColumnTextIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public ThreadSafeMutableRoaringBitmap getValidDocIds() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public ThreadSafeMutableRoaringBitmap getQueryableDocIds() {
+ return null;
+ }
+
+ @Override
+ public GenericRow getRecord(int docId, GenericRow reuse) {
+ GenericRow row = reuse != null ? reuse : new GenericRow();
+ row.getFieldToValueMap().clear();
+ for (String column : _schema.getColumnNames()) {
+ row.putValue(column, getValue(docId, column));
+ }
+ return row;
+ }
+
+ @Override
+ public Object getValue(int docId, String column) {
+ IntFunction<Object> provider = _valueProvidersByColumn.get(column);
+ if (provider == null) {
+ return null;
+ }
+ return provider.apply(docId);
+ }
+
+ @Override
+ public void offload() {
+ }
+
+ @Override
+ public void destroy() {
+ }
+
+ private static Object defaultValue(FieldSpec.DataType dataType) {
+ switch (dataType) {
+ case INT:
+ return 0;
+ case LONG:
+ return 0L;
+ case FLOAT:
+ return 0.0f;
+ case DOUBLE:
+ return 0.0d;
+ case BIG_DECIMAL:
+ return BigDecimal.ZERO;
+ case BOOLEAN:
+ return false;
+ case STRING:
+ return "";
+ case BYTES:
+ return new byte[0];
+ default:
+ return null;
+ }
+ }
+
+ private static final class InMemorySegmentMetadata implements SegmentMetadata {
+ private final String _segmentName;
+ private final Schema _schema;
+ private final int _totalDocs;
+ private final TreeMap<String, ColumnMetadata> _columnMetadataMap;
+
+ private InMemorySegmentMetadata(String segmentName, Schema schema, int totalDocs) {
+ _segmentName = segmentName;
+ _schema = schema;
+ _totalDocs = totalDocs;
+ _columnMetadataMap = new TreeMap<>();
+ for (String column : schema.getColumnNames()) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ if (fieldSpec != null) {
+ _columnMetadataMap.put(column, new InMemoryColumnMetadata(fieldSpec, totalDocs));
+ }
+ }
+ }
+
+ @Deprecated
+ @Override
+ public String getTableName() {
+ return _schema.getSchemaName();
+ }
+
+ @Override
+ public String getName() {
+ return _segmentName;
+ }
+
+ @Override
+ public String getTimeColumn() {
+ return "";
+ }
+
+ @Override
+ public long getStartTime() {
+ return 0;
+ }
+
+ @Override
+ public long getEndTime() {
+ return 0;
+ }
+
+ @Override
+ public TimeUnit getTimeUnit() {
+ return TimeUnit.MILLISECONDS;
+ }
+
+ @Override
+ public Duration getTimeGranularity() {
+ return Duration.ZERO;
+ }
+
+ @Override
+ public Interval getTimeInterval() {
+ return new Interval(0L, 0L);
+ }
+
+ @Override
+ public String getCrc() {
+ return "";
+ }
+
+ @Override
+ public String getDataCrc() {
+ return "";
+ }
+
+ @Override
+ public SegmentVersion getVersion() {
+ return SegmentVersion.v3;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return _schema;
+ }
+
+ @Override
+ public int getTotalDocs() {
+ return _totalDocs;
+ }
+
+ @Override
+ public File getIndexDir() {
+ return new File("");
+ }
+
+ @Nullable
+ @Override
+ public String getCreatorName() {
+ return "systemtable";
+ }
+
+ @Override
+ public long getIndexCreationTime() {
+ return 0;
+ }
+
+ @Override
+ public long getLastIndexedTimestamp() {
+ return 0;
+ }
+
+ @Override
+ public long getLatestIngestionTimestamp() {
+ return Long.MIN_VALUE;
+ }
+
+ @Override
+ public long getMinimumIngestionLagMs() {
+ return Long.MAX_VALUE;
+ }
+
+ @Nullable
+ @Override
+ public List<StarTreeV2Metadata> getStarTreeV2MetadataList() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public MultiColumnTextMetadata getMultiColumnTextMetadata() {
+ return null;
+ }
+
+ @Override
+ public Map<String, String> getCustomMap() {
+ return Collections.emptyMap();
+ }
+
+ @Override
+ public String getStartOffset() {
+ return "";
+ }
+
+ @Override
+ public String getEndOffset() {
+ return "";
+ }
+
+ @Override
+ public TreeMap<String, ColumnMetadata> getColumnMetadataMap() {
+ return _columnMetadataMap;
+ }
+
+ @Override
+ public void removeColumn(String column) {
+ _columnMetadataMap.remove(column);
+ }
+
+ @Override
+ public JsonNode toJson(@Nullable Set<String> columnFilter) {
+ return JsonNodeFactory.instance.objectNode();
+ }
+ }
+
+ private static final class InMemoryColumnMetadata implements ColumnMetadata {
+ private final FieldSpec _fieldSpec;
+ private final int _totalDocs;
+
+ private InMemoryColumnMetadata(FieldSpec fieldSpec, int totalDocs) {
+ _fieldSpec = fieldSpec;
+ _totalDocs = totalDocs;
+ }
+
+ @Override
+ public FieldSpec getFieldSpec() {
+ return _fieldSpec;
+ }
+
+ @Override
+ public int getTotalDocs() {
+ return _totalDocs;
+ }
+
+ @Override
+ public int getCardinality() {
+ return Constants.UNKNOWN_CARDINALITY;
+ }
+
+ @Override
+ public boolean isSorted() {
+ return false;
+ }
+
+ @Override
+ public Comparable getMinValue() {
+ return defaultComparableValue(_fieldSpec.getDataType());
+ }
+
+ @Override
+ public Comparable getMaxValue() {
+ return defaultComparableValue(_fieldSpec.getDataType());
+ }
+
+ @Override
+ public boolean hasDictionary() {
+ return false;
+ }
+
+ @Override
+ public int getColumnMaxLength() {
+ return 0;
+ }
+
+ @Override
+ public int getBitsPerElement() {
+ return 0;
+ }
+
+ @Override
+ public int getMaxNumberOfMultiValues() {
+ return 0;
+ }
+
+ @Override
+ public int getTotalNumberOfEntries() {
+ return _totalDocs;
+ }
+
+ @Nullable
+ @Override
+ public PartitionFunction getPartitionFunction() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public Set<Integer> getPartitions() {
+ return null;
+ }
+
+ @Override
+ public boolean isAutoGenerated() {
+ return false;
+ }
+
+ @Override
+ public Map<IndexType<?, ?, ?>, Long> getIndexSizeMap() {
+ return Collections.emptyMap();
+ }
+
+ private static Comparable defaultComparableValue(FieldSpec.DataType dataType) {
+ switch (dataType) {
+ case INT:
+ return 0;
+ case LONG:
+ return 0L;
+ case FLOAT:
+ return 0.0f;
+ case DOUBLE:
+ return 0.0d;
+ case BIG_DECIMAL:
+ return BigDecimal.ZERO;
+ case BOOLEAN:
+ return false;
+ case STRING:
+ return "";
+ default:
+ return 0;
+ }
+ }
+ }
+
+ private static final class FunctionBasedDataSource implements DataSource {
+ private final DataSourceMetadata _metadata;
+ private final ColumnIndexContainer _indexContainer;
+ private final ForwardIndexReader<?> _forwardIndex;
+
+ private FunctionBasedDataSource(FieldSpec fieldSpec, int numDocs, IntFunction<Object> valueProvider) {
+ _metadata = new FunctionBasedDataSourceMetadata(fieldSpec, numDocs);
+ _indexContainer = ColumnIndexContainer.Empty.INSTANCE;
+ _forwardIndex = new FunctionBasedForwardIndexReader(fieldSpec.getDataType(), valueProvider);
+ }
+
+ @Override
+ public DataSourceMetadata getDataSourceMetadata() {
+ return _metadata;
+ }
+
+ @Override
+ public ColumnIndexContainer getIndexContainer() {
+ return _indexContainer;
+ }
+
+ @Override
+ public <R extends IndexReader> R getIndex(IndexType<?, R, ?> type) {
+ if (type == null) {
+ return null;
+ }
+ if (StandardIndexes.FORWARD_ID.equals(type.getId())) {
+ return (R) _forwardIndex;
+ }
+ return null;
+ }
+
+ @Override
+ public ForwardIndexReader<?> getForwardIndex() {
+ return _forwardIndex;
+ }
+
+ @Nullable
+ @Override
+ public Dictionary getDictionary() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public InvertedIndexReader<?> getInvertedIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public RangeIndexReader<?> getRangeIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public TextIndexReader getTextIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public TextIndexReader getFSTIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public TextIndexReader getIFSTIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public JsonIndexReader getJsonIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public H3IndexReader getH3Index() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public BloomFilterReader getBloomFilter() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public NullValueVectorReader getNullValueVector() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public VectorIndexReader getVectorIndex() {
+ return null;
+ }
+ }
+
+ private static final class FunctionBasedDataSourceMetadata implements DataSourceMetadata {
+ private final FieldSpec _fieldSpec;
+ private final int _numDocs;
+
+ private FunctionBasedDataSourceMetadata(FieldSpec fieldSpec, int numDocs) {
+ _fieldSpec = fieldSpec;
+ _numDocs = numDocs;
+ }
+
+ @Override
+ public FieldSpec getFieldSpec() {
+ return _fieldSpec;
+ }
+
+ @Override
+ public boolean isSorted() {
+ return false;
+ }
+
+ @Override
+ public int getNumDocs() {
+ return _numDocs;
+ }
+
+ @Override
+ public int getNumValues() {
+ return _numDocs;
+ }
+
+ @Override
+ public int getMaxNumValuesPerMVEntry() {
+ return -1;
+ }
+
+ @Nullable
+ @Override
+ public Comparable getMinValue() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public Comparable getMaxValue() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public PartitionFunction getPartitionFunction() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public Set<Integer> getPartitions() {
+ return null;
+ }
+
+ @Override
+ public int getCardinality() {
+ return -1;
+ }
+ }
+
+ private static final class FunctionBasedForwardIndexReader implements ForwardIndexReader<ForwardIndexReaderContext> {
+ private final FieldSpec.DataType _storedType;
+ private final IntFunction<Object> _valueProvider;
+
+ private FunctionBasedForwardIndexReader(FieldSpec.DataType storedType, IntFunction<Object> valueProvider) {
+ _storedType = storedType;
+ _valueProvider = valueProvider;
+ }
+
+ private static <T> T coerceNumber(Object value, Function<Number, T> numberConverter, Function<String, T> parser) {
+ return value instanceof Number ? numberConverter.apply((Number) value) : parser.apply(String.valueOf(value));
+ }
+
+ @Override
+ public boolean isDictionaryEncoded() {
+ return false;
+ }
+
+ @Override
+ public boolean isSingleValue() {
+ return true;
+ }
+
+ @Override
+ public FieldSpec.DataType getStoredType() {
+ return _storedType;
+ }
+
+ @Override
+ public int getInt(int docId, ForwardIndexReaderContext context) {
+ Object value = _valueProvider.apply(docId);
+ return coerceNumber(value, number -> number.intValue(), Integer::parseInt);
+ }
+
+ @Override
+ public long getLong(int docId, ForwardIndexReaderContext context) {
+ Object value = _valueProvider.apply(docId);
+ return coerceNumber(value, number -> number.longValue(), Long::parseLong);
+ }
+
+ @Override
+ public float getFloat(int docId, ForwardIndexReaderContext context) {
+ Object value = _valueProvider.apply(docId);
+ return coerceNumber(value, number -> number.floatValue(), Float::parseFloat);
+ }
+
+ @Override
+ public double getDouble(int docId, ForwardIndexReaderContext context) {
+ Object value = _valueProvider.apply(docId);
+ return coerceNumber(value, number -> number.doubleValue(), Double::parseDouble);
+ }
+
+ @Override
+ public BigDecimal getBigDecimal(int docId, ForwardIndexReaderContext context) {
+ Object value = _valueProvider.apply(docId);
+ if (value instanceof BigDecimal) {
+ return (BigDecimal) value;
+ }
+ if (value instanceof Number) {
+ return BigDecimal.valueOf(((Number) value).doubleValue());
+ }
+ return new BigDecimal(String.valueOf(value));
+ }
Review Comment:
`FunctionBasedForwardIndexReader` numeric getters (`getInt/getLong/getFloat/getDouble/getBigDecimal`) will throw if the value provider returns null (e.g., parsing "null"), which can crash query execution for system tables and plugins. Treat null as the column’s default value (or add a null-value vector implementation) to make the segment robust to missing values.
##########
pinot-plugins/pinot-system-table/src/main/java/org/apache/pinot/systemtable/provider/InstancesSystemTableProvider.java:
##########
@@ -0,0 +1,384 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.systemtable.provider;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.client.admin.PinotAdminClient;
+import org.apache.pinot.client.admin.PinotAdminTransport;
+import org.apache.pinot.common.systemtable.SystemTable;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.SystemTableProviderContext;
+import org.apache.pinot.common.systemtable.datasource.InMemorySystemTableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing Pinot instance metadata.
+ */
+@SystemTable
+public final class InstancesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(InstancesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.instances";
+
+ private static final String CONTROLLER_TIMEOUT_MS_PROPERTY = "pinot.systemtable.instances.controllerTimeoutMs";
+ private static final long DEFAULT_CONTROLLER_TIMEOUT_MS = Duration.ofSeconds(5).toMillis();
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("instanceId", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("host", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("port", FieldSpec.DataType.INT)
+ .addSingleValueDimension("state", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("tags", FieldSpec.DataType.STRING)
+ .build();
+
+ @Nullable
+ private HelixAdmin _helixAdmin;
+ @Nullable
+ private String _clusterName;
+ private List<String> _configuredControllerUrls = List.of();
+ private long _controllerTimeoutMs = DEFAULT_CONTROLLER_TIMEOUT_MS;
+ private final Map<String, PinotAdminClient> _adminClientCache = new ConcurrentHashMap<>();
+
+ public InstancesSystemTableProvider() {
+ }
+
+ @Override
+ public void init(SystemTableProviderContext context) {
+ _helixAdmin = context.getHelixAdmin();
+ _clusterName = context.getClusterName();
+ if (context.getConfig() != null) {
+ _controllerTimeoutMs =
+ context.getConfig().getProperty(CONTROLLER_TIMEOUT_MS_PROPERTY, DEFAULT_CONTROLLER_TIMEOUT_MS);
+ }
+ }
Review Comment:
Similar to `system.tables`, `_configuredControllerUrls` is only set via the test-only constructor and is never loaded from broker config in `init()`. If Helix discovery fails, `system.instances` will always return 0 rows. Consider reading controller URL(s) from `SystemTableProviderContext#getConfig()` as a fallback.
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/SystemTableRegistry.java:
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.IdentityHashMap;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.PinotReflectionUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Registry to hold and lifecycle-manage system table data providers.
+ * <p>
+ * This class is thread-safe and manages the lifecycle of registered providers.
+ */
+public final class SystemTableRegistry implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableRegistry.class);
+ private static final String SYSTEM_TABLE_PREFIX = "system.";
+
+ // Providers are registered once at broker startup.
+ private final Map<String, SystemTableProvider> _providers = new HashMap<>();
+ private final SystemTableProviderContext _context;
+ private volatile boolean _closed = false;
+
+ /**
+ * Creates a new SystemTableRegistry and initializes it with annotated providers.
+ */
+ public SystemTableRegistry(TableCache tableCache, HelixAdmin helixAdmin, String clusterName,
+ @Nullable PinotConfiguration config) {
+ _context = new SystemTableProviderContext(tableCache, helixAdmin, clusterName, config);
+ registerAnnotatedProviders();
+ }
+
+ /**
+ * Registers a system table provider.
+ * <p>
+ * Note: This method does NOT call {@link SystemTableProvider#init(SystemTableProviderContext)} on the provider.
+ * It is intended for registering providers that have already been initialized (e.g. in tests).
+ *
+ * @throws IllegalArgumentException if the table name does not start with "system."
+ */
+ public void register(SystemTableProvider provider) {
+ String tableName = provider.getTableName();
+ if (!tableName.toLowerCase(Locale.ROOT).startsWith(SYSTEM_TABLE_PREFIX)) {
+ throw new IllegalArgumentException(
+ "System table name must start with '" + SYSTEM_TABLE_PREFIX + "', got: " + tableName);
+ }
+ synchronized (_providers) {
+ if (_closed) {
+ throw new IllegalStateException("Cannot register provider after registry is closed");
+ }
+ _providers.put(normalize(tableName), provider);
+ }
+ }
+
+ /**
+ * Retrieves a registered system table provider by table name.
+ *
+ * @return the provider, or null if not found or registry is closed
+ */
+ @Nullable
+ public SystemTableProvider get(String tableName) {
+ synchronized (_providers) {
+ if (_closed) {
+ return null;
+ }
+ return _providers.get(normalize(tableName));
+ }
+ }
+
+ /**
+ * Checks if a system table is registered.
+ */
+ public boolean isRegistered(String tableName) {
+ synchronized (_providers) {
+ if (_closed) {
+ return false;
+ }
+ return _providers.containsKey(normalize(tableName));
+ }
+ }
+
+ /**
+ * Returns all registered providers.
+ */
+ public Collection<SystemTableProvider> getProviders() {
+ synchronized (_providers) {
+ if (_closed) {
+ return Collections.emptyList();
+ }
+ return Collections.unmodifiableCollection(new java.util.ArrayList<>(_providers.values()));
+ }
+ }
+
+ /**
+ * Discover and register providers marked with {@link SystemTable} using the available dependencies.
+ * <p>
+ * Follows the ScalarFunction pattern: any class annotated with @SystemTable under a "*.systemtable.*" package
+ * will be discovered via reflection, instantiated with a no-arg constructor, initialized via
+ * {@link SystemTableProvider#init(SystemTableProviderContext)}, and registered.
+ */
+ private void registerAnnotatedProviders() {
+ Set<Class<?>> classes =
+ PinotReflectionUtils.getClassesThroughReflection(".*\\.systemtable\\..*", SystemTable.class);
+ for (Class<?> clazz : classes) {
+ if (!SystemTableProvider.class.isAssignableFrom(clazz)) {
+ continue;
+ }
+ SystemTableProvider provider = instantiateProvider(clazz.asSubclass(SystemTableProvider.class));
+ if (provider == null) {
+ continue;
+ }
+ if (isRegistered(provider.getTableName())) {
+ continue;
+ }
+ LOGGER.info("Registering system table provider: {}", provider.getTableName());
+ register(provider);
+ }
Review Comment:
`registerAnnotatedProviders()` instantiates and initializes providers before checking for an already-registered table name. If duplicates are present on the classpath, the newly created provider is silently dropped without being closed, which can leak resources created during `init()` (threads, clients, caches, etc.). Consider checking for duplicates earlier (e.g., by class), or explicitly calling `close()` on the skipped provider.
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/SystemTableBrokerRequestHandler.java:
##########
@@ -0,0 +1,585 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.requesthandler;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.google.common.annotations.VisibleForTesting;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.client5.http.io.HttpClientConnectionManager;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.HelixManager;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.pinot.broker.api.AccessControl;
+import org.apache.pinot.broker.broker.AccessControlFactory;
+import org.apache.pinot.broker.querylog.QueryLogger;
+import org.apache.pinot.broker.queryquota.QueryQuotaManager;
+import org.apache.pinot.client.ConnectionTimeouts;
+import org.apache.pinot.client.PinotClientException;
+import org.apache.pinot.client.SystemTableDataTableClient;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.common.datatable.DataTableImplV4;
+import org.apache.pinot.common.metrics.BrokerMeter;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.request.QuerySource;
+import org.apache.pinot.common.response.BrokerResponse;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.SystemTableRegistry;
+import org.apache.pinot.common.utils.NamedThreadFactory;
+import org.apache.pinot.common.utils.config.InstanceUtils;
+import org.apache.pinot.common.utils.config.QueryOptionsUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.core.instance.context.BrokerContext;
+import org.apache.pinot.core.operator.blocks.InstanceResponseBlock;
+import org.apache.pinot.core.plan.Plan;
+import org.apache.pinot.core.plan.maker.InstancePlanMakerImplV2;
+import org.apache.pinot.core.plan.maker.PlanMaker;
+import org.apache.pinot.core.query.reduce.BrokerReduceService;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils;
+import org.apache.pinot.core.routing.MultiClusterRoutingContext;
+import org.apache.pinot.core.routing.RoutingManager;
+import org.apache.pinot.core.transport.ServerRoutingInstance;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.SegmentContext;
+import org.apache.pinot.spi.accounting.ThreadAccountant;
+import org.apache.pinot.spi.auth.AuthorizationResult;
+import org.apache.pinot.spi.auth.broker.RequesterIdentity;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.query.QueryExecutionContext;
+import org.apache.pinot.spi.query.QueryThreadContext;
+import org.apache.pinot.spi.trace.RequestContext;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+import org.apache.pinot.sql.parsers.SqlNodeAndOptions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Broker request handler for system tables (handled entirely on the broker).
+ * <p>
+ * System tables are virtual tables that expose cluster metadata (e.g., system.tables, system.instances).
+ * They are executed using the v1 query engine against in-memory segments generated on-demand by
+ * {@link SystemTableProvider} implementations.
+ */
+public class SystemTableBrokerRequestHandler extends BaseBrokerRequestHandler {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableBrokerRequestHandler.class);
+ // Sentinel values used for broker-local execution of system table queries where no actual server routing is needed.
+ private static final String SYSTEM_TABLE_PSEUDO_HOST = "localhost";
+ private static final int SYSTEM_TABLE_PSEUDO_PORT = 0;
+ private static final String SYSTEM_TABLE_DATATABLE_API_PATH = "/query/systemTable/datatable";
+ // Hop-by-hop headers per RFC 7230 plus content-length/host which are request-specific.
+ private static final Set<String> HOP_BY_HOP_HEADERS_TO_SKIP = Set.of(
+ "connection",
+ "keep-alive",
+ "proxy-authenticate",
+ "proxy-authorization",
+ "te",
+ "trailer",
+ "transfer-encoding",
+ "upgrade",
+ "host",
+ "content-length");
+
+ private final SystemTableRegistry _systemTableRegistry;
+ private final BrokerReduceService _brokerReduceService;
+ private final PlanMaker _planMaker;
+ private final ExecutorService _executorService;
+ private final ExecutorService _scatterGatherExecutorService;
+ private final SystemTableDataTableClient _systemTableDataTableClient;
+ @Nullable
+ private final HelixManager _helixManager;
+
+ public SystemTableBrokerRequestHandler(PinotConfiguration config, String brokerId,
+ BrokerRequestIdGenerator requestIdGenerator, RoutingManager routingManager,
+ AccessControlFactory accessControlFactory, QueryQuotaManager queryQuotaManager, TableCache tableCache,
+ SystemTableRegistry systemTableRegistry, ThreadAccountant threadAccountant,
+ @Nullable MultiClusterRoutingContext multiClusterRoutingContext, @Nullable HelixManager helixManager) {
+ super(config, brokerId, requestIdGenerator, routingManager, accessControlFactory, queryQuotaManager, tableCache,
+ threadAccountant, multiClusterRoutingContext);
+ _systemTableRegistry = systemTableRegistry;
+ _brokerReduceService = new BrokerReduceService(_config);
+ _planMaker = new InstancePlanMakerImplV2();
+ _planMaker.init(_config);
+ _helixManager = helixManager;
+ int executorPoolSize = config.getProperty(CommonConstants.Broker.CONFIG_OF_SYSTEM_TABLE_EXECUTOR_POOL_SIZE,
+ CommonConstants.Broker.DEFAULT_SYSTEM_TABLE_EXECUTOR_POOL_SIZE);
+ executorPoolSize = Math.max(1, executorPoolSize);
+ _executorService = QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(executorPoolSize,
+ new NamedThreadFactory("system-table-query-executor")));
+ _scatterGatherExecutorService =
+ QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(executorPoolSize,
+ new NamedThreadFactory("system-table-scatter-gather-executor")));
+ SSLContext sslContext = BrokerContext.getInstance().getClientHttpsContext();
+ int timeoutMs;
+ if (_brokerTimeoutMs > Integer.MAX_VALUE) {
+ LOGGER.warn("Broker timeout {}ms exceeds Integer.MAX_VALUE; clamping to {} for system table client connections",
+ _brokerTimeoutMs, Integer.MAX_VALUE);
+ timeoutMs = Integer.MAX_VALUE;
+ } else if (_brokerTimeoutMs < 1) {
+ LOGGER.warn("Broker timeout {}ms is non-positive; using default 5000ms for system table client connections",
+ _brokerTimeoutMs);
+ timeoutMs = 5000;
+ } else {
+ timeoutMs = (int) _brokerTimeoutMs;
+ }
+ ConnectionTimeouts connectionTimeouts = ConnectionTimeouts.create(timeoutMs, timeoutMs, timeoutMs);
+ _systemTableDataTableClient =
+ new SystemTableDataTableClient(connectionTimeouts, sslContext);
+ }
+
+ @Override
+ public void start() {
+ }
+
+ @Override
+ public void shutDown() {
+ _executorService.shutdownNow();
+ _scatterGatherExecutorService.shutdownNow();
+ try {
+ _systemTableDataTableClient.close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close system table data table client: {}", e.toString());
+ }
+ _brokerReduceService.shutDown();
+ }
+
+ public boolean canHandle(String tableName) {
+ return isSystemTable(tableName) && _systemTableRegistry.isRegistered(tableName);
+ }
+
+ @Override
+ protected BrokerResponse handleRequest(long requestId, String query, SqlNodeAndOptions sqlNodeAndOptions,
+ JsonNode request, @Nullable RequesterIdentity requesterIdentity, RequestContext requestContext,
+ @Nullable HttpHeaders httpHeaders, AccessControl accessControl)
+ throws Exception {
+ long startTimeMs = requestContext.getRequestArrivalTimeMillis();
+ long deadlineMs = startTimeMs + _brokerTimeoutMs;
+ QueryExecutionContext executionContext =
+ new QueryExecutionContext(QueryExecutionContext.QueryType.STE, requestId, Long.toString(requestId),
+ QueryOptionsUtils.getWorkloadName(sqlNodeAndOptions.getOptions()), startTimeMs, deadlineMs, deadlineMs,
+ _brokerId, _brokerId, org.apache.pinot.spi.utils.CommonConstants.Broker.DEFAULT_QUERY_HASH);
+ try (QueryThreadContext ignore = QueryThreadContext.open(executionContext, _threadAccountant)) {
+ PinotQuery pinotQuery;
+ try {
+ pinotQuery = CalciteSqlParser.compileToPinotQuery(sqlNodeAndOptions);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return new BrokerResponseNative(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ Set<String> tableNames = RequestUtils.getTableNames(pinotQuery);
+ if (tableNames == null || tableNames.isEmpty()) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "Failed to extract table name");
+ }
+ if (tableNames.size() != 1) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "System tables do not support joins");
+ }
+ String tableName = tableNames.iterator().next();
+ if (!isSystemTable(tableName)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "Not a system table query");
+ }
+ AuthorizationResult authorizationResult =
+ hasTableAccess(requesterIdentity, Set.of(tableName), requestContext, httpHeaders);
+ if (!authorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return new BrokerResponseNative(QueryErrorCode.ACCESS_DENIED, authorizationResult.getFailureMessage());
+ }
+
+ boolean queryWasLogged = _queryLogger.logQueryReceived(requestId, query);
+ return handleSystemTableQuery(request, pinotQuery, tableName, requestContext, requesterIdentity, query,
+ httpHeaders, queryWasLogged);
+ }
+ }
+
+ @Override
+ protected boolean handleCancel(long queryId, int timeoutMs, Executor executor,
+ HttpClientConnectionManager connMgr, Map<String, Integer> serverResponses) {
+ return false;
+ }
+
+ @Override
+ public boolean cancelQueryByClientId(String clientQueryId, int timeoutMs, Executor executor,
+ HttpClientConnectionManager connMgr, Map<String, Integer> serverResponses)
+ throws Exception {
+ return false;
+ }
+
+ @Override
+ public Map<Long, String> getRunningQueries() {
+ return Collections.emptyMap();
+ }
+
+ @Override
+ public OptionalLong getRequestIdByClientId(String clientQueryId) {
+ return OptionalLong.empty();
+ }
+
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ /**
+ * Executes a system table query against the local broker and returns the raw {@link DataTable} results.
+ * <p>
+ * This method is used by the internal broker-to-broker scatter-gather endpoint and must never perform fanout.
+ * It is invoked when a broker receives a scatter-gather request from another broker for a system table query.
+ *
+ * @param request the JSON request containing the SQL query and options
+ * @param requesterIdentity the identity of the requester for authorization
+ * @param requestContext the request context for tracking
+ * @param httpHeaders the HTTP headers from the request
+ * @return a DataTable containing the query results from this broker's local shard
+ */
+ public DataTable handleSystemTableDataTableRequest(JsonNode request, @Nullable RequesterIdentity requesterIdentity,
+ RequestContext requestContext, @Nullable HttpHeaders httpHeaders) {
+ long startTimeMs = requestContext.getRequestArrivalTimeMillis();
+ if (startTimeMs <= 0) {
+ startTimeMs = System.currentTimeMillis();
+ requestContext.setRequestArrivalTimeMillis(startTimeMs);
+ }
+ long requestId = _requestIdGenerator.get();
+ long deadlineMs = startTimeMs + _brokerTimeoutMs;
+
+ JsonNode sql = request.get(CommonConstants.Broker.Request.SQL);
+ if (sql == null || !sql.isTextual()) {
+ return exceptionDataTable(QueryErrorCode.JSON_PARSING, "Failed to find 'sql' in the request: " + request);
+ }
+ String query = sql.textValue();
+ requestContext.setQuery(query);
+
+ SqlNodeAndOptions sqlNodeAndOptions;
+ try {
+ sqlNodeAndOptions = RequestUtils.parseQuery(query, request);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return exceptionDataTable(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ QueryExecutionContext executionContext =
+ new QueryExecutionContext(QueryExecutionContext.QueryType.STE, requestId, Long.toString(requestId),
+ QueryOptionsUtils.getWorkloadName(sqlNodeAndOptions.getOptions()), startTimeMs, deadlineMs, deadlineMs,
+ _brokerId, _brokerId, org.apache.pinot.spi.utils.CommonConstants.Broker.DEFAULT_QUERY_HASH);
+ try (QueryThreadContext ignore = QueryThreadContext.open(executionContext, _threadAccountant)) {
+ AccessControl accessControl = _accessControlFactory.create();
+ AuthorizationResult authorizationResult = accessControl.authorize(requesterIdentity);
+ if (!authorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return exceptionDataTable(QueryErrorCode.ACCESS_DENIED, authorizationResult.getFailureMessage());
+ }
+
+ PinotQuery pinotQuery;
+ try {
+ pinotQuery = CalciteSqlParser.compileToPinotQuery(sqlNodeAndOptions);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return exceptionDataTable(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ Set<String> tableNames = RequestUtils.getTableNames(pinotQuery);
+ if (tableNames == null || tableNames.isEmpty()) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, "Failed to extract table name");
+ }
+ if (tableNames.size() != 1) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, "System tables do not support joins");
+ }
+ String tableName = tableNames.iterator().next();
+ if (!isSystemTable(tableName)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, "Not a system table query");
+ }
+
+ AuthorizationResult tableAuthorizationResult =
+ hasTableAccess(requesterIdentity, Set.of(tableName), requestContext, httpHeaders);
+ if (!tableAuthorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return exceptionDataTable(QueryErrorCode.ACCESS_DENIED, tableAuthorizationResult.getFailureMessage());
+ }
+
+ SystemTableProvider provider = _systemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ Collection<SystemTableProvider> availableProviders = _systemTableRegistry.getProviders();
+ String availableTables = availableProviders.isEmpty()
+ ? "No system tables are available"
+ : "Available system tables: "
+ + availableProviders.stream().map(SystemTableProvider::getTableName)
+ .collect(java.util.stream.Collectors.joining(", "));
+ return exceptionDataTable(QueryErrorCode.TABLE_DOES_NOT_EXIST,
+ "System table does not exist: " + tableName + ". " + availableTables);
+ }
+
+ try {
+ return executeLocalSystemTableQuery(pinotQuery, provider);
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table datatable query {}: {}", tableName, e.getMessage(),
+ e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return exceptionDataTable(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+ }
+
+ private BrokerResponse handleSystemTableQuery(JsonNode request, PinotQuery pinotQuery, String tableName,
+ RequestContext requestContext, @Nullable RequesterIdentity requesterIdentity, String query,
+ @Nullable HttpHeaders httpHeaders, boolean queryWasLogged) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableProvider provider = _systemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ Collection<SystemTableProvider> availableProviders = _systemTableRegistry.getProviders();
+ String availableTables = availableProviders.isEmpty()
+ ? "No system tables are available"
+ : "Available system tables: "
+ + availableProviders.stream().map(SystemTableProvider::getTableName)
+ .collect(java.util.stream.Collectors.joining(", "));
+ return new BrokerResponseNative(QueryErrorCode.TABLE_DOES_NOT_EXIST,
+ "System table does not exist: " + tableName + ". " + availableTables);
+ }
+ try {
+ long deadlineMs = requestContext.getRequestArrivalTimeMillis() + _brokerTimeoutMs;
+ Map<ServerRoutingInstance, DataTable> dataTableMap;
+ if (provider.getExecutionMode() == SystemTableProvider.ExecutionMode.BROKER_SCATTER_GATHER) {
+ dataTableMap =
+ scatterGatherSystemTableDataTables(provider, pinotQuery, tableName, request, httpHeaders, deadlineMs);
+ } else {
+ dataTableMap = new HashMap<>(1);
+ // Use a synthetic routing instance for broker-local execution of system table queries.
+ dataTableMap.put(new ServerRoutingInstance(SYSTEM_TABLE_PSEUDO_HOST, SYSTEM_TABLE_PSEUDO_PORT,
+ TableType.OFFLINE), executeLocalSystemTableQuery(pinotQuery, provider));
+ }
+
+ BrokerResponseNative brokerResponse;
+ BrokerRequest brokerRequest = new BrokerRequest();
+ QuerySource querySource = new QuerySource();
+ querySource.setTableName(tableName);
+ brokerRequest.setQuerySource(querySource);
+ brokerRequest.setPinotQuery(pinotQuery);
+ brokerResponse = _brokerReduceService.reduceOnDataTable(brokerRequest, brokerRequest, dataTableMap,
+ _brokerTimeoutMs, _brokerMetrics);
+ brokerResponse.setTablesQueried(Set.of(TableNameBuilder.extractRawTableName(tableName)));
+ brokerResponse.setTimeUsedMs(System.currentTimeMillis() - requestContext.getRequestArrivalTimeMillis());
+ _queryLogger.logQueryCompleted(
+ new QueryLogger.QueryLogParams(requestContext, tableName, brokerResponse,
+ QueryLogger.QueryLogParams.QueryEngine.SINGLE_STAGE, requesterIdentity, null),
+ queryWasLogged);
+ return brokerResponse;
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table query {}: {}", tableName, e.getMessage(), e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+
+ private DataTable executeLocalSystemTableQuery(PinotQuery pinotQuery, SystemTableProvider provider)
+ throws Exception {
+ IndexSegment dataSource = provider.getDataSource();
+ try {
+ QueryContext queryContext = QueryContextConverterUtils.getQueryContext(pinotQuery);
+ queryContext.setSchema(provider.getSchema());
+ queryContext.setEndTimeMs(System.currentTimeMillis() + _brokerTimeoutMs);
+
+ // Pass null for serverMetrics because system table queries run broker-local against an in-memory IndexSegment.
+ Plan plan = _planMaker.makeInstancePlan(List.of(new SegmentContext(dataSource)), queryContext, _executorService,
+ null);
+ InstanceResponseBlock instanceResponse = plan.execute();
+ return instanceResponse.toDataTable();
+ } finally {
+ dataSource.destroy();
+ }
+ }
+
+ private static final class BrokerTarget {
+ final ServerRoutingInstance _routingInstance;
+ final String _dataTableUrl;
+
+ BrokerTarget(ServerRoutingInstance routingInstance, String dataTableUrl) {
+ _routingInstance = routingInstance;
+ _dataTableUrl = dataTableUrl;
+ }
+ }
+
+ @VisibleForTesting
+ protected Map<ServerRoutingInstance, DataTable> scatterGatherSystemTableDataTables(SystemTableProvider provider,
+ PinotQuery pinotQuery, String tableName, JsonNode request, @Nullable HttpHeaders httpHeaders, long deadlineMs) {
+ if (_helixManager == null) {
+ throw new IllegalStateException(
+ "HelixManager is required for scatter-gather execution of system table: " + tableName);
+ }
+
+ HelixDataAccessor dataAccessor = _helixManager.getHelixDataAccessor();
+ List<String> liveInstances = dataAccessor.getChildNames(dataAccessor.keyBuilder().liveInstances());
+ if (liveInstances == null || liveInstances.isEmpty()) {
+ throw new IllegalStateException("No live instances found for scatter-gather execution of system table: "
+ + tableName);
+ }
+
+ String localInstanceId = _brokerId;
+ List<BrokerTarget> remoteTargets = new ArrayList<>();
+ @Nullable ServerRoutingInstance localRoutingInstance = null;
+ for (String instanceId : liveInstances) {
+ if (!InstanceTypeUtils.isBroker(instanceId)) {
+ continue;
+ }
+ InstanceConfig instanceConfig = dataAccessor.getProperty(dataAccessor.keyBuilder().instanceConfig(instanceId));
+ if (instanceConfig == null) {
+ LOGGER.warn("Instance config not found for broker instance: {}. This instance will be skipped in scatter-gather"
+ + " execution for system table: {}", instanceId, tableName);
+ continue;
+ }
+ URI baseUri = URI.create(InstanceUtils.getInstanceBaseUri(instanceConfig));
+ ServerRoutingInstance routingInstance = new ServerRoutingInstance(baseUri.getHost(), baseUri.getPort(),
+ TableType.OFFLINE);
+ if (instanceId.equals(localInstanceId)) {
+ localRoutingInstance = routingInstance;
+ } else {
+ remoteTargets.add(new BrokerTarget(routingInstance, baseUri.toString() + SYSTEM_TABLE_DATATABLE_API_PATH));
+ }
+ }
+
+ Map<ServerRoutingInstance, DataTable> dataTableMap = new HashMap<>(remoteTargets.size() + 1);
+ ServerRoutingInstance routingInstance;
+ if (localRoutingInstance != null) {
+ routingInstance = localRoutingInstance;
+ } else {
+ routingInstance = new ServerRoutingInstance(SYSTEM_TABLE_PSEUDO_HOST, SYSTEM_TABLE_PSEUDO_PORT,
+ TableType.OFFLINE);
+ }
+ try {
+ dataTableMap.put(routingInstance, executeLocalSystemTableQuery(pinotQuery, provider));
+ } catch (Exception e) {
+ LOGGER.error("Failed to execute system table query locally for query: {}", pinotQuery, e);
+ dataTableMap.put(routingInstance,
+ exceptionDataTable(QueryErrorCode.QUERY_EXECUTION, "Failed to execute system table query locally: "
+ + e.getMessage()));
+ }
+
+ if (remoteTargets.isEmpty()) {
+ return dataTableMap;
+ }
+
+ String requestBody = request.toString();
+ @Nullable Map<String, String> requestHeaders = toSingleValueRequestHeaders(httpHeaders);
+ if (requestHeaders == null) {
+ requestHeaders = new HashMap<>();
+ }
+ requestHeaders.putIfAbsent("Content-Type", MediaType.APPLICATION_JSON);
+ Map<String, String> requestHeadersFinal = requestHeaders;
+ List<Pair<BrokerTarget, Future<DataTable>>> futures = new ArrayList<>(remoteTargets.size());
+ for (BrokerTarget target : remoteTargets) {
+ Future<DataTable> future =
+ _scatterGatherExecutorService.submit(() -> fetchDataTableFromBroker(target._dataTableUrl, requestBody,
+ requestHeadersFinal));
+ futures.add(Pair.of(target, future));
+ }
+
+ long remainingMs = Math.max(1, deadlineMs - System.currentTimeMillis());
+ for (Pair<BrokerTarget, Future<DataTable>> pair : futures) {
+ BrokerTarget target = pair.getLeft();
+ try {
+ dataTableMap.put(target._routingInstance, pair.getRight().get(remainingMs, TimeUnit.MILLISECONDS));
+ } catch (TimeoutException e) {
+ pair.getRight().cancel(true);
+ dataTableMap.put(target._routingInstance, exceptionDataTable(QueryErrorCode.BROKER_TIMEOUT,
+ "Timed out waiting for system table response from " + target._dataTableUrl));
Review Comment:
In scatter-gather, `remainingMs` is computed once before iterating futures, then reused for every `Future#get(...)`. This can let total wait time exceed the overall deadline by up to N× (one full timeout per broker). Recompute the remaining time inside the loop (or track an absolute end time) and fail fast once the deadline is exceeded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2748087068
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/SystemTableBrokerRequestHandler.java:
##########
@@ -0,0 +1,537 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.requesthandler;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.google.common.annotations.VisibleForTesting;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.HelixManager;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.pinot.broker.api.AccessControl;
+import org.apache.pinot.broker.broker.AccessControlFactory;
+import org.apache.pinot.broker.querylog.QueryLogger;
+import org.apache.pinot.broker.queryquota.QueryQuotaManager;
+import org.apache.pinot.client.ConnectionTimeouts;
+import org.apache.pinot.client.PinotClientException;
+import org.apache.pinot.client.SystemTableDataTableClient;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.common.datatable.DataTableImplV4;
+import org.apache.pinot.common.metrics.BrokerMeter;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.request.QuerySource;
+import org.apache.pinot.common.response.BrokerResponse;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.SystemTableRegistry;
+import org.apache.pinot.common.utils.NamedThreadFactory;
+import org.apache.pinot.common.utils.config.InstanceUtils;
+import org.apache.pinot.common.utils.config.QueryOptionsUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.core.instance.context.BrokerContext;
+import org.apache.pinot.core.operator.blocks.InstanceResponseBlock;
+import org.apache.pinot.core.plan.Plan;
+import org.apache.pinot.core.plan.maker.InstancePlanMakerImplV2;
+import org.apache.pinot.core.plan.maker.PlanMaker;
+import org.apache.pinot.core.query.reduce.BrokerReduceService;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils;
+import org.apache.pinot.core.routing.MultiClusterRoutingContext;
+import org.apache.pinot.core.routing.RoutingManager;
+import org.apache.pinot.core.transport.ServerRoutingInstance;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.SegmentContext;
+import org.apache.pinot.spi.accounting.ThreadAccountant;
+import org.apache.pinot.spi.auth.AuthorizationResult;
+import org.apache.pinot.spi.auth.broker.RequesterIdentity;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.query.QueryExecutionContext;
+import org.apache.pinot.spi.query.QueryThreadContext;
+import org.apache.pinot.spi.trace.RequestContext;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+import org.apache.pinot.sql.parsers.SqlNodeAndOptions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Broker request handler for system tables (handled entirely on the broker).
+ */
+public class SystemTableBrokerRequestHandler extends BaseBrokerRequestHandler {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableBrokerRequestHandler.class);
+ private static final String SYSTEM_TABLE_PSEUDO_HOST = "localhost";
+ private static final int SYSTEM_TABLE_PSEUDO_PORT = 0;
+ private static final String SYSTEM_TABLE_DATATABLE_API_PATH = "/query/systemTable/datatable";
+ // Hop-by-hop headers per RFC 7230 plus content-length/host which are request-specific.
+ private static final Set<String> HOP_BY_HOP_HEADERS_TO_SKIP = Set.of(
+ "connection",
+ "keep-alive",
+ "proxy-authenticate",
+ "proxy-authorization",
+ "te",
+ "trailer",
+ "transfer-encoding",
+ "upgrade",
+ "host",
+ "content-length");
+
+ private final BrokerReduceService _brokerReduceService;
+ private final PlanMaker _planMaker;
+ private final ExecutorService _executorService;
+ private final ExecutorService _scatterGatherExecutorService;
+ private final SystemTableDataTableClient _systemTableDataTableClient;
+ @Nullable
+ private final HelixManager _helixManager;
+
+ public SystemTableBrokerRequestHandler(PinotConfiguration config, String brokerId,
+ BrokerRequestIdGenerator requestIdGenerator, RoutingManager routingManager,
+ AccessControlFactory accessControlFactory, QueryQuotaManager queryQuotaManager, TableCache tableCache,
+ ThreadAccountant threadAccountant, @Nullable MultiClusterRoutingContext multiClusterRoutingContext,
+ @Nullable HelixManager helixManager) {
+ super(config, brokerId, requestIdGenerator, routingManager, accessControlFactory, queryQuotaManager, tableCache,
+ threadAccountant, multiClusterRoutingContext);
+ _brokerReduceService = new BrokerReduceService(_config);
+ _planMaker = new InstancePlanMakerImplV2();
+ _planMaker.init(_config);
+ _helixManager = helixManager;
+ int executorPoolSize = config.getProperty(CommonConstants.Broker.CONFIG_OF_SYSTEM_TABLE_EXECUTOR_POOL_SIZE,
+ CommonConstants.Broker.DEFAULT_SYSTEM_TABLE_EXECUTOR_POOL_SIZE);
+ executorPoolSize = Math.max(1, executorPoolSize);
+ _executorService = QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(executorPoolSize,
+ new NamedThreadFactory("system-table-query-executor")));
+ _scatterGatherExecutorService =
+ QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(executorPoolSize,
+ new NamedThreadFactory("system-table-scatter-gather-executor")));
+ SSLContext sslContext = BrokerContext.getInstance().getClientHttpsContext();
+ int timeoutMs = (int) Math.min(Integer.MAX_VALUE, _brokerTimeoutMs);
+ if (timeoutMs < 1) {
+ timeoutMs = 1;
+ }
+ ConnectionTimeouts connectionTimeouts = ConnectionTimeouts.create(timeoutMs, timeoutMs, timeoutMs);
+ _systemTableDataTableClient =
+ new SystemTableDataTableClient(connectionTimeouts, sslContext);
Review Comment:
The SystemTableDataTableClient is instantiated with specific connection timeouts and SSL context, but there are no tests covering error handling when SSL context is null or when connection timeouts are invalid. Consider adding test coverage for these edge cases.
##########
pinot-plugins/pinot-system-table/src/main/java/org/apache/pinot/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,662 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.databind.JsonNode;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.client.admin.PinotAdminClient;
+import org.apache.pinot.client.admin.PinotAdminTransport;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.systemtable.SystemTable;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.datasource.InMemorySystemTableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final String SIZE_CACHE_TTL_MS_PROPERTY = "pinot.systemtable.tables.sizeCacheTtlMs";
+ private static final long DEFAULT_SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+ private static final long SIZE_CACHE_TTL_MS = getNonNegativeLongProperty(SIZE_CACHE_TTL_MS_PROPERTY,
+ DEFAULT_SIZE_CACHE_TTL_MS);
+
+ private static final String CONTROLLER_TIMEOUT_MS_PROPERTY = "pinot.systemtable.tables.controllerTimeoutMs";
+ private static final long DEFAULT_CONTROLLER_TIMEOUT_MS = Duration.ofSeconds(5).toMillis();
+ private static final long CONTROLLER_TIMEOUT_MS = getPositiveLongProperty(CONTROLLER_TIMEOUT_MS_PROPERTY,
+ DEFAULT_CONTROLLER_TIMEOUT_MS);
+
+ private static final long SIZE_FETCH_FAILURE_WARN_INTERVAL_MS = Duration.ofHours(1).toMillis();
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
Review Comment:
The column names `reportedSize` and `estimatedSize` are inconsistent with the camelCase convention used by other columns in the schema. Consider renaming them to `reportedSizeBytes` and `estimatedSizeBytes` to clarify the unit and maintain consistency with other size-related fields.
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/SystemTableRegistry.java:
##########
@@ -0,0 +1,202 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.IdentityHashMap;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.PinotReflectionUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Registry to hold and lifecycle-manage system table data providers.
+ */
+public final class SystemTableRegistry {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableRegistry.class);
+
+ // Providers are registered once at broker startup.
+ private static final Map<String, SystemTableProvider> PROVIDERS = new HashMap<>();
+
+ private static final class ConstructorSpec {
+ final Class<?>[] _paramTypes;
+ final Object[] _args;
+
+ ConstructorSpec(Class<?>[] paramTypes, Object[] args) {
+ _paramTypes = paramTypes;
+ _args = args;
+ }
+ }
+
+ private SystemTableRegistry() {
+ throw new UnsupportedOperationException("This is a utility class and cannot be instantiated");
+ }
+
+ public static void register(SystemTableProvider provider) {
+ synchronized (PROVIDERS) {
+ PROVIDERS.put(normalize(provider.getTableName()), provider);
+ }
+ }
+
+ @Nullable
+ public static SystemTableProvider get(String tableName) {
+ synchronized (PROVIDERS) {
+ return PROVIDERS.get(normalize(tableName));
+ }
+ }
+
+ public static boolean isRegistered(String tableName) {
+ synchronized (PROVIDERS) {
+ return PROVIDERS.containsKey(normalize(tableName));
+ }
+ }
+
+ public static Collection<SystemTableProvider> getProviders() {
+ synchronized (PROVIDERS) {
+ return Collections.unmodifiableCollection(new java.util.ArrayList<>(PROVIDERS.values()));
+ }
+ }
+
+ /**
+ * Discover and register providers marked with {@link SystemTable} using the available dependencies.
+ * Follows the ScalarFunction pattern: any class annotated with @SystemTable under a "*.systemtable.*" package
+ * will be discovered via reflection and registered.
+ */
+ public static void registerAnnotatedProviders(TableCache tableCache, HelixAdmin helixAdmin, String clusterName,
+ @Nullable PinotConfiguration config) {
+ Set<Class<?>> classes =
+ PinotReflectionUtils.getClassesThroughReflection(".*\\.systemtable\\..*", SystemTable.class);
+ for (Class<?> clazz : classes) {
+ if (!SystemTableProvider.class.isAssignableFrom(clazz)) {
+ continue;
+ }
+ SystemTableProvider provider =
+ instantiateProvider(clazz.asSubclass(SystemTableProvider.class), tableCache, helixAdmin, clusterName, config);
+ if (provider == null) {
+ continue;
+ }
+ if (isRegistered(provider.getTableName())) {
+ continue;
+ }
+ LOGGER.info("Registering system table provider: {}", provider.getTableName());
+ register(provider);
+ }
+ }
+
+ public static void close()
+ throws Exception {
+ Exception firstException = null;
+ // Snapshot providers to avoid concurrent modifications and to close each provider at most once.
+ Map<SystemTableProvider, Boolean> providersToClose = new IdentityHashMap<>();
+ synchronized (PROVIDERS) {
+ for (SystemTableProvider provider : PROVIDERS.values()) {
+ providersToClose.put(provider, Boolean.TRUE);
+ }
+ }
+ try {
+ for (SystemTableProvider provider : providersToClose.keySet()) {
Review Comment:
Using `IdentityHashMap<SystemTableProvider, Boolean>` to track unique providers for closing is unconventional. Consider using `Collections.newSetFromMap(new IdentityHashMap<>())` to create an identity-based Set, which more clearly expresses the intent of tracking unique provider instances.
```suggestion
Set<SystemTableProvider> providersToClose =
Collections.newSetFromMap(new IdentityHashMap<SystemTableProvider, Boolean>());
synchronized (PROVIDERS) {
for (SystemTableProvider provider : PROVIDERS.values()) {
providersToClose.add(provider);
}
}
try {
for (SystemTableProvider provider : providersToClose) {
```
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/SystemTableBrokerRequestHandler.java:
##########
@@ -0,0 +1,537 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.requesthandler;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.google.common.annotations.VisibleForTesting;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.HelixManager;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.pinot.broker.api.AccessControl;
+import org.apache.pinot.broker.broker.AccessControlFactory;
+import org.apache.pinot.broker.querylog.QueryLogger;
+import org.apache.pinot.broker.queryquota.QueryQuotaManager;
+import org.apache.pinot.client.ConnectionTimeouts;
+import org.apache.pinot.client.PinotClientException;
+import org.apache.pinot.client.SystemTableDataTableClient;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.common.datatable.DataTableImplV4;
+import org.apache.pinot.common.metrics.BrokerMeter;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.request.QuerySource;
+import org.apache.pinot.common.response.BrokerResponse;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.SystemTableRegistry;
+import org.apache.pinot.common.utils.NamedThreadFactory;
+import org.apache.pinot.common.utils.config.InstanceUtils;
+import org.apache.pinot.common.utils.config.QueryOptionsUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.core.instance.context.BrokerContext;
+import org.apache.pinot.core.operator.blocks.InstanceResponseBlock;
+import org.apache.pinot.core.plan.Plan;
+import org.apache.pinot.core.plan.maker.InstancePlanMakerImplV2;
+import org.apache.pinot.core.plan.maker.PlanMaker;
+import org.apache.pinot.core.query.reduce.BrokerReduceService;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils;
+import org.apache.pinot.core.routing.MultiClusterRoutingContext;
+import org.apache.pinot.core.routing.RoutingManager;
+import org.apache.pinot.core.transport.ServerRoutingInstance;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.SegmentContext;
+import org.apache.pinot.spi.accounting.ThreadAccountant;
+import org.apache.pinot.spi.auth.AuthorizationResult;
+import org.apache.pinot.spi.auth.broker.RequesterIdentity;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.query.QueryExecutionContext;
+import org.apache.pinot.spi.query.QueryThreadContext;
+import org.apache.pinot.spi.trace.RequestContext;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+import org.apache.pinot.sql.parsers.SqlNodeAndOptions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Broker request handler for system tables (handled entirely on the broker).
+ */
+public class SystemTableBrokerRequestHandler extends BaseBrokerRequestHandler {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableBrokerRequestHandler.class);
+ private static final String SYSTEM_TABLE_PSEUDO_HOST = "localhost";
+ private static final int SYSTEM_TABLE_PSEUDO_PORT = 0;
+ private static final String SYSTEM_TABLE_DATATABLE_API_PATH = "/query/systemTable/datatable";
+ // Hop-by-hop headers per RFC 7230 plus content-length/host which are request-specific.
+ private static final Set<String> HOP_BY_HOP_HEADERS_TO_SKIP = Set.of(
+ "connection",
+ "keep-alive",
+ "proxy-authenticate",
+ "proxy-authorization",
+ "te",
+ "trailer",
+ "transfer-encoding",
+ "upgrade",
+ "host",
+ "content-length");
+
+ private final BrokerReduceService _brokerReduceService;
+ private final PlanMaker _planMaker;
+ private final ExecutorService _executorService;
+ private final ExecutorService _scatterGatherExecutorService;
+ private final SystemTableDataTableClient _systemTableDataTableClient;
+ @Nullable
+ private final HelixManager _helixManager;
+
+ public SystemTableBrokerRequestHandler(PinotConfiguration config, String brokerId,
+ BrokerRequestIdGenerator requestIdGenerator, RoutingManager routingManager,
+ AccessControlFactory accessControlFactory, QueryQuotaManager queryQuotaManager, TableCache tableCache,
+ ThreadAccountant threadAccountant, @Nullable MultiClusterRoutingContext multiClusterRoutingContext,
+ @Nullable HelixManager helixManager) {
+ super(config, brokerId, requestIdGenerator, routingManager, accessControlFactory, queryQuotaManager, tableCache,
+ threadAccountant, multiClusterRoutingContext);
+ _brokerReduceService = new BrokerReduceService(_config);
+ _planMaker = new InstancePlanMakerImplV2();
+ _planMaker.init(_config);
+ _helixManager = helixManager;
+ int executorPoolSize = config.getProperty(CommonConstants.Broker.CONFIG_OF_SYSTEM_TABLE_EXECUTOR_POOL_SIZE,
+ CommonConstants.Broker.DEFAULT_SYSTEM_TABLE_EXECUTOR_POOL_SIZE);
+ executorPoolSize = Math.max(1, executorPoolSize);
+ _executorService = QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(executorPoolSize,
+ new NamedThreadFactory("system-table-query-executor")));
+ _scatterGatherExecutorService =
+ QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(executorPoolSize,
+ new NamedThreadFactory("system-table-scatter-gather-executor")));
+ SSLContext sslContext = BrokerContext.getInstance().getClientHttpsContext();
+ int timeoutMs = (int) Math.min(Integer.MAX_VALUE, _brokerTimeoutMs);
+ if (timeoutMs < 1) {
Review Comment:
The timeout clamping logic (minimum 1ms) could lead to unexpected behavior if `_brokerTimeoutMs` is 0 or negative, which might be a valid configuration for certain use cases. Consider logging a warning when the timeout is clamped, or documenting why 1ms is the minimum acceptable value.
```suggestion
if (timeoutMs < 1) {
// Enforce a minimum timeout of 1ms; zero or negative timeouts are not meaningful for client connections.
LOGGER.warn("Configured broker timeout {}ms is non-positive; clamping to 1ms for system table client "
+ "connections", _brokerTimeoutMs);
```
##########
pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/admin/PinotAdminPathUtils.java:
##########
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.client.admin;
+
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+
+
+/**
+ * Utility for encoding admin client URL path segments. Stateless and thread-safe.
+ */
+final class PinotAdminPathUtils {
+ private PinotAdminPathUtils() {
+ }
+
Review Comment:
The method lacks a Javadoc comment explaining why the `+` character is replaced with `%20`. This is important because URLEncoder.encode() follows application/x-www-form-urlencoded encoding (where space becomes +), but URL path segments require percent-encoding (where space becomes %20). Add a comment explaining this distinction.
```suggestion
/**
* Encodes a single URL path segment using UTF-8.
* <p>
* {@link URLEncoder} implements {@code application/x-www-form-urlencoded} encoding, where spaces
* are converted to {@code +}. For URL path segments, spaces must instead be percent-encoded as
* {@code %20}, so this method post-processes the encoded value to replace {@code +} with
* {@code %20}.
*/
```
##########
pinot-plugins/pinot-system-table/src/main/java/org/apache/pinot/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,662 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.databind.JsonNode;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.client.admin.PinotAdminClient;
+import org.apache.pinot.client.admin.PinotAdminTransport;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.systemtable.SystemTable;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.datasource.InMemorySystemTableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final String SIZE_CACHE_TTL_MS_PROPERTY = "pinot.systemtable.tables.sizeCacheTtlMs";
+ private static final long DEFAULT_SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+ private static final long SIZE_CACHE_TTL_MS = getNonNegativeLongProperty(SIZE_CACHE_TTL_MS_PROPERTY,
+ DEFAULT_SIZE_CACHE_TTL_MS);
+
+ private static final String CONTROLLER_TIMEOUT_MS_PROPERTY = "pinot.systemtable.tables.controllerTimeoutMs";
+ private static final long DEFAULT_CONTROLLER_TIMEOUT_MS = Duration.ofSeconds(5).toMillis();
+ private static final long CONTROLLER_TIMEOUT_MS = getPositiveLongProperty(CONTROLLER_TIMEOUT_MS_PROPERTY,
+ DEFAULT_CONTROLLER_TIMEOUT_MS);
+
+ private static final long SIZE_FETCH_FAILURE_WARN_INTERVAL_MS = Duration.ofHours(1).toMillis();
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _configuredControllerUrls;
+ private final Map<String, CachedSize> _sizeCache = new ConcurrentHashMap<>();
+ private final Map<String, PinotAdminClient> _adminClientCache = new ConcurrentHashMap<>();
+ private final AtomicLong _lastSizeFetchFailureWarnLogMs = new AtomicLong();
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _configuredControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public TableConfig getTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ }
+
+ @Override
+ public void close()
+ throws Exception {
+ for (Map.Entry<String, PinotAdminClient> entry : _adminClientCache.entrySet()) {
+ try {
+ entry.getValue().close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close admin client for {}: {}", entry.getKey(), e.toString());
+ }
+ }
+ _adminClientCache.clear();
+ }
+
+ @Override
+ public IndexSegment getDataSource() {
+ if (_tableCache == null) {
+ return new InMemorySystemTableSegment(TABLE_NAME, SCHEMA, 0, Collections.emptyMap());
+ }
+
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ List<String> controllerBaseUrls = getControllerBaseUrls();
+ Function<String, TableSize> sizeFetcher = getSizeFetcher();
+ class TableRow {
+ final String _tableNameWithType;
+ final TableType _tableType;
+ final String _rawTableName;
+ final @Nullable TableConfig _tableConfig;
+ private volatile @Nullable String _tableConfigJson;
+ private volatile @Nullable TableSize _tableSize;
+ private volatile boolean _tableSizeFetched;
+
+ private TableRow(String tableNameWithType, TableType tableType, String rawTableName,
+ @Nullable TableConfig tableConfig) {
+ _tableNameWithType = tableNameWithType;
+ _tableType = tableType;
+ _rawTableName = rawTableName;
+ _tableConfig = tableConfig;
+ }
+
+ @Nullable
+ private TableSize getTableSize() {
+ if (_tableSizeFetched) {
+ return _tableSize;
+ }
+ synchronized (this) {
+ if (_tableSizeFetched) {
+ return _tableSize;
+ }
+ _tableSize = fetchTableSize(_tableNameWithType, sizeFetcher, controllerBaseUrls);
+ _tableSizeFetched = true;
+ return _tableSize;
+ }
+ }
+
+ private String getStatus() {
+ if (_tableConfig != null) {
+ return "ONLINE";
+ }
+ TableSize sizeFromController = getTableSize();
+ int segments = sizeFromController != null ? getSegmentCount(sizeFromController, _tableType) : 0;
+ return segments > 0 ? "ONLINE" : "UNKNOWN";
+ }
+
+ private int getSegments() {
+ TableSize sizeFromController = getTableSize();
+ return sizeFromController != null ? getSegmentCount(sizeFromController, _tableType) : 0;
+ }
+
+ private long getTotalDocs() {
+ TableSize sizeFromController = getTableSize();
+ return sizeFromController != null ? TablesSystemTableProvider.this.getTotalDocs(sizeFromController, _tableType,
+ _tableNameWithType, controllerBaseUrls) : 0L;
+ }
+
+ private long getReportedSize() {
+ TableSize sizeFromController = getTableSize();
+ if (sizeFromController == null || sizeFromController._reportedSizeInBytes < 0) {
+ return 0L;
+ }
+ return sizeFromController._reportedSizeInBytes;
+ }
+
+ private long getEstimatedSize() {
+ TableSize sizeFromController = getTableSize();
+ if (sizeFromController == null || sizeFromController._estimatedSizeInBytes < 0) {
+ return 0L;
+ }
+ return sizeFromController._estimatedSizeInBytes;
+ }
+
+ private String getBrokerTenant() {
+ if (_tableConfig != null && _tableConfig.getTenantConfig() != null) {
+ String tenant = _tableConfig.getTenantConfig().getBroker();
+ return tenant != null ? tenant : "";
+ }
+ return "";
+ }
+
+ private String getServerTenant() {
+ if (_tableConfig != null && _tableConfig.getTenantConfig() != null) {
+ String tenant = _tableConfig.getTenantConfig().getServer();
+ return tenant != null ? tenant : "";
+ }
+ return "";
+ }
+
+ private int getReplicas() {
+ if (_tableConfig != null && _tableConfig.getValidationConfig() != null) {
+ Integer replicationNumber = _tableConfig.getValidationConfig().getReplicationNumber();
+ if (replicationNumber != null) {
+ return replicationNumber;
+ }
+ }
+ return 0;
+ }
+
+ private String getTableConfigJson() {
+ String cached = _tableConfigJson;
+ if (cached != null) {
+ return cached;
+ }
+ synchronized (this) {
+ cached = _tableConfigJson;
+ if (cached != null) {
+ return cached;
+ }
+ cached = "";
+ if (_tableConfig != null) {
+ try {
+ cached = JsonUtils.objectToString(_tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", _tableNameWithType, e.toString());
+ cached = _tableConfig.toString();
+ }
+ }
+ _tableConfigJson = cached;
+ return cached;
+ }
+ }
+ }
+
+ List<TableRow> tableRows = new ArrayList<>(sortedTableNames.size());
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ tableRows.add(new TableRow(tableNameWithType, tableType, rawTableName, tableConfig));
+ }
+
+ Map<String, IntFunction<Object>> valueProviders = new java.util.HashMap<>();
+ valueProviders.put("tableName", docId -> tableRows.get(docId)._rawTableName);
+ valueProviders.put("type", docId -> tableRows.get(docId)._tableType.name());
+ valueProviders.put("status", docId -> tableRows.get(docId).getStatus());
+ valueProviders.put("segments", docId -> tableRows.get(docId).getSegments());
+ valueProviders.put("totalDocs", docId -> tableRows.get(docId).getTotalDocs());
+ valueProviders.put("reportedSize", docId -> tableRows.get(docId).getReportedSize());
+ valueProviders.put("estimatedSize", docId -> tableRows.get(docId).getEstimatedSize());
+ valueProviders.put("brokerTenant", docId -> tableRows.get(docId).getBrokerTenant());
+ valueProviders.put("serverTenant", docId -> tableRows.get(docId).getServerTenant());
+ valueProviders.put("replicas", docId -> tableRows.get(docId).getReplicas());
+ valueProviders.put("tableConfig", docId -> tableRows.get(docId).getTableConfigJson());
+
+ return new InMemorySystemTableSegment(TABLE_NAME, SCHEMA, tableRows.size(), valueProviders);
+ }
+
+ @Nullable
+ private TableSize fetchTableSize(String tableNameWithType,
+ @Nullable Function<String, TableSize> fetcher, List<String> controllerBaseUrls) {
+ boolean cacheEnabled = SIZE_CACHE_TTL_MS > 0;
+ TableSize cached = cacheEnabled ? getCachedSize(tableNameWithType) : null;
+ if (cached != null) {
+ return cached;
+ }
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ if (cacheEnabled) {
+ cacheSize(tableNameWithType, fetched);
+ }
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(controllerBaseUrls, rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(controllerBaseUrls, tableNameWithType);
+ if (size == null) {
+ logSizeFetchFailure("{}: failed to fetch size for {} from controllers {} "
+ + "(tried raw table name '{}' and table name with type '{}')",
+ TABLE_NAME, tableNameWithType, controllerBaseUrls, rawTableName, tableNameWithType);
+ }
+ }
+ if (size != null && cacheEnabled) {
+ cacheSize(tableNameWithType, size);
+ }
+ return size;
+ }
+
+ @Nullable
+ private TableSize fetchTableSizeForName(List<String> controllerBaseUrls, String tableName) {
+ for (String baseUrl : controllerBaseUrls) {
+ try {
+ PinotAdminClient adminClient = getOrCreateAdminClient(baseUrl);
+ if (adminClient == null) {
+ continue;
+ }
+
+ JsonNode sizeNode = adminClient.getTableSize(tableName, true, false);
+
+ if (sizeNode == null) {
+ continue;
+ }
+
+ TableSize parsed = JsonUtils.stringToObject(sizeNode.toString(), TableSize.class);
+ LOGGER.debug("{}: controller size response for {} via {} -> segments offline={}, realtime={}, "
+ + "reportedSize={}, estimatedSize={}", TABLE_NAME, tableName, baseUrl,
+ parsed._offlineSegments != null && parsed._offlineSegments._segments != null
+ ? parsed._offlineSegments._segments.size() : 0,
+ parsed._realtimeSegments != null && parsed._realtimeSegments._segments != null
+ ? parsed._realtimeSegments._segments.size() : 0,
+ parsed._reportedSizeInBytes, parsed._estimatedSizeInBytes);
+ return parsed;
+ } catch (Exception e) {
+ logSizeFetchFailure("{}: error fetching table size for {} via {} using admin client", TABLE_NAME, tableName,
+ baseUrl, e);
+ }
+ }
+ return null;
+ }
+
+ private List<String> getControllerBaseUrls() {
+ Set<String> urls = new LinkedHashSet<>();
+ if (_helixAdmin != null) {
+ for (String controller : discoverControllersFromHelix()) {
+ String normalized = normalizeControllerUrl(controller);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ }
+ for (String url : _configuredControllerUrls) {
+ String normalized = normalizeControllerUrl(url);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ return new ArrayList<>(urls);
+ }
+
+ private int getSegmentCount(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.size();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.size();
+ }
+ return 0;
+ }
+
+ private long getTotalDocsFromSize(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ return 0;
+ }
+
+ private long getTotalDocs(TableSize sizeFromController, TableType tableType, String tableNameWithType,
+ List<String> controllerBaseUrls) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ long cached = sizeFromController._offlineTotalDocs;
+ if (cached >= 0) {
+ return cached;
+ }
+ long totalDocsFromSize = getTotalDocsFromSize(sizeFromController, tableType);
+ if (totalDocsFromSize > 0) {
+ synchronized (sizeFromController) {
+ if (sizeFromController._offlineTotalDocs < 0) {
+ sizeFromController._offlineTotalDocs = totalDocsFromSize;
+ }
+ return sizeFromController._offlineTotalDocs;
+ }
+ }
+ long fetched = fetchTotalDocsFromSegmentMetadata(tableNameWithType, sizeFromController._offlineSegments._segments,
+ controllerBaseUrls);
Review Comment:
The method fetches totalDocs by iterating through all segments and making individual controller API calls for each segment's metadata. For tables with many segments, this could result in a large number of network round trips. Consider optimizing by batching segment metadata requests or using a controller API that returns totalDocs for all segments in a single call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2643405446
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/InstancesSystemTableProvider.java:
##########
@@ -0,0 +1,76 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.SystemTableResponseUtils;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+
+
+/**
+ * Basic system table exposing Pinot instance metadata.
+ */
+@SystemTable
+public final class InstancesSystemTableProvider implements SystemTableDataProvider {
+ public static final String TABLE_NAME = "system.instances";
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("instanceId", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("host", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("port", FieldSpec.DataType.INT)
+ .addSingleValueDimension("state", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("tags", FieldSpec.DataType.STRING)
+ .build();
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public TableConfig getTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ }
+
+ @Override
+ public BrokerResponseNative getBrokerResponse(PinotQuery pinotQuery) {
+ // Controller/Helix fetch can be added later; for now return an empty result.
+ List<String> projectionColumns = pinotQuery.getSelectList() != null
+ ? pinotQuery.getSelectList().stream().map(expr -> expr.getIdentifier().getName()).collect(Collectors.toList())
Review Comment:
The code assumes all select expressions have identifiers without null checks. If `expr.getIdentifier()` returns null, this will throw a NullPointerException. Add null checks or use the same pattern as `TablesSystemTableProvider.getProjectionColumns` which handles this case properly.
```suggestion
? pinotQuery.getSelectList().stream()
.map(expr -> expr.getIdentifier())
.filter(identifier -> identifier != null)
.map(identifier -> identifier.getName())
.collect(Collectors.toList())
```
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BrokerRequestHandlerDelegate.java:
##########
@@ -106,7 +115,27 @@ public BrokerResponse handleRequest(JsonNode request, @Nullable SqlNodeAndOption
}
}
+ Set<String> tableNames = null;
+ if (_systemTableBrokerRequestHandler != null) {
+ try {
+ // NOTE: compileToPinotQuery(SqlNodeAndOptions) mutates the SqlNode (e.g. SqlOrderBy -> SqlSelect).
+ // Re-parse from raw SQL so the SqlNodeAndOptions passed to the handler is unchanged (important for
+ // Multi-Stage Engine (MSE)).
+ JsonNode sql = request.get(Request.SQL);
+ String sqlQuery =
+ (sql != null && sql.isTextual()) ? sql.asText() : sqlNodeAndOptions.getSqlNode().toString();
+ tableNames = RequestUtils.getTableNames(CalciteSqlParser.compileToPinotQuery(sqlQuery));
Review Comment:
If `sqlNodeAndOptions` is null (as indicated by the `@Nullable` annotation on the parameter), calling `getSqlNode()` will throw a NullPointerException. Add a null check for `sqlNodeAndOptions` before accessing `getSqlNode()`.
```suggestion
String sqlQuery;
if (sql != null && sql.isTextual()) {
sqlQuery = sql.asText();
} else if (sqlNodeAndOptions != null) {
sqlQuery = sqlNodeAndOptions.getSqlNode().toString();
} else {
// If both the raw SQL and SqlNodeAndOptions are unavailable, skip determining table names.
sqlQuery = null;
}
if (sqlQuery != null) {
tableNames = RequestUtils.getTableNames(CalciteSqlParser.compileToPinotQuery(sqlQuery));
}
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,850 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.SystemTableResponseUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.FilterKind;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableDataProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final String PINOT_ADMIN_CLIENT_CLASS_NAME = "org.apache.pinot.client.admin.PinotAdminClient";
+ private static final String SIZE_CACHE_TTL_MS_PROPERTY = "pinot.systemtable.tables.sizeCacheTtlMs";
+ private static final long DEFAULT_SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+ private static final long SIZE_CACHE_TTL_MS = getNonNegativeLongProperty(SIZE_CACHE_TTL_MS_PROPERTY,
+ DEFAULT_SIZE_CACHE_TTL_MS);
+
+ private static final String CONTROLLER_TIMEOUT_MS_PROPERTY = "pinot.systemtable.tables.controllerTimeoutMs";
+ private static final long DEFAULT_CONTROLLER_TIMEOUT_MS = Duration.ofSeconds(5).toMillis();
+ private static final long CONTROLLER_TIMEOUT_MS = getPositiveLongProperty(CONTROLLER_TIMEOUT_MS_PROPERTY,
+ DEFAULT_CONTROLLER_TIMEOUT_MS);
+
+ private static final String ADMIN_TRANSPORT_REQUEST_TIMEOUT_MS = "pinot.admin.request.timeout.ms";
+ private static final String ADMIN_TRANSPORT_SCHEME = "pinot.admin.scheme";
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final @Nullable Class<?> PINOT_ADMIN_CLIENT_CLASS = loadAdminClientClass();
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+ private final Map<String, CachedSize> _sizeCache = new ConcurrentHashMap<>();
+ private final Map<String, AutoCloseable> _adminClientCache = new ConcurrentHashMap<>();
+ private final AtomicBoolean _loggedSizeFetchFailure = new AtomicBoolean();
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public TableConfig getTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ }
+
+ @Override
+ public void close()
+ throws Exception {
+ for (Map.Entry<String, AutoCloseable> entry : _adminClientCache.entrySet()) {
+ try {
+ entry.getValue().close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close admin client for {}: {}", entry.getKey(), e.toString());
+ }
+ }
+ _adminClientCache.clear();
+ }
+
+ @Override
+ public BrokerResponseNative getBrokerResponse(PinotQuery pinotQuery) {
+ List<String> projectionColumns = getProjectionColumns(pinotQuery);
+ if (_tableCache == null) {
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, List.of(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ int offset = Math.max(0, pinotQuery.getOffset());
+ int limit = pinotQuery.getLimit();
+ boolean hasLimit = limit > 0;
+ org.apache.pinot.common.request.Expression filterExpression = pinotQuery.getFilterExpression();
+ List<String> controllerBaseUrls = getControllerBaseUrls();
+ Function<String, TableSize> sizeFetcher = getSizeFetcher();
+
+ if (filterExpression == null) {
+ int totalRows = sortedTableNames.size();
+ int startIndex = Math.min(offset, totalRows);
+ int endIndex = totalRows;
+ if (limit == 0) {
+ endIndex = startIndex;
+ } else if (hasLimit) {
+ endIndex = Math.min(totalRows, startIndex + limit);
+ }
+ List<GenericRow> rows = new ArrayList<>(Math.max(0, endIndex - startIndex));
+ for (int i = startIndex; i < endIndex; i++) {
+ String tableNameWithType = sortedTableNames.get(i);
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType, sizeFetcher, controllerBaseUrls);
+ rows.add(buildRow(rawTableName, stats));
+ }
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, rows, totalRows);
+ }
+
+ int totalRows = 0;
+ int initialCapacity = hasLimit ? Math.min(sortedTableNames.size(), limit) : sortedTableNames.size();
+ List<GenericRow> rows = new ArrayList<>(initialCapacity);
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType, sizeFetcher, controllerBaseUrls);
+ if (!matchesFilter(filterExpression, stats, rawTableName)) {
+ continue;
+ }
+ totalRows++;
+ if (totalRows <= offset) {
+ continue;
+ }
+ if (limit == 0) {
+ continue;
+ }
+ if (hasLimit && rows.size() >= limit) {
+ continue;
+ }
+ rows.add(buildRow(rawTableName, stats));
+ }
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, rows, totalRows);
+ }
+
+ private GenericRow buildRow(String rawTableName, TableStats stats) {
+ GenericRow row = new GenericRow();
+ row.putValue("tableName", rawTableName);
+ row.putValue("type", stats._type);
+ row.putValue("status", stats._status);
+ row.putValue("segments", stats._segments);
+ row.putValue("totalDocs", stats._totalDocs);
+ row.putValue("reportedSize", stats._reportedSizeInBytes);
+ row.putValue("estimatedSize", stats._estimatedSizeInBytes);
+ row.putValue("storageTier", stats._storageTier);
+ row.putValue("brokerTenant", stats._brokerTenant);
+ row.putValue("serverTenant", stats._serverTenant);
+ row.putValue("replicas", stats._replicas);
+ row.putValue("tableConfig", stats._tableConfig);
+ return row;
+ }
+
+ private TableStats buildStats(String tableNameWithType, TableType tableType,
+ @Nullable Function<String, TableSize> tableSizeFetcher, List<String> controllerBaseUrls) {
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ int segments = 0;
+ long totalDocs = 0;
+ long reportedSize = 0;
+ long estimatedSize = 0;
+ String tierValue = "";
+ String brokerTenant = "";
+ String serverTenant = "";
+ int replicas = 0;
+ if (tableConfig != null && tableConfig.getTenantConfig() != null) {
+ brokerTenant = tableConfig.getTenantConfig().getBroker();
+ serverTenant = tableConfig.getTenantConfig().getServer();
+ }
+ if (tableConfig != null && tableConfig.getValidationConfig() != null) {
+ Integer repl = tableConfig.getValidationConfig().getReplicationNumber();
+ replicas = repl != null ? repl : replicas;
+ }
+ // Use controller API only
+ TableSize sizeFromController = fetchTableSize(tableNameWithType, tableSizeFetcher, controllerBaseUrls);
+ if (sizeFromController != null) {
+ if (sizeFromController._reportedSizeInBytes >= 0) {
+ reportedSize = sizeFromController._reportedSizeInBytes;
+ }
+ if (sizeFromController._estimatedSizeInBytes >= 0) {
+ estimatedSize = sizeFromController._estimatedSizeInBytes;
+ }
+ segments = getSegmentCount(sizeFromController, tableType);
+ totalDocs = getTotalDocs(sizeFromController, tableType);
+ }
+ String status = (tableConfig != null || segments > 0) ? "ONLINE" : "UNKNOWN";
+ String tableConfigJson = "";
+ if (tableConfig != null) {
+ try {
+ tableConfigJson = JsonUtils.objectToString(tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", tableNameWithType, e.toString());
+ tableConfigJson = tableConfig.toString();
+ }
+ }
+ return new TableStats(tableType.name(), status, segments, totalDocs, reportedSize, estimatedSize, tierValue,
+ tableConfigJson, brokerTenant, serverTenant, replicas);
+ }
+
+ private boolean matchesFilter(@Nullable org.apache.pinot.common.request.Expression filterExpression, TableStats stats,
+ String rawTableName) {
+ if (filterExpression == null) {
+ return true;
+ }
+ org.apache.pinot.common.request.Function function = filterExpression.getFunctionCall();
+ if (function == null) {
+ return true;
+ }
+ FilterKind filterKind = toFilterKind(function.getOperator());
+ if (filterKind == null) {
+ return true;
+ }
+ switch (filterKind) {
+ case AND:
+ for (org.apache.pinot.common.request.Expression child : function.getOperands()) {
+ if (!matchesFilter(child, stats, rawTableName)) {
+ return false;
+ }
+ }
+ return true;
+ case OR:
+ for (org.apache.pinot.common.request.Expression child : function.getOperands()) {
+ if (matchesFilter(child, stats, rawTableName)) {
+ return true;
+ }
+ }
+ return false;
+ case NOT:
+ if (function.getOperandsSize() == 0) {
+ return true;
+ }
+ return !matchesFilter(function.getOperands().get(0), stats, rawTableName);
+ default:
+ return matchesLeafFilter(filterKind, function.getOperands(), stats, rawTableName);
+ }
+ }
+
+ private boolean matchesLeafFilter(FilterKind filterKind,
+ List<org.apache.pinot.common.request.Expression> operands, TableStats stats, String rawTableName) {
+ String column = extractIdentifier(operands);
+ if (column == null) {
+ return true;
+ }
+ List<String> values = extractLiteralValues(operands);
+ if (values.isEmpty()) {
+ return true;
+ }
+ switch (column.toLowerCase(Locale.ROOT)) {
+ case "tablename":
+ return matchesString(values, rawTableName, filterKind);
+ case "type":
+ return matchesString(values, stats._type, filterKind);
+ case "status":
+ return matchesString(values, stats._status, filterKind);
+ case "segments":
+ return matchesNumber(values, stats._segments, filterKind);
+ case "reportedsize":
+ return matchesNumber(values, stats._reportedSizeInBytes, filterKind);
+ case "estimatedsize":
+ return matchesNumber(values, stats._estimatedSizeInBytes, filterKind);
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesString(List<String> candidates, String actual, FilterKind filterKind) {
+ switch (filterKind) {
+ case EQUALS:
+ case IN:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case NOT_EQUALS:
+ return candidates.stream().noneMatch(v -> v.equalsIgnoreCase(actual));
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesNumber(List<String> candidates, long actual, FilterKind filterKind) {
+ try {
+ switch (filterKind) {
+ case EQUALS:
+ return candidates.stream().anyMatch(v -> Long.parseLong(v) == actual);
+ case NOT_EQUALS:
+ return candidates.stream().noneMatch(v -> Long.parseLong(v) == actual);
+ case GREATER_THAN:
+ return candidates.stream().anyMatch(v -> actual > Long.parseLong(v));
+ case GREATER_THAN_OR_EQUAL:
+ return candidates.stream().anyMatch(v -> actual >= Long.parseLong(v));
+ case LESS_THAN:
+ return candidates.stream().anyMatch(v -> actual < Long.parseLong(v));
+ case LESS_THAN_OR_EQUAL:
+ return candidates.stream().anyMatch(v -> actual <= Long.parseLong(v));
+ case IN:
+ for (String candidate : candidates) {
+ if (actual == Long.parseLong(candidate)) {
+ return true;
+ }
+ }
+ return false;
+ case RANGE:
+ case BETWEEN:
+ if (candidates.size() >= 2) {
+ long lower = Long.parseLong(candidates.get(0));
+ long upper = Long.parseLong(candidates.get(1));
+ return actual >= lower && actual <= upper;
+ }
+ return true;
+ default:
+ return true;
+ }
+ } catch (NumberFormatException e) {
+ LOGGER.debug("Failed to parse numeric filter value {}: {}", candidates, e.toString());
+ return true;
+ }
+ }
+
+ private @Nullable String extractIdentifier(List<org.apache.pinot.common.request.Expression> operands) {
+ for (org.apache.pinot.common.request.Expression operand : operands) {
+ org.apache.pinot.common.request.Identifier identifier = operand.getIdentifier();
+ if (identifier != null) {
+ return identifier.getName();
+ }
+ }
+ return null;
+ }
+
+ private List<String> extractLiteralValues(List<org.apache.pinot.common.request.Expression> operands) {
+ List<String> values = new ArrayList<>();
+ for (org.apache.pinot.common.request.Expression operand : operands) {
+ org.apache.pinot.common.request.Literal literal = operand.getLiteral();
+ if (literal != null) {
+ if (literal.getSetField() == org.apache.pinot.common.request.Literal._Fields.NULL_VALUE) {
+ values.add("null");
+ continue;
+ }
+ try {
+ values.add(RequestUtils.getLiteralString(literal));
+ } catch (Exception e) {
+ values.add(String.valueOf(RequestUtils.getLiteralValue(literal)));
+ }
+ }
+ }
+ return values;
+ }
+
+ private @Nullable FilterKind toFilterKind(String operator) {
+ String normalized = operator.toUpperCase(Locale.ROOT);
+ switch (normalized) {
+ case "EQ":
+ return FilterKind.EQUALS;
+ case "NEQ":
+ return FilterKind.NOT_EQUALS;
+ case "GT":
+ return FilterKind.GREATER_THAN;
+ case "GTE":
+ return FilterKind.GREATER_THAN_OR_EQUAL;
+ case "LT":
+ return FilterKind.LESS_THAN;
+ case "LTE":
+ return FilterKind.LESS_THAN_OR_EQUAL;
+ default:
+ try {
+ return FilterKind.valueOf(normalized);
+ } catch (Exception e) {
+ return null;
+ }
+ }
+ }
+
+ private @Nullable TableSize fetchTableSize(String tableNameWithType,
+ @Nullable Function<String, TableSize> fetcher, List<String> controllerBaseUrls) {
+ boolean cacheEnabled = SIZE_CACHE_TTL_MS > 0;
+ TableSize cached = cacheEnabled ? getCachedSize(tableNameWithType) : null;
+ if (cached != null) {
+ return cached;
+ }
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ if (cacheEnabled) {
+ cacheSize(tableNameWithType, fetched);
+ }
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(controllerBaseUrls, rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(controllerBaseUrls, tableNameWithType);
+ if (size == null) {
+ logSizeFetchFailure("{}: failed to fetch size for {} from controllers {} (tried '{}' and '{}')", TABLE_NAME,
+ tableNameWithType, controllerBaseUrls, rawTableName, tableNameWithType);
+ }
Review Comment:
The log message format references both `rawTableName` and `tableNameWithType` but only tries `tableNameWithType` in this branch (line 487). The message says 'tried ... and ...' but only one attempt was made. This could confuse someone debugging. Consider revising the message to accurately reflect the attempts made in this specific code path.
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,850 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.SystemTableResponseUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.FilterKind;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableDataProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final String PINOT_ADMIN_CLIENT_CLASS_NAME = "org.apache.pinot.client.admin.PinotAdminClient";
+ private static final String SIZE_CACHE_TTL_MS_PROPERTY = "pinot.systemtable.tables.sizeCacheTtlMs";
+ private static final long DEFAULT_SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+ private static final long SIZE_CACHE_TTL_MS = getNonNegativeLongProperty(SIZE_CACHE_TTL_MS_PROPERTY,
+ DEFAULT_SIZE_CACHE_TTL_MS);
+
+ private static final String CONTROLLER_TIMEOUT_MS_PROPERTY = "pinot.systemtable.tables.controllerTimeoutMs";
+ private static final long DEFAULT_CONTROLLER_TIMEOUT_MS = Duration.ofSeconds(5).toMillis();
+ private static final long CONTROLLER_TIMEOUT_MS = getPositiveLongProperty(CONTROLLER_TIMEOUT_MS_PROPERTY,
+ DEFAULT_CONTROLLER_TIMEOUT_MS);
+
+ private static final String ADMIN_TRANSPORT_REQUEST_TIMEOUT_MS = "pinot.admin.request.timeout.ms";
+ private static final String ADMIN_TRANSPORT_SCHEME = "pinot.admin.scheme";
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final @Nullable Class<?> PINOT_ADMIN_CLIENT_CLASS = loadAdminClientClass();
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+ private final Map<String, CachedSize> _sizeCache = new ConcurrentHashMap<>();
+ private final Map<String, AutoCloseable> _adminClientCache = new ConcurrentHashMap<>();
+ private final AtomicBoolean _loggedSizeFetchFailure = new AtomicBoolean();
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public TableConfig getTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ }
+
+ @Override
+ public void close()
+ throws Exception {
+ for (Map.Entry<String, AutoCloseable> entry : _adminClientCache.entrySet()) {
+ try {
+ entry.getValue().close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close admin client for {}: {}", entry.getKey(), e.toString());
+ }
+ }
+ _adminClientCache.clear();
+ }
+
+ @Override
+ public BrokerResponseNative getBrokerResponse(PinotQuery pinotQuery) {
+ List<String> projectionColumns = getProjectionColumns(pinotQuery);
+ if (_tableCache == null) {
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, List.of(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ int offset = Math.max(0, pinotQuery.getOffset());
+ int limit = pinotQuery.getLimit();
+ boolean hasLimit = limit > 0;
+ org.apache.pinot.common.request.Expression filterExpression = pinotQuery.getFilterExpression();
+ List<String> controllerBaseUrls = getControllerBaseUrls();
+ Function<String, TableSize> sizeFetcher = getSizeFetcher();
+
+ if (filterExpression == null) {
+ int totalRows = sortedTableNames.size();
+ int startIndex = Math.min(offset, totalRows);
+ int endIndex = totalRows;
+ if (limit == 0) {
+ endIndex = startIndex;
+ } else if (hasLimit) {
+ endIndex = Math.min(totalRows, startIndex + limit);
+ }
+ List<GenericRow> rows = new ArrayList<>(Math.max(0, endIndex - startIndex));
+ for (int i = startIndex; i < endIndex; i++) {
+ String tableNameWithType = sortedTableNames.get(i);
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType, sizeFetcher, controllerBaseUrls);
+ rows.add(buildRow(rawTableName, stats));
+ }
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, rows, totalRows);
+ }
+
+ int totalRows = 0;
+ int initialCapacity = hasLimit ? Math.min(sortedTableNames.size(), limit) : sortedTableNames.size();
+ List<GenericRow> rows = new ArrayList<>(initialCapacity);
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType, sizeFetcher, controllerBaseUrls);
+ if (!matchesFilter(filterExpression, stats, rawTableName)) {
+ continue;
+ }
+ totalRows++;
+ if (totalRows <= offset) {
+ continue;
+ }
+ if (limit == 0) {
+ continue;
+ }
+ if (hasLimit && rows.size() >= limit) {
+ continue;
+ }
+ rows.add(buildRow(rawTableName, stats));
+ }
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, rows, totalRows);
+ }
+
+ private GenericRow buildRow(String rawTableName, TableStats stats) {
+ GenericRow row = new GenericRow();
+ row.putValue("tableName", rawTableName);
+ row.putValue("type", stats._type);
+ row.putValue("status", stats._status);
+ row.putValue("segments", stats._segments);
+ row.putValue("totalDocs", stats._totalDocs);
+ row.putValue("reportedSize", stats._reportedSizeInBytes);
+ row.putValue("estimatedSize", stats._estimatedSizeInBytes);
+ row.putValue("storageTier", stats._storageTier);
+ row.putValue("brokerTenant", stats._brokerTenant);
+ row.putValue("serverTenant", stats._serverTenant);
+ row.putValue("replicas", stats._replicas);
+ row.putValue("tableConfig", stats._tableConfig);
+ return row;
+ }
+
+ private TableStats buildStats(String tableNameWithType, TableType tableType,
+ @Nullable Function<String, TableSize> tableSizeFetcher, List<String> controllerBaseUrls) {
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ int segments = 0;
+ long totalDocs = 0;
+ long reportedSize = 0;
+ long estimatedSize = 0;
+ String tierValue = "";
+ String brokerTenant = "";
+ String serverTenant = "";
+ int replicas = 0;
+ if (tableConfig != null && tableConfig.getTenantConfig() != null) {
+ brokerTenant = tableConfig.getTenantConfig().getBroker();
+ serverTenant = tableConfig.getTenantConfig().getServer();
+ }
+ if (tableConfig != null && tableConfig.getValidationConfig() != null) {
+ Integer repl = tableConfig.getValidationConfig().getReplicationNumber();
+ replicas = repl != null ? repl : replicas;
+ }
+ // Use controller API only
+ TableSize sizeFromController = fetchTableSize(tableNameWithType, tableSizeFetcher, controllerBaseUrls);
+ if (sizeFromController != null) {
+ if (sizeFromController._reportedSizeInBytes >= 0) {
+ reportedSize = sizeFromController._reportedSizeInBytes;
+ }
+ if (sizeFromController._estimatedSizeInBytes >= 0) {
+ estimatedSize = sizeFromController._estimatedSizeInBytes;
+ }
+ segments = getSegmentCount(sizeFromController, tableType);
+ totalDocs = getTotalDocs(sizeFromController, tableType);
+ }
+ String status = (tableConfig != null || segments > 0) ? "ONLINE" : "UNKNOWN";
+ String tableConfigJson = "";
+ if (tableConfig != null) {
+ try {
+ tableConfigJson = JsonUtils.objectToString(tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", tableNameWithType, e.toString());
+ tableConfigJson = tableConfig.toString();
+ }
+ }
+ return new TableStats(tableType.name(), status, segments, totalDocs, reportedSize, estimatedSize, tierValue,
+ tableConfigJson, brokerTenant, serverTenant, replicas);
+ }
+
+ private boolean matchesFilter(@Nullable org.apache.pinot.common.request.Expression filterExpression, TableStats stats,
+ String rawTableName) {
+ if (filterExpression == null) {
+ return true;
+ }
+ org.apache.pinot.common.request.Function function = filterExpression.getFunctionCall();
+ if (function == null) {
+ return true;
+ }
+ FilterKind filterKind = toFilterKind(function.getOperator());
+ if (filterKind == null) {
+ return true;
+ }
+ switch (filterKind) {
+ case AND:
+ for (org.apache.pinot.common.request.Expression child : function.getOperands()) {
+ if (!matchesFilter(child, stats, rawTableName)) {
+ return false;
+ }
+ }
+ return true;
+ case OR:
+ for (org.apache.pinot.common.request.Expression child : function.getOperands()) {
+ if (matchesFilter(child, stats, rawTableName)) {
+ return true;
+ }
+ }
+ return false;
+ case NOT:
+ if (function.getOperandsSize() == 0) {
+ return true;
+ }
+ return !matchesFilter(function.getOperands().get(0), stats, rawTableName);
+ default:
+ return matchesLeafFilter(filterKind, function.getOperands(), stats, rawTableName);
+ }
+ }
+
+ private boolean matchesLeafFilter(FilterKind filterKind,
+ List<org.apache.pinot.common.request.Expression> operands, TableStats stats, String rawTableName) {
+ String column = extractIdentifier(operands);
+ if (column == null) {
+ return true;
+ }
+ List<String> values = extractLiteralValues(operands);
+ if (values.isEmpty()) {
+ return true;
+ }
+ switch (column.toLowerCase(Locale.ROOT)) {
+ case "tablename":
+ return matchesString(values, rawTableName, filterKind);
+ case "type":
+ return matchesString(values, stats._type, filterKind);
+ case "status":
+ return matchesString(values, stats._status, filterKind);
+ case "segments":
+ return matchesNumber(values, stats._segments, filterKind);
+ case "reportedsize":
+ return matchesNumber(values, stats._reportedSizeInBytes, filterKind);
+ case "estimatedsize":
+ return matchesNumber(values, stats._estimatedSizeInBytes, filterKind);
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesString(List<String> candidates, String actual, FilterKind filterKind) {
+ switch (filterKind) {
+ case EQUALS:
+ case IN:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case NOT_EQUALS:
+ return candidates.stream().noneMatch(v -> v.equalsIgnoreCase(actual));
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesNumber(List<String> candidates, long actual, FilterKind filterKind) {
+ try {
+ switch (filterKind) {
+ case EQUALS:
+ return candidates.stream().anyMatch(v -> Long.parseLong(v) == actual);
+ case NOT_EQUALS:
+ return candidates.stream().noneMatch(v -> Long.parseLong(v) == actual);
+ case GREATER_THAN:
+ return candidates.stream().anyMatch(v -> actual > Long.parseLong(v));
+ case GREATER_THAN_OR_EQUAL:
+ return candidates.stream().anyMatch(v -> actual >= Long.parseLong(v));
+ case LESS_THAN:
+ return candidates.stream().anyMatch(v -> actual < Long.parseLong(v));
+ case LESS_THAN_OR_EQUAL:
+ return candidates.stream().anyMatch(v -> actual <= Long.parseLong(v));
+ case IN:
+ for (String candidate : candidates) {
+ if (actual == Long.parseLong(candidate)) {
+ return true;
+ }
+ }
+ return false;
+ case RANGE:
+ case BETWEEN:
+ if (candidates.size() >= 2) {
+ long lower = Long.parseLong(candidates.get(0));
+ long upper = Long.parseLong(candidates.get(1));
+ return actual >= lower && actual <= upper;
+ }
+ return true;
+ default:
+ return true;
+ }
+ } catch (NumberFormatException e) {
+ LOGGER.debug("Failed to parse numeric filter value {}: {}", candidates, e.toString());
+ return true;
+ }
+ }
+
+ private @Nullable String extractIdentifier(List<org.apache.pinot.common.request.Expression> operands) {
+ for (org.apache.pinot.common.request.Expression operand : operands) {
+ org.apache.pinot.common.request.Identifier identifier = operand.getIdentifier();
+ if (identifier != null) {
+ return identifier.getName();
+ }
+ }
+ return null;
+ }
+
+ private List<String> extractLiteralValues(List<org.apache.pinot.common.request.Expression> operands) {
+ List<String> values = new ArrayList<>();
+ for (org.apache.pinot.common.request.Expression operand : operands) {
+ org.apache.pinot.common.request.Literal literal = operand.getLiteral();
+ if (literal != null) {
+ if (literal.getSetField() == org.apache.pinot.common.request.Literal._Fields.NULL_VALUE) {
+ values.add("null");
+ continue;
+ }
+ try {
+ values.add(RequestUtils.getLiteralString(literal));
+ } catch (Exception e) {
+ values.add(String.valueOf(RequestUtils.getLiteralValue(literal)));
+ }
+ }
+ }
+ return values;
+ }
+
+ private @Nullable FilterKind toFilterKind(String operator) {
+ String normalized = operator.toUpperCase(Locale.ROOT);
+ switch (normalized) {
+ case "EQ":
+ return FilterKind.EQUALS;
+ case "NEQ":
+ return FilterKind.NOT_EQUALS;
+ case "GT":
+ return FilterKind.GREATER_THAN;
+ case "GTE":
+ return FilterKind.GREATER_THAN_OR_EQUAL;
+ case "LT":
+ return FilterKind.LESS_THAN;
+ case "LTE":
+ return FilterKind.LESS_THAN_OR_EQUAL;
+ default:
+ try {
+ return FilterKind.valueOf(normalized);
+ } catch (Exception e) {
+ return null;
+ }
+ }
+ }
+
+ private @Nullable TableSize fetchTableSize(String tableNameWithType,
+ @Nullable Function<String, TableSize> fetcher, List<String> controllerBaseUrls) {
+ boolean cacheEnabled = SIZE_CACHE_TTL_MS > 0;
+ TableSize cached = cacheEnabled ? getCachedSize(tableNameWithType) : null;
+ if (cached != null) {
+ return cached;
+ }
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ if (cacheEnabled) {
+ cacheSize(tableNameWithType, fetched);
+ }
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(controllerBaseUrls, rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(controllerBaseUrls, tableNameWithType);
+ if (size == null) {
+ logSizeFetchFailure("{}: failed to fetch size for {} from controllers {} (tried '{}' and '{}')", TABLE_NAME,
+ tableNameWithType, controllerBaseUrls, rawTableName, tableNameWithType);
+ }
+ }
+ if (size != null && cacheEnabled) {
+ cacheSize(tableNameWithType, size);
+ }
+ return size;
+ }
+
+ private @Nullable TableSize fetchTableSizeForName(List<String> controllerBaseUrls, String tableName) {
+ Class<?> clientClass = PINOT_ADMIN_CLIENT_CLASS;
+ if (clientClass == null) {
+ return null;
+ }
+ for (String baseUrl : controllerBaseUrls) {
+ try {
+ AutoCloseable adminClient = getOrCreateAdminClient(baseUrl);
+ if (adminClient == null) {
+ continue;
+ }
+
+ Object sizeNode;
+ try {
+ sizeNode = clientClass.getMethod("getTableSize", String.class, boolean.class, boolean.class)
+ .invoke(adminClient, tableName, true, false);
+ } catch (NoSuchMethodException e) {
+ Object tableClient = clientClass.getMethod("getTableClient").invoke(adminClient);
+ sizeNode = tableClient.getClass().getMethod("getTableSize", String.class, boolean.class, boolean.class)
+ .invoke(tableClient, tableName, true, false);
+ }
+
+ if (sizeNode == null) {
+ continue;
+ }
+
+ TableSize parsed = JsonUtils.stringToObject(sizeNode.toString(), TableSize.class);
+ LOGGER.debug("{}: controller size response for {} via {} -> segments offline={}, realtime={}, "
+ + "reportedSize={}, estimatedSize={}", TABLE_NAME, tableName, baseUrl,
+ parsed._offlineSegments != null && parsed._offlineSegments._segments != null
+ ? parsed._offlineSegments._segments.size() : 0,
+ parsed._realtimeSegments != null && parsed._realtimeSegments._segments != null
+ ? parsed._realtimeSegments._segments.size() : 0,
+ parsed._reportedSizeInBytes, parsed._estimatedSizeInBytes);
+ return parsed;
+ } catch (Exception e) {
+ logSizeFetchFailure("{}: error fetching table size for {} via {} using admin client", TABLE_NAME, tableName,
+ baseUrl, e);
+ }
+ }
+ return null;
+ }
+
+ private List<String> getControllerBaseUrls() {
+ Set<String> urls = new LinkedHashSet<>();
+ if (_helixAdmin != null) {
+ for (String controller : discoverControllersFromHelix()) {
+ String normalized = normalizeControllerUrl(controller);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ }
+ for (String url : _staticControllerUrls) {
+ String normalized = normalizeControllerUrl(url);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ return new ArrayList<>(urls);
+ }
+
+ private int getSegmentCount(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.size();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.size();
+ }
+ return 0;
+ }
+
+ private long getTotalDocs(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ return 0;
+ }
+
+ private @Nullable Function<String, TableSize> getSizeFetcher() {
+ if (_tableSizeFetcherOverride != null) {
+ return _tableSizeFetcherOverride;
+ }
+ return null;
+ }
+
+ private List<String> discoverControllersFromHelix() {
+ List<String> urls = new ArrayList<>();
+ try {
+ if (_clusterName == null) {
+ LOGGER.warn("Cannot discover controllers without cluster name");
+ return List.of();
+ }
+ for (String controllerId : _helixAdmin.getInstancesInCluster(_clusterName)) {
+ if (!InstanceTypeUtils.isController(controllerId)) {
+ continue;
+ }
+ int firstUnderscore = controllerId.indexOf('_');
+ int lastUnderscore = controllerId.lastIndexOf('_');
+ if (firstUnderscore > 0 && lastUnderscore > firstUnderscore && lastUnderscore < controllerId.length() - 1) {
+ String host = controllerId.substring(firstUnderscore + 1, lastUnderscore);
+ String port = controllerId.substring(lastUnderscore + 1);
Review Comment:
The controller ID parsing logic duplicates the expected format 'Controller_<host>_<port>' in both code (substring operations) and warning messages (line 617). Consider extracting this parsing logic into a separate utility method with clear documentation of the expected format. This would improve testability and make the format expectation more explicit.
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BrokerRequestHandlerDelegate.java:
##########
@@ -106,7 +115,27 @@ public BrokerResponse handleRequest(JsonNode request, @Nullable SqlNodeAndOption
}
}
+ Set<String> tableNames = null;
+ if (_systemTableBrokerRequestHandler != null) {
+ try {
+ // NOTE: compileToPinotQuery(SqlNodeAndOptions) mutates the SqlNode (e.g. SqlOrderBy -> SqlSelect).
+ // Re-parse from raw SQL so the SqlNodeAndOptions passed to the handler is unchanged (important for
+ // Multi-Stage Engine (MSE)).
+ JsonNode sql = request.get(Request.SQL);
+ String sqlQuery =
+ (sql != null && sql.isTextual()) ? sql.asText() : sqlNodeAndOptions.getSqlNode().toString();
+ tableNames = RequestUtils.getTableNames(CalciteSqlParser.compileToPinotQuery(sqlQuery));
+ } catch (Exception e) {
+ // Ignore compilation exceptions here; the selected request handler will surface them appropriately.
+ LOGGER.debug("Failed to compile query while determining request handler", e);
+ }
Review Comment:
The null check for `_systemTableBrokerRequestHandler` happens here but at lines 134-137 it's checked again. However, the handler is used at line 74 and 86 in `start()` and `shutDown()` without null checks. If the handler can be null, these methods will throw NullPointerException. Either make the handler non-nullable in the constructor or add null checks in the lifecycle methods.
```suggestion
try {
// NOTE: compileToPinotQuery(SqlNodeAndOptions) mutates the SqlNode (e.g. SqlOrderBy -> SqlSelect).
// Re-parse from raw SQL so the SqlNodeAndOptions passed to the handler is unchanged (important for
// Multi-Stage Engine (MSE)).
JsonNode sql = request.get(Request.SQL);
String sqlQuery =
(sql != null && sql.isTextual()) ? sql.asText() : sqlNodeAndOptions.getSqlNode().toString();
tableNames = RequestUtils.getTableNames(CalciteSqlParser.compileToPinotQuery(sqlQuery));
} catch (Exception e) {
// Ignore compilation exceptions here; the selected request handler will surface them appropriately.
LOGGER.debug("Failed to compile query while determining request handler", e);
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/SystemTableRegistry.java:
##########
@@ -0,0 +1,169 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.IdentityHashMap;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.PinotReflectionUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Registry to hold and lifecycle-manage system table data providers.
+ */
+public final class SystemTableRegistry {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableRegistry.class);
+
+ // Providers are registered once at broker startup.
+ private static final Map<String, SystemTableDataProvider> PROVIDERS = new HashMap<>();
+
+ private SystemTableRegistry() {
+ throw new UnsupportedOperationException("This is a utility class and cannot be instantiated");
+ }
+
+ public static void register(SystemTableDataProvider provider) {
+ synchronized (PROVIDERS) {
+ PROVIDERS.put(normalize(provider.getTableName()), provider);
+ }
+ }
+
+ @Nullable
+ public static SystemTableDataProvider get(String tableName) {
+ synchronized (PROVIDERS) {
+ return PROVIDERS.get(normalize(tableName));
+ }
+ }
+
+ public static boolean isRegistered(String tableName) {
+ synchronized (PROVIDERS) {
+ return PROVIDERS.containsKey(normalize(tableName));
+ }
+ }
+
+ public static Collection<SystemTableDataProvider> getProviders() {
+ synchronized (PROVIDERS) {
+ return Collections.unmodifiableCollection(new java.util.ArrayList<>(PROVIDERS.values()));
+ }
+ }
+
+ /**
+ * Discover and register providers marked with {@link SystemTable} using the available dependencies.
+ * Follows the ScalarFunction pattern: any class annotated with @SystemTable under a "*.systemtable.*" package
+ * will be discovered via reflection and registered.
+ */
+ public static void registerAnnotatedProviders(TableCache tableCache, HelixAdmin helixAdmin, String clusterName) {
+ Set<Class<?>> classes =
+ PinotReflectionUtils.getClassesThroughReflection(".*\\.systemtable\\..*", SystemTable.class);
+ for (Class<?> clazz : classes) {
+ if (!SystemTableDataProvider.class.isAssignableFrom(clazz)) {
+ continue;
+ }
+ SystemTableDataProvider provider = instantiateProvider(
+ clazz.asSubclass(SystemTableDataProvider.class), tableCache, helixAdmin, clusterName);
+ if (provider == null) {
+ continue;
+ }
+ if (isRegistered(provider.getTableName())) {
+ continue;
+ }
+ LOGGER.info("Registering system table provider: {}", provider.getTableName());
+ register(provider);
+ }
+ }
+
+ public static void close()
+ throws Exception {
+ Exception firstException = null;
+ // Snapshot providers to avoid concurrent modifications and to close each provider at most once.
+ Map<SystemTableDataProvider, Boolean> providersToClose = new IdentityHashMap<>();
+ synchronized (PROVIDERS) {
+ for (SystemTableDataProvider provider : PROVIDERS.values()) {
+ providersToClose.put(provider, Boolean.TRUE);
+ }
+ }
+ try {
+ for (SystemTableDataProvider provider : providersToClose.keySet()) {
+ try {
+ provider.close();
+ } catch (Exception e) {
+ if (firstException == null) {
+ firstException = e;
+ } else {
+ firstException.addSuppressed(e);
+ }
+ }
+ }
+ } finally {
+ synchronized (PROVIDERS) {
+ PROVIDERS.clear();
+ }
+ }
+ if (firstException != null) {
+ throw firstException;
+ }
+ }
+
+ /**
+ * Initialize and register all annotated system table providers.
+ */
+ public static void init(TableCache tableCache, HelixAdmin helixAdmin, String clusterName) {
+ registerAnnotatedProviders(tableCache, helixAdmin, clusterName);
+ }
+
+ private static String normalize(String tableName) {
+ return tableName.toLowerCase(Locale.ROOT);
+ }
+
+ private static @Nullable SystemTableDataProvider instantiateProvider(Class<? extends SystemTableDataProvider> clazz,
+ TableCache tableCache, HelixAdmin helixAdmin, String clusterName) {
+ try {
+ // Prefer the most specific constructor available.
+ try {
+ return clazz.getDeclaredConstructor(TableCache.class, HelixAdmin.class, String.class)
+ .newInstance(tableCache, helixAdmin, clusterName);
+ } catch (NoSuchMethodException ignored) {
+ // fall through
+ }
+ try {
+ return clazz.getDeclaredConstructor(TableCache.class, HelixAdmin.class).newInstance(tableCache, helixAdmin);
+ } catch (NoSuchMethodException ignored) {
+ // fall through
+ }
+ try {
+ return clazz.getDeclaredConstructor(TableCache.class).newInstance(tableCache);
+ } catch (NoSuchMethodException ignored) {
+ // fall through
+ }
+ return clazz.getDeclaredConstructor().newInstance();
Review Comment:
Using `getDeclaredConstructor()` and `newInstance()` can bypass access controls on private constructors. This could be a security concern if system table providers are expected to control instantiation. Consider documenting that system table provider constructors should be public or use `getConstructor()` instead of `getDeclaredConstructor()` to enforce public constructors.
```suggestion
return clazz.getConstructor(TableCache.class, HelixAdmin.class, String.class)
.newInstance(tableCache, helixAdmin, clusterName);
} catch (NoSuchMethodException ignored) {
// fall through
}
try {
return clazz.getConstructor(TableCache.class, HelixAdmin.class).newInstance(tableCache, helixAdmin);
} catch (NoSuchMethodException ignored) {
// fall through
}
try {
return clazz.getConstructor(TableCache.class).newInstance(tableCache);
} catch (NoSuchMethodException ignored) {
// fall through
}
return clazz.getConstructor().newInstance();
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,850 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.SystemTableResponseUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.FilterKind;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableDataProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final String PINOT_ADMIN_CLIENT_CLASS_NAME = "org.apache.pinot.client.admin.PinotAdminClient";
+ private static final String SIZE_CACHE_TTL_MS_PROPERTY = "pinot.systemtable.tables.sizeCacheTtlMs";
+ private static final long DEFAULT_SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+ private static final long SIZE_CACHE_TTL_MS = getNonNegativeLongProperty(SIZE_CACHE_TTL_MS_PROPERTY,
+ DEFAULT_SIZE_CACHE_TTL_MS);
+
+ private static final String CONTROLLER_TIMEOUT_MS_PROPERTY = "pinot.systemtable.tables.controllerTimeoutMs";
+ private static final long DEFAULT_CONTROLLER_TIMEOUT_MS = Duration.ofSeconds(5).toMillis();
+ private static final long CONTROLLER_TIMEOUT_MS = getPositiveLongProperty(CONTROLLER_TIMEOUT_MS_PROPERTY,
+ DEFAULT_CONTROLLER_TIMEOUT_MS);
+
+ private static final String ADMIN_TRANSPORT_REQUEST_TIMEOUT_MS = "pinot.admin.request.timeout.ms";
+ private static final String ADMIN_TRANSPORT_SCHEME = "pinot.admin.scheme";
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final @Nullable Class<?> PINOT_ADMIN_CLIENT_CLASS = loadAdminClientClass();
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+ private final Map<String, CachedSize> _sizeCache = new ConcurrentHashMap<>();
+ private final Map<String, AutoCloseable> _adminClientCache = new ConcurrentHashMap<>();
+ private final AtomicBoolean _loggedSizeFetchFailure = new AtomicBoolean();
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public TableConfig getTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ }
+
+ @Override
+ public void close()
+ throws Exception {
+ for (Map.Entry<String, AutoCloseable> entry : _adminClientCache.entrySet()) {
+ try {
+ entry.getValue().close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close admin client for {}: {}", entry.getKey(), e.toString());
+ }
+ }
+ _adminClientCache.clear();
+ }
+
+ @Override
+ public BrokerResponseNative getBrokerResponse(PinotQuery pinotQuery) {
+ List<String> projectionColumns = getProjectionColumns(pinotQuery);
+ if (_tableCache == null) {
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, List.of(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ int offset = Math.max(0, pinotQuery.getOffset());
+ int limit = pinotQuery.getLimit();
+ boolean hasLimit = limit > 0;
+ org.apache.pinot.common.request.Expression filterExpression = pinotQuery.getFilterExpression();
+ List<String> controllerBaseUrls = getControllerBaseUrls();
+ Function<String, TableSize> sizeFetcher = getSizeFetcher();
+
+ if (filterExpression == null) {
+ int totalRows = sortedTableNames.size();
+ int startIndex = Math.min(offset, totalRows);
+ int endIndex = totalRows;
+ if (limit == 0) {
+ endIndex = startIndex;
+ } else if (hasLimit) {
+ endIndex = Math.min(totalRows, startIndex + limit);
+ }
+ List<GenericRow> rows = new ArrayList<>(Math.max(0, endIndex - startIndex));
+ for (int i = startIndex; i < endIndex; i++) {
+ String tableNameWithType = sortedTableNames.get(i);
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType, sizeFetcher, controllerBaseUrls);
+ rows.add(buildRow(rawTableName, stats));
+ }
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, rows, totalRows);
+ }
+
+ int totalRows = 0;
+ int initialCapacity = hasLimit ? Math.min(sortedTableNames.size(), limit) : sortedTableNames.size();
+ List<GenericRow> rows = new ArrayList<>(initialCapacity);
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType, sizeFetcher, controllerBaseUrls);
+ if (!matchesFilter(filterExpression, stats, rawTableName)) {
+ continue;
+ }
+ totalRows++;
+ if (totalRows <= offset) {
+ continue;
+ }
+ if (limit == 0) {
+ continue;
+ }
+ if (hasLimit && rows.size() >= limit) {
+ continue;
+ }
+ rows.add(buildRow(rawTableName, stats));
+ }
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, rows, totalRows);
+ }
+
+ private GenericRow buildRow(String rawTableName, TableStats stats) {
+ GenericRow row = new GenericRow();
+ row.putValue("tableName", rawTableName);
+ row.putValue("type", stats._type);
+ row.putValue("status", stats._status);
+ row.putValue("segments", stats._segments);
+ row.putValue("totalDocs", stats._totalDocs);
+ row.putValue("reportedSize", stats._reportedSizeInBytes);
+ row.putValue("estimatedSize", stats._estimatedSizeInBytes);
+ row.putValue("storageTier", stats._storageTier);
+ row.putValue("brokerTenant", stats._brokerTenant);
+ row.putValue("serverTenant", stats._serverTenant);
+ row.putValue("replicas", stats._replicas);
+ row.putValue("tableConfig", stats._tableConfig);
+ return row;
+ }
+
+ private TableStats buildStats(String tableNameWithType, TableType tableType,
+ @Nullable Function<String, TableSize> tableSizeFetcher, List<String> controllerBaseUrls) {
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ int segments = 0;
+ long totalDocs = 0;
+ long reportedSize = 0;
+ long estimatedSize = 0;
+ String tierValue = "";
+ String brokerTenant = "";
+ String serverTenant = "";
+ int replicas = 0;
+ if (tableConfig != null && tableConfig.getTenantConfig() != null) {
+ brokerTenant = tableConfig.getTenantConfig().getBroker();
+ serverTenant = tableConfig.getTenantConfig().getServer();
+ }
+ if (tableConfig != null && tableConfig.getValidationConfig() != null) {
+ Integer repl = tableConfig.getValidationConfig().getReplicationNumber();
+ replicas = repl != null ? repl : replicas;
+ }
+ // Use controller API only
+ TableSize sizeFromController = fetchTableSize(tableNameWithType, tableSizeFetcher, controllerBaseUrls);
+ if (sizeFromController != null) {
+ if (sizeFromController._reportedSizeInBytes >= 0) {
+ reportedSize = sizeFromController._reportedSizeInBytes;
+ }
+ if (sizeFromController._estimatedSizeInBytes >= 0) {
+ estimatedSize = sizeFromController._estimatedSizeInBytes;
+ }
+ segments = getSegmentCount(sizeFromController, tableType);
+ totalDocs = getTotalDocs(sizeFromController, tableType);
+ }
+ String status = (tableConfig != null || segments > 0) ? "ONLINE" : "UNKNOWN";
+ String tableConfigJson = "";
+ if (tableConfig != null) {
+ try {
+ tableConfigJson = JsonUtils.objectToString(tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", tableNameWithType, e.toString());
+ tableConfigJson = tableConfig.toString();
+ }
+ }
+ return new TableStats(tableType.name(), status, segments, totalDocs, reportedSize, estimatedSize, tierValue,
+ tableConfigJson, brokerTenant, serverTenant, replicas);
+ }
+
+ private boolean matchesFilter(@Nullable org.apache.pinot.common.request.Expression filterExpression, TableStats stats,
+ String rawTableName) {
+ if (filterExpression == null) {
+ return true;
+ }
+ org.apache.pinot.common.request.Function function = filterExpression.getFunctionCall();
+ if (function == null) {
+ return true;
+ }
+ FilterKind filterKind = toFilterKind(function.getOperator());
+ if (filterKind == null) {
+ return true;
+ }
+ switch (filterKind) {
+ case AND:
+ for (org.apache.pinot.common.request.Expression child : function.getOperands()) {
+ if (!matchesFilter(child, stats, rawTableName)) {
+ return false;
+ }
+ }
+ return true;
+ case OR:
+ for (org.apache.pinot.common.request.Expression child : function.getOperands()) {
+ if (matchesFilter(child, stats, rawTableName)) {
+ return true;
+ }
+ }
+ return false;
+ case NOT:
+ if (function.getOperandsSize() == 0) {
+ return true;
+ }
+ return !matchesFilter(function.getOperands().get(0), stats, rawTableName);
+ default:
+ return matchesLeafFilter(filterKind, function.getOperands(), stats, rawTableName);
+ }
+ }
+
+ private boolean matchesLeafFilter(FilterKind filterKind,
+ List<org.apache.pinot.common.request.Expression> operands, TableStats stats, String rawTableName) {
+ String column = extractIdentifier(operands);
+ if (column == null) {
+ return true;
+ }
+ List<String> values = extractLiteralValues(operands);
+ if (values.isEmpty()) {
+ return true;
+ }
+ switch (column.toLowerCase(Locale.ROOT)) {
+ case "tablename":
+ return matchesString(values, rawTableName, filterKind);
+ case "type":
+ return matchesString(values, stats._type, filterKind);
+ case "status":
+ return matchesString(values, stats._status, filterKind);
+ case "segments":
+ return matchesNumber(values, stats._segments, filterKind);
+ case "reportedsize":
+ return matchesNumber(values, stats._reportedSizeInBytes, filterKind);
+ case "estimatedsize":
+ return matchesNumber(values, stats._estimatedSizeInBytes, filterKind);
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesString(List<String> candidates, String actual, FilterKind filterKind) {
+ switch (filterKind) {
+ case EQUALS:
+ case IN:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case NOT_EQUALS:
+ return candidates.stream().noneMatch(v -> v.equalsIgnoreCase(actual));
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesNumber(List<String> candidates, long actual, FilterKind filterKind) {
+ try {
+ switch (filterKind) {
+ case EQUALS:
+ return candidates.stream().anyMatch(v -> Long.parseLong(v) == actual);
+ case NOT_EQUALS:
+ return candidates.stream().noneMatch(v -> Long.parseLong(v) == actual);
+ case GREATER_THAN:
+ return candidates.stream().anyMatch(v -> actual > Long.parseLong(v));
+ case GREATER_THAN_OR_EQUAL:
+ return candidates.stream().anyMatch(v -> actual >= Long.parseLong(v));
+ case LESS_THAN:
+ return candidates.stream().anyMatch(v -> actual < Long.parseLong(v));
+ case LESS_THAN_OR_EQUAL:
+ return candidates.stream().anyMatch(v -> actual <= Long.parseLong(v));
+ case IN:
+ for (String candidate : candidates) {
+ if (actual == Long.parseLong(candidate)) {
+ return true;
+ }
+ }
+ return false;
+ case RANGE:
+ case BETWEEN:
+ if (candidates.size() >= 2) {
+ long lower = Long.parseLong(candidates.get(0));
+ long upper = Long.parseLong(candidates.get(1));
+ return actual >= lower && actual <= upper;
+ }
+ return true;
+ default:
+ return true;
+ }
+ } catch (NumberFormatException e) {
+ LOGGER.debug("Failed to parse numeric filter value {}: {}", candidates, e.toString());
+ return true;
+ }
+ }
+
+ private @Nullable String extractIdentifier(List<org.apache.pinot.common.request.Expression> operands) {
+ for (org.apache.pinot.common.request.Expression operand : operands) {
+ org.apache.pinot.common.request.Identifier identifier = operand.getIdentifier();
+ if (identifier != null) {
+ return identifier.getName();
+ }
+ }
+ return null;
+ }
+
+ private List<String> extractLiteralValues(List<org.apache.pinot.common.request.Expression> operands) {
+ List<String> values = new ArrayList<>();
+ for (org.apache.pinot.common.request.Expression operand : operands) {
+ org.apache.pinot.common.request.Literal literal = operand.getLiteral();
+ if (literal != null) {
+ if (literal.getSetField() == org.apache.pinot.common.request.Literal._Fields.NULL_VALUE) {
+ values.add("null");
Review Comment:
The special handling of null literals by converting them to the string \"null\" is inconsistent with how filter matching works. The filter matching logic does string comparisons and numeric parsing but doesn't have special handling for the literal string \"null\". This could lead to unexpected behavior. Consider documenting this behavior or handling null literals consistently throughout the filter evaluation logic.
```suggestion
// Skip NULL_VALUE literals instead of treating them as the string "null"
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,850 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.SystemTableResponseUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.FilterKind;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableDataProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final String PINOT_ADMIN_CLIENT_CLASS_NAME = "org.apache.pinot.client.admin.PinotAdminClient";
+ private static final String SIZE_CACHE_TTL_MS_PROPERTY = "pinot.systemtable.tables.sizeCacheTtlMs";
+ private static final long DEFAULT_SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+ private static final long SIZE_CACHE_TTL_MS = getNonNegativeLongProperty(SIZE_CACHE_TTL_MS_PROPERTY,
+ DEFAULT_SIZE_CACHE_TTL_MS);
+
+ private static final String CONTROLLER_TIMEOUT_MS_PROPERTY = "pinot.systemtable.tables.controllerTimeoutMs";
+ private static final long DEFAULT_CONTROLLER_TIMEOUT_MS = Duration.ofSeconds(5).toMillis();
+ private static final long CONTROLLER_TIMEOUT_MS = getPositiveLongProperty(CONTROLLER_TIMEOUT_MS_PROPERTY,
+ DEFAULT_CONTROLLER_TIMEOUT_MS);
+
+ private static final String ADMIN_TRANSPORT_REQUEST_TIMEOUT_MS = "pinot.admin.request.timeout.ms";
+ private static final String ADMIN_TRANSPORT_SCHEME = "pinot.admin.scheme";
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final @Nullable Class<?> PINOT_ADMIN_CLIENT_CLASS = loadAdminClientClass();
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+ private final Map<String, CachedSize> _sizeCache = new ConcurrentHashMap<>();
+ private final Map<String, AutoCloseable> _adminClientCache = new ConcurrentHashMap<>();
+ private final AtomicBoolean _loggedSizeFetchFailure = new AtomicBoolean();
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public TableConfig getTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ }
+
+ @Override
+ public void close()
+ throws Exception {
+ for (Map.Entry<String, AutoCloseable> entry : _adminClientCache.entrySet()) {
+ try {
+ entry.getValue().close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close admin client for {}: {}", entry.getKey(), e.toString());
+ }
+ }
+ _adminClientCache.clear();
+ }
+
+ @Override
+ public BrokerResponseNative getBrokerResponse(PinotQuery pinotQuery) {
+ List<String> projectionColumns = getProjectionColumns(pinotQuery);
+ if (_tableCache == null) {
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, List.of(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ int offset = Math.max(0, pinotQuery.getOffset());
+ int limit = pinotQuery.getLimit();
+ boolean hasLimit = limit > 0;
+ org.apache.pinot.common.request.Expression filterExpression = pinotQuery.getFilterExpression();
+ List<String> controllerBaseUrls = getControllerBaseUrls();
+ Function<String, TableSize> sizeFetcher = getSizeFetcher();
+
+ if (filterExpression == null) {
+ int totalRows = sortedTableNames.size();
+ int startIndex = Math.min(offset, totalRows);
+ int endIndex = totalRows;
+ if (limit == 0) {
+ endIndex = startIndex;
+ } else if (hasLimit) {
+ endIndex = Math.min(totalRows, startIndex + limit);
+ }
+ List<GenericRow> rows = new ArrayList<>(Math.max(0, endIndex - startIndex));
+ for (int i = startIndex; i < endIndex; i++) {
+ String tableNameWithType = sortedTableNames.get(i);
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType, sizeFetcher, controllerBaseUrls);
+ rows.add(buildRow(rawTableName, stats));
+ }
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, rows, totalRows);
+ }
+
+ int totalRows = 0;
+ int initialCapacity = hasLimit ? Math.min(sortedTableNames.size(), limit) : sortedTableNames.size();
+ List<GenericRow> rows = new ArrayList<>(initialCapacity);
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType, sizeFetcher, controllerBaseUrls);
+ if (!matchesFilter(filterExpression, stats, rawTableName)) {
+ continue;
+ }
+ totalRows++;
+ if (totalRows <= offset) {
+ continue;
+ }
+ if (limit == 0) {
+ continue;
+ }
+ if (hasLimit && rows.size() >= limit) {
+ continue;
+ }
+ rows.add(buildRow(rawTableName, stats));
+ }
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, rows, totalRows);
+ }
+
+ private GenericRow buildRow(String rawTableName, TableStats stats) {
+ GenericRow row = new GenericRow();
+ row.putValue("tableName", rawTableName);
+ row.putValue("type", stats._type);
+ row.putValue("status", stats._status);
+ row.putValue("segments", stats._segments);
+ row.putValue("totalDocs", stats._totalDocs);
+ row.putValue("reportedSize", stats._reportedSizeInBytes);
+ row.putValue("estimatedSize", stats._estimatedSizeInBytes);
+ row.putValue("storageTier", stats._storageTier);
+ row.putValue("brokerTenant", stats._brokerTenant);
+ row.putValue("serverTenant", stats._serverTenant);
+ row.putValue("replicas", stats._replicas);
+ row.putValue("tableConfig", stats._tableConfig);
+ return row;
+ }
+
+ private TableStats buildStats(String tableNameWithType, TableType tableType,
+ @Nullable Function<String, TableSize> tableSizeFetcher, List<String> controllerBaseUrls) {
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ int segments = 0;
+ long totalDocs = 0;
+ long reportedSize = 0;
+ long estimatedSize = 0;
+ String tierValue = "";
+ String brokerTenant = "";
+ String serverTenant = "";
+ int replicas = 0;
+ if (tableConfig != null && tableConfig.getTenantConfig() != null) {
+ brokerTenant = tableConfig.getTenantConfig().getBroker();
+ serverTenant = tableConfig.getTenantConfig().getServer();
+ }
+ if (tableConfig != null && tableConfig.getValidationConfig() != null) {
+ Integer repl = tableConfig.getValidationConfig().getReplicationNumber();
+ replicas = repl != null ? repl : replicas;
+ }
+ // Use controller API only
+ TableSize sizeFromController = fetchTableSize(tableNameWithType, tableSizeFetcher, controllerBaseUrls);
+ if (sizeFromController != null) {
+ if (sizeFromController._reportedSizeInBytes >= 0) {
+ reportedSize = sizeFromController._reportedSizeInBytes;
+ }
+ if (sizeFromController._estimatedSizeInBytes >= 0) {
+ estimatedSize = sizeFromController._estimatedSizeInBytes;
+ }
+ segments = getSegmentCount(sizeFromController, tableType);
+ totalDocs = getTotalDocs(sizeFromController, tableType);
+ }
+ String status = (tableConfig != null || segments > 0) ? "ONLINE" : "UNKNOWN";
+ String tableConfigJson = "";
+ if (tableConfig != null) {
+ try {
+ tableConfigJson = JsonUtils.objectToString(tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", tableNameWithType, e.toString());
+ tableConfigJson = tableConfig.toString();
+ }
+ }
+ return new TableStats(tableType.name(), status, segments, totalDocs, reportedSize, estimatedSize, tierValue,
+ tableConfigJson, brokerTenant, serverTenant, replicas);
+ }
+
+ private boolean matchesFilter(@Nullable org.apache.pinot.common.request.Expression filterExpression, TableStats stats,
+ String rawTableName) {
+ if (filterExpression == null) {
+ return true;
+ }
+ org.apache.pinot.common.request.Function function = filterExpression.getFunctionCall();
+ if (function == null) {
+ return true;
+ }
+ FilterKind filterKind = toFilterKind(function.getOperator());
+ if (filterKind == null) {
+ return true;
+ }
+ switch (filterKind) {
+ case AND:
+ for (org.apache.pinot.common.request.Expression child : function.getOperands()) {
+ if (!matchesFilter(child, stats, rawTableName)) {
+ return false;
+ }
+ }
+ return true;
+ case OR:
+ for (org.apache.pinot.common.request.Expression child : function.getOperands()) {
+ if (matchesFilter(child, stats, rawTableName)) {
+ return true;
+ }
+ }
+ return false;
+ case NOT:
+ if (function.getOperandsSize() == 0) {
+ return true;
+ }
+ return !matchesFilter(function.getOperands().get(0), stats, rawTableName);
+ default:
+ return matchesLeafFilter(filterKind, function.getOperands(), stats, rawTableName);
+ }
+ }
+
+ private boolean matchesLeafFilter(FilterKind filterKind,
+ List<org.apache.pinot.common.request.Expression> operands, TableStats stats, String rawTableName) {
+ String column = extractIdentifier(operands);
+ if (column == null) {
+ return true;
+ }
+ List<String> values = extractLiteralValues(operands);
+ if (values.isEmpty()) {
+ return true;
+ }
+ switch (column.toLowerCase(Locale.ROOT)) {
+ case "tablename":
+ return matchesString(values, rawTableName, filterKind);
+ case "type":
+ return matchesString(values, stats._type, filterKind);
+ case "status":
+ return matchesString(values, stats._status, filterKind);
+ case "segments":
+ return matchesNumber(values, stats._segments, filterKind);
+ case "reportedsize":
+ return matchesNumber(values, stats._reportedSizeInBytes, filterKind);
+ case "estimatedsize":
+ return matchesNumber(values, stats._estimatedSizeInBytes, filterKind);
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesString(List<String> candidates, String actual, FilterKind filterKind) {
+ switch (filterKind) {
+ case EQUALS:
+ case IN:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case NOT_EQUALS:
+ return candidates.stream().noneMatch(v -> v.equalsIgnoreCase(actual));
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesNumber(List<String> candidates, long actual, FilterKind filterKind) {
+ try {
+ switch (filterKind) {
+ case EQUALS:
+ return candidates.stream().anyMatch(v -> Long.parseLong(v) == actual);
+ case NOT_EQUALS:
+ return candidates.stream().noneMatch(v -> Long.parseLong(v) == actual);
+ case GREATER_THAN:
+ return candidates.stream().anyMatch(v -> actual > Long.parseLong(v));
+ case GREATER_THAN_OR_EQUAL:
+ return candidates.stream().anyMatch(v -> actual >= Long.parseLong(v));
+ case LESS_THAN:
+ return candidates.stream().anyMatch(v -> actual < Long.parseLong(v));
+ case LESS_THAN_OR_EQUAL:
+ return candidates.stream().anyMatch(v -> actual <= Long.parseLong(v));
+ case IN:
+ for (String candidate : candidates) {
+ if (actual == Long.parseLong(candidate)) {
+ return true;
+ }
+ }
+ return false;
+ case RANGE:
+ case BETWEEN:
+ if (candidates.size() >= 2) {
+ long lower = Long.parseLong(candidates.get(0));
+ long upper = Long.parseLong(candidates.get(1));
+ return actual >= lower && actual <= upper;
+ }
+ return true;
+ default:
+ return true;
+ }
+ } catch (NumberFormatException e) {
+ LOGGER.debug("Failed to parse numeric filter value {}: {}", candidates, e.toString());
+ return true;
+ }
+ }
+
+ private @Nullable String extractIdentifier(List<org.apache.pinot.common.request.Expression> operands) {
+ for (org.apache.pinot.common.request.Expression operand : operands) {
+ org.apache.pinot.common.request.Identifier identifier = operand.getIdentifier();
+ if (identifier != null) {
+ return identifier.getName();
+ }
+ }
+ return null;
+ }
+
+ private List<String> extractLiteralValues(List<org.apache.pinot.common.request.Expression> operands) {
+ List<String> values = new ArrayList<>();
+ for (org.apache.pinot.common.request.Expression operand : operands) {
+ org.apache.pinot.common.request.Literal literal = operand.getLiteral();
+ if (literal != null) {
+ if (literal.getSetField() == org.apache.pinot.common.request.Literal._Fields.NULL_VALUE) {
+ values.add("null");
+ continue;
+ }
+ try {
+ values.add(RequestUtils.getLiteralString(literal));
+ } catch (Exception e) {
+ values.add(String.valueOf(RequestUtils.getLiteralValue(literal)));
+ }
+ }
+ }
+ return values;
+ }
+
+ private @Nullable FilterKind toFilterKind(String operator) {
+ String normalized = operator.toUpperCase(Locale.ROOT);
+ switch (normalized) {
+ case "EQ":
+ return FilterKind.EQUALS;
+ case "NEQ":
+ return FilterKind.NOT_EQUALS;
+ case "GT":
+ return FilterKind.GREATER_THAN;
+ case "GTE":
+ return FilterKind.GREATER_THAN_OR_EQUAL;
+ case "LT":
+ return FilterKind.LESS_THAN;
+ case "LTE":
+ return FilterKind.LESS_THAN_OR_EQUAL;
+ default:
+ try {
+ return FilterKind.valueOf(normalized);
+ } catch (Exception e) {
+ return null;
+ }
+ }
+ }
+
+ private @Nullable TableSize fetchTableSize(String tableNameWithType,
+ @Nullable Function<String, TableSize> fetcher, List<String> controllerBaseUrls) {
+ boolean cacheEnabled = SIZE_CACHE_TTL_MS > 0;
+ TableSize cached = cacheEnabled ? getCachedSize(tableNameWithType) : null;
+ if (cached != null) {
+ return cached;
+ }
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ if (cacheEnabled) {
+ cacheSize(tableNameWithType, fetched);
+ }
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(controllerBaseUrls, rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(controllerBaseUrls, tableNameWithType);
+ if (size == null) {
+ logSizeFetchFailure("{}: failed to fetch size for {} from controllers {} (tried '{}' and '{}')", TABLE_NAME,
+ tableNameWithType, controllerBaseUrls, rawTableName, tableNameWithType);
+ }
+ }
+ if (size != null && cacheEnabled) {
+ cacheSize(tableNameWithType, size);
+ }
+ return size;
+ }
+
+ private @Nullable TableSize fetchTableSizeForName(List<String> controllerBaseUrls, String tableName) {
+ Class<?> clientClass = PINOT_ADMIN_CLIENT_CLASS;
+ if (clientClass == null) {
+ return null;
+ }
+ for (String baseUrl : controllerBaseUrls) {
+ try {
+ AutoCloseable adminClient = getOrCreateAdminClient(baseUrl);
+ if (adminClient == null) {
+ continue;
+ }
+
+ Object sizeNode;
+ try {
+ sizeNode = clientClass.getMethod("getTableSize", String.class, boolean.class, boolean.class)
+ .invoke(adminClient, tableName, true, false);
+ } catch (NoSuchMethodException e) {
+ Object tableClient = clientClass.getMethod("getTableClient").invoke(adminClient);
+ sizeNode = tableClient.getClass().getMethod("getTableSize", String.class, boolean.class, boolean.class)
+ .invoke(tableClient, tableName, true, false);
+ }
+
+ if (sizeNode == null) {
+ continue;
+ }
+
+ TableSize parsed = JsonUtils.stringToObject(sizeNode.toString(), TableSize.class);
+ LOGGER.debug("{}: controller size response for {} via {} -> segments offline={}, realtime={}, "
+ + "reportedSize={}, estimatedSize={}", TABLE_NAME, tableName, baseUrl,
+ parsed._offlineSegments != null && parsed._offlineSegments._segments != null
+ ? parsed._offlineSegments._segments.size() : 0,
+ parsed._realtimeSegments != null && parsed._realtimeSegments._segments != null
+ ? parsed._realtimeSegments._segments.size() : 0,
+ parsed._reportedSizeInBytes, parsed._estimatedSizeInBytes);
+ return parsed;
+ } catch (Exception e) {
+ logSizeFetchFailure("{}: error fetching table size for {} via {} using admin client", TABLE_NAME, tableName,
+ baseUrl, e);
+ }
+ }
+ return null;
+ }
+
+ private List<String> getControllerBaseUrls() {
+ Set<String> urls = new LinkedHashSet<>();
+ if (_helixAdmin != null) {
+ for (String controller : discoverControllersFromHelix()) {
+ String normalized = normalizeControllerUrl(controller);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ }
+ for (String url : _staticControllerUrls) {
+ String normalized = normalizeControllerUrl(url);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ return new ArrayList<>(urls);
+ }
+
+ private int getSegmentCount(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.size();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.size();
+ }
+ return 0;
+ }
+
+ private long getTotalDocs(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ return 0;
+ }
+
+ private @Nullable Function<String, TableSize> getSizeFetcher() {
+ if (_tableSizeFetcherOverride != null) {
+ return _tableSizeFetcherOverride;
+ }
+ return null;
+ }
+
+ private List<String> discoverControllersFromHelix() {
+ List<String> urls = new ArrayList<>();
+ try {
+ if (_clusterName == null) {
+ LOGGER.warn("Cannot discover controllers without cluster name");
+ return List.of();
+ }
+ for (String controllerId : _helixAdmin.getInstancesInCluster(_clusterName)) {
+ if (!InstanceTypeUtils.isController(controllerId)) {
+ continue;
+ }
+ int firstUnderscore = controllerId.indexOf('_');
+ int lastUnderscore = controllerId.lastIndexOf('_');
+ if (firstUnderscore > 0 && lastUnderscore > firstUnderscore && lastUnderscore < controllerId.length() - 1) {
+ String host = controllerId.substring(firstUnderscore + 1, lastUnderscore);
+ String port = controllerId.substring(lastUnderscore + 1);
+ if (!host.isEmpty() && !port.isEmpty()) {
+ urls.add(host + ":" + port);
+ } else {
+ LOGGER.warn("Unable to parse controller address from instance id (empty host or port): {}", controllerId);
+ }
+ } else {
+ LOGGER.warn("Unable to parse controller address from instance id '{}' (expected 'Controller_<host>_<port>')",
+ controllerId);
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Failed to discover controllers from Helix", e);
+ }
+ return urls;
+ }
+
+ private @Nullable AutoCloseable getOrCreateAdminClient(String controllerBaseUrl) {
+ Class<?> clientClass = PINOT_ADMIN_CLIENT_CLASS;
+ if (clientClass == null) {
+ return null;
+ }
+ String normalized = normalizeControllerUrl(controllerBaseUrl);
+ if (normalized == null) {
+ return null;
+ }
+ return _adminClientCache.computeIfAbsent(normalized, controller -> {
+ try {
+ String controllerAddress = stripScheme(controller);
+ Properties properties = new Properties();
+ properties.setProperty(ADMIN_TRANSPORT_REQUEST_TIMEOUT_MS, String.valueOf(CONTROLLER_TIMEOUT_MS));
+ properties.setProperty(ADMIN_TRANSPORT_SCHEME, controller.startsWith("https://") ? "https" : "http");
Review Comment:
The constant names `ADMIN_TRANSPORT_REQUEST_TIMEOUT_MS` and `ADMIN_TRANSPORT_SCHEME` are defined without a clear prefix indicating they are property keys for the PinotAdminClient. Consider renaming them to something like `PINOT_ADMIN_REQUEST_TIMEOUT_MS_PROPERTY_KEY` and `PINOT_ADMIN_SCHEME_PROPERTY_KEY` for clarity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2870557223
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/SystemTableBrokerRequestHandler.java:
##########
@@ -0,0 +1,585 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.requesthandler;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.google.common.annotations.VisibleForTesting;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.client5.http.io.HttpClientConnectionManager;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.HelixManager;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.pinot.broker.api.AccessControl;
+import org.apache.pinot.broker.broker.AccessControlFactory;
+import org.apache.pinot.broker.querylog.QueryLogger;
+import org.apache.pinot.broker.queryquota.QueryQuotaManager;
+import org.apache.pinot.client.ConnectionTimeouts;
+import org.apache.pinot.client.PinotClientException;
+import org.apache.pinot.client.SystemTableDataTableClient;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.common.datatable.DataTableImplV4;
+import org.apache.pinot.common.metrics.BrokerMeter;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.request.QuerySource;
+import org.apache.pinot.common.response.BrokerResponse;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.SystemTableRegistry;
+import org.apache.pinot.common.utils.NamedThreadFactory;
+import org.apache.pinot.common.utils.config.InstanceUtils;
+import org.apache.pinot.common.utils.config.QueryOptionsUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.core.instance.context.BrokerContext;
+import org.apache.pinot.core.operator.blocks.InstanceResponseBlock;
+import org.apache.pinot.core.plan.Plan;
+import org.apache.pinot.core.plan.maker.InstancePlanMakerImplV2;
+import org.apache.pinot.core.plan.maker.PlanMaker;
+import org.apache.pinot.core.query.reduce.BrokerReduceService;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils;
+import org.apache.pinot.core.routing.MultiClusterRoutingContext;
+import org.apache.pinot.core.routing.RoutingManager;
+import org.apache.pinot.core.transport.ServerRoutingInstance;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.SegmentContext;
+import org.apache.pinot.spi.accounting.ThreadAccountant;
+import org.apache.pinot.spi.auth.AuthorizationResult;
+import org.apache.pinot.spi.auth.broker.RequesterIdentity;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.query.QueryExecutionContext;
+import org.apache.pinot.spi.query.QueryThreadContext;
+import org.apache.pinot.spi.trace.RequestContext;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+import org.apache.pinot.sql.parsers.SqlNodeAndOptions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Broker request handler for system tables (handled entirely on the broker).
+ * <p>
+ * System tables are virtual tables that expose cluster metadata (e.g., system.tables, system.instances).
+ * They are executed using the v1 query engine against in-memory segments generated on-demand by
+ * {@link SystemTableProvider} implementations.
+ */
+public class SystemTableBrokerRequestHandler extends BaseBrokerRequestHandler {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableBrokerRequestHandler.class);
+ // Sentinel values used for broker-local execution of system table queries where no actual server routing is needed.
+ private static final String SYSTEM_TABLE_PSEUDO_HOST = "localhost";
+ private static final int SYSTEM_TABLE_PSEUDO_PORT = 0;
+ private static final String SYSTEM_TABLE_DATATABLE_API_PATH = "/query/systemTable/datatable";
+ // Hop-by-hop headers per RFC 7230 plus content-length/host which are request-specific.
+ private static final Set<String> HOP_BY_HOP_HEADERS_TO_SKIP = Set.of(
+ "connection",
+ "keep-alive",
+ "proxy-authenticate",
+ "proxy-authorization",
+ "te",
+ "trailer",
+ "transfer-encoding",
+ "upgrade",
+ "host",
+ "content-length");
+
+ private final SystemTableRegistry _systemTableRegistry;
+ private final BrokerReduceService _brokerReduceService;
+ private final PlanMaker _planMaker;
+ private final ExecutorService _executorService;
+ private final ExecutorService _scatterGatherExecutorService;
+ private final SystemTableDataTableClient _systemTableDataTableClient;
+ @Nullable
+ private final HelixManager _helixManager;
+
+ public SystemTableBrokerRequestHandler(PinotConfiguration config, String brokerId,
+ BrokerRequestIdGenerator requestIdGenerator, RoutingManager routingManager,
+ AccessControlFactory accessControlFactory, QueryQuotaManager queryQuotaManager, TableCache tableCache,
+ SystemTableRegistry systemTableRegistry, ThreadAccountant threadAccountant,
+ @Nullable MultiClusterRoutingContext multiClusterRoutingContext, @Nullable HelixManager helixManager) {
+ super(config, brokerId, requestIdGenerator, routingManager, accessControlFactory, queryQuotaManager, tableCache,
+ threadAccountant, multiClusterRoutingContext);
+ _systemTableRegistry = systemTableRegistry;
+ _brokerReduceService = new BrokerReduceService(_config);
+ _planMaker = new InstancePlanMakerImplV2();
+ _planMaker.init(_config);
+ _helixManager = helixManager;
+ int executorPoolSize = config.getProperty(CommonConstants.Broker.CONFIG_OF_SYSTEM_TABLE_EXECUTOR_POOL_SIZE,
+ CommonConstants.Broker.DEFAULT_SYSTEM_TABLE_EXECUTOR_POOL_SIZE);
+ executorPoolSize = Math.max(1, executorPoolSize);
+ _executorService = QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(executorPoolSize,
+ new NamedThreadFactory("system-table-query-executor")));
+ _scatterGatherExecutorService =
+ QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(executorPoolSize,
+ new NamedThreadFactory("system-table-scatter-gather-executor")));
+ SSLContext sslContext = BrokerContext.getInstance().getClientHttpsContext();
+ int timeoutMs;
+ if (_brokerTimeoutMs > Integer.MAX_VALUE) {
+ LOGGER.warn("Broker timeout {}ms exceeds Integer.MAX_VALUE; clamping to {} for system table client connections",
+ _brokerTimeoutMs, Integer.MAX_VALUE);
+ timeoutMs = Integer.MAX_VALUE;
+ } else if (_brokerTimeoutMs < 1) {
+ LOGGER.warn("Broker timeout {}ms is non-positive; using default 5000ms for system table client connections",
+ _brokerTimeoutMs);
+ timeoutMs = 5000;
+ } else {
+ timeoutMs = (int) _brokerTimeoutMs;
+ }
+ ConnectionTimeouts connectionTimeouts = ConnectionTimeouts.create(timeoutMs, timeoutMs, timeoutMs);
+ _systemTableDataTableClient =
+ new SystemTableDataTableClient(connectionTimeouts, sslContext);
+ }
+
+ @Override
+ public void start() {
+ }
+
+ @Override
+ public void shutDown() {
+ _executorService.shutdownNow();
+ _scatterGatherExecutorService.shutdownNow();
+ try {
+ _systemTableDataTableClient.close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close system table data table client: {}", e.toString());
+ }
+ _brokerReduceService.shutDown();
+ }
+
+ public boolean canHandle(String tableName) {
+ return isSystemTable(tableName) && _systemTableRegistry.isRegistered(tableName);
+ }
+
+ @Override
+ protected BrokerResponse handleRequest(long requestId, String query, SqlNodeAndOptions sqlNodeAndOptions,
+ JsonNode request, @Nullable RequesterIdentity requesterIdentity, RequestContext requestContext,
+ @Nullable HttpHeaders httpHeaders, AccessControl accessControl)
+ throws Exception {
+ long startTimeMs = requestContext.getRequestArrivalTimeMillis();
+ long deadlineMs = startTimeMs + _brokerTimeoutMs;
+ QueryExecutionContext executionContext =
+ new QueryExecutionContext(QueryExecutionContext.QueryType.STE, requestId, Long.toString(requestId),
+ QueryOptionsUtils.getWorkloadName(sqlNodeAndOptions.getOptions()), startTimeMs, deadlineMs, deadlineMs,
+ _brokerId, _brokerId, org.apache.pinot.spi.utils.CommonConstants.Broker.DEFAULT_QUERY_HASH);
+ try (QueryThreadContext ignore = QueryThreadContext.open(executionContext, _threadAccountant)) {
+ PinotQuery pinotQuery;
+ try {
+ pinotQuery = CalciteSqlParser.compileToPinotQuery(sqlNodeAndOptions);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return new BrokerResponseNative(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ Set<String> tableNames = RequestUtils.getTableNames(pinotQuery);
+ if (tableNames == null || tableNames.isEmpty()) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "Failed to extract table name");
+ }
+ if (tableNames.size() != 1) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "System tables do not support joins");
+ }
+ String tableName = tableNames.iterator().next();
+ if (!isSystemTable(tableName)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "Not a system table query");
+ }
+ AuthorizationResult authorizationResult =
+ hasTableAccess(requesterIdentity, Set.of(tableName), requestContext, httpHeaders);
+ if (!authorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return new BrokerResponseNative(QueryErrorCode.ACCESS_DENIED, authorizationResult.getFailureMessage());
+ }
+
+ boolean queryWasLogged = _queryLogger.logQueryReceived(requestId, query);
+ return handleSystemTableQuery(request, pinotQuery, tableName, requestContext, requesterIdentity, query,
+ httpHeaders, queryWasLogged);
+ }
+ }
+
+ @Override
+ protected boolean handleCancel(long queryId, int timeoutMs, Executor executor,
+ HttpClientConnectionManager connMgr, Map<String, Integer> serverResponses) {
+ return false;
+ }
+
+ @Override
+ public boolean cancelQueryByClientId(String clientQueryId, int timeoutMs, Executor executor,
+ HttpClientConnectionManager connMgr, Map<String, Integer> serverResponses)
+ throws Exception {
+ return false;
+ }
+
+ @Override
+ public Map<Long, String> getRunningQueries() {
+ return Collections.emptyMap();
+ }
+
+ @Override
+ public OptionalLong getRequestIdByClientId(String clientQueryId) {
+ return OptionalLong.empty();
+ }
+
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ /**
+ * Executes a system table query against the local broker and returns the raw {@link DataTable} results.
+ * <p>
+ * This method is used by the internal broker-to-broker scatter-gather endpoint and must never perform fanout.
+ * It is invoked when a broker receives a scatter-gather request from another broker for a system table query.
+ *
+ * @param request the JSON request containing the SQL query and options
+ * @param requesterIdentity the identity of the requester for authorization
+ * @param requestContext the request context for tracking
+ * @param httpHeaders the HTTP headers from the request
+ * @return a DataTable containing the query results from this broker's local shard
+ */
+ public DataTable handleSystemTableDataTableRequest(JsonNode request, @Nullable RequesterIdentity requesterIdentity,
+ RequestContext requestContext, @Nullable HttpHeaders httpHeaders) {
+ long startTimeMs = requestContext.getRequestArrivalTimeMillis();
+ if (startTimeMs <= 0) {
+ startTimeMs = System.currentTimeMillis();
+ requestContext.setRequestArrivalTimeMillis(startTimeMs);
+ }
+ long requestId = _requestIdGenerator.get();
+ long deadlineMs = startTimeMs + _brokerTimeoutMs;
+
+ JsonNode sql = request.get(CommonConstants.Broker.Request.SQL);
+ if (sql == null || !sql.isTextual()) {
+ return exceptionDataTable(QueryErrorCode.JSON_PARSING, "Failed to find 'sql' in the request: " + request);
+ }
+ String query = sql.textValue();
+ requestContext.setQuery(query);
+
+ SqlNodeAndOptions sqlNodeAndOptions;
+ try {
+ sqlNodeAndOptions = RequestUtils.parseQuery(query, request);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return exceptionDataTable(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ QueryExecutionContext executionContext =
+ new QueryExecutionContext(QueryExecutionContext.QueryType.STE, requestId, Long.toString(requestId),
+ QueryOptionsUtils.getWorkloadName(sqlNodeAndOptions.getOptions()), startTimeMs, deadlineMs, deadlineMs,
+ _brokerId, _brokerId, org.apache.pinot.spi.utils.CommonConstants.Broker.DEFAULT_QUERY_HASH);
+ try (QueryThreadContext ignore = QueryThreadContext.open(executionContext, _threadAccountant)) {
+ AccessControl accessControl = _accessControlFactory.create();
+ AuthorizationResult authorizationResult = accessControl.authorize(requesterIdentity);
+ if (!authorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return exceptionDataTable(QueryErrorCode.ACCESS_DENIED, authorizationResult.getFailureMessage());
+ }
+
+ PinotQuery pinotQuery;
+ try {
+ pinotQuery = CalciteSqlParser.compileToPinotQuery(sqlNodeAndOptions);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return exceptionDataTable(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ Set<String> tableNames = RequestUtils.getTableNames(pinotQuery);
+ if (tableNames == null || tableNames.isEmpty()) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, "Failed to extract table name");
+ }
+ if (tableNames.size() != 1) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, "System tables do not support joins");
+ }
+ String tableName = tableNames.iterator().next();
+ if (!isSystemTable(tableName)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, "Not a system table query");
+ }
+
+ AuthorizationResult tableAuthorizationResult =
+ hasTableAccess(requesterIdentity, Set.of(tableName), requestContext, httpHeaders);
+ if (!tableAuthorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return exceptionDataTable(QueryErrorCode.ACCESS_DENIED, tableAuthorizationResult.getFailureMessage());
+ }
+
+ SystemTableProvider provider = _systemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ Collection<SystemTableProvider> availableProviders = _systemTableRegistry.getProviders();
+ String availableTables = availableProviders.isEmpty()
+ ? "No system tables are available"
+ : "Available system tables: "
+ + availableProviders.stream().map(SystemTableProvider::getTableName)
+ .collect(java.util.stream.Collectors.joining(", "));
+ return exceptionDataTable(QueryErrorCode.TABLE_DOES_NOT_EXIST,
+ "System table does not exist: " + tableName + ". " + availableTables);
+ }
+
+ try {
+ return executeLocalSystemTableQuery(pinotQuery, provider);
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table datatable query {}: {}", tableName, e.getMessage(),
+ e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return exceptionDataTable(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+ }
+
+ private BrokerResponse handleSystemTableQuery(JsonNode request, PinotQuery pinotQuery, String tableName,
+ RequestContext requestContext, @Nullable RequesterIdentity requesterIdentity, String query,
+ @Nullable HttpHeaders httpHeaders, boolean queryWasLogged) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableProvider provider = _systemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ Collection<SystemTableProvider> availableProviders = _systemTableRegistry.getProviders();
+ String availableTables = availableProviders.isEmpty()
+ ? "No system tables are available"
+ : "Available system tables: "
+ + availableProviders.stream().map(SystemTableProvider::getTableName)
+ .collect(java.util.stream.Collectors.joining(", "));
+ return new BrokerResponseNative(QueryErrorCode.TABLE_DOES_NOT_EXIST,
+ "System table does not exist: " + tableName + ". " + availableTables);
+ }
+ try {
+ long deadlineMs = requestContext.getRequestArrivalTimeMillis() + _brokerTimeoutMs;
+ Map<ServerRoutingInstance, DataTable> dataTableMap;
+ if (provider.getExecutionMode() == SystemTableProvider.ExecutionMode.BROKER_SCATTER_GATHER) {
+ dataTableMap =
+ scatterGatherSystemTableDataTables(provider, pinotQuery, tableName, request, httpHeaders, deadlineMs);
+ } else {
+ dataTableMap = new HashMap<>(1);
+ // Use a synthetic routing instance for broker-local execution of system table queries.
+ dataTableMap.put(new ServerRoutingInstance(SYSTEM_TABLE_PSEUDO_HOST, SYSTEM_TABLE_PSEUDO_PORT,
+ TableType.OFFLINE), executeLocalSystemTableQuery(pinotQuery, provider));
+ }
+
+ BrokerResponseNative brokerResponse;
+ BrokerRequest brokerRequest = new BrokerRequest();
+ QuerySource querySource = new QuerySource();
+ querySource.setTableName(tableName);
+ brokerRequest.setQuerySource(querySource);
+ brokerRequest.setPinotQuery(pinotQuery);
+ brokerResponse = _brokerReduceService.reduceOnDataTable(brokerRequest, brokerRequest, dataTableMap,
+ _brokerTimeoutMs, _brokerMetrics);
+ brokerResponse.setTablesQueried(Set.of(TableNameBuilder.extractRawTableName(tableName)));
+ brokerResponse.setTimeUsedMs(System.currentTimeMillis() - requestContext.getRequestArrivalTimeMillis());
+ _queryLogger.logQueryCompleted(
+ new QueryLogger.QueryLogParams(requestContext, tableName, brokerResponse,
+ QueryLogger.QueryLogParams.QueryEngine.SINGLE_STAGE, requesterIdentity, null),
+ queryWasLogged);
+ return brokerResponse;
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table query {}: {}", tableName, e.getMessage(), e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+
+ private DataTable executeLocalSystemTableQuery(PinotQuery pinotQuery, SystemTableProvider provider)
+ throws Exception {
+ IndexSegment dataSource = provider.getDataSource();
+ try {
+ QueryContext queryContext = QueryContextConverterUtils.getQueryContext(pinotQuery);
+ queryContext.setSchema(provider.getSchema());
+ queryContext.setEndTimeMs(System.currentTimeMillis() + _brokerTimeoutMs);
+
+ // Pass null for serverMetrics because system table queries run broker-local against an in-memory IndexSegment.
+ Plan plan = _planMaker.makeInstancePlan(List.of(new SegmentContext(dataSource)), queryContext, _executorService,
+ null);
+ InstanceResponseBlock instanceResponse = plan.execute();
+ return instanceResponse.toDataTable();
+ } finally {
+ dataSource.destroy();
+ }
+ }
+
+ private static final class BrokerTarget {
+ final ServerRoutingInstance _routingInstance;
+ final String _dataTableUrl;
+
+ BrokerTarget(ServerRoutingInstance routingInstance, String dataTableUrl) {
+ _routingInstance = routingInstance;
+ _dataTableUrl = dataTableUrl;
+ }
+ }
+
+ @VisibleForTesting
+ protected Map<ServerRoutingInstance, DataTable> scatterGatherSystemTableDataTables(SystemTableProvider provider,
+ PinotQuery pinotQuery, String tableName, JsonNode request, @Nullable HttpHeaders httpHeaders, long deadlineMs) {
+ if (_helixManager == null) {
+ throw new IllegalStateException(
+ "HelixManager is required for scatter-gather execution of system table: " + tableName);
+ }
+
+ HelixDataAccessor dataAccessor = _helixManager.getHelixDataAccessor();
+ List<String> liveInstances = dataAccessor.getChildNames(dataAccessor.keyBuilder().liveInstances());
+ if (liveInstances == null || liveInstances.isEmpty()) {
+ throw new IllegalStateException("No live instances found for scatter-gather execution of system table: "
+ + tableName);
+ }
+
+ String localInstanceId = _brokerId;
+ List<BrokerTarget> remoteTargets = new ArrayList<>();
+ @Nullable ServerRoutingInstance localRoutingInstance = null;
+ for (String instanceId : liveInstances) {
+ if (!InstanceTypeUtils.isBroker(instanceId)) {
+ continue;
+ }
+ InstanceConfig instanceConfig = dataAccessor.getProperty(dataAccessor.keyBuilder().instanceConfig(instanceId));
+ if (instanceConfig == null) {
+ LOGGER.warn("Instance config not found for broker instance: {}. This instance will be skipped in scatter-gather"
+ + " execution for system table: {}", instanceId, tableName);
+ continue;
+ }
+ URI baseUri = URI.create(InstanceUtils.getInstanceBaseUri(instanceConfig));
+ ServerRoutingInstance routingInstance = new ServerRoutingInstance(baseUri.getHost(), baseUri.getPort(),
+ TableType.OFFLINE);
+ if (instanceId.equals(localInstanceId)) {
+ localRoutingInstance = routingInstance;
+ } else {
+ remoteTargets.add(new BrokerTarget(routingInstance, baseUri.toString() + SYSTEM_TABLE_DATATABLE_API_PATH));
+ }
+ }
+
+ Map<ServerRoutingInstance, DataTable> dataTableMap = new HashMap<>(remoteTargets.size() + 1);
+ ServerRoutingInstance routingInstance;
+ if (localRoutingInstance != null) {
+ routingInstance = localRoutingInstance;
+ } else {
+ routingInstance = new ServerRoutingInstance(SYSTEM_TABLE_PSEUDO_HOST, SYSTEM_TABLE_PSEUDO_PORT,
+ TableType.OFFLINE);
+ }
+ try {
+ dataTableMap.put(routingInstance, executeLocalSystemTableQuery(pinotQuery, provider));
+ } catch (Exception e) {
+ LOGGER.error("Failed to execute system table query locally for query: {}", pinotQuery, e);
+ dataTableMap.put(routingInstance,
+ exceptionDataTable(QueryErrorCode.QUERY_EXECUTION, "Failed to execute system table query locally: "
+ + e.getMessage()));
+ }
+
+ if (remoteTargets.isEmpty()) {
+ return dataTableMap;
+ }
+
+ String requestBody = request.toString();
+ @Nullable Map<String, String> requestHeaders = toSingleValueRequestHeaders(httpHeaders);
+ if (requestHeaders == null) {
+ requestHeaders = new HashMap<>();
+ }
+ requestHeaders.putIfAbsent("Content-Type", MediaType.APPLICATION_JSON);
+ Map<String, String> requestHeadersFinal = requestHeaders;
+ List<Pair<BrokerTarget, Future<DataTable>>> futures = new ArrayList<>(remoteTargets.size());
+ for (BrokerTarget target : remoteTargets) {
+ Future<DataTable> future =
+ _scatterGatherExecutorService.submit(() -> fetchDataTableFromBroker(target._dataTableUrl, requestBody,
+ requestHeadersFinal));
+ futures.add(Pair.of(target, future));
+ }
+
+ long remainingMs = Math.max(1, deadlineMs - System.currentTimeMillis());
+ for (Pair<BrokerTarget, Future<DataTable>> pair : futures) {
+ BrokerTarget target = pair.getLeft();
+ try {
+ dataTableMap.put(target._routingInstance, pair.getRight().get(remainingMs, TimeUnit.MILLISECONDS));
Review Comment:
In scatter-gather execution, `remainingMs` is computed once before waiting on all futures, then reused for each `Future#get`. This can exceed the overall `deadlineMs` by up to (numBrokers × remainingMs) because waits are sequential. Recompute the remaining time before each `get()` (and short-circuit once the deadline is reached) so the query respects the configured broker timeout end-to-end.
```suggestion
for (Pair<BrokerTarget, Future<DataTable>> pair : futures) {
BrokerTarget target = pair.getLeft();
long remainingMs = deadlineMs - System.currentTimeMillis();
if (remainingMs <= 0) {
pair.getRight().cancel(true);
dataTableMap.put(target._routingInstance, exceptionDataTable(QueryErrorCode.BROKER_TIMEOUT,
"Timed out waiting for system table response from " + target._dataTableUrl));
continue;
}
try {
dataTableMap.put(target._routingInstance,
pair.getRight().get(Math.max(1, remainingMs), TimeUnit.MILLISECONDS));
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/datasource/InMemorySystemTableSegment.java:
##########
@@ -0,0 +1,734 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.datasource;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.node.JsonNodeFactory;
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.Constants;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.SegmentVersion;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.datasource.DataSourceMetadata;
+import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexType;
+import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.column.ColumnIndexContainer;
+import org.apache.pinot.segment.spi.index.multicolumntext.MultiColumnTextMetadata;
+import org.apache.pinot.segment.spi.index.mutable.ThreadSafeMutableRoaringBitmap;
+import org.apache.pinot.segment.spi.index.reader.BloomFilterReader;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReaderContext;
+import org.apache.pinot.segment.spi.index.reader.H3IndexReader;
+import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
+import org.apache.pinot.segment.spi.index.reader.JsonIndexReader;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+import org.apache.pinot.segment.spi.index.reader.RangeIndexReader;
+import org.apache.pinot.segment.spi.index.reader.TextIndexReader;
+import org.apache.pinot.segment.spi.index.reader.VectorIndexReader;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2Metadata;
+import org.apache.pinot.segment.spi.partition.PartitionFunction;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.joda.time.Duration;
+import org.joda.time.Interval;
+
+
+/**
+ * In-memory {@link IndexSegment} implementation intended for system table queries.
+ * <p>
+ * This segment is backed by per-column value functions (docId -> value) and exposes raw forward indexes (no
+ * dictionaries/inverted indexes) so that the standard v1 query engine can operate on it.
+ */
+public final class InMemorySystemTableSegment implements IndexSegment {
+ private final String _segmentName;
+ private final Schema _schema;
+ private final int _numDocs;
+ private final Map<String, IntFunction<Object>> _valueProvidersByColumn;
+ private final Map<String, DataSource> _dataSourcesByColumn;
+ private final SegmentMetadata _segmentMetadata;
+
+ public InMemorySystemTableSegment(String segmentName, Schema schema, int numDocs,
+ Map<String, IntFunction<Object>> valueProvidersByColumn) {
+ _segmentName = segmentName;
+ _schema = schema;
+ _numDocs = numDocs;
+ _valueProvidersByColumn = new HashMap<>(valueProvidersByColumn);
+ _dataSourcesByColumn = new HashMap<>(schema.getColumnNames().size());
+ for (String column : schema.getColumnNames()) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ if (fieldSpec == null) {
+ continue;
+ }
+ IntFunction<Object> provider = _valueProvidersByColumn.get(column);
+ if (provider == null) {
+ provider = docId -> defaultValue(fieldSpec.getDataType());
+ _valueProvidersByColumn.put(column, provider);
+ }
+ _dataSourcesByColumn.put(column, new FunctionBasedDataSource(fieldSpec, numDocs, provider));
+ }
+ _segmentMetadata = new InMemorySegmentMetadata(segmentName, schema, numDocs);
+ }
+
+ @Override
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ @Override
+ public SegmentMetadata getSegmentMetadata() {
+ return _segmentMetadata;
+ }
+
+ @Override
+ public Set<String> getColumnNames() {
+ return _schema.getColumnNames();
+ }
+
+ @Override
+ public Set<String> getPhysicalColumnNames() {
+ return _schema.getPhysicalColumnNames();
+ }
+
+ @Nullable
+ @Override
+ public DataSource getDataSourceNullable(String column) {
+ return _dataSourcesByColumn.get(column);
+ }
+
+ @Override
+ public DataSource getDataSource(String column, Schema schema) {
+ DataSource dataSource = getDataSourceNullable(column);
+ if (dataSource != null) {
+ return dataSource;
+ }
+ throw new IllegalStateException("Failed to find data source for column: " + column);
+ }
+
+ @Nullable
+ @Override
+ public List<StarTreeV2> getStarTrees() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public TextIndexReader getMultiColumnTextIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public ThreadSafeMutableRoaringBitmap getValidDocIds() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public ThreadSafeMutableRoaringBitmap getQueryableDocIds() {
+ return null;
+ }
+
+ @Override
+ public GenericRow getRecord(int docId, GenericRow reuse) {
+ GenericRow row = reuse != null ? reuse : new GenericRow();
+ row.getFieldToValueMap().clear();
+ for (String column : _schema.getColumnNames()) {
+ row.putValue(column, getValue(docId, column));
+ }
+ return row;
+ }
+
+ @Override
+ public Object getValue(int docId, String column) {
+ IntFunction<Object> provider = _valueProvidersByColumn.get(column);
+ if (provider == null) {
+ return null;
+ }
+ return provider.apply(docId);
+ }
+
+ @Override
+ public void offload() {
+ }
+
+ @Override
+ public void destroy() {
+ }
+
+ private static Object defaultValue(FieldSpec.DataType dataType) {
+ switch (dataType) {
+ case INT:
+ return 0;
+ case LONG:
+ return 0L;
+ case FLOAT:
+ return 0.0f;
+ case DOUBLE:
+ return 0.0d;
+ case BIG_DECIMAL:
+ return BigDecimal.ZERO;
+ case BOOLEAN:
+ return false;
+ case STRING:
+ return "";
+ case BYTES:
+ return new byte[0];
+ default:
+ return null;
+ }
+ }
+
+ private static final class InMemorySegmentMetadata implements SegmentMetadata {
+ private final String _segmentName;
+ private final Schema _schema;
+ private final int _totalDocs;
+ private final TreeMap<String, ColumnMetadata> _columnMetadataMap;
+
+ private InMemorySegmentMetadata(String segmentName, Schema schema, int totalDocs) {
+ _segmentName = segmentName;
+ _schema = schema;
+ _totalDocs = totalDocs;
+ _columnMetadataMap = new TreeMap<>();
+ for (String column : schema.getColumnNames()) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ if (fieldSpec != null) {
+ _columnMetadataMap.put(column, new InMemoryColumnMetadata(fieldSpec, totalDocs));
+ }
+ }
+ }
+
+ @Deprecated
+ @Override
+ public String getTableName() {
+ return _schema.getSchemaName();
+ }
+
+ @Override
+ public String getName() {
+ return _segmentName;
+ }
+
+ @Override
+ public String getTimeColumn() {
+ return "";
+ }
+
+ @Override
+ public long getStartTime() {
+ return 0;
+ }
+
+ @Override
+ public long getEndTime() {
+ return 0;
+ }
+
+ @Override
+ public TimeUnit getTimeUnit() {
+ return TimeUnit.MILLISECONDS;
+ }
+
+ @Override
+ public Duration getTimeGranularity() {
+ return Duration.ZERO;
+ }
+
+ @Override
+ public Interval getTimeInterval() {
+ return new Interval(0L, 0L);
+ }
+
+ @Override
+ public String getCrc() {
+ return "";
+ }
+
+ @Override
+ public String getDataCrc() {
+ return "";
+ }
+
+ @Override
+ public SegmentVersion getVersion() {
+ return SegmentVersion.v3;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return _schema;
+ }
+
+ @Override
+ public int getTotalDocs() {
+ return _totalDocs;
+ }
+
+ @Override
+ public File getIndexDir() {
+ return new File("");
+ }
+
+ @Nullable
+ @Override
+ public String getCreatorName() {
+ return "systemtable";
+ }
+
+ @Override
+ public long getIndexCreationTime() {
+ return 0;
+ }
+
+ @Override
+ public long getLastIndexedTimestamp() {
+ return 0;
+ }
+
+ @Override
+ public long getLatestIngestionTimestamp() {
+ return Long.MIN_VALUE;
+ }
+
+ @Override
+ public long getMinimumIngestionLagMs() {
+ return Long.MAX_VALUE;
+ }
+
+ @Nullable
+ @Override
+ public List<StarTreeV2Metadata> getStarTreeV2MetadataList() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public MultiColumnTextMetadata getMultiColumnTextMetadata() {
+ return null;
+ }
+
+ @Override
+ public Map<String, String> getCustomMap() {
+ return Collections.emptyMap();
+ }
+
+ @Override
+ public String getStartOffset() {
+ return "";
+ }
+
+ @Override
+ public String getEndOffset() {
+ return "";
+ }
+
+ @Override
+ public TreeMap<String, ColumnMetadata> getColumnMetadataMap() {
+ return _columnMetadataMap;
+ }
+
+ @Override
+ public void removeColumn(String column) {
+ _columnMetadataMap.remove(column);
+ }
+
+ @Override
+ public JsonNode toJson(@Nullable Set<String> columnFilter) {
+ return JsonNodeFactory.instance.objectNode();
+ }
+ }
+
+ private static final class InMemoryColumnMetadata implements ColumnMetadata {
+ private final FieldSpec _fieldSpec;
+ private final int _totalDocs;
+
+ private InMemoryColumnMetadata(FieldSpec fieldSpec, int totalDocs) {
+ _fieldSpec = fieldSpec;
+ _totalDocs = totalDocs;
+ }
+
+ @Override
+ public FieldSpec getFieldSpec() {
+ return _fieldSpec;
+ }
+
+ @Override
+ public int getTotalDocs() {
+ return _totalDocs;
+ }
+
+ @Override
+ public int getCardinality() {
+ return Constants.UNKNOWN_CARDINALITY;
+ }
+
+ @Override
+ public boolean isSorted() {
+ return false;
+ }
+
+ @Override
+ public Comparable getMinValue() {
+ return defaultComparableValue(_fieldSpec.getDataType());
+ }
+
+ @Override
+ public Comparable getMaxValue() {
+ return defaultComparableValue(_fieldSpec.getDataType());
+ }
+
+ @Override
+ public boolean hasDictionary() {
+ return false;
+ }
+
+ @Override
+ public int getColumnMaxLength() {
+ return 0;
+ }
+
+ @Override
+ public int getBitsPerElement() {
+ return 0;
+ }
+
+ @Override
+ public int getMaxNumberOfMultiValues() {
+ return 0;
+ }
+
+ @Override
+ public int getTotalNumberOfEntries() {
+ return _totalDocs;
+ }
+
+ @Nullable
+ @Override
+ public PartitionFunction getPartitionFunction() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public Set<Integer> getPartitions() {
+ return null;
+ }
+
+ @Override
+ public boolean isAutoGenerated() {
+ return false;
+ }
+
+ @Override
+ public Map<IndexType<?, ?, ?>, Long> getIndexSizeMap() {
+ return Collections.emptyMap();
+ }
+
+ private static Comparable defaultComparableValue(FieldSpec.DataType dataType) {
+ switch (dataType) {
+ case INT:
+ return 0;
+ case LONG:
+ return 0L;
+ case FLOAT:
+ return 0.0f;
+ case DOUBLE:
+ return 0.0d;
+ case BIG_DECIMAL:
+ return BigDecimal.ZERO;
+ case BOOLEAN:
+ return false;
+ case STRING:
+ return "";
+ default:
+ return 0;
+ }
+ }
+ }
+
+ private static final class FunctionBasedDataSource implements DataSource {
+ private final DataSourceMetadata _metadata;
+ private final ColumnIndexContainer _indexContainer;
+ private final ForwardIndexReader<?> _forwardIndex;
+
+ private FunctionBasedDataSource(FieldSpec fieldSpec, int numDocs, IntFunction<Object> valueProvider) {
+ _metadata = new FunctionBasedDataSourceMetadata(fieldSpec, numDocs);
+ _indexContainer = ColumnIndexContainer.Empty.INSTANCE;
+ _forwardIndex = new FunctionBasedForwardIndexReader(fieldSpec.getDataType(), valueProvider);
+ }
+
+ @Override
+ public DataSourceMetadata getDataSourceMetadata() {
+ return _metadata;
+ }
+
+ @Override
+ public ColumnIndexContainer getIndexContainer() {
+ return _indexContainer;
+ }
+
+ @Override
+ public <R extends IndexReader> R getIndex(IndexType<?, R, ?> type) {
+ if (type == null) {
+ return null;
+ }
+ if (StandardIndexes.FORWARD_ID.equals(type.getId())) {
+ return (R) _forwardIndex;
+ }
+ return null;
+ }
+
+ @Override
+ public ForwardIndexReader<?> getForwardIndex() {
+ return _forwardIndex;
+ }
+
+ @Nullable
+ @Override
+ public Dictionary getDictionary() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public InvertedIndexReader<?> getInvertedIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public RangeIndexReader<?> getRangeIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public TextIndexReader getTextIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public TextIndexReader getFSTIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public TextIndexReader getIFSTIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public JsonIndexReader getJsonIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public H3IndexReader getH3Index() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public BloomFilterReader getBloomFilter() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public NullValueVectorReader getNullValueVector() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public VectorIndexReader getVectorIndex() {
+ return null;
+ }
+ }
+
+ private static final class FunctionBasedDataSourceMetadata implements DataSourceMetadata {
+ private final FieldSpec _fieldSpec;
+ private final int _numDocs;
+
+ private FunctionBasedDataSourceMetadata(FieldSpec fieldSpec, int numDocs) {
+ _fieldSpec = fieldSpec;
+ _numDocs = numDocs;
+ }
+
+ @Override
+ public FieldSpec getFieldSpec() {
+ return _fieldSpec;
+ }
+
+ @Override
+ public boolean isSorted() {
+ return false;
+ }
+
+ @Override
+ public int getNumDocs() {
+ return _numDocs;
+ }
+
+ @Override
+ public int getNumValues() {
+ return _numDocs;
+ }
+
+ @Override
+ public int getMaxNumValuesPerMVEntry() {
+ return -1;
+ }
+
+ @Nullable
+ @Override
+ public Comparable getMinValue() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public Comparable getMaxValue() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public PartitionFunction getPartitionFunction() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public Set<Integer> getPartitions() {
+ return null;
+ }
+
+ @Override
+ public int getCardinality() {
+ return -1;
+ }
+ }
+
+ private static final class FunctionBasedForwardIndexReader implements ForwardIndexReader<ForwardIndexReaderContext> {
+ private final FieldSpec.DataType _storedType;
+ private final IntFunction<Object> _valueProvider;
+
+ private FunctionBasedForwardIndexReader(FieldSpec.DataType storedType, IntFunction<Object> valueProvider) {
+ _storedType = storedType;
+ _valueProvider = valueProvider;
+ }
+
+ private static <T> T coerceNumber(Object value, Function<Number, T> numberConverter, Function<String, T> parser) {
+ return value instanceof Number ? numberConverter.apply((Number) value) : parser.apply(String.valueOf(value));
+ }
Review Comment:
`FunctionBasedForwardIndexReader.coerceNumber()` doesn’t handle null values from the value provider. If a provider returns null for a numeric column, this will try to parse the string "null" and throw a `NumberFormatException` during query execution. Consider treating null as the column default (or rejecting nulls earlier) so providers can safely omit numeric values.
##########
pinot-plugins/pinot-system-table/src/main/java/org/apache/pinot/systemtable/provider/InstancesSystemTableProvider.java:
##########
@@ -0,0 +1,382 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.systemtable.provider;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.client.admin.PinotAdminClient;
+import org.apache.pinot.client.admin.PinotAdminTransport;
+import org.apache.pinot.common.systemtable.SystemTable;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.SystemTableProviderContext;
+import org.apache.pinot.common.systemtable.datasource.InMemorySystemTableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing Pinot instance metadata.
+ */
+@SystemTable
+public final class InstancesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(InstancesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.instances";
+
+ private static final String CONTROLLER_TIMEOUT_MS_PROPERTY = "pinot.systemtable.instances.controllerTimeoutMs";
+ private static final long DEFAULT_CONTROLLER_TIMEOUT_MS = Duration.ofSeconds(5).toMillis();
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("instanceId", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("host", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("port", FieldSpec.DataType.INT)
+ .addSingleValueDimension("state", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("tags", FieldSpec.DataType.STRING)
+ .build();
+
+ private @Nullable HelixAdmin _helixAdmin;
+ private @Nullable String _clusterName;
+ private List<String> _configuredControllerUrls = List.of();
+ private long _controllerTimeoutMs = DEFAULT_CONTROLLER_TIMEOUT_MS;
+ private final Map<String, PinotAdminClient> _adminClientCache = new ConcurrentHashMap<>();
+
+ public InstancesSystemTableProvider() {
+ }
+
+ @Override
+ public void init(SystemTableProviderContext context) {
+ _helixAdmin = context.getHelixAdmin();
+ _clusterName = context.getClusterName();
+ if (context.getConfig() != null) {
+ _controllerTimeoutMs =
+ context.getConfig().getProperty(CONTROLLER_TIMEOUT_MS_PROPERTY, DEFAULT_CONTROLLER_TIMEOUT_MS);
+ }
+ }
+
+ /**
+ * Package-private constructor for testing with overrides.
+ */
+ InstancesSystemTableProvider(@Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable List<String> controllerUrls) {
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _configuredControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public TableConfig getTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ }
+
+ @Override
+ public void close()
+ throws Exception {
+ for (Map.Entry<String, PinotAdminClient> entry : _adminClientCache.entrySet()) {
+ try {
+ entry.getValue().close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close admin client for {}: {}", entry.getKey(), e.toString());
+ }
+ }
+ _adminClientCache.clear();
+ }
+
+ @Override
+ public IndexSegment getDataSource() {
+ List<String> controllerBaseUrls = getControllerBaseUrls();
+ if (controllerBaseUrls.isEmpty()) {
+ return new InMemorySystemTableSegment(TABLE_NAME, SCHEMA, 0, Collections.emptyMap());
+ }
+
+ List<String> instances = fetchInstances(controllerBaseUrls);
+ if (instances.isEmpty()) {
+ return new InMemorySystemTableSegment(TABLE_NAME, SCHEMA, 0, Collections.emptyMap());
+ }
+
+ Set<String> liveInstances = fetchLiveInstances(controllerBaseUrls);
+
+ List<InstanceRow> rows = new ArrayList<>(instances.size());
+ for (String instanceId : instances) {
+ rows.add(fetchInstanceRow(instanceId, liveInstances, controllerBaseUrls));
+ }
Review Comment:
`getDataSource()` calls `fetchInstanceRow()` for every instance, and `fetchInstanceRow()` hits the controller `getInstance()` endpoint. This creates an N+1 request pattern (one list call + one call per instance) on every system.instances query, which can be slow and load the controller on large clusters. Consider using Helix (via the provided `HelixAdmin`/cluster name) to read instance configs/live instances in bulk, or add caching/batching to avoid per-instance controller requests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2575346399
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java:
##########
@@ -1033,6 +1046,246 @@ private void throwAccessDeniedError(long requestId, String query, RequestContext
throw new WebApplicationException("Permission denied." + failureMessage, Response.Status.FORBIDDEN);
}
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ private BrokerResponse handleSystemTableQuery(PinotQuery pinotQuery, String tableName,
+ RequestContext requestContext, @Nullable RequesterIdentity requesterIdentity, String query) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableProvider provider = _systemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return BrokerResponseNative.TABLE_DOES_NOT_EXIST;
+ }
+ try {
+ if (!isSupportedSystemTableQuery(pinotQuery)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION,
+ "System tables only support simple projection/filter/limit queries");
+ }
+ Schema systemSchema = provider.getSchema();
+ List<String> projectionColumns = extractProjectionColumns(pinotQuery, systemSchema);
+ int offset = Math.max(0, pinotQuery.getOffset());
+ int limit = Math.max(0, pinotQuery.getLimit());
+ SystemTableRequest systemTableRequest =
+ new SystemTableRequest(projectionColumns, toSystemTableFilter(pinotQuery.getFilterExpression()), offset,
+ limit);
+ SystemTableResponse systemTableResponse = provider.getRows(systemTableRequest);
+ BrokerResponseNative brokerResponse =
+ buildSystemTableBrokerResponse(tableName, systemSchema, projectionColumns, systemTableResponse,
+ requestContext);
+ brokerResponse.setTimeUsedMs(System.currentTimeMillis() - requestContext.getRequestArrivalTimeMillis());
+ _queryLogger.log(new QueryLogger.QueryLogParams(requestContext, tableName, brokerResponse,
+ QueryLogger.QueryLogParams.QueryEngine.SINGLE_STAGE, requesterIdentity, null));
+ return brokerResponse;
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table query {}: {}", tableName, e.getMessage(), e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+
+ private boolean isSupportedSystemTableQuery(PinotQuery pinotQuery) {
+ return (pinotQuery.getGroupByList() == null || pinotQuery.getGroupByList().isEmpty())
+ && pinotQuery.getHavingExpression() == null
+ && (pinotQuery.getOrderByList() == null || pinotQuery.getOrderByList().isEmpty());
+ }
+
+ private List<String> extractProjectionColumns(PinotQuery pinotQuery, Schema schema)
+ throws BadQueryRequestException {
+ List<String> projections = new ArrayList<>();
+ boolean hasStar = false;
+ List<Expression> selectList = pinotQuery.getSelectList();
+ if (CollectionUtils.isEmpty(selectList)) {
+ throw new BadQueryRequestException("System tables require a projection list");
+ }
+ for (Expression expression : selectList) {
+ Identifier identifier = expression.getIdentifier();
+ if (identifier != null) {
+ if ("*".equals(identifier.getName())) {
+ hasStar = true;
+ } else {
+ projections.add(identifier.getName());
+ }
+ continue;
+ }
+ Function function = expression.getFunctionCall();
+ if (function != null && "AS".equalsIgnoreCase(function.getOperator()) && !function.getOperands().isEmpty()) {
+ Identifier aliased = function.getOperands().get(0).getIdentifier();
+ if (aliased != null) {
+ projections.add(aliased.getName());
+ continue;
+ }
+ }
+ throw new BadQueryRequestException("System tables only support column projections or '*'");
+ }
+ if (hasStar || projections.isEmpty()) {
+ projections = new ArrayList<>(schema.getColumnNames());
+ }
+ List<String> normalized = new ArrayList<>(projections.size());
+ for (String column : projections) {
+ if (schema.hasColumn(column)) {
+ normalized.add(column);
+ continue;
+ }
+ String matched = null;
+ for (String schemaColumn : schema.getColumnNames()) {
+ if (schemaColumn.equalsIgnoreCase(column)) {
+ matched = schemaColumn;
+ break;
+ }
+ }
+ if (matched == null) {
+ throw new BadQueryRequestException("Unknown column in system table: " + column);
+ }
+ normalized.add(matched);
+ }
+ return normalized;
+ }
+
+ private BrokerResponseNative buildSystemTableBrokerResponse(String tableName, Schema schema,
+ List<String> projectionColumns, SystemTableResponse response, RequestContext requestContext) {
+ DataSchema dataSchema = buildSystemTableDataSchema(schema, projectionColumns);
+ List<GenericRow> rows = response != null ? response.getRows() : Collections.emptyList();
+ List<Object[]> resultRows = new ArrayList<>();
+ if (rows != null) {
+ for (GenericRow row : rows) {
+ Object[] values = new Object[projectionColumns.size()];
+ for (int i = 0; i < projectionColumns.size(); i++) {
+ values[i] = row != null ? row.getValue(projectionColumns.get(i)) : null;
Review Comment:
The null check for individual row elements (line 1162) is redundant since rows is already checked to be non-null and would not contain null elements. If null rows are possible, they should be filtered when building the list rather than handled during iteration.
```suggestion
values[i] = row.getValue(projectionColumns.get(i));
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,584 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTableFilter;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.apache.pinot.spi.systemtable.SystemTableRequest;
+import org.apache.pinot.spi.systemtable.SystemTableResponse;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("system.tables")
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return "system.tables";
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public SystemTableResponse getRows(SystemTableRequest request) {
+ if (_tableCache == null) {
+ return new SystemTableResponse(List.of(), System.currentTimeMillis(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ List<GenericRow> rows = new ArrayList<>(sortedTableNames.size());
+ int offset = request.getOffset();
+ int limit = request.getLimit();
+ if (limit == 0) {
+ return new SystemTableResponse(List.of(), System.currentTimeMillis(), 0);
+ }
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType);
+ if (!matchesFilter(request.getFilter(), stats, rawTableName)) {
+ continue;
+ }
+ GenericRow row = new GenericRow();
+ row.putValue("tableName", rawTableName);
+ row.putValue("type", stats._type);
+ row.putValue("status", stats._status);
+ row.putValue("segments", stats._segments);
+ row.putValue("totalDocs", stats._totalDocs);
+ row.putValue("reportedSize", stats._reportedSizeInBytes);
+ row.putValue("estimatedSize", stats._estimatedSizeInBytes);
+ row.putValue("storageTier", stats._storageTier);
+ row.putValue("brokerTenant", stats._brokerTenant);
+ row.putValue("serverTenant", stats._serverTenant);
+ row.putValue("replicas", stats._replicas);
+ row.putValue("tableConfig", stats._tableConfig);
+ if (offset > 0) {
+ offset--;
+ continue;
+ }
+ rows.add(row);
+ if (limit > 0 && rows.size() >= limit) {
+ break;
+ }
+ }
+ return new SystemTableResponse(rows, System.currentTimeMillis(), rows.size());
Review Comment:
The totalRows returned should be the count of all matching rows before applying offset/limit, but this returns rows.size() which is the count after applying limit. This will cause incorrect pagination metadata when offset/limit are used.
```suggestion
return new SystemTableResponse(rows, System.currentTimeMillis(), sortedTableNames.size());
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,584 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTableFilter;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.apache.pinot.spi.systemtable.SystemTableRequest;
+import org.apache.pinot.spi.systemtable.SystemTableResponse;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("system.tables")
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return "system.tables";
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public SystemTableResponse getRows(SystemTableRequest request) {
+ if (_tableCache == null) {
+ return new SystemTableResponse(List.of(), System.currentTimeMillis(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ List<GenericRow> rows = new ArrayList<>(sortedTableNames.size());
+ int offset = request.getOffset();
+ int limit = request.getLimit();
+ if (limit == 0) {
+ return new SystemTableResponse(List.of(), System.currentTimeMillis(), 0);
+ }
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType);
+ if (!matchesFilter(request.getFilter(), stats, rawTableName)) {
+ continue;
+ }
+ GenericRow row = new GenericRow();
+ row.putValue("tableName", rawTableName);
+ row.putValue("type", stats._type);
+ row.putValue("status", stats._status);
+ row.putValue("segments", stats._segments);
+ row.putValue("totalDocs", stats._totalDocs);
+ row.putValue("reportedSize", stats._reportedSizeInBytes);
+ row.putValue("estimatedSize", stats._estimatedSizeInBytes);
+ row.putValue("storageTier", stats._storageTier);
+ row.putValue("brokerTenant", stats._brokerTenant);
+ row.putValue("serverTenant", stats._serverTenant);
+ row.putValue("replicas", stats._replicas);
+ row.putValue("tableConfig", stats._tableConfig);
+ if (offset > 0) {
+ offset--;
+ continue;
+ }
Review Comment:
The offset logic is applied after the filter check, which means the offset is counting filtered rows. However, the totalRows count returned in line 171 uses rows.size() after pagination, which creates inconsistent pagination behavior. The totalRows should reflect all filtered rows before offset/limit, but currently it's rows after limit.
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,584 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTableFilter;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.apache.pinot.spi.systemtable.SystemTableRequest;
+import org.apache.pinot.spi.systemtable.SystemTableResponse;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("system.tables")
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return "system.tables";
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public SystemTableResponse getRows(SystemTableRequest request) {
+ if (_tableCache == null) {
+ return new SystemTableResponse(List.of(), System.currentTimeMillis(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ List<GenericRow> rows = new ArrayList<>(sortedTableNames.size());
+ int offset = request.getOffset();
+ int limit = request.getLimit();
+ if (limit == 0) {
+ return new SystemTableResponse(List.of(), System.currentTimeMillis(), 0);
+ }
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType);
+ if (!matchesFilter(request.getFilter(), stats, rawTableName)) {
+ continue;
+ }
+ GenericRow row = new GenericRow();
+ row.putValue("tableName", rawTableName);
+ row.putValue("type", stats._type);
+ row.putValue("status", stats._status);
+ row.putValue("segments", stats._segments);
+ row.putValue("totalDocs", stats._totalDocs);
+ row.putValue("reportedSize", stats._reportedSizeInBytes);
+ row.putValue("estimatedSize", stats._estimatedSizeInBytes);
+ row.putValue("storageTier", stats._storageTier);
+ row.putValue("brokerTenant", stats._brokerTenant);
+ row.putValue("serverTenant", stats._serverTenant);
+ row.putValue("replicas", stats._replicas);
+ row.putValue("tableConfig", stats._tableConfig);
+ if (offset > 0) {
+ offset--;
+ continue;
+ }
+ rows.add(row);
+ if (limit > 0 && rows.size() >= limit) {
+ break;
+ }
+ }
+ return new SystemTableResponse(rows, System.currentTimeMillis(), rows.size());
+ }
+
+ private TableStats buildStats(String tableNameWithType, TableType tableType) {
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ int segments = 0;
+ long totalDocs = 0;
+ long reportedSize = 0;
+ long estimatedSize = 0;
+ String tierValue = "";
+ String brokerTenant = "";
+ String serverTenant = "";
+ int replicas = 0;
+ if (tableConfig != null && tableConfig.getTenantConfig() != null) {
+ brokerTenant = tableConfig.getTenantConfig().getBroker();
+ serverTenant = tableConfig.getTenantConfig().getServer();
+ }
+ if (tableConfig != null && tableConfig.getValidationConfig() != null) {
+ Integer repl = tableConfig.getValidationConfig().getReplicationNumber();
+ replicas = repl != null ? repl : replicas;
+ }
+ // Use controller API only
+ TableSize sizeFromController = fetchTableSize(tableNameWithType);
+ if (sizeFromController != null) {
+ if (sizeFromController._reportedSizeInBytes >= 0) {
+ reportedSize = sizeFromController._reportedSizeInBytes;
+ }
+ if (sizeFromController._estimatedSizeInBytes >= 0) {
+ estimatedSize = sizeFromController._estimatedSizeInBytes;
+ }
+ segments = getSegmentCount(sizeFromController, tableType);
+ totalDocs = getTotalDocs(sizeFromController, tableType);
+ }
+ String status = tableConfig != null ? "ONLINE" : (segments > 0 ? "ONLINE" : "UNKNOWN");
+ String tableConfigJson = "";
+ if (tableConfig != null) {
+ try {
+ tableConfigJson = JsonUtils.objectToString(tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", tableNameWithType, e.toString());
+ tableConfigJson = tableConfig.toString();
+ }
+ }
+ return new TableStats(tableType.name(), status, segments, totalDocs, reportedSize, estimatedSize, tierValue,
+ tableConfigJson, brokerTenant, serverTenant, replicas);
+ }
+
+ private boolean matchesFilter(SystemTableFilter filter, TableStats stats, String rawTableName) {
+ if (filter == null) {
+ return true;
+ }
+ if (filter.getChildren().isEmpty()) {
+ return matchesLeafFilter(filter, stats, rawTableName);
+ }
+ switch (filter.getOperator()) {
+ case AND:
+ for (SystemTableFilter child : filter.getChildren()) {
+ if (!matchesFilter(child, stats, rawTableName)) {
+ return false;
+ }
+ }
+ return true;
+ case OR:
+ for (SystemTableFilter child : filter.getChildren()) {
+ if (matchesFilter(child, stats, rawTableName)) {
+ return true;
+ }
+ }
+ return false;
+ case NOT:
+ return filter.getChildren().isEmpty() || !matchesFilter(filter.getChildren().get(0), stats, rawTableName);
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesLeafFilter(SystemTableFilter filter, TableStats stats, String rawTableName) {
+ String column = filter.getColumn();
+ if (column == null) {
+ return true;
+ }
+ List<String> values = filter.getValues();
+ if (values == null || values.isEmpty()) {
+ return true;
+ }
+ switch (column.toLowerCase()) {
+ case "tablename":
+ return matchesString(values, rawTableName, filter.getOperator());
+ case "type":
+ return matchesString(values, stats._type, filter.getOperator());
+ case "status":
+ return matchesString(values, stats._status, filter.getOperator());
+ case "segments":
+ return matchesNumber(values, stats._segments, filter.getOperator());
+ case "reportedsize":
+ return matchesNumber(values, stats._reportedSizeInBytes, filter.getOperator());
+ case "estimatedsize":
+ return matchesNumber(values, stats._estimatedSizeInBytes, filter.getOperator());
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesString(List<String> candidates, String actual, SystemTableFilter.Operator operator) {
+ switch (operator) {
+ case EQ:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case IN:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case NEQ:
+ return candidates.stream().noneMatch(v -> v.equalsIgnoreCase(actual));
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesNumber(List<String> candidates, long actual, SystemTableFilter.Operator operator) {
+ try {
+ switch (operator) {
+ case EQ:
+ return candidates.stream().anyMatch(v -> Long.parseLong(v) == actual);
+ case NEQ:
+ return candidates.stream().noneMatch(v -> Long.parseLong(v) == actual);
+ case GT:
+ return candidates.stream().anyMatch(v -> actual > Long.parseLong(v));
+ case GTE:
+ return candidates.stream().anyMatch(v -> actual >= Long.parseLong(v));
+ case LT:
+ return candidates.stream().anyMatch(v -> actual < Long.parseLong(v));
+ case LTE:
+ return candidates.stream().anyMatch(v -> actual <= Long.parseLong(v));
+ case IN:
+ for (String candidate : candidates) {
+ if (actual == Long.parseLong(candidate)) {
+ return true;
+ }
+ }
+ return false;
+ default:
+ return true;
+ }
+ } catch (NumberFormatException e) {
+ LOGGER.debug("Failed to parse numeric filter value {}: {}", candidates, e.toString());
+ return true;
+ }
+ }
+
+ private @Nullable TableSize fetchTableSize(String tableNameWithType) {
+ Function<String, TableSize> fetcher = getSizeFetcher();
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(tableNameWithType);
+ if (size == null) {
+ LOGGER.warn("system.tables: failed to fetch size for {} (raw: {}), falling back to zeros",
+ tableNameWithType, rawTableName);
+ }
+ }
+ return size;
+ }
+
+ private @Nullable TableSize fetchTableSizeForName(String tableName) {
+ for (String baseUrl : getControllerBaseUrls()) {
+ try {
+ String url = baseUrl + "/tables/" + tableName + "/size?verbose=true&includeReplacedSegments=false";
+ HttpRequest request = HttpRequest.newBuilder(URI.create(url))
+ .timeout(Duration.ofSeconds(5))
+ .GET()
+ .header("Accept", "application/json")
+ .build();
+ HttpResponse<String> response = _httpClient.send(request, HttpResponse.BodyHandlers.ofString());
+ if (response.statusCode() >= 200 && response.statusCode() < 300) {
+ TableSize parsed = JsonUtils.stringToObject(response.body(), TableSize.class);
+ LOGGER.info("system.tables: controller size response for {} via {} -> segments offline={}, realtime={}, "
+ + "reportedSize={}, estimatedSize={}", tableName, baseUrl,
+ parsed._offlineSegments != null && parsed._offlineSegments._segments != null
+ ? parsed._offlineSegments._segments.size() : 0,
+ parsed._realtimeSegments != null && parsed._realtimeSegments._segments != null
+ ? parsed._realtimeSegments._segments.size() : 0,
+ parsed._reportedSizeInBytes, parsed._estimatedSizeInBytes);
Review Comment:
[nitpick] This info-level log on every table size fetch could be noisy in production with many tables. Consider using debug level or adding a sample rate to reduce log volume.
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,584 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTableFilter;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.apache.pinot.spi.systemtable.SystemTableRequest;
+import org.apache.pinot.spi.systemtable.SystemTableResponse;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("system.tables")
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return "system.tables";
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public SystemTableResponse getRows(SystemTableRequest request) {
+ if (_tableCache == null) {
+ return new SystemTableResponse(List.of(), System.currentTimeMillis(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ List<GenericRow> rows = new ArrayList<>(sortedTableNames.size());
+ int offset = request.getOffset();
+ int limit = request.getLimit();
+ if (limit == 0) {
+ return new SystemTableResponse(List.of(), System.currentTimeMillis(), 0);
+ }
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType);
+ if (!matchesFilter(request.getFilter(), stats, rawTableName)) {
+ continue;
+ }
+ GenericRow row = new GenericRow();
+ row.putValue("tableName", rawTableName);
+ row.putValue("type", stats._type);
+ row.putValue("status", stats._status);
+ row.putValue("segments", stats._segments);
+ row.putValue("totalDocs", stats._totalDocs);
+ row.putValue("reportedSize", stats._reportedSizeInBytes);
+ row.putValue("estimatedSize", stats._estimatedSizeInBytes);
+ row.putValue("storageTier", stats._storageTier);
+ row.putValue("brokerTenant", stats._brokerTenant);
+ row.putValue("serverTenant", stats._serverTenant);
+ row.putValue("replicas", stats._replicas);
+ row.putValue("tableConfig", stats._tableConfig);
+ if (offset > 0) {
+ offset--;
+ continue;
+ }
+ rows.add(row);
+ if (limit > 0 && rows.size() >= limit) {
+ break;
+ }
+ }
+ return new SystemTableResponse(rows, System.currentTimeMillis(), rows.size());
+ }
+
+ private TableStats buildStats(String tableNameWithType, TableType tableType) {
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ int segments = 0;
+ long totalDocs = 0;
+ long reportedSize = 0;
+ long estimatedSize = 0;
+ String tierValue = "";
+ String brokerTenant = "";
+ String serverTenant = "";
+ int replicas = 0;
+ if (tableConfig != null && tableConfig.getTenantConfig() != null) {
+ brokerTenant = tableConfig.getTenantConfig().getBroker();
+ serverTenant = tableConfig.getTenantConfig().getServer();
+ }
+ if (tableConfig != null && tableConfig.getValidationConfig() != null) {
+ Integer repl = tableConfig.getValidationConfig().getReplicationNumber();
+ replicas = repl != null ? repl : replicas;
+ }
+ // Use controller API only
+ TableSize sizeFromController = fetchTableSize(tableNameWithType);
+ if (sizeFromController != null) {
+ if (sizeFromController._reportedSizeInBytes >= 0) {
+ reportedSize = sizeFromController._reportedSizeInBytes;
+ }
+ if (sizeFromController._estimatedSizeInBytes >= 0) {
+ estimatedSize = sizeFromController._estimatedSizeInBytes;
+ }
+ segments = getSegmentCount(sizeFromController, tableType);
+ totalDocs = getTotalDocs(sizeFromController, tableType);
+ }
+ String status = tableConfig != null ? "ONLINE" : (segments > 0 ? "ONLINE" : "UNKNOWN");
+ String tableConfigJson = "";
+ if (tableConfig != null) {
+ try {
+ tableConfigJson = JsonUtils.objectToString(tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", tableNameWithType, e.toString());
+ tableConfigJson = tableConfig.toString();
+ }
+ }
+ return new TableStats(tableType.name(), status, segments, totalDocs, reportedSize, estimatedSize, tierValue,
+ tableConfigJson, brokerTenant, serverTenant, replicas);
+ }
+
+ private boolean matchesFilter(SystemTableFilter filter, TableStats stats, String rawTableName) {
+ if (filter == null) {
+ return true;
+ }
+ if (filter.getChildren().isEmpty()) {
+ return matchesLeafFilter(filter, stats, rawTableName);
+ }
+ switch (filter.getOperator()) {
+ case AND:
+ for (SystemTableFilter child : filter.getChildren()) {
+ if (!matchesFilter(child, stats, rawTableName)) {
+ return false;
+ }
+ }
+ return true;
+ case OR:
+ for (SystemTableFilter child : filter.getChildren()) {
+ if (matchesFilter(child, stats, rawTableName)) {
+ return true;
+ }
+ }
+ return false;
+ case NOT:
+ return filter.getChildren().isEmpty() || !matchesFilter(filter.getChildren().get(0), stats, rawTableName);
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesLeafFilter(SystemTableFilter filter, TableStats stats, String rawTableName) {
+ String column = filter.getColumn();
+ if (column == null) {
+ return true;
+ }
+ List<String> values = filter.getValues();
+ if (values == null || values.isEmpty()) {
+ return true;
+ }
+ switch (column.toLowerCase()) {
+ case "tablename":
+ return matchesString(values, rawTableName, filter.getOperator());
+ case "type":
+ return matchesString(values, stats._type, filter.getOperator());
+ case "status":
+ return matchesString(values, stats._status, filter.getOperator());
+ case "segments":
+ return matchesNumber(values, stats._segments, filter.getOperator());
+ case "reportedsize":
+ return matchesNumber(values, stats._reportedSizeInBytes, filter.getOperator());
+ case "estimatedsize":
+ return matchesNumber(values, stats._estimatedSizeInBytes, filter.getOperator());
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesString(List<String> candidates, String actual, SystemTableFilter.Operator operator) {
+ switch (operator) {
+ case EQ:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case IN:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case NEQ:
+ return candidates.stream().noneMatch(v -> v.equalsIgnoreCase(actual));
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesNumber(List<String> candidates, long actual, SystemTableFilter.Operator operator) {
+ try {
+ switch (operator) {
+ case EQ:
+ return candidates.stream().anyMatch(v -> Long.parseLong(v) == actual);
+ case NEQ:
+ return candidates.stream().noneMatch(v -> Long.parseLong(v) == actual);
+ case GT:
+ return candidates.stream().anyMatch(v -> actual > Long.parseLong(v));
+ case GTE:
+ return candidates.stream().anyMatch(v -> actual >= Long.parseLong(v));
+ case LT:
+ return candidates.stream().anyMatch(v -> actual < Long.parseLong(v));
+ case LTE:
+ return candidates.stream().anyMatch(v -> actual <= Long.parseLong(v));
+ case IN:
+ for (String candidate : candidates) {
+ if (actual == Long.parseLong(candidate)) {
+ return true;
+ }
+ }
+ return false;
+ default:
+ return true;
+ }
+ } catch (NumberFormatException e) {
+ LOGGER.debug("Failed to parse numeric filter value {}: {}", candidates, e.toString());
+ return true;
+ }
+ }
+
+ private @Nullable TableSize fetchTableSize(String tableNameWithType) {
+ Function<String, TableSize> fetcher = getSizeFetcher();
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(tableNameWithType);
+ if (size == null) {
+ LOGGER.warn("system.tables: failed to fetch size for {} (raw: {}), falling back to zeros",
+ tableNameWithType, rawTableName);
+ }
+ }
+ return size;
+ }
+
+ private @Nullable TableSize fetchTableSizeForName(String tableName) {
+ for (String baseUrl : getControllerBaseUrls()) {
+ try {
+ String url = baseUrl + "/tables/" + tableName + "/size?verbose=true&includeReplacedSegments=false";
+ HttpRequest request = HttpRequest.newBuilder(URI.create(url))
+ .timeout(Duration.ofSeconds(5))
+ .GET()
+ .header("Accept", "application/json")
+ .build();
+ HttpResponse<String> response = _httpClient.send(request, HttpResponse.BodyHandlers.ofString());
+ if (response.statusCode() >= 200 && response.statusCode() < 300) {
+ TableSize parsed = JsonUtils.stringToObject(response.body(), TableSize.class);
+ LOGGER.info("system.tables: controller size response for {} via {} -> segments offline={}, realtime={}, "
+ + "reportedSize={}, estimatedSize={}", tableName, baseUrl,
+ parsed._offlineSegments != null && parsed._offlineSegments._segments != null
+ ? parsed._offlineSegments._segments.size() : 0,
+ parsed._realtimeSegments != null && parsed._realtimeSegments._segments != null
+ ? parsed._realtimeSegments._segments.size() : 0,
+ parsed._reportedSizeInBytes, parsed._estimatedSizeInBytes);
+ return parsed;
+ } else {
+ LOGGER.warn("system.tables: failed to fetch table size for {} via {}: status {}, body={}", tableName,
+ baseUrl, response.statusCode(), response.body());
+ }
+ } catch (Exception e) {
+ LOGGER.warn("system.tables: error fetching table size for {} via {}: {}", tableName, baseUrl, e.toString(), e);
Review Comment:
The error message format string has 3 placeholders but 4 arguments are provided (tableName, baseUrl, e.toString(), e). The last 'e' argument for stack trace won't match any placeholder, causing a formatting error.
```suggestion
LOGGER.warn("system.tables: error fetching table size for {} via {}", tableName, baseUrl, e);
```
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java:
##########
@@ -1033,6 +1046,246 @@ private void throwAccessDeniedError(long requestId, String query, RequestContext
throw new WebApplicationException("Permission denied." + failureMessage, Response.Status.FORBIDDEN);
}
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ private BrokerResponse handleSystemTableQuery(PinotQuery pinotQuery, String tableName,
+ RequestContext requestContext, @Nullable RequesterIdentity requesterIdentity, String query) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableProvider provider = _systemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return BrokerResponseNative.TABLE_DOES_NOT_EXIST;
+ }
+ try {
+ if (!isSupportedSystemTableQuery(pinotQuery)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION,
+ "System tables only support simple projection/filter/limit queries");
+ }
+ Schema systemSchema = provider.getSchema();
+ List<String> projectionColumns = extractProjectionColumns(pinotQuery, systemSchema);
+ int offset = Math.max(0, pinotQuery.getOffset());
+ int limit = Math.max(0, pinotQuery.getLimit());
+ SystemTableRequest systemTableRequest =
+ new SystemTableRequest(projectionColumns, toSystemTableFilter(pinotQuery.getFilterExpression()), offset,
+ limit);
+ SystemTableResponse systemTableResponse = provider.getRows(systemTableRequest);
+ BrokerResponseNative brokerResponse =
+ buildSystemTableBrokerResponse(tableName, systemSchema, projectionColumns, systemTableResponse,
+ requestContext);
+ brokerResponse.setTimeUsedMs(System.currentTimeMillis() - requestContext.getRequestArrivalTimeMillis());
+ _queryLogger.log(new QueryLogger.QueryLogParams(requestContext, tableName, brokerResponse,
+ QueryLogger.QueryLogParams.QueryEngine.SINGLE_STAGE, requesterIdentity, null));
+ return brokerResponse;
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table query {}: {}", tableName, e.getMessage(), e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+
+ private boolean isSupportedSystemTableQuery(PinotQuery pinotQuery) {
+ return (pinotQuery.getGroupByList() == null || pinotQuery.getGroupByList().isEmpty())
+ && pinotQuery.getHavingExpression() == null
+ && (pinotQuery.getOrderByList() == null || pinotQuery.getOrderByList().isEmpty());
+ }
+
+ private List<String> extractProjectionColumns(PinotQuery pinotQuery, Schema schema)
+ throws BadQueryRequestException {
+ List<String> projections = new ArrayList<>();
+ boolean hasStar = false;
+ List<Expression> selectList = pinotQuery.getSelectList();
+ if (CollectionUtils.isEmpty(selectList)) {
+ throw new BadQueryRequestException("System tables require a projection list");
+ }
+ for (Expression expression : selectList) {
+ Identifier identifier = expression.getIdentifier();
+ if (identifier != null) {
+ if ("*".equals(identifier.getName())) {
+ hasStar = true;
+ } else {
+ projections.add(identifier.getName());
+ }
+ continue;
+ }
+ Function function = expression.getFunctionCall();
+ if (function != null && "AS".equalsIgnoreCase(function.getOperator()) && !function.getOperands().isEmpty()) {
+ Identifier aliased = function.getOperands().get(0).getIdentifier();
+ if (aliased != null) {
+ projections.add(aliased.getName());
+ continue;
+ }
+ }
+ throw new BadQueryRequestException("System tables only support column projections or '*'");
+ }
+ if (hasStar || projections.isEmpty()) {
+ projections = new ArrayList<>(schema.getColumnNames());
+ }
+ List<String> normalized = new ArrayList<>(projections.size());
+ for (String column : projections) {
+ if (schema.hasColumn(column)) {
+ normalized.add(column);
+ continue;
+ }
+ String matched = null;
+ for (String schemaColumn : schema.getColumnNames()) {
+ if (schemaColumn.equalsIgnoreCase(column)) {
+ matched = schemaColumn;
+ break;
+ }
+ }
+ if (matched == null) {
+ throw new BadQueryRequestException("Unknown column in system table: " + column);
+ }
+ normalized.add(matched);
+ }
+ return normalized;
+ }
+
+ private BrokerResponseNative buildSystemTableBrokerResponse(String tableName, Schema schema,
+ List<String> projectionColumns, SystemTableResponse response, RequestContext requestContext) {
+ DataSchema dataSchema = buildSystemTableDataSchema(schema, projectionColumns);
+ List<GenericRow> rows = response != null ? response.getRows() : Collections.emptyList();
+ List<Object[]> resultRows = new ArrayList<>();
+ if (rows != null) {
+ for (GenericRow row : rows) {
+ Object[] values = new Object[projectionColumns.size()];
+ for (int i = 0; i < projectionColumns.size(); i++) {
+ values[i] = row != null ? row.getValue(projectionColumns.get(i)) : null;
+ }
+ resultRows.add(values);
+ }
+ }
+ BrokerResponseNative brokerResponse = new BrokerResponseNative();
+ brokerResponse.setResultTable(new ResultTable(dataSchema, resultRows));
+ brokerResponse.setNumDocsScanned(resultRows.size());
+ brokerResponse.setNumEntriesScannedPostFilter(resultRows.size());
Review Comment:
When response is null (line 1156), resultRows will be empty but numEntriesScannedPostFilter should be 0 by default. However, if response is non-null but contains null rows, this could be misleading. Consider using response.getTotalRows() when response is not null for consistency with totalDocs.
```suggestion
brokerResponse.setNumEntriesScannedPostFilter(response != null ? response.getTotalRows() : 0);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 closed pull request #17291: Add system tables support with registry and broker wiring
URL: https://github.com/apache/pinot/pull/17291
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2573627906
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java:
##########
@@ -1033,6 +1046,254 @@ private void throwAccessDeniedError(long requestId, String query, RequestContext
throw new WebApplicationException("Permission denied." + failureMessage, Response.Status.FORBIDDEN);
}
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ private BrokerResponse handleSystemTableQuery(PinotQuery pinotQuery, String tableName,
+ RequestContext requestContext, @Nullable RequesterIdentity requesterIdentity, String query) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableProvider provider = _systemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return BrokerResponseNative.TABLE_DOES_NOT_EXIST;
+ }
+ try {
+ if (!isSupportedSystemTableQuery(pinotQuery)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION,
+ "System tables only support simple projection/filter/limit queries");
+ }
+ Schema systemSchema = provider.getSchema();
+ List<String> projectionColumns = extractProjectionColumns(pinotQuery, systemSchema);
+ int offset = Math.max(0, pinotQuery.getOffset());
+ int limit = Math.max(0, pinotQuery.getLimit());
+ SystemTableRequest systemTableRequest =
+ new SystemTableRequest(projectionColumns, toSystemTableFilter(pinotQuery.getFilterExpression()), offset,
+ limit);
+ SystemTableResponse systemTableResponse = provider.getRows(systemTableRequest);
+ BrokerResponseNative brokerResponse =
+ buildSystemTableBrokerResponse(tableName, systemSchema, projectionColumns, systemTableResponse, offset,
+ limit, requestContext);
+ brokerResponse.setTimeUsedMs(System.currentTimeMillis() - requestContext.getRequestArrivalTimeMillis());
+ _queryLogger.log(new QueryLogger.QueryLogParams(requestContext, tableName, brokerResponse,
+ QueryLogger.QueryLogParams.QueryEngine.SINGLE_STAGE, requesterIdentity, null));
+ return brokerResponse;
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table query {}: {}", tableName, e.getMessage(), e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+
+ private boolean isSupportedSystemTableQuery(PinotQuery pinotQuery) {
+ return (pinotQuery.getGroupByList() == null || pinotQuery.getGroupByList().isEmpty())
+ && pinotQuery.getHavingExpression() == null
+ && (pinotQuery.getOrderByList() == null || pinotQuery.getOrderByList().isEmpty());
+ }
+
+ private List<String> extractProjectionColumns(PinotQuery pinotQuery, Schema schema)
+ throws BadQueryRequestException {
+ List<String> projections = new ArrayList<>();
+ boolean hasStar = false;
+ List<Expression> selectList = pinotQuery.getSelectList();
+ if (CollectionUtils.isEmpty(selectList)) {
+ throw new BadQueryRequestException("System tables require a projection list");
+ }
+ for (Expression expression : selectList) {
+ Identifier identifier = expression.getIdentifier();
+ if (identifier != null) {
+ if ("*".equals(identifier.getName())) {
+ hasStar = true;
+ } else {
+ projections.add(identifier.getName());
+ }
+ continue;
+ }
+ Function function = expression.getFunctionCall();
+ if (function != null && "AS".equalsIgnoreCase(function.getOperator()) && !function.getOperands().isEmpty()) {
+ Identifier aliased = function.getOperands().get(0).getIdentifier();
+ if (aliased != null) {
+ projections.add(aliased.getName());
+ continue;
+ }
+ }
+ throw new BadQueryRequestException("System tables only support column projections or '*'");
+ }
+ if (hasStar || projections.isEmpty()) {
+ projections = new ArrayList<>(schema.getColumnNames());
+ }
+ List<String> normalized = new ArrayList<>(projections.size());
+ for (String column : projections) {
+ if (schema.hasColumn(column)) {
+ normalized.add(column);
+ continue;
+ }
+ String matched = null;
+ for (String schemaColumn : schema.getColumnNames()) {
+ if (schemaColumn.equalsIgnoreCase(column)) {
+ matched = schemaColumn;
+ break;
+ }
+ }
+ if (matched == null) {
+ throw new BadQueryRequestException("Unknown column in system table: " + column);
+ }
+ normalized.add(matched);
+ }
+ return normalized;
+ }
+
+ private BrokerResponseNative buildSystemTableBrokerResponse(String tableName, Schema schema,
+ List<String> projectionColumns, SystemTableResponse response, int offset, int limit,
+ RequestContext requestContext) {
+ DataSchema dataSchema = buildSystemTableDataSchema(schema, projectionColumns);
+ List<GenericRow> rows = response != null ? response.getRows() : Collections.emptyList();
+ List<Object[]> resultRows = new ArrayList<>();
+ if (rows != null && limit != 0) {
+ int skipped = 0;
+ for (GenericRow row : rows) {
+ if (skipped++ < offset) {
+ continue;
+ }
+ if (limit > 0 && resultRows.size() >= limit) {
+ break;
+ }
Review Comment:
The offset is being applied in-memory after fetching all rows from the provider. Since SystemTableRequest includes offset, providers should handle offset to avoid fetching and iterating over rows that will be skipped.
```suggestion
List<String> projectionColumns, SystemTableResponse response,
RequestContext requestContext) {
DataSchema dataSchema = buildSystemTableDataSchema(schema, projectionColumns);
List<GenericRow> rows = response != null ? response.getRows() : Collections.emptyList();
List<Object[]> resultRows = new ArrayList<>();
if (rows != null) {
for (GenericRow row : rows) {
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,573 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTableFilter;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.apache.pinot.spi.systemtable.SystemTableRequest;
+import org.apache.pinot.spi.systemtable.SystemTableResponse;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("system.tables")
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return "system.tables";
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public SystemTableResponse getRows(SystemTableRequest request) {
+ if (_tableCache == null) {
+ return new SystemTableResponse(List.of(), System.currentTimeMillis(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ List<GenericRow> rows = new ArrayList<>(sortedTableNames.size());
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType);
+ if (!matchesFilter(request.getFilter(), stats, rawTableName)) {
+ continue;
+ }
+ GenericRow row = new GenericRow();
+ row.putValue("tableName", rawTableName);
+ row.putValue("type", stats._type);
+ row.putValue("status", stats._status);
+ row.putValue("segments", stats._segments);
+ row.putValue("totalDocs", stats._totalDocs);
+ row.putValue("reportedSize", stats._reportedSizeInBytes);
+ row.putValue("estimatedSize", stats._estimatedSizeInBytes);
+ row.putValue("storageTier", stats._storageTier);
+ row.putValue("brokerTenant", stats._brokerTenant);
+ row.putValue("serverTenant", stats._serverTenant);
+ row.putValue("replicas", stats._replicas);
+ row.putValue("tableConfig", stats._tableConfig);
+ rows.add(row);
+ }
+ return new SystemTableResponse(rows, System.currentTimeMillis(), rows.size());
+ }
+
+ private TableStats buildStats(String tableNameWithType, TableType tableType) {
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ int segments = 0;
+ long totalDocs = 0;
+ long reportedSize = 0;
+ long estimatedSize = 0;
+ String tierValue = "";
+ String brokerTenant = "";
+ String serverTenant = "";
+ int replicas = 0;
+ if (tableConfig != null && tableConfig.getTenantConfig() != null) {
+ brokerTenant = tableConfig.getTenantConfig().getBroker();
+ serverTenant = tableConfig.getTenantConfig().getServer();
+ }
+ if (tableConfig != null && tableConfig.getValidationConfig() != null) {
+ Integer repl = tableConfig.getValidationConfig().getReplicationNumber();
+ replicas = repl != null ? repl : replicas;
+ }
+ // Use controller API only
+ TableSize sizeFromController = fetchTableSize(tableNameWithType);
+ if (sizeFromController != null) {
+ if (sizeFromController._reportedSizeInBytes >= 0) {
+ reportedSize = sizeFromController._reportedSizeInBytes;
+ }
+ if (sizeFromController._estimatedSizeInBytes >= 0) {
+ estimatedSize = sizeFromController._estimatedSizeInBytes;
+ }
+ segments = getSegmentCount(sizeFromController, tableType);
+ totalDocs = getTotalDocs(sizeFromController, tableType);
+ }
+ String status = tableConfig != null ? "ONLINE" : (segments > 0 ? "ONLINE" : "UNKNOWN");
+ String tableConfigJson = "";
+ if (tableConfig != null) {
+ try {
+ tableConfigJson = JsonUtils.objectToString(tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", tableNameWithType, e.toString());
+ tableConfigJson = tableConfig.toString();
+ }
+ }
+ return new TableStats(tableType.name(), status, segments, totalDocs, reportedSize, estimatedSize, tierValue,
+ tableConfigJson, brokerTenant, serverTenant, replicas);
+ }
+
+ private boolean matchesFilter(SystemTableFilter filter, TableStats stats, String rawTableName) {
+ if (filter == null) {
+ return true;
+ }
+ if (filter.getChildren().isEmpty()) {
+ return matchesLeafFilter(filter, stats, rawTableName);
+ }
+ switch (filter.getOperator()) {
+ case AND:
+ for (SystemTableFilter child : filter.getChildren()) {
+ if (!matchesFilter(child, stats, rawTableName)) {
+ return false;
+ }
+ }
+ return true;
+ case OR:
+ for (SystemTableFilter child : filter.getChildren()) {
+ if (matchesFilter(child, stats, rawTableName)) {
+ return true;
+ }
+ }
+ return false;
+ case NOT:
+ return filter.getChildren().isEmpty() || !matchesFilter(filter.getChildren().get(0), stats, rawTableName);
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesLeafFilter(SystemTableFilter filter, TableStats stats, String rawTableName) {
+ String column = filter.getColumn();
+ if (column == null) {
+ return true;
+ }
+ List<String> values = filter.getValues();
+ if (values == null || values.isEmpty()) {
+ return true;
+ }
+ switch (column.toLowerCase()) {
+ case "tablename":
+ return matchesString(values, rawTableName, filter.getOperator());
+ case "type":
+ return matchesString(values, stats._type, filter.getOperator());
+ case "status":
+ return matchesString(values, stats._status, filter.getOperator());
+ case "segments":
+ return matchesNumber(values, stats._segments, filter.getOperator());
+ case "reportedsize":
+ return matchesNumber(values, stats._reportedSizeInBytes, filter.getOperator());
+ case "estimatedsize":
+ return matchesNumber(values, stats._estimatedSizeInBytes, filter.getOperator());
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesString(List<String> candidates, String actual, SystemTableFilter.Operator operator) {
+ switch (operator) {
+ case EQ:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case IN:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case NEQ:
+ return candidates.stream().noneMatch(v -> v.equalsIgnoreCase(actual));
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesNumber(List<String> candidates, long actual, SystemTableFilter.Operator operator) {
+ try {
+ switch (operator) {
+ case EQ:
+ return candidates.stream().anyMatch(v -> Long.parseLong(v) == actual);
+ case NEQ:
+ return candidates.stream().noneMatch(v -> Long.parseLong(v) == actual);
+ case GT:
+ return candidates.stream().anyMatch(v -> actual > Long.parseLong(v));
+ case GTE:
+ return candidates.stream().anyMatch(v -> actual >= Long.parseLong(v));
+ case LT:
+ return candidates.stream().anyMatch(v -> actual < Long.parseLong(v));
+ case LTE:
+ return candidates.stream().anyMatch(v -> actual <= Long.parseLong(v));
+ case IN:
+ for (String candidate : candidates) {
+ if (actual == Long.parseLong(candidate)) {
+ return true;
+ }
+ }
+ return false;
+ default:
+ return true;
+ }
+ } catch (NumberFormatException e) {
+ LOGGER.debug("Failed to parse numeric filter value {}: {}", candidates, e.toString());
+ return true;
+ }
+ }
+
+ private @Nullable TableSize fetchTableSize(String tableNameWithType) {
+ Function<String, TableSize> fetcher = getSizeFetcher();
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(tableNameWithType);
+ if (size == null) {
+ LOGGER.warn("system.tables: failed to fetch size for {} (raw: {}), falling back to zeros",
+ tableNameWithType, rawTableName);
+ }
+ }
+ return size;
+ }
+
+ private @Nullable TableSize fetchTableSizeForName(String tableName) {
+ for (String baseUrl : getControllerBaseUrls()) {
+ try {
+ String url = baseUrl + "/tables/" + tableName + "/size?verbose=true&includeReplacedSegments=false";
+ HttpRequest request = HttpRequest.newBuilder(URI.create(url))
+ .timeout(Duration.ofSeconds(5))
+ .GET()
+ .header("Accept", "application/json")
+ .build();
+ HttpResponse<String> response = _httpClient.send(request, HttpResponse.BodyHandlers.ofString());
+ if (response.statusCode() >= 200 && response.statusCode() < 300) {
+ TableSize parsed = JsonUtils.stringToObject(response.body(), TableSize.class);
+ LOGGER.info("system.tables: controller size response for {} via {} -> segments offline={}, realtime={}, "
+ + "reportedSize={}, estimatedSize={}", tableName, baseUrl,
+ parsed._offlineSegments != null && parsed._offlineSegments._segments != null
+ ? parsed._offlineSegments._segments.size() : 0,
+ parsed._realtimeSegments != null && parsed._realtimeSegments._segments != null
+ ? parsed._realtimeSegments._segments.size() : 0,
+ parsed._reportedSizeInBytes, parsed._estimatedSizeInBytes);
+ return parsed;
+ } else {
+ LOGGER.warn("system.tables: failed to fetch table size for {} via {}: status {}, body={}", tableName,
+ baseUrl, response.statusCode(), response.body());
+ }
+ } catch (Exception e) {
+ LOGGER.warn("system.tables: error fetching table size for {} via {}: {}", tableName, baseUrl, e.toString(), e);
+ System.out.println("system.tables fetch error for " + tableName + " via " + baseUrl + ": " + e);
Review Comment:
Debug output should use SLF4J logging instead of System.out. A warning is already logged on line 355, making this print redundant and inconsistent with the project's logging standards.
```suggestion
```
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java:
##########
@@ -1033,6 +1046,254 @@ private void throwAccessDeniedError(long requestId, String query, RequestContext
throw new WebApplicationException("Permission denied." + failureMessage, Response.Status.FORBIDDEN);
}
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ private BrokerResponse handleSystemTableQuery(PinotQuery pinotQuery, String tableName,
+ RequestContext requestContext, @Nullable RequesterIdentity requesterIdentity, String query) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableProvider provider = _systemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return BrokerResponseNative.TABLE_DOES_NOT_EXIST;
+ }
+ try {
+ if (!isSupportedSystemTableQuery(pinotQuery)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION,
+ "System tables only support simple projection/filter/limit queries");
+ }
+ Schema systemSchema = provider.getSchema();
+ List<String> projectionColumns = extractProjectionColumns(pinotQuery, systemSchema);
+ int offset = Math.max(0, pinotQuery.getOffset());
+ int limit = Math.max(0, pinotQuery.getLimit());
+ SystemTableRequest systemTableRequest =
+ new SystemTableRequest(projectionColumns, toSystemTableFilter(pinotQuery.getFilterExpression()), offset,
+ limit);
+ SystemTableResponse systemTableResponse = provider.getRows(systemTableRequest);
+ BrokerResponseNative brokerResponse =
+ buildSystemTableBrokerResponse(tableName, systemSchema, projectionColumns, systemTableResponse, offset,
+ limit, requestContext);
+ brokerResponse.setTimeUsedMs(System.currentTimeMillis() - requestContext.getRequestArrivalTimeMillis());
+ _queryLogger.log(new QueryLogger.QueryLogParams(requestContext, tableName, brokerResponse,
+ QueryLogger.QueryLogParams.QueryEngine.SINGLE_STAGE, requesterIdentity, null));
+ return brokerResponse;
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table query {}: {}", tableName, e.getMessage(), e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+
+ private boolean isSupportedSystemTableQuery(PinotQuery pinotQuery) {
+ return (pinotQuery.getGroupByList() == null || pinotQuery.getGroupByList().isEmpty())
+ && pinotQuery.getHavingExpression() == null
+ && (pinotQuery.getOrderByList() == null || pinotQuery.getOrderByList().isEmpty());
+ }
+
+ private List<String> extractProjectionColumns(PinotQuery pinotQuery, Schema schema)
+ throws BadQueryRequestException {
+ List<String> projections = new ArrayList<>();
+ boolean hasStar = false;
+ List<Expression> selectList = pinotQuery.getSelectList();
+ if (CollectionUtils.isEmpty(selectList)) {
+ throw new BadQueryRequestException("System tables require a projection list");
+ }
+ for (Expression expression : selectList) {
+ Identifier identifier = expression.getIdentifier();
+ if (identifier != null) {
+ if ("*".equals(identifier.getName())) {
+ hasStar = true;
+ } else {
+ projections.add(identifier.getName());
+ }
+ continue;
+ }
+ Function function = expression.getFunctionCall();
+ if (function != null && "AS".equalsIgnoreCase(function.getOperator()) && !function.getOperands().isEmpty()) {
+ Identifier aliased = function.getOperands().get(0).getIdentifier();
+ if (aliased != null) {
+ projections.add(aliased.getName());
+ continue;
+ }
+ }
+ throw new BadQueryRequestException("System tables only support column projections or '*'");
+ }
+ if (hasStar || projections.isEmpty()) {
+ projections = new ArrayList<>(schema.getColumnNames());
+ }
+ List<String> normalized = new ArrayList<>(projections.size());
+ for (String column : projections) {
+ if (schema.hasColumn(column)) {
+ normalized.add(column);
+ continue;
+ }
+ String matched = null;
+ for (String schemaColumn : schema.getColumnNames()) {
+ if (schemaColumn.equalsIgnoreCase(column)) {
+ matched = schemaColumn;
+ break;
+ }
+ }
+ if (matched == null) {
+ throw new BadQueryRequestException("Unknown column in system table: " + column);
+ }
+ normalized.add(matched);
+ }
+ return normalized;
+ }
+
+ private BrokerResponseNative buildSystemTableBrokerResponse(String tableName, Schema schema,
+ List<String> projectionColumns, SystemTableResponse response, int offset, int limit,
+ RequestContext requestContext) {
+ DataSchema dataSchema = buildSystemTableDataSchema(schema, projectionColumns);
+ List<GenericRow> rows = response != null ? response.getRows() : Collections.emptyList();
+ List<Object[]> resultRows = new ArrayList<>();
+ if (rows != null && limit != 0) {
+ int skipped = 0;
+ for (GenericRow row : rows) {
+ if (skipped++ < offset) {
+ continue;
+ }
+ if (limit > 0 && resultRows.size() >= limit) {
+ break;
+ }
Review Comment:
The limit is being applied in-memory after fetching all rows from the provider. Since SystemTableRequest includes limit, providers should handle limit to avoid fetching unnecessary rows.
```suggestion
if (rows != null) {
for (GenericRow row : rows) {
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,573 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTableFilter;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.apache.pinot.spi.systemtable.SystemTableRequest;
+import org.apache.pinot.spi.systemtable.SystemTableResponse;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("system.tables")
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return "system.tables";
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public SystemTableResponse getRows(SystemTableRequest request) {
+ if (_tableCache == null) {
+ return new SystemTableResponse(List.of(), System.currentTimeMillis(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ List<GenericRow> rows = new ArrayList<>(sortedTableNames.size());
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType);
+ if (!matchesFilter(request.getFilter(), stats, rawTableName)) {
+ continue;
+ }
+ GenericRow row = new GenericRow();
+ row.putValue("tableName", rawTableName);
+ row.putValue("type", stats._type);
+ row.putValue("status", stats._status);
+ row.putValue("segments", stats._segments);
+ row.putValue("totalDocs", stats._totalDocs);
+ row.putValue("reportedSize", stats._reportedSizeInBytes);
+ row.putValue("estimatedSize", stats._estimatedSizeInBytes);
+ row.putValue("storageTier", stats._storageTier);
+ row.putValue("brokerTenant", stats._brokerTenant);
+ row.putValue("serverTenant", stats._serverTenant);
+ row.putValue("replicas", stats._replicas);
+ row.putValue("tableConfig", stats._tableConfig);
+ rows.add(row);
+ }
+ return new SystemTableResponse(rows, System.currentTimeMillis(), rows.size());
+ }
+
+ private TableStats buildStats(String tableNameWithType, TableType tableType) {
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ int segments = 0;
+ long totalDocs = 0;
+ long reportedSize = 0;
+ long estimatedSize = 0;
+ String tierValue = "";
+ String brokerTenant = "";
+ String serverTenant = "";
+ int replicas = 0;
+ if (tableConfig != null && tableConfig.getTenantConfig() != null) {
+ brokerTenant = tableConfig.getTenantConfig().getBroker();
+ serverTenant = tableConfig.getTenantConfig().getServer();
+ }
+ if (tableConfig != null && tableConfig.getValidationConfig() != null) {
+ Integer repl = tableConfig.getValidationConfig().getReplicationNumber();
+ replicas = repl != null ? repl : replicas;
+ }
+ // Use controller API only
+ TableSize sizeFromController = fetchTableSize(tableNameWithType);
+ if (sizeFromController != null) {
+ if (sizeFromController._reportedSizeInBytes >= 0) {
+ reportedSize = sizeFromController._reportedSizeInBytes;
+ }
+ if (sizeFromController._estimatedSizeInBytes >= 0) {
+ estimatedSize = sizeFromController._estimatedSizeInBytes;
+ }
+ segments = getSegmentCount(sizeFromController, tableType);
+ totalDocs = getTotalDocs(sizeFromController, tableType);
+ }
+ String status = tableConfig != null ? "ONLINE" : (segments > 0 ? "ONLINE" : "UNKNOWN");
+ String tableConfigJson = "";
+ if (tableConfig != null) {
+ try {
+ tableConfigJson = JsonUtils.objectToString(tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", tableNameWithType, e.toString());
+ tableConfigJson = tableConfig.toString();
+ }
+ }
+ return new TableStats(tableType.name(), status, segments, totalDocs, reportedSize, estimatedSize, tierValue,
+ tableConfigJson, brokerTenant, serverTenant, replicas);
+ }
+
+ private boolean matchesFilter(SystemTableFilter filter, TableStats stats, String rawTableName) {
+ if (filter == null) {
+ return true;
+ }
+ if (filter.getChildren().isEmpty()) {
+ return matchesLeafFilter(filter, stats, rawTableName);
+ }
+ switch (filter.getOperator()) {
+ case AND:
+ for (SystemTableFilter child : filter.getChildren()) {
+ if (!matchesFilter(child, stats, rawTableName)) {
+ return false;
+ }
+ }
+ return true;
+ case OR:
+ for (SystemTableFilter child : filter.getChildren()) {
+ if (matchesFilter(child, stats, rawTableName)) {
+ return true;
+ }
+ }
+ return false;
+ case NOT:
+ return filter.getChildren().isEmpty() || !matchesFilter(filter.getChildren().get(0), stats, rawTableName);
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesLeafFilter(SystemTableFilter filter, TableStats stats, String rawTableName) {
+ String column = filter.getColumn();
+ if (column == null) {
+ return true;
+ }
+ List<String> values = filter.getValues();
+ if (values == null || values.isEmpty()) {
+ return true;
+ }
+ switch (column.toLowerCase()) {
+ case "tablename":
+ return matchesString(values, rawTableName, filter.getOperator());
+ case "type":
+ return matchesString(values, stats._type, filter.getOperator());
+ case "status":
+ return matchesString(values, stats._status, filter.getOperator());
+ case "segments":
+ return matchesNumber(values, stats._segments, filter.getOperator());
+ case "reportedsize":
+ return matchesNumber(values, stats._reportedSizeInBytes, filter.getOperator());
+ case "estimatedsize":
+ return matchesNumber(values, stats._estimatedSizeInBytes, filter.getOperator());
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesString(List<String> candidates, String actual, SystemTableFilter.Operator operator) {
+ switch (operator) {
+ case EQ:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case IN:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case NEQ:
+ return candidates.stream().noneMatch(v -> v.equalsIgnoreCase(actual));
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesNumber(List<String> candidates, long actual, SystemTableFilter.Operator operator) {
+ try {
+ switch (operator) {
+ case EQ:
+ return candidates.stream().anyMatch(v -> Long.parseLong(v) == actual);
+ case NEQ:
+ return candidates.stream().noneMatch(v -> Long.parseLong(v) == actual);
+ case GT:
+ return candidates.stream().anyMatch(v -> actual > Long.parseLong(v));
+ case GTE:
+ return candidates.stream().anyMatch(v -> actual >= Long.parseLong(v));
+ case LT:
+ return candidates.stream().anyMatch(v -> actual < Long.parseLong(v));
+ case LTE:
+ return candidates.stream().anyMatch(v -> actual <= Long.parseLong(v));
+ case IN:
+ for (String candidate : candidates) {
+ if (actual == Long.parseLong(candidate)) {
+ return true;
+ }
+ }
+ return false;
+ default:
+ return true;
+ }
+ } catch (NumberFormatException e) {
+ LOGGER.debug("Failed to parse numeric filter value {}: {}", candidates, e.toString());
+ return true;
+ }
+ }
+
+ private @Nullable TableSize fetchTableSize(String tableNameWithType) {
+ Function<String, TableSize> fetcher = getSizeFetcher();
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(tableNameWithType);
+ if (size == null) {
+ LOGGER.warn("system.tables: failed to fetch size for {} (raw: {}), falling back to zeros",
+ tableNameWithType, rawTableName);
+ }
+ }
+ return size;
+ }
+
+ private @Nullable TableSize fetchTableSizeForName(String tableName) {
+ for (String baseUrl : getControllerBaseUrls()) {
+ try {
+ String url = baseUrl + "/tables/" + tableName + "/size?verbose=true&includeReplacedSegments=false";
+ HttpRequest request = HttpRequest.newBuilder(URI.create(url))
+ .timeout(Duration.ofSeconds(5))
+ .GET()
+ .header("Accept", "application/json")
+ .build();
+ HttpResponse<String> response = _httpClient.send(request, HttpResponse.BodyHandlers.ofString());
+ if (response.statusCode() >= 200 && response.statusCode() < 300) {
+ TableSize parsed = JsonUtils.stringToObject(response.body(), TableSize.class);
+ LOGGER.info("system.tables: controller size response for {} via {} -> segments offline={}, realtime={}, "
+ + "reportedSize={}, estimatedSize={}", tableName, baseUrl,
+ parsed._offlineSegments != null && parsed._offlineSegments._segments != null
+ ? parsed._offlineSegments._segments.size() : 0,
+ parsed._realtimeSegments != null && parsed._realtimeSegments._segments != null
+ ? parsed._realtimeSegments._segments.size() : 0,
+ parsed._reportedSizeInBytes, parsed._estimatedSizeInBytes);
+ return parsed;
+ } else {
+ LOGGER.warn("system.tables: failed to fetch table size for {} via {}: status {}, body={}", tableName,
+ baseUrl, response.statusCode(), response.body());
+ }
+ } catch (Exception e) {
+ LOGGER.warn("system.tables: error fetching table size for {} via {}: {}", tableName, baseUrl, e.toString(), e);
+ System.out.println("system.tables fetch error for " + tableName + " via " + baseUrl + ": " + e);
+ }
+ }
+ return null;
+ }
+
+ private List<String> getControllerBaseUrls() {
+ Set<String> urls = new LinkedHashSet<>();
+ if (_helixAdmin != null) {
+ for (String controller : discoverControllersFromHelix()) {
+ String normalized = normalizeControllerUrl(controller);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ }
+ for (String url : _staticControllerUrls) {
+ String normalized = normalizeControllerUrl(url);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ return new ArrayList<>(urls);
+ }
+
+ private int getSegmentCount(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.size();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.size();
+ }
+ return 0;
+ }
+
+ private long getTotalDocs(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ return 0;
+ }
+
+ private @Nullable Function<String, TableSize> getSizeFetcher() {
+ if (_tableSizeFetcherOverride != null) {
+ return _tableSizeFetcherOverride;
+ }
+ List<String> controllers = getControllerBaseUrls();
+ if (controllers.isEmpty() || !isAdminClientAvailable()) {
+ return null;
+ }
+ return tableNameWithType -> fetchWithAdminClient(controllers, tableNameWithType);
+ }
+
+ private List<String> discoverControllersFromHelix() {
+ List<String> urls = new ArrayList<>();
+ try {
+ if (_clusterName == null) {
+ LOGGER.warn("Cannot discover controllers without cluster name");
+ return List.of();
+ }
+ for (String controllerId : _helixAdmin.getInstancesInCluster(_clusterName)) {
+ if (!InstanceTypeUtils.isController(controllerId)) {
+ continue;
+ }
+ int idx = controllerId.lastIndexOf('_');
+ if (idx > 0 && idx < controllerId.length() - 1) {
+ String host = controllerId.substring(controllerId.indexOf('_') + 1, idx);
+ String port = controllerId.substring(idx + 1);
+ urls.add(host + ":" + port);
+ } else {
+ LOGGER.warn("Unable to parse controller address from instance id: {}", controllerId);
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Failed to discover controllers from Helix", e);
+ }
+ return urls;
+ }
+
+ private boolean isAdminClientAvailable() {
+ try {
+ Class.forName("org.apache.pinot.client.admin.PinotAdminClient");
+ return true;
+ } catch (ClassNotFoundException e) {
+ LOGGER.debug("PinotAdminClient not on classpath; falling back to HTTP for table size fetch");
+ return false;
+ }
Review Comment:
The method performs a reflective class lookup on every call to check classpath availability. Cache the result in a static final field to avoid repeated Class.forName operations.
```suggestion
private static final boolean IS_ADMIN_CLIENT_AVAILABLE;
static {
boolean available;
try {
Class.forName("org.apache.pinot.client.admin.PinotAdminClient");
available = true;
} catch (ClassNotFoundException e) {
available = false;
}
IS_ADMIN_CLIENT_AVAILABLE = available;
}
private boolean isAdminClientAvailable() {
if (!IS_ADMIN_CLIENT_AVAILABLE) {
LOGGER.debug("PinotAdminClient not on classpath; falling back to HTTP for table size fetch");
}
return IS_ADMIN_CLIENT_AVAILABLE;
```
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/DatabaseUtils.java:
##########
@@ -67,6 +67,10 @@ public static String translateTableName(String tableName, @Nullable String datab
case 2:
Preconditions.checkArgument(!tableSplit[1].isEmpty(), "Invalid table name '%s'", tableName);
String databasePrefix = tableSplit[0];
+ // Allow system tables to bypass database prefix validation so they can be queried regardless of header.
Review Comment:
[nitpick] Add a comment explaining that this bypass allows system tables to be queried regardless of database header, as mentioned in the inline comment on line 70 but would benefit from additional context about security implications or expected usage patterns.
```suggestion
/**
* Allow system tables to bypass database prefix validation so they can be queried regardless of header.
* System tables are globally accessible and are not subject to database-level access controls.
* This is intentional, as system tables typically provide metadata or monitoring information
* that should be available across all databases. Care should be taken to ensure that system tables
* do not expose sensitive information, as they are accessible regardless of the database context
* provided in the request header.
*/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #17291:
URL: https://github.com/apache/pinot/pull/17291#issuecomment-3592542443
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/17291?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `28.90625%` with `273 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 55.54%. Comparing base ([`0f1db95`](https://app.codecov.io/gh/apache/pinot/commit/0f1db956502cf98cce9bd1a4173b0967501dc707?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`2d58d97`](https://app.codecov.io/gh/apache/pinot/commit/2d58d97490f7c171dfb5d62f2fe0dae4f596509d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/17291?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ystemtable/provider/TablesSystemTableProvider.java](https://app.codecov.io/gh/apache/pinot/pull/17291?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Fsystemtable%2Fprovider%2FTablesSystemTableProvider.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vc3lzdGVtdGFibGUvcHJvdmlkZXIvVGFibGVzU3lzdGVtVGFibGVQcm92aWRlci5qYXZh) | 31.90% | [188 Missing and 19 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/17291?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...pache/pinot/spi/systemtable/SystemTableFilter.java](https://app.codecov.io/gh/apache/pinot/pull/17291?src=pr&el=tree&filepath=pinot-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fspi%2Fsystemtable%2FSystemTableFilter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvc3lzdGVtdGFibGUvU3lzdGVtVGFibGVGaWx0ZXIuamF2YQ==) | 0.00% | [27 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/17291?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../pinot/common/systemtable/SystemTableRegistry.java](https://app.codecov.io/gh/apache/pinot/pull/17291?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Fsystemtable%2FSystemTableRegistry.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vc3lzdGVtdGFibGUvU3lzdGVtVGFibGVSZWdpc3RyeS5qYXZh) | 0.00% | [13 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/17291?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...emtable/provider/InstancesSystemTableProvider.java](https://app.codecov.io/gh/apache/pinot/pull/17291?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Fsystemtable%2Fprovider%2FInstancesSystemTableProvider.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vc3lzdGVtdGFibGUvcHJvdmlkZXIvSW5zdGFuY2VzU3lzdGVtVGFibGVQcm92aWRlci5qYXZh) | 0.00% | [12 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/17291?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../pinot/spi/systemtable/SystemTableConfigUtils.java](https://app.codecov.io/gh/apache/pinot/pull/17291?src=pr&el=tree&filepath=pinot-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fspi%2Fsystemtable%2FSystemTableConfigUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvc3lzdGVtdGFibGUvU3lzdGVtVGFibGVDb25maWdVdGlscy5qYXZh) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/17291?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/pinot/spi/systemtable/SystemTableRequest.java](https://app.codecov.io/gh/apache/pinot/pull/17291?src=pr&el=tree&filepath=pinot-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fspi%2Fsystemtable%2FSystemTableRequest.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvc3lzdGVtdGFibGUvU3lzdGVtVGFibGVSZXF1ZXN0LmphdmE=) | 72.72% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/17291?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...a/org/apache/pinot/common/utils/DatabaseUtils.java](https://app.codecov.io/gh/apache/pinot/pull/17291?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Futils%2FDatabaseUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRGF0YWJhc2VVdGlscy5qYXZh) | 0.00% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/17291?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/pinot/spi/systemtable/SystemTableProvider.java](https://app.codecov.io/gh/apache/pinot/pull/17291?src=pr&el=tree&filepath=pinot-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fspi%2Fsystemtable%2FSystemTableProvider.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvc3lzdGVtdGFibGUvU3lzdGVtVGFibGVQcm92aWRlci5qYXZh) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/17291?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/pinot/spi/systemtable/SystemTableResponse.java](https://app.codecov.io/gh/apache/pinot/pull/17291?src=pr&el=tree&filepath=pinot-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fspi%2Fsystemtable%2FSystemTableResponse.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvc3lzdGVtdGFibGUvU3lzdGVtVGFibGVSZXNwb25zZS5qYXZh) | 75.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/17291?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (0f1db95) and HEAD (2d58d97). Click for more details.
>
> <details><summary>HEAD has 32 uploads less than BASE</summary>
>
>| Flag | BASE (0f1db95) | HEAD (2d58d97) |
>|------|------|------|
>|unittests|3|1|
>|java-11|5|0|
>|temurin|9|1|
>|unittests2|2|0|
>|java-21|4|1|
>|integration|6|0|
>|integration2|2|0|
>|integration1|2|0|
>|custom-integration1|2|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #17291 +/- ##
============================================
- Coverage 63.20% 55.54% -7.67%
+ Complexity 1430 702 -728
============================================
Files 3133 2454 -679
Lines 186079 138156 -47923
Branches 28408 21944 -6464
============================================
- Hits 117611 76738 -40873
+ Misses 59398 54977 -4421
+ Partials 9070 6441 -2629
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/17291/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/17291/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/17291/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/17291/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/17291/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/17291/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/17291/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.54% <28.90%> (+21.68%)` | :arrow_up: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/17291/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.54% <28.90%> (-7.67%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/17291/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.54% <28.90%> (-7.66%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/17291/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.54% <28.90%> (-0.07%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/17291/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/17291?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2617586921
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,597 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTableFilter;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.apache.pinot.spi.systemtable.SystemTableRequest;
+import org.apache.pinot.spi.systemtable.SystemTableResponse;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("system.tables")
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final boolean IS_ADMIN_CLIENT_AVAILABLE;
+ static {
+ boolean available;
+ try {
+ Class.forName("org.apache.pinot.client.admin.PinotAdminClient");
+ available = true;
+ } catch (ClassNotFoundException e) {
+ available = false;
+ }
+ IS_ADMIN_CLIENT_AVAILABLE = available;
Review Comment:
The class name check uses reflection to verify PinotAdminClient availability, but this is fragile and could break if the class is relocated or renamed. Consider using a more robust mechanism such as checking for the presence of a specific method or using a feature flag/configuration property instead of relying on ClassNotFoundException.
```suggestion
String adminClientEnabled = System.getProperty("pinot.admin.client.enabled");
if (adminClientEnabled == null) {
LOGGER.warn("System property 'pinot.admin.client.enabled' is not set. Defaulting PinotAdminClient availability to false.");
IS_ADMIN_CLIENT_AVAILABLE = false;
} else {
IS_ADMIN_CLIENT_AVAILABLE = Boolean.parseBoolean(adminClientEnabled);
}
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,597 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTableFilter;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.apache.pinot.spi.systemtable.SystemTableRequest;
+import org.apache.pinot.spi.systemtable.SystemTableResponse;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("system.tables")
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final boolean IS_ADMIN_CLIENT_AVAILABLE;
+ static {
+ boolean available;
+ try {
+ Class.forName("org.apache.pinot.client.admin.PinotAdminClient");
+ available = true;
+ } catch (ClassNotFoundException e) {
+ available = false;
+ }
+ IS_ADMIN_CLIENT_AVAILABLE = available;
+ }
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return "system.tables";
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public SystemTableResponse getRows(SystemTableRequest request) {
+ if (_tableCache == null) {
+ return new SystemTableResponse(List.of(), System.currentTimeMillis(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ int offset = Math.max(0, request.getOffset());
+ int limit = request.getLimit();
+ boolean hasLimit = limit > 0;
+ int totalRows = 0;
+ int initialCapacity = hasLimit ? Math.min(sortedTableNames.size(), limit) : 0;
Review Comment:
The initial capacity is set to 0 when there's no limit, which will cause the ArrayList to resize multiple times as elements are added. This is inefficient for large table lists. Set initialCapacity to sortedTableNames.size() when hasLimit is false to avoid unnecessary resizing.
```suggestion
int initialCapacity = hasLimit ? Math.min(sortedTableNames.size(), limit) : sortedTableNames.size();
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,597 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTableFilter;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.apache.pinot.spi.systemtable.SystemTableRequest;
+import org.apache.pinot.spi.systemtable.SystemTableResponse;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("system.tables")
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final boolean IS_ADMIN_CLIENT_AVAILABLE;
+ static {
+ boolean available;
+ try {
+ Class.forName("org.apache.pinot.client.admin.PinotAdminClient");
+ available = true;
+ } catch (ClassNotFoundException e) {
+ available = false;
+ }
+ IS_ADMIN_CLIENT_AVAILABLE = available;
+ }
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return "system.tables";
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public SystemTableResponse getRows(SystemTableRequest request) {
+ if (_tableCache == null) {
+ return new SystemTableResponse(List.of(), System.currentTimeMillis(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ int offset = Math.max(0, request.getOffset());
+ int limit = request.getLimit();
+ boolean hasLimit = limit > 0;
+ int totalRows = 0;
+ int initialCapacity = hasLimit ? Math.min(sortedTableNames.size(), limit) : 0;
+ List<GenericRow> rows = new ArrayList<>(initialCapacity);
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType);
+ if (!matchesFilter(request.getFilter(), stats, rawTableName)) {
+ continue;
+ }
+ totalRows++;
+ if (offset > 0) {
+ offset--;
+ continue;
+ }
+ if (limit == 0) {
+ continue;
+ }
+ if (hasLimit && rows.size() >= limit) {
+ continue;
+ }
+ GenericRow row = new GenericRow();
+ row.putValue("tableName", rawTableName);
+ row.putValue("type", stats._type);
+ row.putValue("status", stats._status);
+ row.putValue("segments", stats._segments);
+ row.putValue("totalDocs", stats._totalDocs);
+ row.putValue("reportedSize", stats._reportedSizeInBytes);
+ row.putValue("estimatedSize", stats._estimatedSizeInBytes);
+ row.putValue("storageTier", stats._storageTier);
+ row.putValue("brokerTenant", stats._brokerTenant);
+ row.putValue("serverTenant", stats._serverTenant);
+ row.putValue("replicas", stats._replicas);
+ row.putValue("tableConfig", stats._tableConfig);
+ rows.add(row);
+ }
+ return new SystemTableResponse(rows, System.currentTimeMillis(), totalRows);
+ }
+
+ private TableStats buildStats(String tableNameWithType, TableType tableType) {
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ int segments = 0;
+ long totalDocs = 0;
+ long reportedSize = 0;
+ long estimatedSize = 0;
+ String tierValue = "";
+ String brokerTenant = "";
+ String serverTenant = "";
+ int replicas = 0;
+ if (tableConfig != null && tableConfig.getTenantConfig() != null) {
+ brokerTenant = tableConfig.getTenantConfig().getBroker();
+ serverTenant = tableConfig.getTenantConfig().getServer();
+ }
+ if (tableConfig != null && tableConfig.getValidationConfig() != null) {
+ Integer repl = tableConfig.getValidationConfig().getReplicationNumber();
+ replicas = repl != null ? repl : replicas;
+ }
+ // Use controller API only
+ TableSize sizeFromController = fetchTableSize(tableNameWithType);
+ if (sizeFromController != null) {
+ if (sizeFromController._reportedSizeInBytes >= 0) {
+ reportedSize = sizeFromController._reportedSizeInBytes;
+ }
+ if (sizeFromController._estimatedSizeInBytes >= 0) {
+ estimatedSize = sizeFromController._estimatedSizeInBytes;
+ }
+ segments = getSegmentCount(sizeFromController, tableType);
+ totalDocs = getTotalDocs(sizeFromController, tableType);
+ }
+ String status = tableConfig != null ? "ONLINE" : (segments > 0 ? "ONLINE" : "UNKNOWN");
+ String tableConfigJson = "";
+ if (tableConfig != null) {
+ try {
+ tableConfigJson = JsonUtils.objectToString(tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", tableNameWithType, e.toString());
+ tableConfigJson = tableConfig.toString();
+ }
+ }
+ return new TableStats(tableType.name(), status, segments, totalDocs, reportedSize, estimatedSize, tierValue,
+ tableConfigJson, brokerTenant, serverTenant, replicas);
+ }
+
+ private boolean matchesFilter(SystemTableFilter filter, TableStats stats, String rawTableName) {
+ if (filter == null) {
+ return true;
+ }
+ if (filter.getChildren().isEmpty()) {
+ return matchesLeafFilter(filter, stats, rawTableName);
+ }
+ switch (filter.getOperator()) {
+ case AND:
+ for (SystemTableFilter child : filter.getChildren()) {
+ if (!matchesFilter(child, stats, rawTableName)) {
+ return false;
+ }
+ }
+ return true;
+ case OR:
+ for (SystemTableFilter child : filter.getChildren()) {
+ if (matchesFilter(child, stats, rawTableName)) {
+ return true;
+ }
+ }
+ return false;
+ case NOT:
+ return filter.getChildren().isEmpty() || !matchesFilter(filter.getChildren().get(0), stats, rawTableName);
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesLeafFilter(SystemTableFilter filter, TableStats stats, String rawTableName) {
+ String column = filter.getColumn();
+ if (column == null) {
+ return true;
+ }
+ List<String> values = filter.getValues();
+ if (values == null || values.isEmpty()) {
+ return true;
+ }
+ switch (column.toLowerCase()) {
+ case "tablename":
+ return matchesString(values, rawTableName, filter.getOperator());
+ case "type":
+ return matchesString(values, stats._type, filter.getOperator());
+ case "status":
+ return matchesString(values, stats._status, filter.getOperator());
+ case "segments":
+ return matchesNumber(values, stats._segments, filter.getOperator());
+ case "reportedsize":
+ return matchesNumber(values, stats._reportedSizeInBytes, filter.getOperator());
+ case "estimatedsize":
+ return matchesNumber(values, stats._estimatedSizeInBytes, filter.getOperator());
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesString(List<String> candidates, String actual, SystemTableFilter.Operator operator) {
+ switch (operator) {
+ case EQ:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case IN:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case NEQ:
+ return candidates.stream().noneMatch(v -> v.equalsIgnoreCase(actual));
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesNumber(List<String> candidates, long actual, SystemTableFilter.Operator operator) {
+ try {
+ switch (operator) {
+ case EQ:
+ return candidates.stream().anyMatch(v -> Long.parseLong(v) == actual);
+ case NEQ:
+ return candidates.stream().noneMatch(v -> Long.parseLong(v) == actual);
+ case GT:
+ return candidates.stream().anyMatch(v -> actual > Long.parseLong(v));
+ case GTE:
+ return candidates.stream().anyMatch(v -> actual >= Long.parseLong(v));
+ case LT:
+ return candidates.stream().anyMatch(v -> actual < Long.parseLong(v));
+ case LTE:
+ return candidates.stream().anyMatch(v -> actual <= Long.parseLong(v));
+ case IN:
+ for (String candidate : candidates) {
+ if (actual == Long.parseLong(candidate)) {
+ return true;
+ }
+ }
+ return false;
+ default:
+ return true;
+ }
+ } catch (NumberFormatException e) {
+ LOGGER.debug("Failed to parse numeric filter value {}: {}", candidates, e.toString());
+ return true;
+ }
+ }
+
+ private @Nullable TableSize fetchTableSize(String tableNameWithType) {
+ Function<String, TableSize> fetcher = getSizeFetcher();
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(tableNameWithType);
+ if (size == null) {
+ LOGGER.warn("system.tables: failed to fetch size for {} (raw: {}), falling back to zeros",
+ tableNameWithType, rawTableName);
+ }
+ }
+ return size;
+ }
+
+ private @Nullable TableSize fetchTableSizeForName(String tableName) {
+ for (String baseUrl : getControllerBaseUrls()) {
+ try {
+ String url = baseUrl + "/tables/" + tableName + "/size?verbose=true&includeReplacedSegments=false";
+ HttpRequest request = HttpRequest.newBuilder(URI.create(url))
+ .timeout(Duration.ofSeconds(5))
+ .GET()
+ .header("Accept", "application/json")
+ .build();
+ HttpResponse<String> response = _httpClient.send(request, HttpResponse.BodyHandlers.ofString());
+ if (response.statusCode() >= 200 && response.statusCode() < 300) {
+ TableSize parsed = JsonUtils.stringToObject(response.body(), TableSize.class);
+ LOGGER.debug("system.tables: controller size response for {} via {} -> segments offline={}, realtime={}, "
+ + "reportedSize={}, estimatedSize={}", tableName, baseUrl,
+ parsed._offlineSegments != null && parsed._offlineSegments._segments != null
+ ? parsed._offlineSegments._segments.size() : 0,
+ parsed._realtimeSegments != null && parsed._realtimeSegments._segments != null
+ ? parsed._realtimeSegments._segments.size() : 0,
+ parsed._reportedSizeInBytes, parsed._estimatedSizeInBytes);
+ return parsed;
+ } else {
+ LOGGER.warn("system.tables: failed to fetch table size for {} via {}: status {}, body={}", tableName,
+ baseUrl, response.statusCode(), response.body());
+ }
+ } catch (Exception e) {
+ LOGGER.warn("system.tables: error fetching table size for {} via {}", tableName, baseUrl, e);
+ }
+ }
+ return null;
+ }
+
+ private List<String> getControllerBaseUrls() {
+ Set<String> urls = new LinkedHashSet<>();
+ if (_helixAdmin != null) {
+ for (String controller : discoverControllersFromHelix()) {
+ String normalized = normalizeControllerUrl(controller);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ }
+ for (String url : _staticControllerUrls) {
+ String normalized = normalizeControllerUrl(url);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ return new ArrayList<>(urls);
+ }
+
+ private int getSegmentCount(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.size();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.size();
+ }
+ return 0;
+ }
+
+ private long getTotalDocs(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ return 0;
+ }
+
+ private @Nullable Function<String, TableSize> getSizeFetcher() {
+ if (_tableSizeFetcherOverride != null) {
+ return _tableSizeFetcherOverride;
+ }
+ List<String> controllers = getControllerBaseUrls();
+ if (controllers.isEmpty() || !isAdminClientAvailable()) {
+ return null;
+ }
+ return tableNameWithType -> fetchWithAdminClient(controllers, tableNameWithType);
+ }
+
+ private List<String> discoverControllersFromHelix() {
+ List<String> urls = new ArrayList<>();
+ try {
+ if (_clusterName == null) {
+ LOGGER.warn("Cannot discover controllers without cluster name");
+ return List.of();
+ }
+ for (String controllerId : _helixAdmin.getInstancesInCluster(_clusterName)) {
+ if (!InstanceTypeUtils.isController(controllerId)) {
+ continue;
+ }
+ int idx = controllerId.lastIndexOf('_');
+ if (idx > 0 && idx < controllerId.length() - 1) {
+ String host = controllerId.substring(controllerId.indexOf('_') + 1, idx);
+ String port = controllerId.substring(idx + 1);
+ urls.add(host + ":" + port);
+ } else {
+ LOGGER.warn("Unable to parse controller address from instance id: {}", controllerId);
+ }
+ }
Review Comment:
The parsing logic uses `indexOf('_')` for the host extraction but `lastIndexOf('_')` for validation and port extraction. If the controller ID has multiple underscores (e.g., "Controller_host_name_8080"), the host will be incorrectly extracted starting from the first underscore instead of after "Controller_". Use `idx` consistently or validate the controller ID format more carefully.
```suggestion
int firstUnderscoreIdx = controllerId.indexOf('_');
int lastUnderscoreIdx = controllerId.lastIndexOf('_');
// Ensure the controllerId starts with "Controller_", and has at least two underscores
if (firstUnderscoreIdx > 0 && lastUnderscoreIdx > firstUnderscoreIdx && lastUnderscoreIdx < controllerId.length() - 1) {
String host = controllerId.substring(firstUnderscoreIdx + 1, lastUnderscoreIdx);
String port = controllerId.substring(lastUnderscoreIdx + 1);
urls.add(host + ":" + port);
} else {
LOGGER.warn("Unable to parse controller address from instance id: {}", controllerId);
}
```
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java:
##########
@@ -1033,6 +1046,244 @@ private void throwAccessDeniedError(long requestId, String query, RequestContext
throw new WebApplicationException("Permission denied." + failureMessage, Response.Status.FORBIDDEN);
}
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ private BrokerResponse handleSystemTableQuery(PinotQuery pinotQuery, String tableName,
+ RequestContext requestContext, @Nullable RequesterIdentity requesterIdentity, String query) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableProvider provider = _systemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return BrokerResponseNative.TABLE_DOES_NOT_EXIST;
+ }
+ try {
+ if (!isSupportedSystemTableQuery(pinotQuery)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION,
+ "System tables only support simple projection/filter/limit queries");
+ }
+ Schema systemSchema = provider.getSchema();
+ List<String> projectionColumns = extractProjectionColumns(pinotQuery, systemSchema);
+ int offset = Math.max(0, pinotQuery.getOffset());
+ int limit = Math.max(0, pinotQuery.getLimit());
Review Comment:
When offset or limit are negative in the PinotQuery, they are clamped to 0. However, a limit of 0 has special semantics (no results returned) which may not be the intended behavior when the original limit was negative (typically meaning "no limit"). Consider using -1 or Integer.MAX_VALUE to represent "no limit" rather than 0.
```suggestion
int limit = pinotQuery.getLimit() < 0 ? -1 : pinotQuery.getLimit();
```
##########
pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BaseBrokerStarter.java:
##########
@@ -354,6 +357,12 @@ public void start()
boolean caseInsensitive =
_brokerConf.getProperty(Helix.ENABLE_CASE_INSENSITIVE_KEY, Helix.DEFAULT_ENABLE_CASE_INSENSITIVE);
TableCache tableCache = new ZkTableCache(_propertyStore, caseInsensitive);
+ if (!SystemTableRegistry.INSTANCE.isRegistered("system.tables")) {
+ SystemTableRegistry.INSTANCE.register(new TablesSystemTableProvider(tableCache, _helixAdmin, _clusterName));
+ }
+ if (!SystemTableRegistry.INSTANCE.isRegistered("system.instances")) {
+ SystemTableRegistry.INSTANCE.register(new InstancesSystemTableProvider());
Review Comment:
Using hardcoded string literals "system.tables" and "system.instances" for registration checks creates a risk of typos and makes the code harder to maintain. Define constants for these table names (e.g., in the provider classes or a shared constants file) and reference them here.
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java:
##########
@@ -1033,6 +1046,244 @@ private void throwAccessDeniedError(long requestId, String query, RequestContext
throw new WebApplicationException("Permission denied." + failureMessage, Response.Status.FORBIDDEN);
}
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ private BrokerResponse handleSystemTableQuery(PinotQuery pinotQuery, String tableName,
+ RequestContext requestContext, @Nullable RequesterIdentity requesterIdentity, String query) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableProvider provider = _systemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return BrokerResponseNative.TABLE_DOES_NOT_EXIST;
+ }
+ try {
+ if (!isSupportedSystemTableQuery(pinotQuery)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION,
+ "System tables only support simple projection/filter/limit queries");
+ }
+ Schema systemSchema = provider.getSchema();
+ List<String> projectionColumns = extractProjectionColumns(pinotQuery, systemSchema);
+ int offset = Math.max(0, pinotQuery.getOffset());
+ int limit = Math.max(0, pinotQuery.getLimit());
+ SystemTableRequest systemTableRequest =
+ new SystemTableRequest(projectionColumns, toSystemTableFilter(pinotQuery.getFilterExpression()), offset,
+ limit);
+ SystemTableResponse systemTableResponse = provider.getRows(systemTableRequest);
+ BrokerResponseNative brokerResponse =
+ buildSystemTableBrokerResponse(tableName, systemSchema, projectionColumns, systemTableResponse,
+ requestContext);
+ brokerResponse.setTimeUsedMs(System.currentTimeMillis() - requestContext.getRequestArrivalTimeMillis());
+ _queryLogger.log(new QueryLogger.QueryLogParams(requestContext, tableName, brokerResponse,
+ QueryLogger.QueryLogParams.QueryEngine.SINGLE_STAGE, requesterIdentity, null));
+ return brokerResponse;
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table query {}: {}", tableName, e.getMessage(), e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+
+ private boolean isSupportedSystemTableQuery(PinotQuery pinotQuery) {
+ return (pinotQuery.getGroupByList() == null || pinotQuery.getGroupByList().isEmpty())
+ && pinotQuery.getHavingExpression() == null
+ && (pinotQuery.getOrderByList() == null || pinotQuery.getOrderByList().isEmpty());
+ }
+
+ private List<String> extractProjectionColumns(PinotQuery pinotQuery, Schema schema)
+ throws BadQueryRequestException {
+ List<String> projections = new ArrayList<>();
+ boolean hasStar = false;
+ List<Expression> selectList = pinotQuery.getSelectList();
+ if (CollectionUtils.isEmpty(selectList)) {
+ throw new BadQueryRequestException("System tables require a projection list");
+ }
+ for (Expression expression : selectList) {
+ Identifier identifier = expression.getIdentifier();
+ if (identifier != null) {
+ if ("*".equals(identifier.getName())) {
+ hasStar = true;
+ } else {
+ projections.add(identifier.getName());
+ }
+ continue;
+ }
+ Function function = expression.getFunctionCall();
+ if (function != null && "AS".equalsIgnoreCase(function.getOperator()) && !function.getOperands().isEmpty()) {
+ Identifier aliased = function.getOperands().get(0).getIdentifier();
+ if (aliased != null) {
+ projections.add(aliased.getName());
+ continue;
+ }
+ }
+ throw new BadQueryRequestException("System tables only support column projections or '*'");
Review Comment:
The error message doesn't explain what was actually encountered that caused the failure. Include the problematic expression in the error message (e.g., "System tables only support column projections or '*', but found: " + expression) to help users debug their queries.
```suggestion
throw new BadQueryRequestException("System tables only support column projections or '*', but found: " + expression);
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,597 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTableFilter;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.apache.pinot.spi.systemtable.SystemTableRequest;
+import org.apache.pinot.spi.systemtable.SystemTableResponse;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("system.tables")
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final boolean IS_ADMIN_CLIENT_AVAILABLE;
+ static {
+ boolean available;
+ try {
+ Class.forName("org.apache.pinot.client.admin.PinotAdminClient");
+ available = true;
+ } catch (ClassNotFoundException e) {
+ available = false;
+ }
+ IS_ADMIN_CLIENT_AVAILABLE = available;
+ }
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return "system.tables";
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public SystemTableResponse getRows(SystemTableRequest request) {
+ if (_tableCache == null) {
+ return new SystemTableResponse(List.of(), System.currentTimeMillis(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ int offset = Math.max(0, request.getOffset());
+ int limit = request.getLimit();
+ boolean hasLimit = limit > 0;
+ int totalRows = 0;
+ int initialCapacity = hasLimit ? Math.min(sortedTableNames.size(), limit) : 0;
+ List<GenericRow> rows = new ArrayList<>(initialCapacity);
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType);
+ if (!matchesFilter(request.getFilter(), stats, rawTableName)) {
+ continue;
+ }
+ totalRows++;
+ if (offset > 0) {
+ offset--;
+ continue;
+ }
+ if (limit == 0) {
+ continue;
Review Comment:
When limit is 0, the loop continues to process remaining tables even though no more rows should be added. This is inefficient and wastes CPU cycles. Add a break statement here instead of continue to exit the loop early once the limit is reached.
```suggestion
break;
```
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java:
##########
@@ -1033,6 +1046,244 @@ private void throwAccessDeniedError(long requestId, String query, RequestContext
throw new WebApplicationException("Permission denied." + failureMessage, Response.Status.FORBIDDEN);
}
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ private BrokerResponse handleSystemTableQuery(PinotQuery pinotQuery, String tableName,
+ RequestContext requestContext, @Nullable RequesterIdentity requesterIdentity, String query) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableProvider provider = _systemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return BrokerResponseNative.TABLE_DOES_NOT_EXIST;
+ }
+ try {
+ if (!isSupportedSystemTableQuery(pinotQuery)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION,
+ "System tables only support simple projection/filter/limit queries");
+ }
+ Schema systemSchema = provider.getSchema();
+ List<String> projectionColumns = extractProjectionColumns(pinotQuery, systemSchema);
+ int offset = Math.max(0, pinotQuery.getOffset());
+ int limit = Math.max(0, pinotQuery.getLimit());
+ SystemTableRequest systemTableRequest =
+ new SystemTableRequest(projectionColumns, toSystemTableFilter(pinotQuery.getFilterExpression()), offset,
+ limit);
+ SystemTableResponse systemTableResponse = provider.getRows(systemTableRequest);
+ BrokerResponseNative brokerResponse =
+ buildSystemTableBrokerResponse(tableName, systemSchema, projectionColumns, systemTableResponse,
+ requestContext);
+ brokerResponse.setTimeUsedMs(System.currentTimeMillis() - requestContext.getRequestArrivalTimeMillis());
+ _queryLogger.log(new QueryLogger.QueryLogParams(requestContext, tableName, brokerResponse,
+ QueryLogger.QueryLogParams.QueryEngine.SINGLE_STAGE, requesterIdentity, null));
+ return brokerResponse;
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table query {}: {}", tableName, e.getMessage(), e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+
+ private boolean isSupportedSystemTableQuery(PinotQuery pinotQuery) {
+ return (pinotQuery.getGroupByList() == null || pinotQuery.getGroupByList().isEmpty())
+ && pinotQuery.getHavingExpression() == null
+ && (pinotQuery.getOrderByList() == null || pinotQuery.getOrderByList().isEmpty());
+ }
+
+ private List<String> extractProjectionColumns(PinotQuery pinotQuery, Schema schema)
+ throws BadQueryRequestException {
+ List<String> projections = new ArrayList<>();
+ boolean hasStar = false;
+ List<Expression> selectList = pinotQuery.getSelectList();
+ if (CollectionUtils.isEmpty(selectList)) {
+ throw new BadQueryRequestException("System tables require a projection list");
Review Comment:
The error message "System tables require a projection list" is unclear because empty select lists are already handled by the Pinot query parser and would fail earlier. If this check is defensive, clarify the message to indicate this is an internal error (e.g., "Internal error: system table query missing projection list").
```suggestion
throw new BadQueryRequestException("Internal error: system table query missing projection list");
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,597 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTableFilter;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.apache.pinot.spi.systemtable.SystemTableRequest;
+import org.apache.pinot.spi.systemtable.SystemTableResponse;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("system.tables")
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final boolean IS_ADMIN_CLIENT_AVAILABLE;
+ static {
+ boolean available;
+ try {
+ Class.forName("org.apache.pinot.client.admin.PinotAdminClient");
+ available = true;
+ } catch (ClassNotFoundException e) {
+ available = false;
+ }
+ IS_ADMIN_CLIENT_AVAILABLE = available;
+ }
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return "system.tables";
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public SystemTableResponse getRows(SystemTableRequest request) {
+ if (_tableCache == null) {
+ return new SystemTableResponse(List.of(), System.currentTimeMillis(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ int offset = Math.max(0, request.getOffset());
+ int limit = request.getLimit();
+ boolean hasLimit = limit > 0;
+ int totalRows = 0;
+ int initialCapacity = hasLimit ? Math.min(sortedTableNames.size(), limit) : 0;
+ List<GenericRow> rows = new ArrayList<>(initialCapacity);
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType);
+ if (!matchesFilter(request.getFilter(), stats, rawTableName)) {
+ continue;
+ }
+ totalRows++;
+ if (offset > 0) {
+ offset--;
+ continue;
+ }
+ if (limit == 0) {
+ continue;
+ }
+ if (hasLimit && rows.size() >= limit) {
+ continue;
+ }
+ GenericRow row = new GenericRow();
+ row.putValue("tableName", rawTableName);
+ row.putValue("type", stats._type);
+ row.putValue("status", stats._status);
+ row.putValue("segments", stats._segments);
+ row.putValue("totalDocs", stats._totalDocs);
+ row.putValue("reportedSize", stats._reportedSizeInBytes);
+ row.putValue("estimatedSize", stats._estimatedSizeInBytes);
+ row.putValue("storageTier", stats._storageTier);
+ row.putValue("brokerTenant", stats._brokerTenant);
+ row.putValue("serverTenant", stats._serverTenant);
+ row.putValue("replicas", stats._replicas);
+ row.putValue("tableConfig", stats._tableConfig);
+ rows.add(row);
+ }
+ return new SystemTableResponse(rows, System.currentTimeMillis(), totalRows);
+ }
+
+ private TableStats buildStats(String tableNameWithType, TableType tableType) {
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ int segments = 0;
+ long totalDocs = 0;
+ long reportedSize = 0;
+ long estimatedSize = 0;
+ String tierValue = "";
+ String brokerTenant = "";
+ String serverTenant = "";
+ int replicas = 0;
+ if (tableConfig != null && tableConfig.getTenantConfig() != null) {
+ brokerTenant = tableConfig.getTenantConfig().getBroker();
+ serverTenant = tableConfig.getTenantConfig().getServer();
+ }
+ if (tableConfig != null && tableConfig.getValidationConfig() != null) {
+ Integer repl = tableConfig.getValidationConfig().getReplicationNumber();
+ replicas = repl != null ? repl : replicas;
+ }
+ // Use controller API only
+ TableSize sizeFromController = fetchTableSize(tableNameWithType);
+ if (sizeFromController != null) {
+ if (sizeFromController._reportedSizeInBytes >= 0) {
+ reportedSize = sizeFromController._reportedSizeInBytes;
+ }
+ if (sizeFromController._estimatedSizeInBytes >= 0) {
+ estimatedSize = sizeFromController._estimatedSizeInBytes;
+ }
+ segments = getSegmentCount(sizeFromController, tableType);
+ totalDocs = getTotalDocs(sizeFromController, tableType);
+ }
+ String status = tableConfig != null ? "ONLINE" : (segments > 0 ? "ONLINE" : "UNKNOWN");
Review Comment:
The status determination logic is duplicated (both branches return "ONLINE" when their condition is true). Simplify this to `String status = (tableConfig != null || segments > 0) ? "ONLINE" : "UNKNOWN";` for better readability and maintainability.
```suggestion
String status = (tableConfig != null || segments > 0) ? "ONLINE" : "UNKNOWN";
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,597 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTableFilter;
+import org.apache.pinot.spi.systemtable.SystemTableProvider;
+import org.apache.pinot.spi.systemtable.SystemTableRequest;
+import org.apache.pinot.spi.systemtable.SystemTableResponse;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("system.tables")
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final boolean IS_ADMIN_CLIENT_AVAILABLE;
+ static {
+ boolean available;
+ try {
+ Class.forName("org.apache.pinot.client.admin.PinotAdminClient");
+ available = true;
+ } catch (ClassNotFoundException e) {
+ available = false;
+ }
+ IS_ADMIN_CLIENT_AVAILABLE = available;
+ }
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return "system.tables";
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public SystemTableResponse getRows(SystemTableRequest request) {
+ if (_tableCache == null) {
+ return new SystemTableResponse(List.of(), System.currentTimeMillis(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ int offset = Math.max(0, request.getOffset());
+ int limit = request.getLimit();
+ boolean hasLimit = limit > 0;
+ int totalRows = 0;
+ int initialCapacity = hasLimit ? Math.min(sortedTableNames.size(), limit) : 0;
+ List<GenericRow> rows = new ArrayList<>(initialCapacity);
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType);
+ if (!matchesFilter(request.getFilter(), stats, rawTableName)) {
+ continue;
+ }
+ totalRows++;
+ if (offset > 0) {
+ offset--;
+ continue;
+ }
+ if (limit == 0) {
+ continue;
+ }
+ if (hasLimit && rows.size() >= limit) {
+ continue;
+ }
+ GenericRow row = new GenericRow();
+ row.putValue("tableName", rawTableName);
+ row.putValue("type", stats._type);
+ row.putValue("status", stats._status);
+ row.putValue("segments", stats._segments);
+ row.putValue("totalDocs", stats._totalDocs);
+ row.putValue("reportedSize", stats._reportedSizeInBytes);
+ row.putValue("estimatedSize", stats._estimatedSizeInBytes);
+ row.putValue("storageTier", stats._storageTier);
+ row.putValue("brokerTenant", stats._brokerTenant);
+ row.putValue("serverTenant", stats._serverTenant);
+ row.putValue("replicas", stats._replicas);
+ row.putValue("tableConfig", stats._tableConfig);
+ rows.add(row);
+ }
+ return new SystemTableResponse(rows, System.currentTimeMillis(), totalRows);
+ }
+
+ private TableStats buildStats(String tableNameWithType, TableType tableType) {
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ int segments = 0;
+ long totalDocs = 0;
+ long reportedSize = 0;
+ long estimatedSize = 0;
+ String tierValue = "";
+ String brokerTenant = "";
+ String serverTenant = "";
+ int replicas = 0;
+ if (tableConfig != null && tableConfig.getTenantConfig() != null) {
+ brokerTenant = tableConfig.getTenantConfig().getBroker();
+ serverTenant = tableConfig.getTenantConfig().getServer();
+ }
+ if (tableConfig != null && tableConfig.getValidationConfig() != null) {
+ Integer repl = tableConfig.getValidationConfig().getReplicationNumber();
+ replicas = repl != null ? repl : replicas;
+ }
+ // Use controller API only
+ TableSize sizeFromController = fetchTableSize(tableNameWithType);
+ if (sizeFromController != null) {
+ if (sizeFromController._reportedSizeInBytes >= 0) {
+ reportedSize = sizeFromController._reportedSizeInBytes;
+ }
+ if (sizeFromController._estimatedSizeInBytes >= 0) {
+ estimatedSize = sizeFromController._estimatedSizeInBytes;
+ }
+ segments = getSegmentCount(sizeFromController, tableType);
+ totalDocs = getTotalDocs(sizeFromController, tableType);
+ }
+ String status = tableConfig != null ? "ONLINE" : (segments > 0 ? "ONLINE" : "UNKNOWN");
+ String tableConfigJson = "";
+ if (tableConfig != null) {
+ try {
+ tableConfigJson = JsonUtils.objectToString(tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", tableNameWithType, e.toString());
+ tableConfigJson = tableConfig.toString();
+ }
+ }
+ return new TableStats(tableType.name(), status, segments, totalDocs, reportedSize, estimatedSize, tierValue,
+ tableConfigJson, brokerTenant, serverTenant, replicas);
+ }
+
+ private boolean matchesFilter(SystemTableFilter filter, TableStats stats, String rawTableName) {
+ if (filter == null) {
+ return true;
+ }
+ if (filter.getChildren().isEmpty()) {
+ return matchesLeafFilter(filter, stats, rawTableName);
+ }
+ switch (filter.getOperator()) {
+ case AND:
+ for (SystemTableFilter child : filter.getChildren()) {
+ if (!matchesFilter(child, stats, rawTableName)) {
+ return false;
+ }
+ }
+ return true;
+ case OR:
+ for (SystemTableFilter child : filter.getChildren()) {
+ if (matchesFilter(child, stats, rawTableName)) {
+ return true;
+ }
+ }
+ return false;
+ case NOT:
+ return filter.getChildren().isEmpty() || !matchesFilter(filter.getChildren().get(0), stats, rawTableName);
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesLeafFilter(SystemTableFilter filter, TableStats stats, String rawTableName) {
+ String column = filter.getColumn();
+ if (column == null) {
+ return true;
+ }
+ List<String> values = filter.getValues();
+ if (values == null || values.isEmpty()) {
+ return true;
+ }
+ switch (column.toLowerCase()) {
+ case "tablename":
+ return matchesString(values, rawTableName, filter.getOperator());
+ case "type":
+ return matchesString(values, stats._type, filter.getOperator());
+ case "status":
+ return matchesString(values, stats._status, filter.getOperator());
+ case "segments":
+ return matchesNumber(values, stats._segments, filter.getOperator());
+ case "reportedsize":
+ return matchesNumber(values, stats._reportedSizeInBytes, filter.getOperator());
+ case "estimatedsize":
+ return matchesNumber(values, stats._estimatedSizeInBytes, filter.getOperator());
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesString(List<String> candidates, String actual, SystemTableFilter.Operator operator) {
+ switch (operator) {
+ case EQ:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case IN:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case NEQ:
+ return candidates.stream().noneMatch(v -> v.equalsIgnoreCase(actual));
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesNumber(List<String> candidates, long actual, SystemTableFilter.Operator operator) {
+ try {
+ switch (operator) {
+ case EQ:
+ return candidates.stream().anyMatch(v -> Long.parseLong(v) == actual);
+ case NEQ:
+ return candidates.stream().noneMatch(v -> Long.parseLong(v) == actual);
+ case GT:
+ return candidates.stream().anyMatch(v -> actual > Long.parseLong(v));
+ case GTE:
+ return candidates.stream().anyMatch(v -> actual >= Long.parseLong(v));
+ case LT:
+ return candidates.stream().anyMatch(v -> actual < Long.parseLong(v));
+ case LTE:
+ return candidates.stream().anyMatch(v -> actual <= Long.parseLong(v));
+ case IN:
+ for (String candidate : candidates) {
+ if (actual == Long.parseLong(candidate)) {
+ return true;
+ }
+ }
+ return false;
+ default:
+ return true;
+ }
+ } catch (NumberFormatException e) {
+ LOGGER.debug("Failed to parse numeric filter value {}: {}", candidates, e.toString());
+ return true;
+ }
+ }
+
+ private @Nullable TableSize fetchTableSize(String tableNameWithType) {
+ Function<String, TableSize> fetcher = getSizeFetcher();
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(tableNameWithType);
+ if (size == null) {
+ LOGGER.warn("system.tables: failed to fetch size for {} (raw: {}), falling back to zeros",
Review Comment:
The warning message says "falling back to zeros" but the code actually returns null when size fetching fails, not zero values. Update the message to accurately reflect the behavior (e.g., "failed to fetch size, returning null") or change the code to return a TableSize object with zero values as the message suggests.
```suggestion
LOGGER.warn("system.tables: failed to fetch size for {} (raw: {}), returning null",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2640570977
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/SystemTableBrokerRequestHandler.java:
##########
@@ -0,0 +1,188 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.requesthandler;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.util.Collections;
+import java.util.Locale;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.Executor;
+import javax.annotation.Nullable;
+import javax.ws.rs.core.HttpHeaders;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.hc.client5.http.io.HttpClientConnectionManager;
+import org.apache.pinot.broker.api.AccessControl;
+import org.apache.pinot.broker.broker.AccessControlFactory;
+import org.apache.pinot.broker.querylog.QueryLogger;
+import org.apache.pinot.broker.queryquota.QueryQuotaManager;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.metrics.BrokerMeter;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.response.BrokerResponse;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.SystemTableRegistry;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.core.routing.RoutingManager;
+import org.apache.pinot.spi.accounting.ThreadAccountant;
+import org.apache.pinot.spi.auth.AuthorizationResult;
+import org.apache.pinot.spi.auth.broker.RequesterIdentity;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.trace.RequestContext;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+import org.apache.pinot.sql.parsers.SqlNodeAndOptions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Broker request handler for system tables (handled entirely on the broker).
+ */
+public class SystemTableBrokerRequestHandler extends BaseBrokerRequestHandler {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableBrokerRequestHandler.class);
+
+ public SystemTableBrokerRequestHandler(PinotConfiguration config, String brokerId,
+ BrokerRequestIdGenerator requestIdGenerator, RoutingManager routingManager,
+ AccessControlFactory accessControlFactory, QueryQuotaManager queryQuotaManager, TableCache tableCache,
+ ThreadAccountant threadAccountant) {
+ super(config, brokerId, requestIdGenerator, routingManager, accessControlFactory, queryQuotaManager, tableCache,
+ threadAccountant);
+ }
+
+ @Override
+ public void start() {
+ }
+
+ @Override
+ public void shutDown() {
+ }
+
+ public boolean canHandle(String tableName) {
+ return isSystemTable(tableName) && SystemTableRegistry.isRegistered(tableName);
+ }
+
+ @Override
+ protected BrokerResponse handleRequest(long requestId, String query, SqlNodeAndOptions sqlNodeAndOptions,
+ JsonNode request, @Nullable RequesterIdentity requesterIdentity, RequestContext requestContext,
+ @Nullable HttpHeaders httpHeaders, AccessControl accessControl)
+ throws Exception {
+ PinotQuery pinotQuery;
+ try {
+ pinotQuery = CalciteSqlParser.compileToPinotQuery(sqlNodeAndOptions);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return new BrokerResponseNative(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ Set<String> tableNames = RequestUtils.getTableNames(pinotQuery);
+ if (tableNames == null || tableNames.isEmpty()) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "Failed to extract table name");
+ }
+ if (tableNames.size() != 1) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "System tables do not support joins");
+ }
+ String tableName = tableNames.iterator().next();
+ if (!isSystemTable(tableName)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "Not a system table query");
+ }
+ AuthorizationResult authorizationResult =
+ hasTableAccess(requesterIdentity, Set.of(tableName), requestContext, httpHeaders);
+ if (!authorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return new BrokerResponseNative(QueryErrorCode.ACCESS_DENIED, authorizationResult.getFailureMessage());
+ }
+
+ return handleSystemTableQuery(pinotQuery, tableName, requestContext, requesterIdentity, query);
+ }
+
+ @Override
+ protected boolean handleCancel(long queryId, int timeoutMs, Executor executor,
+ HttpClientConnectionManager connMgr, Map<String, Integer> serverResponses) {
+ return false;
+ }
+
+ @Override
+ public boolean cancelQueryByClientId(String clientQueryId, int timeoutMs, Executor executor,
+ HttpClientConnectionManager connMgr, Map<String, Integer> serverResponses)
+ throws Exception {
+ return false;
+ }
+
+ @Override
+ public Map<Long, String> getRunningQueries() {
+ return Collections.emptyMap();
+ }
+
+ @Override
+ public OptionalLong getRequestIdByClientId(String clientQueryId) {
+ return OptionalLong.empty();
+ }
+
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ private BrokerResponse handleSystemTableQuery(PinotQuery pinotQuery, String tableName, RequestContext requestContext,
+ @Nullable RequesterIdentity requesterIdentity, String query) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableDataProvider provider = SystemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return BrokerResponseNative.TABLE_DOES_NOT_EXIST;
+ }
+ try {
+ if (!isSupportedSystemTableQuery(pinotQuery)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION,
+ "System tables only support simple projection/filter/limit queries");
Review Comment:
The error message does not specify which unsupported features were used (e.g., GROUP BY, ORDER BY, HAVING). Include the specific unsupported clause(s) found to help users understand the issue.
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BrokerRequestHandlerDelegate.java:
##########
@@ -106,7 +113,25 @@ public BrokerResponse handleRequest(JsonNode request, @Nullable SqlNodeAndOption
}
}
+ Set<String> tableNames = null;
+ if (_systemTableBrokerRequestHandler != null) {
+ try {
+ // NOTE: compileToPinotQuery(SqlNodeAndOptions) mutates the SqlNode (e.g. SqlOrderBy -> SqlSelect).
+ // Re-parse from raw SQL so the SqlNodeAndOptions passed to the handler is unchanged (important for MSE).
Review Comment:
The comment references 'MSE' without defining the acronym. Add '(Multi-Stage Engine)' after the first usage to clarify for future maintainers unfamiliar with this shorthand.
```suggestion
// Re-parse from raw SQL so the SqlNodeAndOptions passed to the handler is unchanged (important for MSE (Multi-Stage Engine)).
```
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/DatabaseUtils.java:
##########
@@ -67,6 +67,19 @@ public static String translateTableName(String tableName, @Nullable String datab
case 2:
Preconditions.checkArgument(!tableSplit[1].isEmpty(), "Invalid table name '%s'", tableName);
String databasePrefix = tableSplit[0];
+ /*
+ * Allow system tables to bypass database prefix validation so they can be queried regardless of the database
+ * header. System tables are intended to be globally accessible and are not subject to database-level access
+ * controls. Ensure system tables do not expose sensitive information because they can be queried without a
+ * matching database context.
+ *
+ * Security note: system tables are intentionally exempt from database-scoped access control; broader Pinot
+ * security is handled by AccessControl implementations (see org.apache.pinot.spi.security.AccessControl) and
+ * the Pinot security model documentation. New system tables should be vetted for sensitive content.
+ */
+ if ("system".equalsIgnoreCase(databasePrefix)) {
+ return tableName;
Review Comment:
The newly added system table bypass logic lacks corresponding test coverage. Add a test case verifying that system.* tables are correctly exempted from database prefix validation.
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,726 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.SystemTableResponseUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.FilterKind;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableDataProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final long SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final boolean IS_ADMIN_CLIENT_AVAILABLE;
+ static {
+ boolean available;
+ try {
+ Class.forName("org.apache.pinot.client.admin.PinotAdminClient");
+ available = true;
+ } catch (ClassNotFoundException e) {
+ available = false;
+ }
+ IS_ADMIN_CLIENT_AVAILABLE = available;
+ }
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+ private final Map<String, CachedSize> _sizeCache = new ConcurrentHashMap<>();
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public TableConfig getTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ }
+
+ @Override
+ public BrokerResponseNative getBrokerResponse(PinotQuery pinotQuery) {
+ List<String> projectionColumns = pinotQuery.getSelectList() != null
+ ? pinotQuery.getSelectList().stream().map(expr -> expr.getIdentifier().getName()).collect(Collectors.toList())
+ : List.of();
+ if (_tableCache == null) {
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, List.of(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ int offset = Math.max(0, pinotQuery.getOffset());
+ int limit = pinotQuery.getLimit();
+ boolean hasLimit = limit > 0;
+ int totalRows = 0;
+ int initialCapacity = hasLimit ? Math.min(sortedTableNames.size(), limit) : 0;
+ List<GenericRow> rows = new ArrayList<>(initialCapacity);
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType);
+ if (!matchesFilter(pinotQuery.getFilterExpression(), stats, rawTableName)) {
+ continue;
+ }
+ if (offset > 0) {
+ offset--;
+ totalRows++;
+ continue;
+ }
+ totalRows++;
+ if (limit == 0) {
+ continue;
+ }
+ if (hasLimit && rows.size() >= limit) {
+ continue;
+ }
+ GenericRow row = new GenericRow();
+ row.putValue("tableName", rawTableName);
+ row.putValue("type", stats._type);
+ row.putValue("status", stats._status);
+ row.putValue("segments", stats._segments);
+ row.putValue("totalDocs", stats._totalDocs);
+ row.putValue("reportedSize", stats._reportedSizeInBytes);
+ row.putValue("estimatedSize", stats._estimatedSizeInBytes);
+ row.putValue("storageTier", stats._storageTier);
+ row.putValue("brokerTenant", stats._brokerTenant);
+ row.putValue("serverTenant", stats._serverTenant);
+ row.putValue("replicas", stats._replicas);
+ row.putValue("tableConfig", stats._tableConfig);
+ rows.add(row);
+ }
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, rows, totalRows);
+ }
+
+ private TableStats buildStats(String tableNameWithType, TableType tableType) {
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ int segments = 0;
+ long totalDocs = 0;
+ long reportedSize = 0;
+ long estimatedSize = 0;
+ String tierValue = "";
+ String brokerTenant = "";
+ String serverTenant = "";
+ int replicas = 0;
+ if (tableConfig != null && tableConfig.getTenantConfig() != null) {
+ brokerTenant = tableConfig.getTenantConfig().getBroker();
+ serverTenant = tableConfig.getTenantConfig().getServer();
+ }
+ if (tableConfig != null && tableConfig.getValidationConfig() != null) {
+ Integer repl = tableConfig.getValidationConfig().getReplicationNumber();
+ replicas = repl != null ? repl : replicas;
+ }
+ // Use controller API only
+ TableSize sizeFromController = fetchTableSize(tableNameWithType);
+ if (sizeFromController != null) {
+ if (sizeFromController._reportedSizeInBytes >= 0) {
+ reportedSize = sizeFromController._reportedSizeInBytes;
+ }
+ if (sizeFromController._estimatedSizeInBytes >= 0) {
+ estimatedSize = sizeFromController._estimatedSizeInBytes;
+ }
+ segments = getSegmentCount(sizeFromController, tableType);
+ totalDocs = getTotalDocs(sizeFromController, tableType);
+ }
+ String status = tableConfig != null ? "ONLINE" : (segments > 0 ? "ONLINE" : "UNKNOWN");
+ String tableConfigJson = "";
+ if (tableConfig != null) {
+ try {
+ tableConfigJson = JsonUtils.objectToString(tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", tableNameWithType, e.toString());
+ tableConfigJson = tableConfig.toString();
+ }
+ }
+ return new TableStats(tableType.name(), status, segments, totalDocs, reportedSize, estimatedSize, tierValue,
+ tableConfigJson, brokerTenant, serverTenant, replicas);
+ }
+
+ private boolean matchesFilter(@Nullable org.apache.pinot.common.request.Expression filterExpression, TableStats stats,
+ String rawTableName) {
+ if (filterExpression == null) {
+ return true;
+ }
+ org.apache.pinot.common.request.Function function = filterExpression.getFunctionCall();
+ if (function == null) {
+ return true;
+ }
+ FilterKind filterKind = toFilterKind(function.getOperator());
+ if (filterKind == null) {
+ return true;
+ }
+ switch (filterKind) {
+ case AND:
+ for (org.apache.pinot.common.request.Expression child : function.getOperands()) {
+ if (!matchesFilter(child, stats, rawTableName)) {
+ return false;
+ }
+ }
+ return true;
+ case OR:
+ for (org.apache.pinot.common.request.Expression child : function.getOperands()) {
+ if (matchesFilter(child, stats, rawTableName)) {
+ return true;
+ }
+ }
+ return false;
+ case NOT:
+ if (function.getOperandsSize() == 0) {
+ return true;
+ }
+ return !matchesFilter(function.getOperands().get(0), stats, rawTableName);
+ default:
+ return matchesLeafFilter(filterKind, function.getOperands(), stats, rawTableName);
+ }
+ }
+
+ private boolean matchesLeafFilter(FilterKind filterKind,
+ List<org.apache.pinot.common.request.Expression> operands, TableStats stats, String rawTableName) {
+ String column = extractIdentifier(operands);
+ if (column == null) {
+ return true;
+ }
+ List<String> values = extractLiteralValues(operands);
+ if (values.isEmpty()) {
+ return true;
+ }
+ switch (column.toLowerCase(Locale.ROOT)) {
+ case "tablename":
+ return matchesString(values, rawTableName, filterKind);
+ case "type":
+ return matchesString(values, stats._type, filterKind);
+ case "status":
+ return matchesString(values, stats._status, filterKind);
+ case "segments":
+ return matchesNumber(values, stats._segments, filterKind);
+ case "reportedsize":
+ return matchesNumber(values, stats._reportedSizeInBytes, filterKind);
+ case "estimatedsize":
+ return matchesNumber(values, stats._estimatedSizeInBytes, filterKind);
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesString(List<String> candidates, String actual, FilterKind filterKind) {
+ switch (filterKind) {
+ case EQUALS:
+ case IN:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case NOT_EQUALS:
+ return candidates.stream().noneMatch(v -> v.equalsIgnoreCase(actual));
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesNumber(List<String> candidates, long actual, FilterKind filterKind) {
+ try {
+ switch (filterKind) {
+ case EQUALS:
+ return candidates.stream().anyMatch(v -> Long.parseLong(v) == actual);
+ case NOT_EQUALS:
+ return candidates.stream().noneMatch(v -> Long.parseLong(v) == actual);
+ case GREATER_THAN:
+ return candidates.stream().anyMatch(v -> actual > Long.parseLong(v));
+ case GREATER_THAN_OR_EQUAL:
+ return candidates.stream().anyMatch(v -> actual >= Long.parseLong(v));
+ case LESS_THAN:
+ return candidates.stream().anyMatch(v -> actual < Long.parseLong(v));
+ case LESS_THAN_OR_EQUAL:
+ return candidates.stream().anyMatch(v -> actual <= Long.parseLong(v));
+ case IN:
+ for (String candidate : candidates) {
+ if (actual == Long.parseLong(candidate)) {
+ return true;
+ }
+ }
+ return false;
+ case RANGE:
+ case BETWEEN:
+ if (candidates.size() >= 2) {
+ long lower = Long.parseLong(candidates.get(0));
+ long upper = Long.parseLong(candidates.get(1));
+ return actual >= lower && actual <= upper;
+ }
+ return true;
+ default:
+ return true;
+ }
+ } catch (NumberFormatException e) {
+ LOGGER.debug("Failed to parse numeric filter value {}: {}", candidates, e.toString());
+ return true;
+ }
+ }
+
+ private @Nullable String extractIdentifier(List<org.apache.pinot.common.request.Expression> operands) {
+ for (org.apache.pinot.common.request.Expression operand : operands) {
+ org.apache.pinot.common.request.Identifier identifier = operand.getIdentifier();
+ if (identifier != null) {
+ return identifier.getName();
+ }
+ }
+ return null;
+ }
+
+ private List<String> extractLiteralValues(List<org.apache.pinot.common.request.Expression> operands) {
+ List<String> values = new ArrayList<>();
+ for (org.apache.pinot.common.request.Expression operand : operands) {
+ org.apache.pinot.common.request.Literal literal = operand.getLiteral();
+ if (literal != null) {
+ if (literal.getSetField() == org.apache.pinot.common.request.Literal._Fields.NULL_VALUE) {
+ values.add("null");
+ continue;
+ }
+ try {
+ values.add(RequestUtils.getLiteralString(literal));
+ } catch (Exception e) {
+ values.add(String.valueOf(RequestUtils.getLiteralValue(literal)));
+ }
+ }
+ }
+ return values;
+ }
+
+ private @Nullable FilterKind toFilterKind(String operator) {
+ String normalized = operator.toUpperCase(Locale.ROOT);
+ switch (normalized) {
+ case "EQ":
+ return FilterKind.EQUALS;
+ case "NEQ":
+ return FilterKind.NOT_EQUALS;
+ case "GT":
+ return FilterKind.GREATER_THAN;
+ case "GTE":
+ return FilterKind.GREATER_THAN_OR_EQUAL;
+ case "LT":
+ return FilterKind.LESS_THAN;
+ case "LTE":
+ return FilterKind.LESS_THAN_OR_EQUAL;
+ default:
+ try {
+ return FilterKind.valueOf(normalized);
+ } catch (Exception e) {
+ return null;
+ }
+ }
+ }
+
+ private @Nullable TableSize fetchTableSize(String tableNameWithType) {
+ TableSize cached = getCachedSize(tableNameWithType);
+ if (cached != null) {
+ return cached;
+ }
+ Function<String, TableSize> fetcher = getSizeFetcher();
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ cacheSize(tableNameWithType, fetched);
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(tableNameWithType);
+ if (size == null) {
+ LOGGER.warn("{}: failed to fetch size for {} (raw: {}), returning null", TABLE_NAME, tableNameWithType,
+ rawTableName);
+ }
+ }
+ if (size != null) {
+ cacheSize(tableNameWithType, size);
+ }
+ return size;
+ }
+
+ private @Nullable TableSize fetchTableSizeForName(String tableName) {
+ for (String baseUrl : getControllerBaseUrls()) {
+ try {
+ String url = baseUrl + "/tables/" + tableName + "/size?verbose=true&includeReplacedSegments=false";
+ HttpRequest request = HttpRequest.newBuilder(URI.create(url))
+ .timeout(Duration.ofSeconds(5))
+ .GET()
+ .header("Accept", "application/json")
+ .build();
+ HttpResponse<String> response = _httpClient.send(request, HttpResponse.BodyHandlers.ofString());
+ if (response.statusCode() >= 200 && response.statusCode() < 300) {
+ TableSize parsed = JsonUtils.stringToObject(response.body(), TableSize.class);
+ LOGGER.debug("{}: controller size response for {} via {} -> segments offline={}, realtime={}, "
+ + "reportedSize={}, estimatedSize={}", TABLE_NAME, tableName, baseUrl,
+ parsed._offlineSegments != null && parsed._offlineSegments._segments != null
+ ? parsed._offlineSegments._segments.size() : 0,
+ parsed._realtimeSegments != null && parsed._realtimeSegments._segments != null
+ ? parsed._realtimeSegments._segments.size() : 0,
+ parsed._reportedSizeInBytes, parsed._estimatedSizeInBytes);
+ return parsed;
+ } else {
+ LOGGER.warn("{}: failed to fetch table size for {} via {}: status {}, body={}", TABLE_NAME, tableName,
+ baseUrl, response.statusCode(), response.body());
+ }
+ } catch (Exception e) {
+ LOGGER.warn("{}: error fetching table size for {} via {}", TABLE_NAME, tableName, baseUrl, e);
+ }
+ }
+ return null;
+ }
+
+ private List<String> getControllerBaseUrls() {
+ Set<String> urls = new LinkedHashSet<>();
+ if (_helixAdmin != null) {
+ for (String controller : discoverControllersFromHelix()) {
+ String normalized = normalizeControllerUrl(controller);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ }
+ for (String url : _staticControllerUrls) {
+ String normalized = normalizeControllerUrl(url);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ return new ArrayList<>(urls);
+ }
+
+ private int getSegmentCount(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.size();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.size();
+ }
+ return 0;
+ }
+
+ private long getTotalDocs(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ return 0;
+ }
+
+ private @Nullable Function<String, TableSize> getSizeFetcher() {
+ if (_tableSizeFetcherOverride != null) {
+ return _tableSizeFetcherOverride;
+ }
+ List<String> controllers = getControllerBaseUrls();
+ if (controllers.isEmpty() || !isAdminClientAvailable()) {
+ return null;
+ }
+ return tableNameWithType -> fetchWithAdminClient(controllers, tableNameWithType);
+ }
+
+ private List<String> discoverControllersFromHelix() {
+ List<String> urls = new ArrayList<>();
+ try {
+ if (_clusterName == null) {
+ LOGGER.warn("Cannot discover controllers without cluster name");
+ return List.of();
+ }
+ for (String controllerId : _helixAdmin.getInstancesInCluster(_clusterName)) {
+ if (!InstanceTypeUtils.isController(controllerId)) {
+ continue;
+ }
+ int firstUnderscore = controllerId.indexOf('_');
+ int lastUnderscore = controllerId.lastIndexOf('_');
+ if (firstUnderscore > 0 && lastUnderscore > firstUnderscore && lastUnderscore < controllerId.length() - 1) {
+ String host = controllerId.substring(firstUnderscore + 1, lastUnderscore);
+ String port = controllerId.substring(lastUnderscore + 1);
+ if (!host.isEmpty() && !port.isEmpty()) {
+ urls.add(host + ":" + port);
+ } else {
+ LOGGER.warn("Unable to parse controller address from instance id (empty host/port): {}", controllerId);
+ }
+ } else {
+ LOGGER.warn("Unable to parse controller address from instance id: {}", controllerId);
Review Comment:
The error message does not explain *why* parsing failed. Consider including details about the expected format or the actual structure encountered to aid debugging.
```suggestion
LOGGER.warn(
"Unable to parse controller address from instance id '{}': expected format '<prefix>_<host>_<port>' "
+ "but got firstUnderscore={}, lastUnderscore={}, length={}",
controllerId, firstUnderscore, lastUnderscore, controllerId.length());
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,726 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.SystemTableResponseUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.FilterKind;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableDataProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final long SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final boolean IS_ADMIN_CLIENT_AVAILABLE;
+ static {
+ boolean available;
+ try {
+ Class.forName("org.apache.pinot.client.admin.PinotAdminClient");
+ available = true;
+ } catch (ClassNotFoundException e) {
+ available = false;
+ }
+ IS_ADMIN_CLIENT_AVAILABLE = available;
+ }
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+ private final Map<String, CachedSize> _sizeCache = new ConcurrentHashMap<>();
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public TableConfig getTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ }
+
+ @Override
+ public BrokerResponseNative getBrokerResponse(PinotQuery pinotQuery) {
+ List<String> projectionColumns = pinotQuery.getSelectList() != null
+ ? pinotQuery.getSelectList().stream().map(expr -> expr.getIdentifier().getName()).collect(Collectors.toList())
+ : List.of();
+ if (_tableCache == null) {
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, List.of(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ int offset = Math.max(0, pinotQuery.getOffset());
+ int limit = pinotQuery.getLimit();
+ boolean hasLimit = limit > 0;
+ int totalRows = 0;
+ int initialCapacity = hasLimit ? Math.min(sortedTableNames.size(), limit) : 0;
+ List<GenericRow> rows = new ArrayList<>(initialCapacity);
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType);
+ if (!matchesFilter(pinotQuery.getFilterExpression(), stats, rawTableName)) {
+ continue;
+ }
+ if (offset > 0) {
+ offset--;
+ totalRows++;
+ continue;
+ }
+ totalRows++;
+ if (limit == 0) {
+ continue;
+ }
+ if (hasLimit && rows.size() >= limit) {
+ continue;
+ }
+ GenericRow row = new GenericRow();
+ row.putValue("tableName", rawTableName);
+ row.putValue("type", stats._type);
+ row.putValue("status", stats._status);
+ row.putValue("segments", stats._segments);
+ row.putValue("totalDocs", stats._totalDocs);
+ row.putValue("reportedSize", stats._reportedSizeInBytes);
+ row.putValue("estimatedSize", stats._estimatedSizeInBytes);
+ row.putValue("storageTier", stats._storageTier);
+ row.putValue("brokerTenant", stats._brokerTenant);
+ row.putValue("serverTenant", stats._serverTenant);
+ row.putValue("replicas", stats._replicas);
+ row.putValue("tableConfig", stats._tableConfig);
+ rows.add(row);
+ }
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, rows, totalRows);
+ }
+
+ private TableStats buildStats(String tableNameWithType, TableType tableType) {
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ int segments = 0;
+ long totalDocs = 0;
+ long reportedSize = 0;
+ long estimatedSize = 0;
+ String tierValue = "";
+ String brokerTenant = "";
+ String serverTenant = "";
+ int replicas = 0;
+ if (tableConfig != null && tableConfig.getTenantConfig() != null) {
+ brokerTenant = tableConfig.getTenantConfig().getBroker();
+ serverTenant = tableConfig.getTenantConfig().getServer();
+ }
+ if (tableConfig != null && tableConfig.getValidationConfig() != null) {
+ Integer repl = tableConfig.getValidationConfig().getReplicationNumber();
+ replicas = repl != null ? repl : replicas;
+ }
+ // Use controller API only
+ TableSize sizeFromController = fetchTableSize(tableNameWithType);
+ if (sizeFromController != null) {
+ if (sizeFromController._reportedSizeInBytes >= 0) {
+ reportedSize = sizeFromController._reportedSizeInBytes;
+ }
+ if (sizeFromController._estimatedSizeInBytes >= 0) {
+ estimatedSize = sizeFromController._estimatedSizeInBytes;
+ }
+ segments = getSegmentCount(sizeFromController, tableType);
+ totalDocs = getTotalDocs(sizeFromController, tableType);
+ }
+ String status = tableConfig != null ? "ONLINE" : (segments > 0 ? "ONLINE" : "UNKNOWN");
+ String tableConfigJson = "";
+ if (tableConfig != null) {
+ try {
+ tableConfigJson = JsonUtils.objectToString(tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", tableNameWithType, e.toString());
+ tableConfigJson = tableConfig.toString();
+ }
+ }
+ return new TableStats(tableType.name(), status, segments, totalDocs, reportedSize, estimatedSize, tierValue,
+ tableConfigJson, brokerTenant, serverTenant, replicas);
+ }
+
+ private boolean matchesFilter(@Nullable org.apache.pinot.common.request.Expression filterExpression, TableStats stats,
+ String rawTableName) {
+ if (filterExpression == null) {
+ return true;
+ }
+ org.apache.pinot.common.request.Function function = filterExpression.getFunctionCall();
+ if (function == null) {
+ return true;
+ }
+ FilterKind filterKind = toFilterKind(function.getOperator());
+ if (filterKind == null) {
+ return true;
+ }
+ switch (filterKind) {
+ case AND:
+ for (org.apache.pinot.common.request.Expression child : function.getOperands()) {
+ if (!matchesFilter(child, stats, rawTableName)) {
+ return false;
+ }
+ }
+ return true;
+ case OR:
+ for (org.apache.pinot.common.request.Expression child : function.getOperands()) {
+ if (matchesFilter(child, stats, rawTableName)) {
+ return true;
+ }
+ }
+ return false;
+ case NOT:
+ if (function.getOperandsSize() == 0) {
+ return true;
+ }
+ return !matchesFilter(function.getOperands().get(0), stats, rawTableName);
+ default:
+ return matchesLeafFilter(filterKind, function.getOperands(), stats, rawTableName);
+ }
+ }
+
+ private boolean matchesLeafFilter(FilterKind filterKind,
+ List<org.apache.pinot.common.request.Expression> operands, TableStats stats, String rawTableName) {
+ String column = extractIdentifier(operands);
+ if (column == null) {
+ return true;
+ }
+ List<String> values = extractLiteralValues(operands);
+ if (values.isEmpty()) {
+ return true;
+ }
+ switch (column.toLowerCase(Locale.ROOT)) {
+ case "tablename":
+ return matchesString(values, rawTableName, filterKind);
+ case "type":
+ return matchesString(values, stats._type, filterKind);
+ case "status":
+ return matchesString(values, stats._status, filterKind);
+ case "segments":
+ return matchesNumber(values, stats._segments, filterKind);
+ case "reportedsize":
+ return matchesNumber(values, stats._reportedSizeInBytes, filterKind);
+ case "estimatedsize":
+ return matchesNumber(values, stats._estimatedSizeInBytes, filterKind);
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesString(List<String> candidates, String actual, FilterKind filterKind) {
+ switch (filterKind) {
+ case EQUALS:
+ case IN:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case NOT_EQUALS:
+ return candidates.stream().noneMatch(v -> v.equalsIgnoreCase(actual));
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesNumber(List<String> candidates, long actual, FilterKind filterKind) {
+ try {
+ switch (filterKind) {
+ case EQUALS:
+ return candidates.stream().anyMatch(v -> Long.parseLong(v) == actual);
+ case NOT_EQUALS:
+ return candidates.stream().noneMatch(v -> Long.parseLong(v) == actual);
+ case GREATER_THAN:
+ return candidates.stream().anyMatch(v -> actual > Long.parseLong(v));
+ case GREATER_THAN_OR_EQUAL:
+ return candidates.stream().anyMatch(v -> actual >= Long.parseLong(v));
+ case LESS_THAN:
+ return candidates.stream().anyMatch(v -> actual < Long.parseLong(v));
+ case LESS_THAN_OR_EQUAL:
+ return candidates.stream().anyMatch(v -> actual <= Long.parseLong(v));
+ case IN:
+ for (String candidate : candidates) {
+ if (actual == Long.parseLong(candidate)) {
+ return true;
+ }
+ }
+ return false;
+ case RANGE:
+ case BETWEEN:
+ if (candidates.size() >= 2) {
+ long lower = Long.parseLong(candidates.get(0));
+ long upper = Long.parseLong(candidates.get(1));
+ return actual >= lower && actual <= upper;
+ }
+ return true;
+ default:
+ return true;
+ }
+ } catch (NumberFormatException e) {
+ LOGGER.debug("Failed to parse numeric filter value {}: {}", candidates, e.toString());
+ return true;
+ }
+ }
+
+ private @Nullable String extractIdentifier(List<org.apache.pinot.common.request.Expression> operands) {
+ for (org.apache.pinot.common.request.Expression operand : operands) {
+ org.apache.pinot.common.request.Identifier identifier = operand.getIdentifier();
+ if (identifier != null) {
+ return identifier.getName();
+ }
+ }
+ return null;
+ }
+
+ private List<String> extractLiteralValues(List<org.apache.pinot.common.request.Expression> operands) {
+ List<String> values = new ArrayList<>();
+ for (org.apache.pinot.common.request.Expression operand : operands) {
+ org.apache.pinot.common.request.Literal literal = operand.getLiteral();
+ if (literal != null) {
+ if (literal.getSetField() == org.apache.pinot.common.request.Literal._Fields.NULL_VALUE) {
+ values.add("null");
+ continue;
+ }
+ try {
+ values.add(RequestUtils.getLiteralString(literal));
+ } catch (Exception e) {
+ values.add(String.valueOf(RequestUtils.getLiteralValue(literal)));
+ }
+ }
+ }
+ return values;
+ }
+
+ private @Nullable FilterKind toFilterKind(String operator) {
+ String normalized = operator.toUpperCase(Locale.ROOT);
+ switch (normalized) {
+ case "EQ":
+ return FilterKind.EQUALS;
+ case "NEQ":
+ return FilterKind.NOT_EQUALS;
+ case "GT":
+ return FilterKind.GREATER_THAN;
+ case "GTE":
+ return FilterKind.GREATER_THAN_OR_EQUAL;
+ case "LT":
+ return FilterKind.LESS_THAN;
+ case "LTE":
+ return FilterKind.LESS_THAN_OR_EQUAL;
+ default:
+ try {
+ return FilterKind.valueOf(normalized);
+ } catch (Exception e) {
+ return null;
+ }
+ }
+ }
+
+ private @Nullable TableSize fetchTableSize(String tableNameWithType) {
+ TableSize cached = getCachedSize(tableNameWithType);
+ if (cached != null) {
+ return cached;
+ }
+ Function<String, TableSize> fetcher = getSizeFetcher();
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ cacheSize(tableNameWithType, fetched);
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(tableNameWithType);
+ if (size == null) {
+ LOGGER.warn("{}: failed to fetch size for {} (raw: {}), returning null", TABLE_NAME, tableNameWithType,
+ rawTableName);
+ }
+ }
+ if (size != null) {
+ cacheSize(tableNameWithType, size);
+ }
+ return size;
+ }
+
+ private @Nullable TableSize fetchTableSizeForName(String tableName) {
+ for (String baseUrl : getControllerBaseUrls()) {
+ try {
+ String url = baseUrl + "/tables/" + tableName + "/size?verbose=true&includeReplacedSegments=false";
+ HttpRequest request = HttpRequest.newBuilder(URI.create(url))
+ .timeout(Duration.ofSeconds(5))
+ .GET()
+ .header("Accept", "application/json")
+ .build();
+ HttpResponse<String> response = _httpClient.send(request, HttpResponse.BodyHandlers.ofString());
+ if (response.statusCode() >= 200 && response.statusCode() < 300) {
+ TableSize parsed = JsonUtils.stringToObject(response.body(), TableSize.class);
+ LOGGER.debug("{}: controller size response for {} via {} -> segments offline={}, realtime={}, "
+ + "reportedSize={}, estimatedSize={}", TABLE_NAME, tableName, baseUrl,
+ parsed._offlineSegments != null && parsed._offlineSegments._segments != null
+ ? parsed._offlineSegments._segments.size() : 0,
+ parsed._realtimeSegments != null && parsed._realtimeSegments._segments != null
+ ? parsed._realtimeSegments._segments.size() : 0,
+ parsed._reportedSizeInBytes, parsed._estimatedSizeInBytes);
Review Comment:
The nested null checks for segment counts make the logging statement difficult to read. Extract these calculations into helper methods like getOfflineSegmentCount() and getRealtimeSegmentCount() to improve readability.
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,726 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.SystemTableResponseUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.FilterKind;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableDataProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final long SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final boolean IS_ADMIN_CLIENT_AVAILABLE;
+ static {
+ boolean available;
+ try {
+ Class.forName("org.apache.pinot.client.admin.PinotAdminClient");
+ available = true;
+ } catch (ClassNotFoundException e) {
+ available = false;
+ }
+ IS_ADMIN_CLIENT_AVAILABLE = available;
+ }
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+ private final Map<String, CachedSize> _sizeCache = new ConcurrentHashMap<>();
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public TableConfig getTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ }
+
+ @Override
+ public BrokerResponseNative getBrokerResponse(PinotQuery pinotQuery) {
+ List<String> projectionColumns = pinotQuery.getSelectList() != null
+ ? pinotQuery.getSelectList().stream().map(expr -> expr.getIdentifier().getName()).collect(Collectors.toList())
+ : List.of();
+ if (_tableCache == null) {
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, List.of(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ int offset = Math.max(0, pinotQuery.getOffset());
+ int limit = pinotQuery.getLimit();
+ boolean hasLimit = limit > 0;
+ int totalRows = 0;
+ int initialCapacity = hasLimit ? Math.min(sortedTableNames.size(), limit) : 0;
+ List<GenericRow> rows = new ArrayList<>(initialCapacity);
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType);
+ if (!matchesFilter(pinotQuery.getFilterExpression(), stats, rawTableName)) {
+ continue;
+ }
+ if (offset > 0) {
+ offset--;
+ totalRows++;
+ continue;
+ }
+ totalRows++;
+ if (limit == 0) {
+ continue;
+ }
+ if (hasLimit && rows.size() >= limit) {
+ continue;
+ }
+ GenericRow row = new GenericRow();
+ row.putValue("tableName", rawTableName);
+ row.putValue("type", stats._type);
+ row.putValue("status", stats._status);
+ row.putValue("segments", stats._segments);
+ row.putValue("totalDocs", stats._totalDocs);
+ row.putValue("reportedSize", stats._reportedSizeInBytes);
+ row.putValue("estimatedSize", stats._estimatedSizeInBytes);
+ row.putValue("storageTier", stats._storageTier);
+ row.putValue("brokerTenant", stats._brokerTenant);
+ row.putValue("serverTenant", stats._serverTenant);
+ row.putValue("replicas", stats._replicas);
+ row.putValue("tableConfig", stats._tableConfig);
+ rows.add(row);
+ }
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, rows, totalRows);
+ }
+
+ private TableStats buildStats(String tableNameWithType, TableType tableType) {
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ int segments = 0;
+ long totalDocs = 0;
+ long reportedSize = 0;
+ long estimatedSize = 0;
+ String tierValue = "";
+ String brokerTenant = "";
+ String serverTenant = "";
+ int replicas = 0;
+ if (tableConfig != null && tableConfig.getTenantConfig() != null) {
+ brokerTenant = tableConfig.getTenantConfig().getBroker();
+ serverTenant = tableConfig.getTenantConfig().getServer();
+ }
+ if (tableConfig != null && tableConfig.getValidationConfig() != null) {
+ Integer repl = tableConfig.getValidationConfig().getReplicationNumber();
+ replicas = repl != null ? repl : replicas;
+ }
+ // Use controller API only
+ TableSize sizeFromController = fetchTableSize(tableNameWithType);
+ if (sizeFromController != null) {
+ if (sizeFromController._reportedSizeInBytes >= 0) {
+ reportedSize = sizeFromController._reportedSizeInBytes;
+ }
+ if (sizeFromController._estimatedSizeInBytes >= 0) {
+ estimatedSize = sizeFromController._estimatedSizeInBytes;
+ }
+ segments = getSegmentCount(sizeFromController, tableType);
+ totalDocs = getTotalDocs(sizeFromController, tableType);
+ }
+ String status = tableConfig != null ? "ONLINE" : (segments > 0 ? "ONLINE" : "UNKNOWN");
+ String tableConfigJson = "";
+ if (tableConfig != null) {
+ try {
+ tableConfigJson = JsonUtils.objectToString(tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", tableNameWithType, e.toString());
+ tableConfigJson = tableConfig.toString();
+ }
+ }
+ return new TableStats(tableType.name(), status, segments, totalDocs, reportedSize, estimatedSize, tierValue,
+ tableConfigJson, brokerTenant, serverTenant, replicas);
+ }
+
+ private boolean matchesFilter(@Nullable org.apache.pinot.common.request.Expression filterExpression, TableStats stats,
+ String rawTableName) {
+ if (filterExpression == null) {
+ return true;
+ }
+ org.apache.pinot.common.request.Function function = filterExpression.getFunctionCall();
+ if (function == null) {
+ return true;
+ }
+ FilterKind filterKind = toFilterKind(function.getOperator());
+ if (filterKind == null) {
+ return true;
+ }
+ switch (filterKind) {
+ case AND:
+ for (org.apache.pinot.common.request.Expression child : function.getOperands()) {
+ if (!matchesFilter(child, stats, rawTableName)) {
+ return false;
+ }
+ }
+ return true;
+ case OR:
+ for (org.apache.pinot.common.request.Expression child : function.getOperands()) {
+ if (matchesFilter(child, stats, rawTableName)) {
+ return true;
+ }
+ }
+ return false;
+ case NOT:
+ if (function.getOperandsSize() == 0) {
+ return true;
+ }
+ return !matchesFilter(function.getOperands().get(0), stats, rawTableName);
+ default:
+ return matchesLeafFilter(filterKind, function.getOperands(), stats, rawTableName);
+ }
+ }
+
+ private boolean matchesLeafFilter(FilterKind filterKind,
+ List<org.apache.pinot.common.request.Expression> operands, TableStats stats, String rawTableName) {
+ String column = extractIdentifier(operands);
+ if (column == null) {
+ return true;
+ }
+ List<String> values = extractLiteralValues(operands);
+ if (values.isEmpty()) {
+ return true;
+ }
+ switch (column.toLowerCase(Locale.ROOT)) {
+ case "tablename":
+ return matchesString(values, rawTableName, filterKind);
+ case "type":
+ return matchesString(values, stats._type, filterKind);
+ case "status":
+ return matchesString(values, stats._status, filterKind);
+ case "segments":
+ return matchesNumber(values, stats._segments, filterKind);
+ case "reportedsize":
+ return matchesNumber(values, stats._reportedSizeInBytes, filterKind);
+ case "estimatedsize":
+ return matchesNumber(values, stats._estimatedSizeInBytes, filterKind);
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesString(List<String> candidates, String actual, FilterKind filterKind) {
+ switch (filterKind) {
+ case EQUALS:
+ case IN:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case NOT_EQUALS:
+ return candidates.stream().noneMatch(v -> v.equalsIgnoreCase(actual));
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesNumber(List<String> candidates, long actual, FilterKind filterKind) {
+ try {
+ switch (filterKind) {
+ case EQUALS:
+ return candidates.stream().anyMatch(v -> Long.parseLong(v) == actual);
+ case NOT_EQUALS:
+ return candidates.stream().noneMatch(v -> Long.parseLong(v) == actual);
+ case GREATER_THAN:
+ return candidates.stream().anyMatch(v -> actual > Long.parseLong(v));
+ case GREATER_THAN_OR_EQUAL:
+ return candidates.stream().anyMatch(v -> actual >= Long.parseLong(v));
+ case LESS_THAN:
+ return candidates.stream().anyMatch(v -> actual < Long.parseLong(v));
+ case LESS_THAN_OR_EQUAL:
+ return candidates.stream().anyMatch(v -> actual <= Long.parseLong(v));
+ case IN:
+ for (String candidate : candidates) {
+ if (actual == Long.parseLong(candidate)) {
+ return true;
+ }
+ }
+ return false;
+ case RANGE:
+ case BETWEEN:
+ if (candidates.size() >= 2) {
+ long lower = Long.parseLong(candidates.get(0));
+ long upper = Long.parseLong(candidates.get(1));
+ return actual >= lower && actual <= upper;
+ }
+ return true;
+ default:
+ return true;
+ }
+ } catch (NumberFormatException e) {
+ LOGGER.debug("Failed to parse numeric filter value {}: {}", candidates, e.toString());
+ return true;
+ }
+ }
+
+ private @Nullable String extractIdentifier(List<org.apache.pinot.common.request.Expression> operands) {
+ for (org.apache.pinot.common.request.Expression operand : operands) {
+ org.apache.pinot.common.request.Identifier identifier = operand.getIdentifier();
+ if (identifier != null) {
+ return identifier.getName();
+ }
+ }
+ return null;
+ }
+
+ private List<String> extractLiteralValues(List<org.apache.pinot.common.request.Expression> operands) {
+ List<String> values = new ArrayList<>();
+ for (org.apache.pinot.common.request.Expression operand : operands) {
+ org.apache.pinot.common.request.Literal literal = operand.getLiteral();
+ if (literal != null) {
+ if (literal.getSetField() == org.apache.pinot.common.request.Literal._Fields.NULL_VALUE) {
+ values.add("null");
+ continue;
+ }
+ try {
+ values.add(RequestUtils.getLiteralString(literal));
+ } catch (Exception e) {
+ values.add(String.valueOf(RequestUtils.getLiteralValue(literal)));
+ }
+ }
+ }
+ return values;
+ }
+
+ private @Nullable FilterKind toFilterKind(String operator) {
+ String normalized = operator.toUpperCase(Locale.ROOT);
+ switch (normalized) {
+ case "EQ":
+ return FilterKind.EQUALS;
+ case "NEQ":
+ return FilterKind.NOT_EQUALS;
+ case "GT":
+ return FilterKind.GREATER_THAN;
+ case "GTE":
+ return FilterKind.GREATER_THAN_OR_EQUAL;
+ case "LT":
+ return FilterKind.LESS_THAN;
+ case "LTE":
+ return FilterKind.LESS_THAN_OR_EQUAL;
+ default:
+ try {
+ return FilterKind.valueOf(normalized);
+ } catch (Exception e) {
+ return null;
+ }
+ }
+ }
+
+ private @Nullable TableSize fetchTableSize(String tableNameWithType) {
+ TableSize cached = getCachedSize(tableNameWithType);
+ if (cached != null) {
+ return cached;
+ }
+ Function<String, TableSize> fetcher = getSizeFetcher();
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ cacheSize(tableNameWithType, fetched);
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(tableNameWithType);
+ if (size == null) {
+ LOGGER.warn("{}: failed to fetch size for {} (raw: {}), returning null", TABLE_NAME, tableNameWithType,
+ rawTableName);
+ }
+ }
+ if (size != null) {
+ cacheSize(tableNameWithType, size);
+ }
+ return size;
+ }
+
+ private @Nullable TableSize fetchTableSizeForName(String tableName) {
+ for (String baseUrl : getControllerBaseUrls()) {
+ try {
+ String url = baseUrl + "/tables/" + tableName + "/size?verbose=true&includeReplacedSegments=false";
+ HttpRequest request = HttpRequest.newBuilder(URI.create(url))
+ .timeout(Duration.ofSeconds(5))
+ .GET()
+ .header("Accept", "application/json")
+ .build();
+ HttpResponse<String> response = _httpClient.send(request, HttpResponse.BodyHandlers.ofString());
+ if (response.statusCode() >= 200 && response.statusCode() < 300) {
+ TableSize parsed = JsonUtils.stringToObject(response.body(), TableSize.class);
+ LOGGER.debug("{}: controller size response for {} via {} -> segments offline={}, realtime={}, "
+ + "reportedSize={}, estimatedSize={}", TABLE_NAME, tableName, baseUrl,
+ parsed._offlineSegments != null && parsed._offlineSegments._segments != null
+ ? parsed._offlineSegments._segments.size() : 0,
+ parsed._realtimeSegments != null && parsed._realtimeSegments._segments != null
+ ? parsed._realtimeSegments._segments.size() : 0,
+ parsed._reportedSizeInBytes, parsed._estimatedSizeInBytes);
+ return parsed;
+ } else {
+ LOGGER.warn("{}: failed to fetch table size for {} via {}: status {}, body={}", TABLE_NAME, tableName,
+ baseUrl, response.statusCode(), response.body());
+ }
+ } catch (Exception e) {
+ LOGGER.warn("{}: error fetching table size for {} via {}", TABLE_NAME, tableName, baseUrl, e);
+ }
+ }
+ return null;
+ }
+
+ private List<String> getControllerBaseUrls() {
+ Set<String> urls = new LinkedHashSet<>();
+ if (_helixAdmin != null) {
+ for (String controller : discoverControllersFromHelix()) {
+ String normalized = normalizeControllerUrl(controller);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ }
+ for (String url : _staticControllerUrls) {
+ String normalized = normalizeControllerUrl(url);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ return new ArrayList<>(urls);
+ }
+
+ private int getSegmentCount(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.size();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.size();
+ }
+ return 0;
+ }
+
+ private long getTotalDocs(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
+ }
+ return 0;
+ }
+
+ private @Nullable Function<String, TableSize> getSizeFetcher() {
+ if (_tableSizeFetcherOverride != null) {
+ return _tableSizeFetcherOverride;
+ }
+ List<String> controllers = getControllerBaseUrls();
+ if (controllers.isEmpty() || !isAdminClientAvailable()) {
+ return null;
+ }
+ return tableNameWithType -> fetchWithAdminClient(controllers, tableNameWithType);
+ }
+
+ private List<String> discoverControllersFromHelix() {
+ List<String> urls = new ArrayList<>();
+ try {
+ if (_clusterName == null) {
+ LOGGER.warn("Cannot discover controllers without cluster name");
+ return List.of();
+ }
+ for (String controllerId : _helixAdmin.getInstancesInCluster(_clusterName)) {
+ if (!InstanceTypeUtils.isController(controllerId)) {
+ continue;
+ }
+ int firstUnderscore = controllerId.indexOf('_');
+ int lastUnderscore = controllerId.lastIndexOf('_');
+ if (firstUnderscore > 0 && lastUnderscore > firstUnderscore && lastUnderscore < controllerId.length() - 1) {
+ String host = controllerId.substring(firstUnderscore + 1, lastUnderscore);
+ String port = controllerId.substring(lastUnderscore + 1);
+ if (!host.isEmpty() && !port.isEmpty()) {
+ urls.add(host + ":" + port);
+ } else {
+ LOGGER.warn("Unable to parse controller address from instance id (empty host/port): {}", controllerId);
Review Comment:
The error message is unclear—'empty host/port' suggests both are missing, but the condition checks if *either* is empty. Reword to 'empty host or port' for accuracy.
```suggestion
LOGGER.warn("Unable to parse controller address from instance id (empty host or port): {}", controllerId);
```
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BrokerRequestHandlerDelegate.java:
##########
@@ -106,7 +113,25 @@ public BrokerResponse handleRequest(JsonNode request, @Nullable SqlNodeAndOption
}
}
+ Set<String> tableNames = null;
+ if (_systemTableBrokerRequestHandler != null) {
+ try {
+ // NOTE: compileToPinotQuery(SqlNodeAndOptions) mutates the SqlNode (e.g. SqlOrderBy -> SqlSelect).
+ // Re-parse from raw SQL so the SqlNodeAndOptions passed to the handler is unchanged (important for MSE).
+ JsonNode sql = request.get(Request.SQL);
+ String sqlQuery =
+ (sql != null && sql.isTextual()) ? sql.asText() : sqlNodeAndOptions.getSqlNode().toString();
+ tableNames = RequestUtils.getTableNames(CalciteSqlParser.compileToPinotQuery(sqlQuery));
+ } catch (Exception e) {
+ // Ignore compilation exceptions here; the selected request handler will surface them appropriately.
Review Comment:
The comment claims the handler will surface errors, but swallowing exceptions without logging may hide issues during debugging. Consider logging at debug level or explain in the comment why silent failure is acceptable.
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,726 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.SystemTableResponseUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.FilterKind;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableDataProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final long SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final boolean IS_ADMIN_CLIENT_AVAILABLE;
+ static {
+ boolean available;
+ try {
+ Class.forName("org.apache.pinot.client.admin.PinotAdminClient");
+ available = true;
+ } catch (ClassNotFoundException e) {
+ available = false;
+ }
+ IS_ADMIN_CLIENT_AVAILABLE = available;
+ }
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+ private final Map<String, CachedSize> _sizeCache = new ConcurrentHashMap<>();
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
Review Comment:
The 5-second connect timeout is a magic number and could vary based on deployment. Extract it as a named constant or configuration property to improve maintainability.
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,726 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.SystemTableResponseUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.FilterKind;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableDataProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final long SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
Review Comment:
The 1-minute cache TTL may not suit all deployment scenarios. Consider making this configurable via a system property or constructor parameter for flexibility.
```suggestion
private static final String SIZE_CACHE_TTL_MS_PROPERTY = "pinot.system.tables.sizeCacheTtlMs";
private static final long DEFAULT_SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
private static final long SIZE_CACHE_TTL_MS =
Long.getLong(SIZE_CACHE_TTL_MS_PROPERTY, DEFAULT_SIZE_CACHE_TTL_MS);
```
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,726 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.SystemTableResponseUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.FilterKind;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableDataProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final long SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final boolean IS_ADMIN_CLIENT_AVAILABLE;
+ static {
+ boolean available;
+ try {
+ Class.forName("org.apache.pinot.client.admin.PinotAdminClient");
Review Comment:
The hardcoded class name 'org.apache.pinot.client.admin.PinotAdminClient' appears both in the static initializer and in fetchWithAdminClient (line 586). Define it as a constant to avoid duplication and ease future refactoring.
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,726 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableDataProvider;
+import org.apache.pinot.common.systemtable.SystemTableResponseUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.systemtable.SystemTable;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.FilterKind;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableDataProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final long SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("storageTier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private static final boolean IS_ADMIN_CLIENT_AVAILABLE;
+ static {
+ boolean available;
+ try {
+ Class.forName("org.apache.pinot.client.admin.PinotAdminClient");
+ available = true;
+ } catch (ClassNotFoundException e) {
+ available = false;
+ }
+ IS_ADMIN_CLIENT_AVAILABLE = available;
+ }
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final HttpClient _httpClient;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _staticControllerUrls;
+ private final Map<String, CachedSize> _sizeCache = new ConcurrentHashMap<>();
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _staticControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public TableConfig getTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ }
+
+ @Override
+ public BrokerResponseNative getBrokerResponse(PinotQuery pinotQuery) {
+ List<String> projectionColumns = pinotQuery.getSelectList() != null
+ ? pinotQuery.getSelectList().stream().map(expr -> expr.getIdentifier().getName()).collect(Collectors.toList())
+ : List.of();
+ if (_tableCache == null) {
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, List.of(), 0);
+ }
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ int offset = Math.max(0, pinotQuery.getOffset());
+ int limit = pinotQuery.getLimit();
+ boolean hasLimit = limit > 0;
+ int totalRows = 0;
+ int initialCapacity = hasLimit ? Math.min(sortedTableNames.size(), limit) : 0;
+ List<GenericRow> rows = new ArrayList<>(initialCapacity);
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableStats stats = buildStats(tableNameWithType, tableType);
+ if (!matchesFilter(pinotQuery.getFilterExpression(), stats, rawTableName)) {
+ continue;
+ }
+ if (offset > 0) {
+ offset--;
+ totalRows++;
+ continue;
+ }
+ totalRows++;
+ if (limit == 0) {
+ continue;
+ }
+ if (hasLimit && rows.size() >= limit) {
+ continue;
+ }
+ GenericRow row = new GenericRow();
+ row.putValue("tableName", rawTableName);
+ row.putValue("type", stats._type);
+ row.putValue("status", stats._status);
+ row.putValue("segments", stats._segments);
+ row.putValue("totalDocs", stats._totalDocs);
+ row.putValue("reportedSize", stats._reportedSizeInBytes);
+ row.putValue("estimatedSize", stats._estimatedSizeInBytes);
+ row.putValue("storageTier", stats._storageTier);
+ row.putValue("brokerTenant", stats._brokerTenant);
+ row.putValue("serverTenant", stats._serverTenant);
+ row.putValue("replicas", stats._replicas);
+ row.putValue("tableConfig", stats._tableConfig);
+ rows.add(row);
+ }
+ return SystemTableResponseUtils.buildBrokerResponse(TABLE_NAME, SCHEMA, projectionColumns, rows, totalRows);
+ }
+
+ private TableStats buildStats(String tableNameWithType, TableType tableType) {
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ int segments = 0;
+ long totalDocs = 0;
+ long reportedSize = 0;
+ long estimatedSize = 0;
+ String tierValue = "";
+ String brokerTenant = "";
+ String serverTenant = "";
+ int replicas = 0;
+ if (tableConfig != null && tableConfig.getTenantConfig() != null) {
+ brokerTenant = tableConfig.getTenantConfig().getBroker();
+ serverTenant = tableConfig.getTenantConfig().getServer();
+ }
+ if (tableConfig != null && tableConfig.getValidationConfig() != null) {
+ Integer repl = tableConfig.getValidationConfig().getReplicationNumber();
+ replicas = repl != null ? repl : replicas;
+ }
+ // Use controller API only
+ TableSize sizeFromController = fetchTableSize(tableNameWithType);
+ if (sizeFromController != null) {
+ if (sizeFromController._reportedSizeInBytes >= 0) {
+ reportedSize = sizeFromController._reportedSizeInBytes;
+ }
+ if (sizeFromController._estimatedSizeInBytes >= 0) {
+ estimatedSize = sizeFromController._estimatedSizeInBytes;
+ }
+ segments = getSegmentCount(sizeFromController, tableType);
+ totalDocs = getTotalDocs(sizeFromController, tableType);
+ }
+ String status = tableConfig != null ? "ONLINE" : (segments > 0 ? "ONLINE" : "UNKNOWN");
+ String tableConfigJson = "";
+ if (tableConfig != null) {
+ try {
+ tableConfigJson = JsonUtils.objectToString(tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", tableNameWithType, e.toString());
+ tableConfigJson = tableConfig.toString();
+ }
+ }
+ return new TableStats(tableType.name(), status, segments, totalDocs, reportedSize, estimatedSize, tierValue,
+ tableConfigJson, brokerTenant, serverTenant, replicas);
+ }
+
+ private boolean matchesFilter(@Nullable org.apache.pinot.common.request.Expression filterExpression, TableStats stats,
+ String rawTableName) {
+ if (filterExpression == null) {
+ return true;
+ }
+ org.apache.pinot.common.request.Function function = filterExpression.getFunctionCall();
+ if (function == null) {
+ return true;
+ }
+ FilterKind filterKind = toFilterKind(function.getOperator());
+ if (filterKind == null) {
+ return true;
+ }
+ switch (filterKind) {
+ case AND:
+ for (org.apache.pinot.common.request.Expression child : function.getOperands()) {
+ if (!matchesFilter(child, stats, rawTableName)) {
+ return false;
+ }
+ }
+ return true;
+ case OR:
+ for (org.apache.pinot.common.request.Expression child : function.getOperands()) {
+ if (matchesFilter(child, stats, rawTableName)) {
+ return true;
+ }
+ }
+ return false;
+ case NOT:
+ if (function.getOperandsSize() == 0) {
+ return true;
+ }
+ return !matchesFilter(function.getOperands().get(0), stats, rawTableName);
+ default:
+ return matchesLeafFilter(filterKind, function.getOperands(), stats, rawTableName);
+ }
+ }
+
+ private boolean matchesLeafFilter(FilterKind filterKind,
+ List<org.apache.pinot.common.request.Expression> operands, TableStats stats, String rawTableName) {
+ String column = extractIdentifier(operands);
+ if (column == null) {
+ return true;
+ }
+ List<String> values = extractLiteralValues(operands);
+ if (values.isEmpty()) {
+ return true;
+ }
+ switch (column.toLowerCase(Locale.ROOT)) {
+ case "tablename":
+ return matchesString(values, rawTableName, filterKind);
+ case "type":
+ return matchesString(values, stats._type, filterKind);
+ case "status":
+ return matchesString(values, stats._status, filterKind);
+ case "segments":
+ return matchesNumber(values, stats._segments, filterKind);
+ case "reportedsize":
+ return matchesNumber(values, stats._reportedSizeInBytes, filterKind);
+ case "estimatedsize":
+ return matchesNumber(values, stats._estimatedSizeInBytes, filterKind);
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesString(List<String> candidates, String actual, FilterKind filterKind) {
+ switch (filterKind) {
+ case EQUALS:
+ case IN:
+ return candidates.stream().anyMatch(v -> v.equalsIgnoreCase(actual));
+ case NOT_EQUALS:
+ return candidates.stream().noneMatch(v -> v.equalsIgnoreCase(actual));
+ default:
+ return true;
+ }
+ }
+
+ private boolean matchesNumber(List<String> candidates, long actual, FilterKind filterKind) {
+ try {
+ switch (filterKind) {
+ case EQUALS:
+ return candidates.stream().anyMatch(v -> Long.parseLong(v) == actual);
+ case NOT_EQUALS:
+ return candidates.stream().noneMatch(v -> Long.parseLong(v) == actual);
+ case GREATER_THAN:
+ return candidates.stream().anyMatch(v -> actual > Long.parseLong(v));
+ case GREATER_THAN_OR_EQUAL:
+ return candidates.stream().anyMatch(v -> actual >= Long.parseLong(v));
+ case LESS_THAN:
+ return candidates.stream().anyMatch(v -> actual < Long.parseLong(v));
+ case LESS_THAN_OR_EQUAL:
+ return candidates.stream().anyMatch(v -> actual <= Long.parseLong(v));
+ case IN:
+ for (String candidate : candidates) {
+ if (actual == Long.parseLong(candidate)) {
+ return true;
+ }
+ }
+ return false;
+ case RANGE:
+ case BETWEEN:
+ if (candidates.size() >= 2) {
+ long lower = Long.parseLong(candidates.get(0));
+ long upper = Long.parseLong(candidates.get(1));
+ return actual >= lower && actual <= upper;
+ }
+ return true;
+ default:
+ return true;
+ }
+ } catch (NumberFormatException e) {
+ LOGGER.debug("Failed to parse numeric filter value {}: {}", candidates, e.toString());
+ return true;
+ }
+ }
+
+ private @Nullable String extractIdentifier(List<org.apache.pinot.common.request.Expression> operands) {
+ for (org.apache.pinot.common.request.Expression operand : operands) {
+ org.apache.pinot.common.request.Identifier identifier = operand.getIdentifier();
+ if (identifier != null) {
+ return identifier.getName();
+ }
+ }
+ return null;
+ }
+
+ private List<String> extractLiteralValues(List<org.apache.pinot.common.request.Expression> operands) {
+ List<String> values = new ArrayList<>();
+ for (org.apache.pinot.common.request.Expression operand : operands) {
+ org.apache.pinot.common.request.Literal literal = operand.getLiteral();
+ if (literal != null) {
+ if (literal.getSetField() == org.apache.pinot.common.request.Literal._Fields.NULL_VALUE) {
+ values.add("null");
+ continue;
+ }
+ try {
+ values.add(RequestUtils.getLiteralString(literal));
+ } catch (Exception e) {
+ values.add(String.valueOf(RequestUtils.getLiteralValue(literal)));
+ }
+ }
+ }
+ return values;
+ }
+
+ private @Nullable FilterKind toFilterKind(String operator) {
+ String normalized = operator.toUpperCase(Locale.ROOT);
+ switch (normalized) {
+ case "EQ":
+ return FilterKind.EQUALS;
+ case "NEQ":
+ return FilterKind.NOT_EQUALS;
+ case "GT":
+ return FilterKind.GREATER_THAN;
+ case "GTE":
+ return FilterKind.GREATER_THAN_OR_EQUAL;
+ case "LT":
+ return FilterKind.LESS_THAN;
+ case "LTE":
+ return FilterKind.LESS_THAN_OR_EQUAL;
+ default:
+ try {
+ return FilterKind.valueOf(normalized);
+ } catch (Exception e) {
+ return null;
+ }
+ }
+ }
+
+ private @Nullable TableSize fetchTableSize(String tableNameWithType) {
+ TableSize cached = getCachedSize(tableNameWithType);
+ if (cached != null) {
+ return cached;
+ }
+ Function<String, TableSize> fetcher = getSizeFetcher();
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ cacheSize(tableNameWithType, fetched);
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(tableNameWithType);
+ if (size == null) {
+ LOGGER.warn("{}: failed to fetch size for {} (raw: {}), returning null", TABLE_NAME, tableNameWithType,
+ rawTableName);
Review Comment:
The message states 'returning null' but does not explain *why* both raw and type-suffixed names failed or what the user can do. Consider adding context about potential causes (e.g., table not found, controller unreachable).
```suggestion
LOGGER.warn(
"{}: failed to fetch table size for {} (raw: {}); no successful response from any controller. This may "
+ "indicate that the table does not exist or that the controller is unreachable; returning null size",
TABLE_NAME, tableNameWithType, rawTableName);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2649760745
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/SystemTableBrokerRequestHandler.java:
##########
@@ -0,0 +1,242 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.requesthandler;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import javax.annotation.Nullable;
+import javax.ws.rs.core.HttpHeaders;
+import org.apache.hc.client5.http.io.HttpClientConnectionManager;
+import org.apache.pinot.broker.api.AccessControl;
+import org.apache.pinot.broker.broker.AccessControlFactory;
+import org.apache.pinot.broker.querylog.QueryLogger;
+import org.apache.pinot.broker.queryquota.QueryQuotaManager;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.common.metrics.BrokerMeter;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.request.QuerySource;
+import org.apache.pinot.common.response.BrokerResponse;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.SystemTableRegistry;
+import org.apache.pinot.common.utils.NamedThreadFactory;
+import org.apache.pinot.common.utils.config.QueryOptionsUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.core.operator.blocks.InstanceResponseBlock;
+import org.apache.pinot.core.plan.Plan;
+import org.apache.pinot.core.plan.maker.InstancePlanMakerImplV2;
+import org.apache.pinot.core.plan.maker.PlanMaker;
+import org.apache.pinot.core.query.reduce.BrokerReduceService;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils;
+import org.apache.pinot.core.routing.RoutingManager;
+import org.apache.pinot.core.transport.ServerRoutingInstance;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.SegmentContext;
+import org.apache.pinot.spi.accounting.ThreadAccountant;
+import org.apache.pinot.spi.auth.AuthorizationResult;
+import org.apache.pinot.spi.auth.broker.RequesterIdentity;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.query.QueryExecutionContext;
+import org.apache.pinot.spi.query.QueryThreadContext;
+import org.apache.pinot.spi.trace.RequestContext;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+import org.apache.pinot.sql.parsers.SqlNodeAndOptions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Broker request handler for system tables (handled entirely on the broker).
+ */
+public class SystemTableBrokerRequestHandler extends BaseBrokerRequestHandler {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableBrokerRequestHandler.class);
+
+ private final BrokerReduceService _brokerReduceService;
+ private final PlanMaker _planMaker;
+ private final ExecutorService _executorService;
+
+ public SystemTableBrokerRequestHandler(PinotConfiguration config, String brokerId,
+ BrokerRequestIdGenerator requestIdGenerator, RoutingManager routingManager,
+ AccessControlFactory accessControlFactory, QueryQuotaManager queryQuotaManager, TableCache tableCache,
+ ThreadAccountant threadAccountant) {
+ super(config, brokerId, requestIdGenerator, routingManager, accessControlFactory, queryQuotaManager, tableCache,
+ threadAccountant);
+ _brokerReduceService = new BrokerReduceService(_config);
+ _planMaker = new InstancePlanMakerImplV2();
+ _planMaker.init(_config);
+ int executorPoolSize = config.getProperty(CommonConstants.Broker.CONFIG_OF_SYSTEM_TABLE_EXECUTOR_POOL_SIZE,
+ CommonConstants.Broker.DEFAULT_SYSTEM_TABLE_EXECUTOR_POOL_SIZE);
+ executorPoolSize = Math.max(1, executorPoolSize);
+ _executorService = QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(executorPoolSize,
+ new NamedThreadFactory("system-table-query-executor")));
+ }
+
+ @Override
+ public void start() {
+ }
+
+ @Override
+ public void shutDown() {
+ _executorService.shutdownNow();
+ _brokerReduceService.shutDown();
+ }
+
+ public boolean canHandle(String tableName) {
+ return isSystemTable(tableName) && SystemTableRegistry.isRegistered(tableName);
+ }
+
+ @Override
+ protected BrokerResponse handleRequest(long requestId, String query, SqlNodeAndOptions sqlNodeAndOptions,
+ JsonNode request, @Nullable RequesterIdentity requesterIdentity, RequestContext requestContext,
+ @Nullable HttpHeaders httpHeaders, AccessControl accessControl)
+ throws Exception {
+ long startTimeMs = requestContext.getRequestArrivalTimeMillis();
+ long deadlineMs = startTimeMs + _brokerTimeoutMs;
+ QueryExecutionContext executionContext =
+ new QueryExecutionContext(QueryExecutionContext.QueryType.SSE, requestId, Long.toString(requestId),
+ QueryOptionsUtils.getWorkloadName(sqlNodeAndOptions.getOptions()), startTimeMs, deadlineMs, deadlineMs,
+ _brokerId, _brokerId, org.apache.pinot.spi.utils.CommonConstants.Broker.DEFAULT_QUERY_HASH);
+ try (QueryThreadContext ignore = QueryThreadContext.open(executionContext, _threadAccountant)) {
+ PinotQuery pinotQuery;
+ try {
+ pinotQuery = CalciteSqlParser.compileToPinotQuery(sqlNodeAndOptions);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return new BrokerResponseNative(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ Set<String> tableNames = RequestUtils.getTableNames(pinotQuery);
+ if (tableNames == null || tableNames.isEmpty()) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "Failed to extract table name");
+ }
+ if (tableNames.size() != 1) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "System tables do not support joins");
+ }
+ String tableName = tableNames.iterator().next();
+ if (!isSystemTable(tableName)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "Not a system table query");
+ }
+ AuthorizationResult authorizationResult =
+ hasTableAccess(requesterIdentity, Set.of(tableName), requestContext, httpHeaders);
+ if (!authorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return new BrokerResponseNative(QueryErrorCode.ACCESS_DENIED, authorizationResult.getFailureMessage());
+ }
+
+ return handleSystemTableQuery(pinotQuery, tableName, requestContext, requesterIdentity, query);
+ }
+ }
+
+ @Override
+ protected boolean handleCancel(long queryId, int timeoutMs, Executor executor,
+ HttpClientConnectionManager connMgr, Map<String, Integer> serverResponses) {
+ return false;
+ }
+
+ @Override
+ public boolean cancelQueryByClientId(String clientQueryId, int timeoutMs, Executor executor,
+ HttpClientConnectionManager connMgr, Map<String, Integer> serverResponses)
+ throws Exception {
+ return false;
+ }
+
+ @Override
+ public Map<Long, String> getRunningQueries() {
+ return Collections.emptyMap();
+ }
+
+ @Override
+ public OptionalLong getRequestIdByClientId(String clientQueryId) {
+ return OptionalLong.empty();
+ }
+
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ private BrokerResponse handleSystemTableQuery(PinotQuery pinotQuery, String tableName, RequestContext requestContext,
+ @Nullable RequesterIdentity requesterIdentity, String query) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableProvider provider = SystemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return BrokerResponseNative.TABLE_DOES_NOT_EXIST;
+ }
+ try {
+ IndexSegment dataSource = provider.getDataSource();
+ BrokerResponseNative brokerResponse;
+ try {
+ QueryContext queryContext = QueryContextConverterUtils.getQueryContext(pinotQuery);
+ queryContext.setSchema(provider.getSchema());
+ queryContext.setEndTimeMs(System.currentTimeMillis() + _brokerTimeoutMs);
+
Review Comment:
Passing `null` as the last argument to `makeInstancePlan` suggests a missing parameter or configuration. Add a comment explaining what this parameter represents and why `null` is appropriate here, or consider using a named constant like `NO_INDEX_SEGMENT_PRELOADER` for clarity.
```suggestion
// Pass null for the index segment preloader because system tables use an in-memory IndexSegment
// that does not require additional segment preloading.
```
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BrokerRequestHandlerDelegate.java:
##########
@@ -108,11 +111,18 @@ public BrokerResponse handleRequest(JsonNode request, @Nullable SqlNodeAndOption
BaseBrokerRequestHandler requestHandler = _singleStageBrokerRequestHandler;
if (QueryOptionsUtils.isUseMultistageEngine(sqlNodeAndOptions.getOptions())) {
+ if (isSystemTableQuery(sqlNodeAndOptions)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION,
+ "System tables are not supported by the multi-stage query engine; please disable useMultistageEngine.");
Review Comment:
The error message 'please disable useMultistageEngine' is not actionable for users who may not control this query option directly. Consider rephrasing to 'System tables require the single-stage query engine. Remove the `useMultistageEngine=true` query option to execute this query.' to provide clearer guidance.
```suggestion
"System tables require the single-stage query engine. Remove the `useMultistageEngine=true` query option to execute this query.");
```
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/SystemTableBrokerRequestHandler.java:
##########
@@ -0,0 +1,242 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.requesthandler;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import javax.annotation.Nullable;
+import javax.ws.rs.core.HttpHeaders;
+import org.apache.hc.client5.http.io.HttpClientConnectionManager;
+import org.apache.pinot.broker.api.AccessControl;
+import org.apache.pinot.broker.broker.AccessControlFactory;
+import org.apache.pinot.broker.querylog.QueryLogger;
+import org.apache.pinot.broker.queryquota.QueryQuotaManager;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.common.metrics.BrokerMeter;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.request.QuerySource;
+import org.apache.pinot.common.response.BrokerResponse;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.SystemTableRegistry;
+import org.apache.pinot.common.utils.NamedThreadFactory;
+import org.apache.pinot.common.utils.config.QueryOptionsUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.core.operator.blocks.InstanceResponseBlock;
+import org.apache.pinot.core.plan.Plan;
+import org.apache.pinot.core.plan.maker.InstancePlanMakerImplV2;
+import org.apache.pinot.core.plan.maker.PlanMaker;
+import org.apache.pinot.core.query.reduce.BrokerReduceService;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils;
+import org.apache.pinot.core.routing.RoutingManager;
+import org.apache.pinot.core.transport.ServerRoutingInstance;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.SegmentContext;
+import org.apache.pinot.spi.accounting.ThreadAccountant;
+import org.apache.pinot.spi.auth.AuthorizationResult;
+import org.apache.pinot.spi.auth.broker.RequesterIdentity;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.query.QueryExecutionContext;
+import org.apache.pinot.spi.query.QueryThreadContext;
+import org.apache.pinot.spi.trace.RequestContext;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+import org.apache.pinot.sql.parsers.SqlNodeAndOptions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Broker request handler for system tables (handled entirely on the broker).
+ */
+public class SystemTableBrokerRequestHandler extends BaseBrokerRequestHandler {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableBrokerRequestHandler.class);
+
+ private final BrokerReduceService _brokerReduceService;
+ private final PlanMaker _planMaker;
+ private final ExecutorService _executorService;
+
+ public SystemTableBrokerRequestHandler(PinotConfiguration config, String brokerId,
+ BrokerRequestIdGenerator requestIdGenerator, RoutingManager routingManager,
+ AccessControlFactory accessControlFactory, QueryQuotaManager queryQuotaManager, TableCache tableCache,
+ ThreadAccountant threadAccountant) {
+ super(config, brokerId, requestIdGenerator, routingManager, accessControlFactory, queryQuotaManager, tableCache,
+ threadAccountant);
+ _brokerReduceService = new BrokerReduceService(_config);
+ _planMaker = new InstancePlanMakerImplV2();
+ _planMaker.init(_config);
+ int executorPoolSize = config.getProperty(CommonConstants.Broker.CONFIG_OF_SYSTEM_TABLE_EXECUTOR_POOL_SIZE,
+ CommonConstants.Broker.DEFAULT_SYSTEM_TABLE_EXECUTOR_POOL_SIZE);
+ executorPoolSize = Math.max(1, executorPoolSize);
+ _executorService = QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(executorPoolSize,
+ new NamedThreadFactory("system-table-query-executor")));
+ }
+
+ @Override
+ public void start() {
+ }
+
+ @Override
+ public void shutDown() {
+ _executorService.shutdownNow();
+ _brokerReduceService.shutDown();
+ }
+
+ public boolean canHandle(String tableName) {
+ return isSystemTable(tableName) && SystemTableRegistry.isRegistered(tableName);
+ }
+
+ @Override
+ protected BrokerResponse handleRequest(long requestId, String query, SqlNodeAndOptions sqlNodeAndOptions,
+ JsonNode request, @Nullable RequesterIdentity requesterIdentity, RequestContext requestContext,
+ @Nullable HttpHeaders httpHeaders, AccessControl accessControl)
+ throws Exception {
+ long startTimeMs = requestContext.getRequestArrivalTimeMillis();
+ long deadlineMs = startTimeMs + _brokerTimeoutMs;
+ QueryExecutionContext executionContext =
+ new QueryExecutionContext(QueryExecutionContext.QueryType.SSE, requestId, Long.toString(requestId),
+ QueryOptionsUtils.getWorkloadName(sqlNodeAndOptions.getOptions()), startTimeMs, deadlineMs, deadlineMs,
+ _brokerId, _brokerId, org.apache.pinot.spi.utils.CommonConstants.Broker.DEFAULT_QUERY_HASH);
+ try (QueryThreadContext ignore = QueryThreadContext.open(executionContext, _threadAccountant)) {
+ PinotQuery pinotQuery;
+ try {
+ pinotQuery = CalciteSqlParser.compileToPinotQuery(sqlNodeAndOptions);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return new BrokerResponseNative(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ Set<String> tableNames = RequestUtils.getTableNames(pinotQuery);
+ if (tableNames == null || tableNames.isEmpty()) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "Failed to extract table name");
+ }
+ if (tableNames.size() != 1) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "System tables do not support joins");
+ }
+ String tableName = tableNames.iterator().next();
+ if (!isSystemTable(tableName)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "Not a system table query");
+ }
+ AuthorizationResult authorizationResult =
+ hasTableAccess(requesterIdentity, Set.of(tableName), requestContext, httpHeaders);
+ if (!authorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return new BrokerResponseNative(QueryErrorCode.ACCESS_DENIED, authorizationResult.getFailureMessage());
+ }
+
+ return handleSystemTableQuery(pinotQuery, tableName, requestContext, requesterIdentity, query);
+ }
+ }
+
+ @Override
+ protected boolean handleCancel(long queryId, int timeoutMs, Executor executor,
+ HttpClientConnectionManager connMgr, Map<String, Integer> serverResponses) {
+ return false;
+ }
+
+ @Override
+ public boolean cancelQueryByClientId(String clientQueryId, int timeoutMs, Executor executor,
+ HttpClientConnectionManager connMgr, Map<String, Integer> serverResponses)
+ throws Exception {
+ return false;
+ }
+
+ @Override
+ public Map<Long, String> getRunningQueries() {
+ return Collections.emptyMap();
+ }
+
+ @Override
+ public OptionalLong getRequestIdByClientId(String clientQueryId) {
+ return OptionalLong.empty();
+ }
+
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ private BrokerResponse handleSystemTableQuery(PinotQuery pinotQuery, String tableName, RequestContext requestContext,
+ @Nullable RequesterIdentity requesterIdentity, String query) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableProvider provider = SystemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return BrokerResponseNative.TABLE_DOES_NOT_EXIST;
+ }
+ try {
+ IndexSegment dataSource = provider.getDataSource();
+ BrokerResponseNative brokerResponse;
+ try {
+ QueryContext queryContext = QueryContextConverterUtils.getQueryContext(pinotQuery);
+ queryContext.setSchema(provider.getSchema());
+ queryContext.setEndTimeMs(System.currentTimeMillis() + _brokerTimeoutMs);
+
+ Plan plan = _planMaker.makeInstancePlan(List.of(new SegmentContext(dataSource)), queryContext, _executorService,
+ null);
+ InstanceResponseBlock instanceResponse = plan.execute();
+
+ Map<ServerRoutingInstance, DataTable> dataTableMap = new HashMap<>(1);
+ dataTableMap.put(new ServerRoutingInstance("localhost", 0, TableType.OFFLINE), instanceResponse.toDataTable());
Review Comment:
The hardcoded `\"localhost\"` and port `0` values in `ServerRoutingInstance` are placeholder values that may be confusing. Consider using named constants like `SYSTEM_TABLE_PSEUDO_HOST = \"localhost\"` and `SYSTEM_TABLE_PSEUDO_PORT = 0` with a comment explaining these are synthetic routing instances for broker-local execution.
```suggestion
// Use a synthetic routing instance for broker-local execution of system table queries.
final String systemTablePseudoHost = "localhost";
final int systemTablePseudoPort = 0;
dataTableMap.put(
new ServerRoutingInstance(systemTablePseudoHost, systemTablePseudoPort, TableType.OFFLINE),
instanceResponse.toDataTable());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add system tables support with registry and broker wiring [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17291:
URL: https://github.com/apache/pinot/pull/17291#discussion_r2651350768
##########
pinot-plugins/pinot-system-table/src/main/java/org/apache/pinot/systemtable/provider/TablesSystemTableProvider.java:
##########
@@ -0,0 +1,662 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.systemtable.provider;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.databind.JsonNode;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.client.admin.PinotAdminClient;
+import org.apache.pinot.client.admin.PinotAdminTransport;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.systemtable.SystemTable;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.datasource.InMemorySystemTableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Basic system table exposing table-level metadata populated from the broker {@link TableCache}.
+ */
+@SystemTable
+public final class TablesSystemTableProvider implements SystemTableProvider {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TablesSystemTableProvider.class);
+ public static final String TABLE_NAME = "system.tables";
+ private static final String SIZE_CACHE_TTL_MS_PROPERTY = "pinot.systemtable.tables.sizeCacheTtlMs";
+ private static final long DEFAULT_SIZE_CACHE_TTL_MS = Duration.ofMinutes(1).toMillis();
+ private static final long SIZE_CACHE_TTL_MS = getNonNegativeLongProperty(SIZE_CACHE_TTL_MS_PROPERTY,
+ DEFAULT_SIZE_CACHE_TTL_MS);
+
+ private static final String CONTROLLER_TIMEOUT_MS_PROPERTY = "pinot.systemtable.tables.controllerTimeoutMs";
+ private static final long DEFAULT_CONTROLLER_TIMEOUT_MS = Duration.ofSeconds(5).toMillis();
+ private static final long CONTROLLER_TIMEOUT_MS = getPositiveLongProperty(CONTROLLER_TIMEOUT_MS_PROPERTY,
+ DEFAULT_CONTROLLER_TIMEOUT_MS);
+
+ private static final long SIZE_FETCH_FAILURE_WARN_INTERVAL_MS = Duration.ofHours(1).toMillis();
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension("tableName", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("type", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("status", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("segments", FieldSpec.DataType.INT)
+ .addSingleValueDimension("totalDocs", FieldSpec.DataType.LONG)
+ .addMetric("reportedSize", FieldSpec.DataType.LONG)
+ .addMetric("estimatedSize", FieldSpec.DataType.LONG)
+ .addSingleValueDimension("brokerTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("serverTenant", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("replicas", FieldSpec.DataType.INT)
+ .addSingleValueDimension("tableConfig", FieldSpec.DataType.STRING)
+ .build();
+
+ private final TableCache _tableCache;
+ private final @Nullable HelixAdmin _helixAdmin;
+ private final @Nullable String _clusterName;
+ private final @Nullable Function<String, TableSize> _tableSizeFetcherOverride;
+ private final List<String> _configuredControllerUrls;
+ private final Map<String, CachedSize> _sizeCache = new ConcurrentHashMap<>();
+ private final Map<String, PinotAdminClient> _adminClientCache = new ConcurrentHashMap<>();
+ private final AtomicLong _lastSizeFetchFailureWarnLogMs = new AtomicLong();
+
+ public TablesSystemTableProvider() {
+ this(null, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache) {
+ this(tableCache, null, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin) {
+ this(tableCache, helixAdmin, null, null, null);
+ }
+
+ public TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin,
+ @Nullable String clusterName) {
+ this(tableCache, helixAdmin, clusterName, null, null);
+ }
+
+ TablesSystemTableProvider(TableCache tableCache, @Nullable HelixAdmin helixAdmin, @Nullable String clusterName,
+ @Nullable Function<String, TableSize> tableSizeFetcherOverride, @Nullable List<String> controllerUrls) {
+ _tableCache = tableCache;
+ _helixAdmin = helixAdmin;
+ _clusterName = clusterName;
+ _tableSizeFetcherOverride = tableSizeFetcherOverride;
+ _configuredControllerUrls = controllerUrls != null ? new ArrayList<>(controllerUrls) : List.of();
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return SCHEMA;
+ }
+
+ @Override
+ public TableConfig getTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ }
+
+ @Override
+ public void close()
+ throws Exception {
+ for (Map.Entry<String, PinotAdminClient> entry : _adminClientCache.entrySet()) {
+ try {
+ entry.getValue().close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close admin client for {}: {}", entry.getKey(), e.toString());
+ }
+ }
+ _adminClientCache.clear();
+ }
+
+ @Override
+ public IndexSegment getDataSource() {
+ if (_tableCache == null) {
+ return new InMemorySystemTableSegment(TABLE_NAME, SCHEMA, 0, Collections.emptyMap());
+ }
+
+ Set<String> tableNamesWithType = new LinkedHashSet<>();
+ for (String tableName : _tableCache.getTableNameMap().values()) {
+ if (TableNameBuilder.getTableTypeFromTableName(tableName) != null) {
+ tableNamesWithType.add(tableName);
+ }
+ }
+ List<String> sortedTableNames = new ArrayList<>(tableNamesWithType);
+ sortedTableNames.sort(Comparator.naturalOrder());
+
+ List<String> controllerBaseUrls = getControllerBaseUrls();
+ Function<String, TableSize> sizeFetcher = getSizeFetcher();
+ class TableRow {
+ final String _tableNameWithType;
+ final TableType _tableType;
+ final String _rawTableName;
+ final @Nullable TableConfig _tableConfig;
+ private volatile @Nullable String _tableConfigJson;
+ private volatile @Nullable TableSize _tableSize;
+ private volatile boolean _tableSizeFetched;
+
+ private TableRow(String tableNameWithType, TableType tableType, String rawTableName,
+ @Nullable TableConfig tableConfig) {
+ _tableNameWithType = tableNameWithType;
+ _tableType = tableType;
+ _rawTableName = rawTableName;
+ _tableConfig = tableConfig;
+ }
+
+ @Nullable
+ private TableSize getTableSize() {
+ if (_tableSizeFetched) {
+ return _tableSize;
+ }
+ synchronized (this) {
+ if (_tableSizeFetched) {
+ return _tableSize;
+ }
+ _tableSize = fetchTableSize(_tableNameWithType, sizeFetcher, controllerBaseUrls);
+ _tableSizeFetched = true;
+ return _tableSize;
+ }
+ }
+
+ private String getStatus() {
+ if (_tableConfig != null) {
+ return "ONLINE";
+ }
+ TableSize sizeFromController = getTableSize();
+ int segments = sizeFromController != null ? getSegmentCount(sizeFromController, _tableType) : 0;
+ return segments > 0 ? "ONLINE" : "UNKNOWN";
+ }
+
+ private int getSegments() {
+ TableSize sizeFromController = getTableSize();
+ return sizeFromController != null ? getSegmentCount(sizeFromController, _tableType) : 0;
+ }
+
+ private long getTotalDocs() {
+ TableSize sizeFromController = getTableSize();
+ return sizeFromController != null ? TablesSystemTableProvider.this.getTotalDocs(sizeFromController, _tableType,
+ _tableNameWithType, controllerBaseUrls) : 0L;
+ }
+
+ private long getReportedSize() {
+ TableSize sizeFromController = getTableSize();
+ if (sizeFromController == null || sizeFromController._reportedSizeInBytes < 0) {
+ return 0L;
+ }
+ return sizeFromController._reportedSizeInBytes;
+ }
+
+ private long getEstimatedSize() {
+ TableSize sizeFromController = getTableSize();
+ if (sizeFromController == null || sizeFromController._estimatedSizeInBytes < 0) {
+ return 0L;
+ }
+ return sizeFromController._estimatedSizeInBytes;
+ }
+
+ private String getBrokerTenant() {
+ if (_tableConfig != null && _tableConfig.getTenantConfig() != null) {
+ String tenant = _tableConfig.getTenantConfig().getBroker();
+ return tenant != null ? tenant : "";
+ }
+ return "";
+ }
+
+ private String getServerTenant() {
+ if (_tableConfig != null && _tableConfig.getTenantConfig() != null) {
+ String tenant = _tableConfig.getTenantConfig().getServer();
+ return tenant != null ? tenant : "";
+ }
+ return "";
+ }
+
+ private int getReplicas() {
+ if (_tableConfig != null && _tableConfig.getValidationConfig() != null) {
+ Integer replicationNumber = _tableConfig.getValidationConfig().getReplicationNumber();
+ if (replicationNumber != null) {
+ return replicationNumber;
+ }
+ }
+ return 0;
+ }
+
+ private String getTableConfigJson() {
+ String cached = _tableConfigJson;
+ if (cached != null) {
+ return cached;
+ }
+ synchronized (this) {
+ cached = _tableConfigJson;
+ if (cached != null) {
+ return cached;
+ }
+ cached = "";
+ if (_tableConfig != null) {
+ try {
+ cached = JsonUtils.objectToString(_tableConfig);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to serialize table config for {}: {}", _tableNameWithType, e.toString());
+ cached = _tableConfig.toString();
+ }
+ }
+ _tableConfigJson = cached;
+ return cached;
+ }
+ }
+ }
+
+ List<TableRow> tableRows = new ArrayList<>(sortedTableNames.size());
+ for (String tableNameWithType : sortedTableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+ if (tableType == null) {
+ continue;
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableConfig tableConfig = _tableCache.getTableConfig(tableNameWithType);
+ tableRows.add(new TableRow(tableNameWithType, tableType, rawTableName, tableConfig));
+ }
+
+ Map<String, IntFunction<Object>> valueProviders = new java.util.HashMap<>();
+ valueProviders.put("tableName", docId -> tableRows.get(docId)._rawTableName);
+ valueProviders.put("type", docId -> tableRows.get(docId)._tableType.name());
+ valueProviders.put("status", docId -> tableRows.get(docId).getStatus());
+ valueProviders.put("segments", docId -> tableRows.get(docId).getSegments());
+ valueProviders.put("totalDocs", docId -> tableRows.get(docId).getTotalDocs());
+ valueProviders.put("reportedSize", docId -> tableRows.get(docId).getReportedSize());
+ valueProviders.put("estimatedSize", docId -> tableRows.get(docId).getEstimatedSize());
+ valueProviders.put("brokerTenant", docId -> tableRows.get(docId).getBrokerTenant());
+ valueProviders.put("serverTenant", docId -> tableRows.get(docId).getServerTenant());
+ valueProviders.put("replicas", docId -> tableRows.get(docId).getReplicas());
+ valueProviders.put("tableConfig", docId -> tableRows.get(docId).getTableConfigJson());
+
+ return new InMemorySystemTableSegment(TABLE_NAME, SCHEMA, tableRows.size(), valueProviders);
+ }
+
+ @Nullable
+ private TableSize fetchTableSize(String tableNameWithType,
+ @Nullable Function<String, TableSize> fetcher, List<String> controllerBaseUrls) {
+ boolean cacheEnabled = SIZE_CACHE_TTL_MS > 0;
+ TableSize cached = cacheEnabled ? getCachedSize(tableNameWithType) : null;
+ if (cached != null) {
+ return cached;
+ }
+ if (fetcher != null) {
+ try {
+ TableSize fetched = fetcher.apply(tableNameWithType);
+ if (fetched != null) {
+ if (cacheEnabled) {
+ cacheSize(tableNameWithType, fetched);
+ }
+ return fetched;
+ }
+ } catch (Exception e) {
+ LOGGER.warn("Table size fetcher failed for {}: {}", tableNameWithType, e.toString());
+ }
+ }
+ String rawTableName = TableNameBuilder.extractRawTableName(tableNameWithType);
+ TableSize size = fetchTableSizeForName(controllerBaseUrls, rawTableName);
+ if (size == null) {
+ size = fetchTableSizeForName(controllerBaseUrls, tableNameWithType);
+ if (size == null) {
+ logSizeFetchFailure("{}: failed to fetch size for {} from controllers {} "
+ + "(tried raw table name '{}' and table name with type '{}')",
+ TABLE_NAME, tableNameWithType, controllerBaseUrls, rawTableName, tableNameWithType);
+ }
+ }
+ if (size != null && cacheEnabled) {
+ cacheSize(tableNameWithType, size);
+ }
+ return size;
+ }
+
+ @Nullable
+ private TableSize fetchTableSizeForName(List<String> controllerBaseUrls, String tableName) {
+ for (String baseUrl : controllerBaseUrls) {
+ try {
+ PinotAdminClient adminClient = getOrCreateAdminClient(baseUrl);
+ if (adminClient == null) {
+ continue;
+ }
+
+ JsonNode sizeNode = adminClient.getTableSize(tableName, true, false);
+
+ if (sizeNode == null) {
+ continue;
+ }
+
+ TableSize parsed = JsonUtils.stringToObject(sizeNode.toString(), TableSize.class);
+ LOGGER.debug("{}: controller size response for {} via {} -> segments offline={}, realtime={}, "
+ + "reportedSize={}, estimatedSize={}", TABLE_NAME, tableName, baseUrl,
+ parsed._offlineSegments != null && parsed._offlineSegments._segments != null
+ ? parsed._offlineSegments._segments.size() : 0,
+ parsed._realtimeSegments != null && parsed._realtimeSegments._segments != null
+ ? parsed._realtimeSegments._segments.size() : 0,
+ parsed._reportedSizeInBytes, parsed._estimatedSizeInBytes);
+ return parsed;
+ } catch (Exception e) {
+ logSizeFetchFailure("{}: error fetching table size for {} via {} using admin client", TABLE_NAME, tableName,
+ baseUrl, e);
+ }
+ }
+ return null;
+ }
+
+ private List<String> getControllerBaseUrls() {
+ Set<String> urls = new LinkedHashSet<>();
+ if (_helixAdmin != null) {
+ for (String controller : discoverControllersFromHelix()) {
+ String normalized = normalizeControllerUrl(controller);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ }
+ for (String url : _configuredControllerUrls) {
+ String normalized = normalizeControllerUrl(url);
+ if (normalized != null) {
+ urls.add(normalized);
+ }
+ }
+ return new ArrayList<>(urls);
+ }
+
+ private int getSegmentCount(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.size();
+ }
+ if (tableType == TableType.REALTIME && sizeFromController._realtimeSegments != null
+ && sizeFromController._realtimeSegments._segments != null) {
+ return sizeFromController._realtimeSegments._segments.size();
+ }
+ return 0;
+ }
+
+ private long getTotalDocsFromSize(TableSize sizeFromController, TableType tableType) {
+ if (tableType == TableType.OFFLINE && sizeFromController._offlineSegments != null
+ && sizeFromController._offlineSegments._segments != null) {
+ return sizeFromController._offlineSegments._segments.values().stream()
+ .mapToLong(segmentSize -> segmentSize._totalDocs).sum();
Review Comment:
The stream operation iterates over all segment sizes to compute the sum. For tables with thousands of segments, this could be inefficient. Consider caching the computed sum or computing it once during the controller API response deserialization.
##########
pinot-common/src/main/java/org/apache/pinot/common/systemtable/SystemTableRegistry.java:
##########
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.IdentityHashMap;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.helix.HelixAdmin;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.PinotReflectionUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Registry to hold and lifecycle-manage system table data providers.
+ */
+public final class SystemTableRegistry {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableRegistry.class);
+
+ // Providers are registered once at broker startup.
+ private static final Map<String, SystemTableProvider> PROVIDERS = new HashMap<>();
+
+ private SystemTableRegistry() {
+ throw new UnsupportedOperationException("This is a utility class and cannot be instantiated");
+ }
+
+ public static void register(SystemTableProvider provider) {
+ synchronized (PROVIDERS) {
+ PROVIDERS.put(normalize(provider.getTableName()), provider);
+ }
+ }
+
+ @Nullable
+ public static SystemTableProvider get(String tableName) {
+ synchronized (PROVIDERS) {
+ return PROVIDERS.get(normalize(tableName));
+ }
+ }
+
+ public static boolean isRegistered(String tableName) {
+ synchronized (PROVIDERS) {
+ return PROVIDERS.containsKey(normalize(tableName));
+ }
+ }
+
+ public static Collection<SystemTableProvider> getProviders() {
+ synchronized (PROVIDERS) {
+ return Collections.unmodifiableCollection(new java.util.ArrayList<>(PROVIDERS.values()));
+ }
+ }
+
+ /**
+ * Discover and register providers marked with {@link SystemTable} using the available dependencies.
+ * Follows the ScalarFunction pattern: any class annotated with @SystemTable under a "*.systemtable.*" package
+ * will be discovered via reflection and registered.
+ */
+ public static void registerAnnotatedProviders(TableCache tableCache, HelixAdmin helixAdmin, String clusterName,
+ @Nullable PinotConfiguration config) {
+ Set<Class<?>> classes =
+ PinotReflectionUtils.getClassesThroughReflection(".*\\.systemtable\\..*", SystemTable.class);
+ for (Class<?> clazz : classes) {
+ if (!SystemTableProvider.class.isAssignableFrom(clazz)) {
+ continue;
+ }
+ SystemTableProvider provider =
+ instantiateProvider(clazz.asSubclass(SystemTableProvider.class), tableCache, helixAdmin, clusterName, config);
+ if (provider == null) {
+ continue;
+ }
+ if (isRegistered(provider.getTableName())) {
+ continue;
+ }
+ LOGGER.info("Registering system table provider: {}", provider.getTableName());
+ register(provider);
+ }
+ }
+
+ public static void close()
+ throws Exception {
+ Exception firstException = null;
+ // Snapshot providers to avoid concurrent modifications and to close each provider at most once.
+ Map<SystemTableProvider, Boolean> providersToClose = new IdentityHashMap<>();
+ synchronized (PROVIDERS) {
+ for (SystemTableProvider provider : PROVIDERS.values()) {
+ providersToClose.put(provider, Boolean.TRUE);
+ }
+ }
+ try {
+ for (SystemTableProvider provider : providersToClose.keySet()) {
+ try {
+ provider.close();
+ } catch (Exception e) {
+ if (firstException == null) {
+ firstException = e;
+ } else {
+ firstException.addSuppressed(e);
+ }
+ }
+ }
+ } finally {
+ synchronized (PROVIDERS) {
+ PROVIDERS.clear();
+ }
+ }
+ if (firstException != null) {
+ throw firstException;
+ }
+ }
+
+ /**
+ * Initialize and register all annotated system table providers.
+ */
+ public static void init(TableCache tableCache, HelixAdmin helixAdmin, String clusterName) {
+ init(tableCache, helixAdmin, clusterName, null);
+ }
+
+ /**
+ * Initialize and register all annotated system table providers.
+ */
+ public static void init(TableCache tableCache, HelixAdmin helixAdmin, String clusterName,
+ @Nullable PinotConfiguration config) {
+ registerAnnotatedProviders(tableCache, helixAdmin, clusterName, config);
+ }
+
+ private static String normalize(String tableName) {
+ return tableName.toLowerCase(Locale.ROOT);
+ }
+
+ @Nullable
+ private static SystemTableProvider instantiateProvider(Class<? extends SystemTableProvider> clazz,
+ TableCache tableCache, HelixAdmin helixAdmin, String clusterName, @Nullable PinotConfiguration config) {
+ try {
+ // Prefer the most specific constructor available.
+ // System table providers may declare any of the following constructors, in decreasing order of specificity:
+ // (TableCache, HelixAdmin, String, PinotConfiguration), (TableCache, HelixAdmin, String),
+ // (TableCache, HelixAdmin), (TableCache), (PinotConfiguration), or a no-arg constructor.
+ // The registry will select the most specific available constructor so providers can opt in to the dependencies
+ // they need without forcing all implementations to depend on Helix or cluster metadata.
+ if (config != null) {
+ try {
+ return clazz.getConstructor(TableCache.class, HelixAdmin.class, String.class, PinotConfiguration.class)
+ .newInstance(tableCache, helixAdmin, clusterName, config);
+ } catch (NoSuchMethodException ignored) {
+ // fall through
+ }
+ }
Review Comment:
The constructor resolution logic contains deeply nested try-catch blocks, making it difficult to follow. Consider extracting the constructor lookup logic into a separate method or using a more declarative approach with an ordered list of constructor signatures to try.
##########
pinot-plugins/pinot-system-table/src/main/java/org/apache/pinot/common/systemtable/datasource/InMemorySystemTableSegment.java:
##########
@@ -0,0 +1,724 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.systemtable.datasource;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.node.JsonNodeFactory;
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.IntFunction;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.Constants;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.SegmentVersion;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.datasource.DataSourceMetadata;
+import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexType;
+import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.column.ColumnIndexContainer;
+import org.apache.pinot.segment.spi.index.multicolumntext.MultiColumnTextMetadata;
+import org.apache.pinot.segment.spi.index.mutable.ThreadSafeMutableRoaringBitmap;
+import org.apache.pinot.segment.spi.index.reader.BloomFilterReader;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReaderContext;
+import org.apache.pinot.segment.spi.index.reader.H3IndexReader;
+import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
+import org.apache.pinot.segment.spi.index.reader.JsonIndexReader;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+import org.apache.pinot.segment.spi.index.reader.RangeIndexReader;
+import org.apache.pinot.segment.spi.index.reader.TextIndexReader;
+import org.apache.pinot.segment.spi.index.reader.VectorIndexReader;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2Metadata;
+import org.apache.pinot.segment.spi.partition.PartitionFunction;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.joda.time.Duration;
+import org.joda.time.Interval;
+
+
+/**
+ * In-memory {@link IndexSegment} implementation intended for system table queries.
+ * <p>
+ * This segment is backed by per-column value functions (docId -> value) and exposes raw forward indexes (no
+ * dictionaries/inverted indexes) so that the standard v1 query engine can operate on it.
+ */
+public final class InMemorySystemTableSegment implements IndexSegment {
+ private final String _segmentName;
+ private final Schema _schema;
+ private final int _numDocs;
+ private final Map<String, IntFunction<Object>> _valueProvidersByColumn;
+ private final Map<String, DataSource> _dataSourcesByColumn;
+ private final SegmentMetadata _segmentMetadata;
+
+ public InMemorySystemTableSegment(String segmentName, Schema schema, int numDocs,
+ Map<String, IntFunction<Object>> valueProvidersByColumn) {
+ _segmentName = segmentName;
+ _schema = schema;
+ _numDocs = numDocs;
+ _valueProvidersByColumn = new HashMap<>(valueProvidersByColumn);
+ _dataSourcesByColumn = new HashMap<>(schema.getColumnNames().size());
+ for (String column : schema.getColumnNames()) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ if (fieldSpec == null) {
+ continue;
+ }
+ IntFunction<Object> provider = _valueProvidersByColumn.get(column);
+ if (provider == null) {
+ provider = docId -> defaultValue(fieldSpec.getDataType());
+ _valueProvidersByColumn.put(column, provider);
+ }
+ _dataSourcesByColumn.put(column, new FunctionBasedDataSource(fieldSpec, numDocs, provider));
+ }
+ _segmentMetadata = new InMemorySegmentMetadata(segmentName, schema, numDocs);
+ }
+
+ @Override
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ @Override
+ public SegmentMetadata getSegmentMetadata() {
+ return _segmentMetadata;
+ }
+
+ @Override
+ public Set<String> getColumnNames() {
+ return _schema.getColumnNames();
+ }
+
+ @Override
+ public Set<String> getPhysicalColumnNames() {
+ return _schema.getPhysicalColumnNames();
+ }
+
+ @Nullable
+ @Override
+ public DataSource getDataSourceNullable(String column) {
+ return _dataSourcesByColumn.get(column);
+ }
+
+ @Override
+ public DataSource getDataSource(String column, Schema schema) {
+ DataSource dataSource = getDataSourceNullable(column);
+ if (dataSource != null) {
+ return dataSource;
+ }
+ throw new IllegalStateException("Failed to find data source for column: " + column);
+ }
+
+ @Nullable
+ @Override
+ public List<StarTreeV2> getStarTrees() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public TextIndexReader getMultiColumnTextIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public ThreadSafeMutableRoaringBitmap getValidDocIds() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public ThreadSafeMutableRoaringBitmap getQueryableDocIds() {
+ return null;
+ }
+
+ @Override
+ public GenericRow getRecord(int docId, GenericRow reuse) {
+ GenericRow row = reuse != null ? reuse : new GenericRow();
+ row.getFieldToValueMap().clear();
+ for (String column : _schema.getColumnNames()) {
+ row.putValue(column, getValue(docId, column));
+ }
+ return row;
+ }
+
+ @Override
+ public Object getValue(int docId, String column) {
+ IntFunction<Object> provider = _valueProvidersByColumn.get(column);
+ if (provider == null) {
+ return null;
+ }
+ return provider.apply(docId);
+ }
+
+ @Override
+ public void offload() {
+ }
+
+ @Override
+ public void destroy() {
+ }
+
+ private static Object defaultValue(FieldSpec.DataType dataType) {
+ switch (dataType) {
+ case INT:
+ return 0;
+ case LONG:
+ return 0L;
+ case FLOAT:
+ return 0.0f;
+ case DOUBLE:
+ return 0.0d;
+ case BIG_DECIMAL:
+ return BigDecimal.ZERO;
+ case BOOLEAN:
+ return false;
+ case STRING:
+ return "";
+ case BYTES:
+ return new byte[0];
+ default:
+ return null;
+ }
+ }
+
+ private static final class InMemorySegmentMetadata implements SegmentMetadata {
+ private final String _segmentName;
+ private final Schema _schema;
+ private final int _totalDocs;
+ private final TreeMap<String, ColumnMetadata> _columnMetadataMap;
+
+ private InMemorySegmentMetadata(String segmentName, Schema schema, int totalDocs) {
+ _segmentName = segmentName;
+ _schema = schema;
+ _totalDocs = totalDocs;
+ _columnMetadataMap = new TreeMap<>();
+ for (String column : schema.getColumnNames()) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ if (fieldSpec != null) {
+ _columnMetadataMap.put(column, new InMemoryColumnMetadata(fieldSpec, totalDocs));
+ }
+ }
+ }
+
+ @Deprecated
+ @Override
+ public String getTableName() {
+ return _schema.getSchemaName();
+ }
+
+ @Override
+ public String getName() {
+ return _segmentName;
+ }
+
+ @Override
+ public String getTimeColumn() {
+ return "";
+ }
+
+ @Override
+ public long getStartTime() {
+ return 0;
+ }
+
+ @Override
+ public long getEndTime() {
+ return 0;
+ }
+
+ @Override
+ public TimeUnit getTimeUnit() {
+ return TimeUnit.MILLISECONDS;
+ }
+
+ @Override
+ public Duration getTimeGranularity() {
+ return Duration.ZERO;
+ }
+
+ @Override
+ public Interval getTimeInterval() {
+ return new Interval(0L, 0L);
+ }
+
+ @Override
+ public String getCrc() {
+ return "";
+ }
+
+ @Override
+ public String getDataCrc() {
+ return "";
+ }
+
+ @Override
+ public SegmentVersion getVersion() {
+ return SegmentVersion.v3;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return _schema;
+ }
+
+ @Override
+ public int getTotalDocs() {
+ return _totalDocs;
+ }
+
+ @Override
+ public File getIndexDir() {
+ return new File("");
+ }
+
+ @Nullable
+ @Override
+ public String getCreatorName() {
+ return "systemtable";
+ }
+
+ @Override
+ public long getIndexCreationTime() {
+ return 0;
+ }
+
+ @Override
+ public long getLastIndexedTimestamp() {
+ return 0;
+ }
+
+ @Override
+ public long getLatestIngestionTimestamp() {
+ return Long.MIN_VALUE;
+ }
+
+ @Nullable
+ @Override
+ public List<StarTreeV2Metadata> getStarTreeV2MetadataList() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public MultiColumnTextMetadata getMultiColumnTextMetadata() {
+ return null;
+ }
+
+ @Override
+ public Map<String, String> getCustomMap() {
+ return Collections.emptyMap();
+ }
+
+ @Override
+ public String getStartOffset() {
+ return "";
+ }
+
+ @Override
+ public String getEndOffset() {
+ return "";
+ }
+
+ @Override
+ public TreeMap<String, ColumnMetadata> getColumnMetadataMap() {
+ return _columnMetadataMap;
+ }
+
+ @Override
+ public void removeColumn(String column) {
+ _columnMetadataMap.remove(column);
+ }
+
+ @Override
+ public JsonNode toJson(@Nullable Set<String> columnFilter) {
+ return JsonNodeFactory.instance.objectNode();
+ }
+ }
+
+ private static final class InMemoryColumnMetadata implements ColumnMetadata {
+ private final FieldSpec _fieldSpec;
+ private final int _totalDocs;
+
+ private InMemoryColumnMetadata(FieldSpec fieldSpec, int totalDocs) {
+ _fieldSpec = fieldSpec;
+ _totalDocs = totalDocs;
+ }
+
+ @Override
+ public FieldSpec getFieldSpec() {
+ return _fieldSpec;
+ }
+
+ @Override
+ public int getTotalDocs() {
+ return _totalDocs;
+ }
+
+ @Override
+ public int getCardinality() {
+ return Constants.UNKNOWN_CARDINALITY;
+ }
+
+ @Override
+ public boolean isSorted() {
+ return false;
+ }
+
+ @Override
+ public Comparable getMinValue() {
+ return defaultComparableValue(_fieldSpec.getDataType());
+ }
+
+ @Override
+ public Comparable getMaxValue() {
+ return defaultComparableValue(_fieldSpec.getDataType());
+ }
+
+ @Override
+ public boolean hasDictionary() {
+ return false;
+ }
+
+ @Override
+ public int getColumnMaxLength() {
+ return 0;
+ }
+
+ @Override
+ public int getBitsPerElement() {
+ return 0;
+ }
+
+ @Override
+ public int getMaxNumberOfMultiValues() {
+ return 0;
+ }
+
+ @Override
+ public int getTotalNumberOfEntries() {
+ return _totalDocs;
+ }
+
+ @Nullable
+ @Override
+ public PartitionFunction getPartitionFunction() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public Set<Integer> getPartitions() {
+ return null;
+ }
+
+ @Override
+ public boolean isAutoGenerated() {
+ return false;
+ }
+
+ @Override
+ public Map<IndexType<?, ?, ?>, Long> getIndexSizeMap() {
+ return Collections.emptyMap();
+ }
+
+ private static Comparable defaultComparableValue(FieldSpec.DataType dataType) {
+ switch (dataType) {
+ case INT:
+ return 0;
+ case LONG:
+ return 0L;
+ case FLOAT:
+ return 0.0f;
+ case DOUBLE:
+ return 0.0d;
+ case BIG_DECIMAL:
+ return BigDecimal.ZERO;
+ case BOOLEAN:
+ return false;
+ case STRING:
+ return "";
+ default:
+ return 0;
+ }
+ }
+ }
+
+ private static final class FunctionBasedDataSource implements DataSource {
+ private final DataSourceMetadata _metadata;
+ private final ColumnIndexContainer _indexContainer;
+ private final ForwardIndexReader<?> _forwardIndex;
+
+ private FunctionBasedDataSource(FieldSpec fieldSpec, int numDocs, IntFunction<Object> valueProvider) {
+ _metadata = new FunctionBasedDataSourceMetadata(fieldSpec, numDocs);
+ _indexContainer = ColumnIndexContainer.Empty.INSTANCE;
+ _forwardIndex = new FunctionBasedForwardIndexReader(fieldSpec.getDataType(), valueProvider);
+ }
+
+ @Override
+ public DataSourceMetadata getDataSourceMetadata() {
+ return _metadata;
+ }
+
+ @Override
+ public ColumnIndexContainer getIndexContainer() {
+ return _indexContainer;
+ }
+
+ @Override
+ public <R extends IndexReader> R getIndex(IndexType<?, R, ?> type) {
+ if (type == null) {
+ return null;
+ }
+ if (StandardIndexes.FORWARD_ID.equals(type.getId())) {
+ return (R) _forwardIndex;
+ }
+ return null;
+ }
+
+ @Override
+ public ForwardIndexReader<?> getForwardIndex() {
+ return _forwardIndex;
+ }
+
+ @Nullable
+ @Override
+ public Dictionary getDictionary() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public InvertedIndexReader<?> getInvertedIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public RangeIndexReader<?> getRangeIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public TextIndexReader getTextIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public TextIndexReader getFSTIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public TextIndexReader getIFSTIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public JsonIndexReader getJsonIndex() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public H3IndexReader getH3Index() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public BloomFilterReader getBloomFilter() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public NullValueVectorReader getNullValueVector() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public VectorIndexReader getVectorIndex() {
+ return null;
+ }
+ }
+
+ private static final class FunctionBasedDataSourceMetadata implements DataSourceMetadata {
+ private final FieldSpec _fieldSpec;
+ private final int _numDocs;
+
+ private FunctionBasedDataSourceMetadata(FieldSpec fieldSpec, int numDocs) {
+ _fieldSpec = fieldSpec;
+ _numDocs = numDocs;
+ }
+
+ @Override
+ public FieldSpec getFieldSpec() {
+ return _fieldSpec;
+ }
+
+ @Override
+ public boolean isSorted() {
+ return false;
+ }
+
+ @Override
+ public int getNumDocs() {
+ return _numDocs;
+ }
+
+ @Override
+ public int getNumValues() {
+ return _numDocs;
+ }
+
+ @Override
+ public int getMaxNumValuesPerMVEntry() {
+ return -1;
+ }
+
+ @Nullable
+ @Override
+ public Comparable getMinValue() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public Comparable getMaxValue() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public PartitionFunction getPartitionFunction() {
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public Set<Integer> getPartitions() {
+ return null;
+ }
+
+ @Override
+ public int getCardinality() {
+ return -1;
+ }
+ }
+
+ private static final class FunctionBasedForwardIndexReader implements ForwardIndexReader<ForwardIndexReaderContext> {
+ private final FieldSpec.DataType _storedType;
+ private final IntFunction<Object> _valueProvider;
+
+ private FunctionBasedForwardIndexReader(FieldSpec.DataType storedType, IntFunction<Object> valueProvider) {
+ _storedType = storedType;
+ _valueProvider = valueProvider;
+ }
+
+ @Override
+ public boolean isDictionaryEncoded() {
+ return false;
+ }
+
+ @Override
+ public boolean isSingleValue() {
+ return true;
+ }
+
+ @Override
+ public FieldSpec.DataType getStoredType() {
+ return _storedType;
+ }
+
+ @Override
+ public int getInt(int docId, ForwardIndexReaderContext context) {
+ Object value = _valueProvider.apply(docId);
+ return value instanceof Number ? ((Number) value).intValue() : Integer.parseInt(String.valueOf(value));
Review Comment:
The type conversion pattern with `String.valueOf()` fallback is repeated across multiple methods (getInt, getLong, getFloat, getDouble). Extract this into a reusable helper method to reduce duplication.
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/SystemTableBrokerRequestHandler.java:
##########
@@ -0,0 +1,546 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.requesthandler;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.Nullable;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.client5.http.classic.methods.HttpPost;
+import org.apache.hc.client5.http.config.RequestConfig;
+import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
+import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse;
+import org.apache.hc.client5.http.impl.classic.HttpClients;
+import org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager;
+import org.apache.hc.client5.http.io.HttpClientConnectionManager;
+import org.apache.hc.core5.http.io.entity.EntityUtils;
+import org.apache.hc.core5.http.io.entity.StringEntity;
+import org.apache.hc.core5.util.Timeout;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.HelixManager;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.pinot.broker.api.AccessControl;
+import org.apache.pinot.broker.broker.AccessControlFactory;
+import org.apache.pinot.broker.querylog.QueryLogger;
+import org.apache.pinot.broker.queryquota.QueryQuotaManager;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.common.datatable.DataTableFactory;
+import org.apache.pinot.common.datatable.DataTableImplV4;
+import org.apache.pinot.common.http.PoolingHttpClientConnectionManagerHelper;
+import org.apache.pinot.common.metrics.BrokerMeter;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.request.QuerySource;
+import org.apache.pinot.common.response.BrokerResponse;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.SystemTableRegistry;
+import org.apache.pinot.common.utils.NamedThreadFactory;
+import org.apache.pinot.common.utils.config.InstanceUtils;
+import org.apache.pinot.common.utils.config.QueryOptionsUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.core.operator.blocks.InstanceResponseBlock;
+import org.apache.pinot.core.plan.Plan;
+import org.apache.pinot.core.plan.maker.InstancePlanMakerImplV2;
+import org.apache.pinot.core.plan.maker.PlanMaker;
+import org.apache.pinot.core.query.reduce.BrokerReduceService;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils;
+import org.apache.pinot.core.routing.RoutingManager;
+import org.apache.pinot.core.transport.ServerRoutingInstance;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.SegmentContext;
+import org.apache.pinot.spi.accounting.ThreadAccountant;
+import org.apache.pinot.spi.auth.AuthorizationResult;
+import org.apache.pinot.spi.auth.broker.RequesterIdentity;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.query.QueryExecutionContext;
+import org.apache.pinot.spi.query.QueryThreadContext;
+import org.apache.pinot.spi.trace.RequestContext;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+import org.apache.pinot.sql.parsers.SqlNodeAndOptions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Broker request handler for system tables (handled entirely on the broker).
+ */
+public class SystemTableBrokerRequestHandler extends BaseBrokerRequestHandler {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableBrokerRequestHandler.class);
+ private static final String SYSTEM_TABLE_PSEUDO_HOST = "localhost";
+ private static final int SYSTEM_TABLE_PSEUDO_PORT = 0;
+ private static final String SYSTEM_TABLE_DATATABLE_API_PATH = "/query/systemTable/datatable";
+
+ private final BrokerReduceService _brokerReduceService;
+ private final PlanMaker _planMaker;
+ private final ExecutorService _executorService;
+ private final ExecutorService _scatterGatherExecutorService;
+ private final PoolingHttpClientConnectionManager _scatterGatherConnMgr;
+ private final CloseableHttpClient _scatterGatherHttpClient;
+ @Nullable
+ private final HelixManager _helixManager;
+
+ public SystemTableBrokerRequestHandler(PinotConfiguration config, String brokerId,
+ BrokerRequestIdGenerator requestIdGenerator, RoutingManager routingManager,
+ AccessControlFactory accessControlFactory, QueryQuotaManager queryQuotaManager, TableCache tableCache,
+ ThreadAccountant threadAccountant, @Nullable HelixManager helixManager) {
+ super(config, brokerId, requestIdGenerator, routingManager, accessControlFactory, queryQuotaManager, tableCache,
+ threadAccountant);
+ _brokerReduceService = new BrokerReduceService(_config);
+ _planMaker = new InstancePlanMakerImplV2();
+ _planMaker.init(_config);
+ _helixManager = helixManager;
+ int executorPoolSize = config.getProperty(CommonConstants.Broker.CONFIG_OF_SYSTEM_TABLE_EXECUTOR_POOL_SIZE,
+ CommonConstants.Broker.DEFAULT_SYSTEM_TABLE_EXECUTOR_POOL_SIZE);
+ executorPoolSize = Math.max(1, executorPoolSize);
+ _executorService = QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(executorPoolSize,
+ new NamedThreadFactory("system-table-query-executor")));
+ _scatterGatherExecutorService =
+ QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(executorPoolSize,
+ new NamedThreadFactory("system-table-scatter-gather-executor")));
+ _scatterGatherConnMgr = PoolingHttpClientConnectionManagerHelper.createWithSocketFactory();
+ Timeout timeout = Timeout.of(_brokerTimeoutMs, TimeUnit.MILLISECONDS);
+ RequestConfig defaultRequestConfig =
+ RequestConfig.custom().setConnectionRequestTimeout(timeout).setResponseTimeout(timeout).build();
+ _scatterGatherHttpClient =
+ HttpClients.custom().setConnectionManager(_scatterGatherConnMgr).setDefaultRequestConfig(defaultRequestConfig)
+ .build();
+ }
+
+ @Override
+ public void start() {
+ }
+
+ @Override
+ public void shutDown() {
+ _executorService.shutdownNow();
+ _scatterGatherExecutorService.shutdownNow();
+ try {
+ _scatterGatherHttpClient.close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close system table scatter-gather http client: {}", e.toString());
+ }
+ try {
+ _scatterGatherConnMgr.close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close system table scatter-gather connection manager: {}", e.toString());
+ }
+ _brokerReduceService.shutDown();
+ }
+
+ public boolean canHandle(String tableName) {
+ return isSystemTable(tableName) && SystemTableRegistry.isRegistered(tableName);
+ }
+
+ @Override
+ protected BrokerResponse handleRequest(long requestId, String query, SqlNodeAndOptions sqlNodeAndOptions,
+ JsonNode request, @Nullable RequesterIdentity requesterIdentity, RequestContext requestContext,
+ @Nullable HttpHeaders httpHeaders, AccessControl accessControl)
+ throws Exception {
+ long startTimeMs = requestContext.getRequestArrivalTimeMillis();
+ long deadlineMs = startTimeMs + _brokerTimeoutMs;
+ QueryExecutionContext executionContext =
+ new QueryExecutionContext(QueryExecutionContext.QueryType.STE, requestId, Long.toString(requestId),
+ QueryOptionsUtils.getWorkloadName(sqlNodeAndOptions.getOptions()), startTimeMs, deadlineMs, deadlineMs,
+ _brokerId, _brokerId, org.apache.pinot.spi.utils.CommonConstants.Broker.DEFAULT_QUERY_HASH);
+ try (QueryThreadContext ignore = QueryThreadContext.open(executionContext, _threadAccountant)) {
+ PinotQuery pinotQuery;
+ try {
+ pinotQuery = CalciteSqlParser.compileToPinotQuery(sqlNodeAndOptions);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return new BrokerResponseNative(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ Set<String> tableNames = RequestUtils.getTableNames(pinotQuery);
+ if (tableNames == null || tableNames.isEmpty()) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "Failed to extract table name");
+ }
+ if (tableNames.size() != 1) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "System tables do not support joins");
+ }
+ String tableName = tableNames.iterator().next();
+ if (!isSystemTable(tableName)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "Not a system table query");
+ }
+ AuthorizationResult authorizationResult =
+ hasTableAccess(requesterIdentity, Set.of(tableName), requestContext, httpHeaders);
+ if (!authorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return new BrokerResponseNative(QueryErrorCode.ACCESS_DENIED, authorizationResult.getFailureMessage());
+ }
+
+ return handleSystemTableQuery(request, pinotQuery, tableName, requestContext, requesterIdentity, query,
+ httpHeaders);
+ }
+ }
+
+ @Override
+ protected boolean handleCancel(long queryId, int timeoutMs, Executor executor,
+ HttpClientConnectionManager connMgr, Map<String, Integer> serverResponses) {
+ return false;
+ }
+
+ @Override
+ public boolean cancelQueryByClientId(String clientQueryId, int timeoutMs, Executor executor,
+ HttpClientConnectionManager connMgr, Map<String, Integer> serverResponses)
+ throws Exception {
+ return false;
+ }
+
+ @Override
+ public Map<Long, String> getRunningQueries() {
+ return Collections.emptyMap();
+ }
+
+ @Override
+ public OptionalLong getRequestIdByClientId(String clientQueryId) {
+ return OptionalLong.empty();
+ }
+
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ /**
+ * Executes a system table query against the local broker and returns the raw {@link DataTable} results.
+ * <p>
+ * This method is used by the internal broker-to-broker scatter-gather endpoint and must never perform fanout.
+ */
+ public DataTable handleSystemTableDataTableRequest(JsonNode request, @Nullable RequesterIdentity requesterIdentity,
+ RequestContext requestContext, @Nullable HttpHeaders httpHeaders) {
+ long startTimeMs = requestContext.getRequestArrivalTimeMillis();
+ if (startTimeMs <= 0) {
+ startTimeMs = System.currentTimeMillis();
+ requestContext.setRequestArrivalTimeMillis(startTimeMs);
+ }
+ long requestId = _requestIdGenerator.get();
+ long deadlineMs = startTimeMs + _brokerTimeoutMs;
+
+ JsonNode sql = request.get(CommonConstants.Broker.Request.SQL);
+ if (sql == null || !sql.isTextual()) {
+ return exceptionDataTable(QueryErrorCode.JSON_PARSING, "Failed to find 'sql' in the request: " + request);
+ }
+ String query = sql.textValue();
+ requestContext.setQuery(query);
+
+ SqlNodeAndOptions sqlNodeAndOptions;
+ try {
+ sqlNodeAndOptions = RequestUtils.parseQuery(query, request);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return exceptionDataTable(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ QueryExecutionContext executionContext =
+ new QueryExecutionContext(QueryExecutionContext.QueryType.STE, requestId, Long.toString(requestId),
+ QueryOptionsUtils.getWorkloadName(sqlNodeAndOptions.getOptions()), startTimeMs, deadlineMs, deadlineMs,
+ _brokerId, _brokerId, org.apache.pinot.spi.utils.CommonConstants.Broker.DEFAULT_QUERY_HASH);
+ try (QueryThreadContext ignore = QueryThreadContext.open(executionContext, _threadAccountant)) {
+ AccessControl accessControl = _accessControlFactory.create();
+ AuthorizationResult authorizationResult = accessControl.authorize(requesterIdentity);
+ if (!authorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return exceptionDataTable(QueryErrorCode.ACCESS_DENIED, authorizationResult.getFailureMessage());
+ }
+
+ PinotQuery pinotQuery;
+ try {
+ pinotQuery = CalciteSqlParser.compileToPinotQuery(sqlNodeAndOptions);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return exceptionDataTable(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ Set<String> tableNames = RequestUtils.getTableNames(pinotQuery);
+ if (tableNames == null || tableNames.isEmpty()) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, "Failed to extract table name");
+ }
+ if (tableNames.size() != 1) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, "System tables do not support joins");
+ }
+ String tableName = tableNames.iterator().next();
+ if (!isSystemTable(tableName)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, "Not a system table query");
+ }
+
+ AuthorizationResult tableAuthorizationResult =
+ hasTableAccess(requesterIdentity, Set.of(tableName), requestContext, httpHeaders);
+ if (!tableAuthorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return exceptionDataTable(QueryErrorCode.ACCESS_DENIED, tableAuthorizationResult.getFailureMessage());
+ }
+
+ SystemTableProvider provider = SystemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return exceptionDataTable(QueryErrorCode.TABLE_DOES_NOT_EXIST, "System table does not exist: " + tableName);
+ }
+
+ try {
+ return executeLocalSystemTableQuery(pinotQuery, provider);
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table datatable query {}: {}", tableName, e.getMessage(),
+ e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return exceptionDataTable(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+ }
+
+ private BrokerResponse handleSystemTableQuery(JsonNode request, PinotQuery pinotQuery, String tableName,
+ RequestContext requestContext, @Nullable RequesterIdentity requesterIdentity, String query,
+ @Nullable HttpHeaders httpHeaders) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableProvider provider = SystemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return BrokerResponseNative.TABLE_DOES_NOT_EXIST;
+ }
+ try {
+ Map<ServerRoutingInstance, DataTable> dataTableMap;
+ if (provider.getExecutionMode() == SystemTableProvider.ExecutionMode.BROKER_SCATTER_GATHER) {
+ dataTableMap = scatterGatherSystemTableDataTables(provider, pinotQuery, tableName, request, httpHeaders);
+ } else {
+ dataTableMap = new HashMap<>(1);
+ // Use a synthetic routing instance for broker-local execution of system table queries.
+ dataTableMap.put(new ServerRoutingInstance(SYSTEM_TABLE_PSEUDO_HOST, SYSTEM_TABLE_PSEUDO_PORT,
+ TableType.OFFLINE), executeLocalSystemTableQuery(pinotQuery, provider));
+ }
+
+ BrokerResponseNative brokerResponse;
+ BrokerRequest brokerRequest = new BrokerRequest();
+ QuerySource querySource = new QuerySource();
+ querySource.setTableName(tableName);
+ brokerRequest.setQuerySource(querySource);
+ brokerRequest.setPinotQuery(pinotQuery);
+ brokerResponse = _brokerReduceService.reduceOnDataTable(brokerRequest, brokerRequest, dataTableMap,
+ _brokerTimeoutMs, _brokerMetrics);
+ brokerResponse.setTablesQueried(Set.of(TableNameBuilder.extractRawTableName(tableName)));
+ brokerResponse.setTimeUsedMs(System.currentTimeMillis() - requestContext.getRequestArrivalTimeMillis());
+ _queryLogger.log(new QueryLogger.QueryLogParams(requestContext, tableName, brokerResponse,
+ QueryLogger.QueryLogParams.QueryEngine.SINGLE_STAGE, requesterIdentity, null));
+ return brokerResponse;
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table query {}: {}", tableName, e.getMessage(), e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+
+ private DataTable executeLocalSystemTableQuery(PinotQuery pinotQuery, SystemTableProvider provider)
+ throws Exception {
+ IndexSegment dataSource = provider.getDataSource();
+ try {
+ QueryContext queryContext = QueryContextConverterUtils.getQueryContext(pinotQuery);
+ queryContext.setSchema(provider.getSchema());
+ queryContext.setEndTimeMs(System.currentTimeMillis() + _brokerTimeoutMs);
+
+ // Pass null for serverMetrics because system table queries run broker-local against an in-memory IndexSegment.
+ Plan plan = _planMaker.makeInstancePlan(List.of(new SegmentContext(dataSource)), queryContext, _executorService,
+ null);
+ InstanceResponseBlock instanceResponse = plan.execute();
+ return instanceResponse.toDataTable();
+ } finally {
+ dataSource.destroy();
+ }
+ }
+
+ static final class BrokerTarget {
+ final ServerRoutingInstance _routingInstance;
+ final String _dataTableUrl;
+
+ BrokerTarget(ServerRoutingInstance routingInstance, String dataTableUrl) {
+ _routingInstance = routingInstance;
+ _dataTableUrl = dataTableUrl;
+ }
+ }
+
+ @com.google.common.annotations.VisibleForTesting
+ protected Map<ServerRoutingInstance, DataTable> scatterGatherSystemTableDataTables(SystemTableProvider provider,
+ PinotQuery pinotQuery, String tableName, JsonNode request, @Nullable HttpHeaders httpHeaders) {
+ if (_helixManager == null) {
+ throw new IllegalStateException(
+ "HelixManager is required for scatter-gather execution of system table: " + tableName);
+ }
+
+ HelixDataAccessor dataAccessor = _helixManager.getHelixDataAccessor();
+ List<String> liveInstances = dataAccessor.getChildNames(dataAccessor.keyBuilder().liveInstances());
+ if (liveInstances == null || liveInstances.isEmpty()) {
+ throw new IllegalStateException("No live instances found for scatter-gather execution of system table: "
+ + tableName);
+ }
+
+ String localInstanceId = _brokerId;
+ List<BrokerTarget> remoteTargets = new ArrayList<>();
+ @Nullable ServerRoutingInstance localRoutingInstance = null;
+ for (String instanceId : liveInstances) {
+ if (!InstanceTypeUtils.isBroker(instanceId)) {
+ continue;
+ }
+ InstanceConfig instanceConfig = dataAccessor.getProperty(dataAccessor.keyBuilder().instanceConfig(instanceId));
+ if (instanceConfig == null) {
+ continue;
+ }
+ URI baseUri = URI.create(InstanceUtils.getInstanceBaseUri(instanceConfig));
+ ServerRoutingInstance routingInstance = new ServerRoutingInstance(baseUri.getHost(), baseUri.getPort(),
+ TableType.OFFLINE);
+ if (instanceId.equals(localInstanceId)) {
+ localRoutingInstance = routingInstance;
+ } else {
+ remoteTargets.add(new BrokerTarget(routingInstance, baseUri.toString() + SYSTEM_TABLE_DATATABLE_API_PATH));
+ }
+ }
+
+ Map<ServerRoutingInstance, DataTable> dataTableMap = new HashMap<>(remoteTargets.size() + 1);
+ ServerRoutingInstance routingInstance = localRoutingInstance != null ? localRoutingInstance
+ : new ServerRoutingInstance(SYSTEM_TABLE_PSEUDO_HOST, SYSTEM_TABLE_PSEUDO_PORT, TableType.OFFLINE);
+ try {
+ dataTableMap.put(routingInstance, executeLocalSystemTableQuery(pinotQuery, provider));
+ } catch (Exception e) {
+ dataTableMap.put(routingInstance,
+ exceptionDataTable(QueryErrorCode.QUERY_EXECUTION, "Failed to execute system table query locally: "
+ + e.getMessage()));
+ }
+
+ if (remoteTargets.isEmpty()) {
+ return dataTableMap;
+ }
+
+ String requestBody = request.toString();
+ @Nullable Map<String, String> requestHeaders = toSingleValueRequestHeaders(httpHeaders);
+ if (requestHeaders == null) {
+ requestHeaders = new HashMap<>();
+ }
+ requestHeaders.putIfAbsent("Content-Type", MediaType.APPLICATION_JSON);
+ Map<String, String> requestHeadersFinal = requestHeaders;
+ int timeoutMs = (int) Math.min(Integer.MAX_VALUE, _brokerTimeoutMs);
+
+ List<Pair<BrokerTarget, Future<DataTable>>> futures = new ArrayList<>(remoteTargets.size());
+ for (BrokerTarget target : remoteTargets) {
+ Future<DataTable> future =
+ _scatterGatherExecutorService.submit(() -> fetchDataTableFromBroker(target._dataTableUrl, requestBody,
+ requestHeadersFinal, timeoutMs));
+ futures.add(Pair.of(target, future));
+ }
+
+ for (Pair<BrokerTarget, Future<DataTable>> pair : futures) {
+ BrokerTarget target = pair.getLeft();
+ try {
+ dataTableMap.put(target._routingInstance, pair.getRight().get());
+ } catch (Exception e) {
+ // Unexpected errors should be surfaced in the merged response.
+ dataTableMap.put(target._routingInstance, exceptionDataTable(QueryErrorCode.BROKER_REQUEST_SEND,
+ "Failed to gather system table response from " + target._dataTableUrl + ": " + e.getMessage()));
+ }
+ }
+ return dataTableMap;
+ }
+
+ private DataTable fetchDataTableFromBroker(String url, String requestBody, Map<String, String> requestHeaders,
+ int timeoutMs) {
+ HttpPost httpPost = new HttpPost(url);
+ Timeout timeout = Timeout.of(timeoutMs, TimeUnit.MILLISECONDS);
+ httpPost.setConfig(RequestConfig.custom().setConnectionRequestTimeout(timeout).setResponseTimeout(timeout).build());
+ httpPost.setEntity(new StringEntity(requestBody));
+ requestHeaders.forEach(httpPost::setHeader);
+ try (CloseableHttpResponse response = _scatterGatherHttpClient.execute(httpPost)) {
+ int status = response.getCode();
+ if (status != 200) {
+ String error = EntityUtils.toString(response.getEntity());
+ return exceptionDataTable(QueryErrorCode.BROKER_REQUEST_SEND,
+ String.format("Scatter-gather system table request failed for %s (status=%d): %s", url, status, error));
+ }
+ byte[] bytes = EntityUtils.toByteArray(response.getEntity());
+ if (bytes.length == 0) {
+ return exceptionDataTable(QueryErrorCode.BROKER_REQUEST_SEND,
+ "Scatter-gather system table request returned an empty response for: " + url);
+ }
+ return DataTableFactory.getDataTable(bytes);
+ } catch (Exception e) {
+ return exceptionDataTable(QueryErrorCode.BROKER_REQUEST_SEND,
+ String.format("Scatter-gather system table request failed for %s: %s", url, e.getMessage()));
+ }
+ }
+
+ private static @Nullable Map<String, String> toSingleValueRequestHeaders(@Nullable HttpHeaders httpHeaders) {
+ if (httpHeaders == null) {
+ return null;
+ }
+ Map<String, String> requestHeaders = new HashMap<>();
+ for (Map.Entry<String, List<String>> entry : httpHeaders.getRequestHeaders().entrySet()) {
+ String headerName = entry.getKey();
+ // Do not forward hop-by-hop headers or content-specific headers that may be invalid for the new request body.
+ // See https://www.rfc-editor.org/rfc/rfc7230#section-6.1
+ String headerNameLower = headerName.toLowerCase(Locale.ROOT);
+ if ("connection".equals(headerNameLower) || "content-length".equals(headerNameLower)
+ || "transfer-encoding".equals(headerNameLower) || "host".equals(headerNameLower)) {
+ continue;
Review Comment:
The list of hop-by-hop headers is hardcoded in the method. Extract this into a static final Set<String> constant for better maintainability and clarity.
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/SystemTableBrokerRequestHandler.java:
##########
@@ -0,0 +1,546 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.requesthandler;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.Nullable;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.MediaType;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hc.client5.http.classic.methods.HttpPost;
+import org.apache.hc.client5.http.config.RequestConfig;
+import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
+import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse;
+import org.apache.hc.client5.http.impl.classic.HttpClients;
+import org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager;
+import org.apache.hc.client5.http.io.HttpClientConnectionManager;
+import org.apache.hc.core5.http.io.entity.EntityUtils;
+import org.apache.hc.core5.http.io.entity.StringEntity;
+import org.apache.hc.core5.util.Timeout;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.HelixManager;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.pinot.broker.api.AccessControl;
+import org.apache.pinot.broker.broker.AccessControlFactory;
+import org.apache.pinot.broker.querylog.QueryLogger;
+import org.apache.pinot.broker.queryquota.QueryQuotaManager;
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.common.datatable.DataTableFactory;
+import org.apache.pinot.common.datatable.DataTableImplV4;
+import org.apache.pinot.common.http.PoolingHttpClientConnectionManagerHelper;
+import org.apache.pinot.common.metrics.BrokerMeter;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.request.QuerySource;
+import org.apache.pinot.common.response.BrokerResponse;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.systemtable.SystemTableProvider;
+import org.apache.pinot.common.systemtable.SystemTableRegistry;
+import org.apache.pinot.common.utils.NamedThreadFactory;
+import org.apache.pinot.common.utils.config.InstanceUtils;
+import org.apache.pinot.common.utils.config.QueryOptionsUtils;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.core.operator.blocks.InstanceResponseBlock;
+import org.apache.pinot.core.plan.Plan;
+import org.apache.pinot.core.plan.maker.InstancePlanMakerImplV2;
+import org.apache.pinot.core.plan.maker.PlanMaker;
+import org.apache.pinot.core.query.reduce.BrokerReduceService;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils;
+import org.apache.pinot.core.routing.RoutingManager;
+import org.apache.pinot.core.transport.ServerRoutingInstance;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.SegmentContext;
+import org.apache.pinot.spi.accounting.ThreadAccountant;
+import org.apache.pinot.spi.auth.AuthorizationResult;
+import org.apache.pinot.spi.auth.broker.RequesterIdentity;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.query.QueryExecutionContext;
+import org.apache.pinot.spi.query.QueryThreadContext;
+import org.apache.pinot.spi.trace.RequestContext;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.InstanceTypeUtils;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+import org.apache.pinot.sql.parsers.SqlNodeAndOptions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Broker request handler for system tables (handled entirely on the broker).
+ */
+public class SystemTableBrokerRequestHandler extends BaseBrokerRequestHandler {
+ private static final Logger LOGGER = LoggerFactory.getLogger(SystemTableBrokerRequestHandler.class);
+ private static final String SYSTEM_TABLE_PSEUDO_HOST = "localhost";
+ private static final int SYSTEM_TABLE_PSEUDO_PORT = 0;
+ private static final String SYSTEM_TABLE_DATATABLE_API_PATH = "/query/systemTable/datatable";
+
+ private final BrokerReduceService _brokerReduceService;
+ private final PlanMaker _planMaker;
+ private final ExecutorService _executorService;
+ private final ExecutorService _scatterGatherExecutorService;
+ private final PoolingHttpClientConnectionManager _scatterGatherConnMgr;
+ private final CloseableHttpClient _scatterGatherHttpClient;
+ @Nullable
+ private final HelixManager _helixManager;
+
+ public SystemTableBrokerRequestHandler(PinotConfiguration config, String brokerId,
+ BrokerRequestIdGenerator requestIdGenerator, RoutingManager routingManager,
+ AccessControlFactory accessControlFactory, QueryQuotaManager queryQuotaManager, TableCache tableCache,
+ ThreadAccountant threadAccountant, @Nullable HelixManager helixManager) {
+ super(config, brokerId, requestIdGenerator, routingManager, accessControlFactory, queryQuotaManager, tableCache,
+ threadAccountant);
+ _brokerReduceService = new BrokerReduceService(_config);
+ _planMaker = new InstancePlanMakerImplV2();
+ _planMaker.init(_config);
+ _helixManager = helixManager;
+ int executorPoolSize = config.getProperty(CommonConstants.Broker.CONFIG_OF_SYSTEM_TABLE_EXECUTOR_POOL_SIZE,
+ CommonConstants.Broker.DEFAULT_SYSTEM_TABLE_EXECUTOR_POOL_SIZE);
+ executorPoolSize = Math.max(1, executorPoolSize);
+ _executorService = QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(executorPoolSize,
+ new NamedThreadFactory("system-table-query-executor")));
+ _scatterGatherExecutorService =
+ QueryThreadContext.contextAwareExecutorService(Executors.newFixedThreadPool(executorPoolSize,
+ new NamedThreadFactory("system-table-scatter-gather-executor")));
+ _scatterGatherConnMgr = PoolingHttpClientConnectionManagerHelper.createWithSocketFactory();
+ Timeout timeout = Timeout.of(_brokerTimeoutMs, TimeUnit.MILLISECONDS);
+ RequestConfig defaultRequestConfig =
+ RequestConfig.custom().setConnectionRequestTimeout(timeout).setResponseTimeout(timeout).build();
+ _scatterGatherHttpClient =
+ HttpClients.custom().setConnectionManager(_scatterGatherConnMgr).setDefaultRequestConfig(defaultRequestConfig)
+ .build();
+ }
+
+ @Override
+ public void start() {
+ }
+
+ @Override
+ public void shutDown() {
+ _executorService.shutdownNow();
+ _scatterGatherExecutorService.shutdownNow();
+ try {
+ _scatterGatherHttpClient.close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close system table scatter-gather http client: {}", e.toString());
+ }
+ try {
+ _scatterGatherConnMgr.close();
+ } catch (Exception e) {
+ LOGGER.debug("Failed to close system table scatter-gather connection manager: {}", e.toString());
+ }
+ _brokerReduceService.shutDown();
+ }
+
+ public boolean canHandle(String tableName) {
+ return isSystemTable(tableName) && SystemTableRegistry.isRegistered(tableName);
+ }
+
+ @Override
+ protected BrokerResponse handleRequest(long requestId, String query, SqlNodeAndOptions sqlNodeAndOptions,
+ JsonNode request, @Nullable RequesterIdentity requesterIdentity, RequestContext requestContext,
+ @Nullable HttpHeaders httpHeaders, AccessControl accessControl)
+ throws Exception {
+ long startTimeMs = requestContext.getRequestArrivalTimeMillis();
+ long deadlineMs = startTimeMs + _brokerTimeoutMs;
+ QueryExecutionContext executionContext =
+ new QueryExecutionContext(QueryExecutionContext.QueryType.STE, requestId, Long.toString(requestId),
+ QueryOptionsUtils.getWorkloadName(sqlNodeAndOptions.getOptions()), startTimeMs, deadlineMs, deadlineMs,
+ _brokerId, _brokerId, org.apache.pinot.spi.utils.CommonConstants.Broker.DEFAULT_QUERY_HASH);
+ try (QueryThreadContext ignore = QueryThreadContext.open(executionContext, _threadAccountant)) {
+ PinotQuery pinotQuery;
+ try {
+ pinotQuery = CalciteSqlParser.compileToPinotQuery(sqlNodeAndOptions);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return new BrokerResponseNative(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ Set<String> tableNames = RequestUtils.getTableNames(pinotQuery);
+ if (tableNames == null || tableNames.isEmpty()) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "Failed to extract table name");
+ }
+ if (tableNames.size() != 1) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "System tables do not support joins");
+ }
+ String tableName = tableNames.iterator().next();
+ if (!isSystemTable(tableName)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, "Not a system table query");
+ }
+ AuthorizationResult authorizationResult =
+ hasTableAccess(requesterIdentity, Set.of(tableName), requestContext, httpHeaders);
+ if (!authorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return new BrokerResponseNative(QueryErrorCode.ACCESS_DENIED, authorizationResult.getFailureMessage());
+ }
+
+ return handleSystemTableQuery(request, pinotQuery, tableName, requestContext, requesterIdentity, query,
+ httpHeaders);
+ }
+ }
+
+ @Override
+ protected boolean handleCancel(long queryId, int timeoutMs, Executor executor,
+ HttpClientConnectionManager connMgr, Map<String, Integer> serverResponses) {
+ return false;
+ }
+
+ @Override
+ public boolean cancelQueryByClientId(String clientQueryId, int timeoutMs, Executor executor,
+ HttpClientConnectionManager connMgr, Map<String, Integer> serverResponses)
+ throws Exception {
+ return false;
+ }
+
+ @Override
+ public Map<Long, String> getRunningQueries() {
+ return Collections.emptyMap();
+ }
+
+ @Override
+ public OptionalLong getRequestIdByClientId(String clientQueryId) {
+ return OptionalLong.empty();
+ }
+
+ private boolean isSystemTable(String tableName) {
+ return tableName != null && tableName.toLowerCase(Locale.ROOT).startsWith("system.");
+ }
+
+ /**
+ * Executes a system table query against the local broker and returns the raw {@link DataTable} results.
+ * <p>
+ * This method is used by the internal broker-to-broker scatter-gather endpoint and must never perform fanout.
+ */
+ public DataTable handleSystemTableDataTableRequest(JsonNode request, @Nullable RequesterIdentity requesterIdentity,
+ RequestContext requestContext, @Nullable HttpHeaders httpHeaders) {
+ long startTimeMs = requestContext.getRequestArrivalTimeMillis();
+ if (startTimeMs <= 0) {
+ startTimeMs = System.currentTimeMillis();
+ requestContext.setRequestArrivalTimeMillis(startTimeMs);
+ }
+ long requestId = _requestIdGenerator.get();
+ long deadlineMs = startTimeMs + _brokerTimeoutMs;
+
+ JsonNode sql = request.get(CommonConstants.Broker.Request.SQL);
+ if (sql == null || !sql.isTextual()) {
+ return exceptionDataTable(QueryErrorCode.JSON_PARSING, "Failed to find 'sql' in the request: " + request);
+ }
+ String query = sql.textValue();
+ requestContext.setQuery(query);
+
+ SqlNodeAndOptions sqlNodeAndOptions;
+ try {
+ sqlNodeAndOptions = RequestUtils.parseQuery(query, request);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return exceptionDataTable(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ QueryExecutionContext executionContext =
+ new QueryExecutionContext(QueryExecutionContext.QueryType.STE, requestId, Long.toString(requestId),
+ QueryOptionsUtils.getWorkloadName(sqlNodeAndOptions.getOptions()), startTimeMs, deadlineMs, deadlineMs,
+ _brokerId, _brokerId, org.apache.pinot.spi.utils.CommonConstants.Broker.DEFAULT_QUERY_HASH);
+ try (QueryThreadContext ignore = QueryThreadContext.open(executionContext, _threadAccountant)) {
+ AccessControl accessControl = _accessControlFactory.create();
+ AuthorizationResult authorizationResult = accessControl.authorize(requesterIdentity);
+ if (!authorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return exceptionDataTable(QueryErrorCode.ACCESS_DENIED, authorizationResult.getFailureMessage());
+ }
+
+ PinotQuery pinotQuery;
+ try {
+ pinotQuery = CalciteSqlParser.compileToPinotQuery(sqlNodeAndOptions);
+ } catch (Exception e) {
+ requestContext.setErrorCode(QueryErrorCode.SQL_PARSING);
+ return exceptionDataTable(QueryErrorCode.SQL_PARSING, e.getMessage());
+ }
+
+ Set<String> tableNames = RequestUtils.getTableNames(pinotQuery);
+ if (tableNames == null || tableNames.isEmpty()) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, "Failed to extract table name");
+ }
+ if (tableNames.size() != 1) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, "System tables do not support joins");
+ }
+ String tableName = tableNames.iterator().next();
+ if (!isSystemTable(tableName)) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, "Not a system table query");
+ }
+
+ AuthorizationResult tableAuthorizationResult =
+ hasTableAccess(requesterIdentity, Set.of(tableName), requestContext, httpHeaders);
+ if (!tableAuthorizationResult.hasAccess()) {
+ requestContext.setErrorCode(QueryErrorCode.ACCESS_DENIED);
+ return exceptionDataTable(QueryErrorCode.ACCESS_DENIED, tableAuthorizationResult.getFailureMessage());
+ }
+
+ SystemTableProvider provider = SystemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return exceptionDataTable(QueryErrorCode.TABLE_DOES_NOT_EXIST, "System table does not exist: " + tableName);
+ }
+
+ try {
+ return executeLocalSystemTableQuery(pinotQuery, provider);
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return exceptionDataTable(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table datatable query {}: {}", tableName, e.getMessage(),
+ e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return exceptionDataTable(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+ }
+
+ private BrokerResponse handleSystemTableQuery(JsonNode request, PinotQuery pinotQuery, String tableName,
+ RequestContext requestContext, @Nullable RequesterIdentity requesterIdentity, String query,
+ @Nullable HttpHeaders httpHeaders) {
+ if (pinotQuery.isExplain()) {
+ return BrokerResponseNative.BROKER_ONLY_EXPLAIN_PLAN_OUTPUT;
+ }
+ SystemTableProvider provider = SystemTableRegistry.get(tableName);
+ if (provider == null) {
+ requestContext.setErrorCode(QueryErrorCode.TABLE_DOES_NOT_EXIST);
+ return BrokerResponseNative.TABLE_DOES_NOT_EXIST;
+ }
+ try {
+ Map<ServerRoutingInstance, DataTable> dataTableMap;
+ if (provider.getExecutionMode() == SystemTableProvider.ExecutionMode.BROKER_SCATTER_GATHER) {
+ dataTableMap = scatterGatherSystemTableDataTables(provider, pinotQuery, tableName, request, httpHeaders);
+ } else {
+ dataTableMap = new HashMap<>(1);
+ // Use a synthetic routing instance for broker-local execution of system table queries.
+ dataTableMap.put(new ServerRoutingInstance(SYSTEM_TABLE_PSEUDO_HOST, SYSTEM_TABLE_PSEUDO_PORT,
+ TableType.OFFLINE), executeLocalSystemTableQuery(pinotQuery, provider));
+ }
+
+ BrokerResponseNative brokerResponse;
+ BrokerRequest brokerRequest = new BrokerRequest();
+ QuerySource querySource = new QuerySource();
+ querySource.setTableName(tableName);
+ brokerRequest.setQuerySource(querySource);
+ brokerRequest.setPinotQuery(pinotQuery);
+ brokerResponse = _brokerReduceService.reduceOnDataTable(brokerRequest, brokerRequest, dataTableMap,
+ _brokerTimeoutMs, _brokerMetrics);
+ brokerResponse.setTablesQueried(Set.of(TableNameBuilder.extractRawTableName(tableName)));
+ brokerResponse.setTimeUsedMs(System.currentTimeMillis() - requestContext.getRequestArrivalTimeMillis());
+ _queryLogger.log(new QueryLogger.QueryLogParams(requestContext, tableName, brokerResponse,
+ QueryLogger.QueryLogParams.QueryEngine.SINGLE_STAGE, requesterIdentity, null));
+ return brokerResponse;
+ } catch (BadQueryRequestException e) {
+ requestContext.setErrorCode(QueryErrorCode.QUERY_VALIDATION);
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.QUERY_VALIDATION_EXCEPTIONS, 1);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_VALIDATION, e.getMessage());
+ } catch (Exception e) {
+ LOGGER.warn("Caught exception while handling system table query {}: {}", tableName, e.getMessage(), e);
+ requestContext.setErrorCode(QueryErrorCode.QUERY_EXECUTION);
+ return new BrokerResponseNative(QueryErrorCode.QUERY_EXECUTION, e.getMessage());
+ }
+ }
+
+ private DataTable executeLocalSystemTableQuery(PinotQuery pinotQuery, SystemTableProvider provider)
+ throws Exception {
+ IndexSegment dataSource = provider.getDataSource();
+ try {
+ QueryContext queryContext = QueryContextConverterUtils.getQueryContext(pinotQuery);
+ queryContext.setSchema(provider.getSchema());
+ queryContext.setEndTimeMs(System.currentTimeMillis() + _brokerTimeoutMs);
+
+ // Pass null for serverMetrics because system table queries run broker-local against an in-memory IndexSegment.
+ Plan plan = _planMaker.makeInstancePlan(List.of(new SegmentContext(dataSource)), queryContext, _executorService,
+ null);
+ InstanceResponseBlock instanceResponse = plan.execute();
+ return instanceResponse.toDataTable();
+ } finally {
+ dataSource.destroy();
+ }
+ }
+
+ static final class BrokerTarget {
+ final ServerRoutingInstance _routingInstance;
+ final String _dataTableUrl;
+
+ BrokerTarget(ServerRoutingInstance routingInstance, String dataTableUrl) {
+ _routingInstance = routingInstance;
+ _dataTableUrl = dataTableUrl;
+ }
+ }
+
+ @com.google.common.annotations.VisibleForTesting
+ protected Map<ServerRoutingInstance, DataTable> scatterGatherSystemTableDataTables(SystemTableProvider provider,
+ PinotQuery pinotQuery, String tableName, JsonNode request, @Nullable HttpHeaders httpHeaders) {
+ if (_helixManager == null) {
+ throw new IllegalStateException(
+ "HelixManager is required for scatter-gather execution of system table: " + tableName);
+ }
+
+ HelixDataAccessor dataAccessor = _helixManager.getHelixDataAccessor();
+ List<String> liveInstances = dataAccessor.getChildNames(dataAccessor.keyBuilder().liveInstances());
+ if (liveInstances == null || liveInstances.isEmpty()) {
+ throw new IllegalStateException("No live instances found for scatter-gather execution of system table: "
+ + tableName);
+ }
+
+ String localInstanceId = _brokerId;
+ List<BrokerTarget> remoteTargets = new ArrayList<>();
+ @Nullable ServerRoutingInstance localRoutingInstance = null;
+ for (String instanceId : liveInstances) {
+ if (!InstanceTypeUtils.isBroker(instanceId)) {
+ continue;
+ }
+ InstanceConfig instanceConfig = dataAccessor.getProperty(dataAccessor.keyBuilder().instanceConfig(instanceId));
+ if (instanceConfig == null) {
+ continue;
+ }
+ URI baseUri = URI.create(InstanceUtils.getInstanceBaseUri(instanceConfig));
+ ServerRoutingInstance routingInstance = new ServerRoutingInstance(baseUri.getHost(), baseUri.getPort(),
+ TableType.OFFLINE);
+ if (instanceId.equals(localInstanceId)) {
+ localRoutingInstance = routingInstance;
+ } else {
+ remoteTargets.add(new BrokerTarget(routingInstance, baseUri.toString() + SYSTEM_TABLE_DATATABLE_API_PATH));
+ }
+ }
+
+ Map<ServerRoutingInstance, DataTable> dataTableMap = new HashMap<>(remoteTargets.size() + 1);
+ ServerRoutingInstance routingInstance = localRoutingInstance != null ? localRoutingInstance
+ : new ServerRoutingInstance(SYSTEM_TABLE_PSEUDO_HOST, SYSTEM_TABLE_PSEUDO_PORT, TableType.OFFLINE);
+ try {
+ dataTableMap.put(routingInstance, executeLocalSystemTableQuery(pinotQuery, provider));
+ } catch (Exception e) {
Review Comment:
When local system table query execution fails, the exception is caught and wrapped in line 456, but the error message doesn't include the original exception details. Consider including the full exception message and stack trace for better debugging.
```suggestion
} catch (Exception e) {
LOGGER.error("Failed to execute system table query locally for query: {}", pinotQuery, e);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org