You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by xi...@apache.org on 2026/05/25 15:18:40 UTC
(pinot) branch master updated: [Feature] SSE Materialized View Query Rewrite (#18529)
This is an automated email from the ASF dual-hosted git repository.
xiangfu0 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 26e7d1211ec [Feature] SSE Materialized View Query Rewrite (#18529)
26e7d1211ec is described below
commit 26e7d1211ec8401bd7681589c265fcd74335f131
Author: Hongkun Xu <xu...@163.com>
AuthorDate: Mon May 25 23:17:45 2026 +0800
[Feature] SSE Materialized View Query Rewrite (#18529)
* [feature] SSE Materialized View — broker query rewrite (PR 2 of 2)
Layers on top of PR 1 (view creation + ingestion). Wires the broker so user
queries against a base table are transparently rewritten to use any
materialized view whose subsumption conditions are met.
**Scope of PR 2 (delta on top of PR 1):**
- pinot-materialized-view: rewrite/ engine + Exact/Aggregation/Scan
subsumption strategies, MaterializedViewMetadataCache (ZK definition +
runtime listeners, base-table reverse index, refresh+close lifecycle),
plan/result types, ExecutionMode, MatchType, MaterializedViewQueryShape
- pinot-materialized-view: handler/ (DefaultMaterializedViewHandler,
MaterializedViewSplitDispatcher, MaterializedViewSplitExecutionContext,
MaterializedViewCompileContext, MaterializedViewHandler SPI with init/
compile/executeSplit/annotateResponse/invalidateBaseTable/refreshTable/
getCacheEntryCount/close hooks)
- pinot-materialized-view: context/MaterializedViewContext immutable
compile-time carrier with forFullRewrite / forSplitRewrite / empty()
factories and accessors that surface pre-rewrite query/table/raw-name
across the FULL_REWRITE swap
- pinot-broker: BaseSingleStageBrokerRequestHandler MV-rewrite path
(compile-time match, FULL_REWRITE swap, SPLIT_REWRITE dual scatter-
gather + reduce, response annotation, defense-in-depth fallbacks with
QUERY_REWRITE_EXCEPTIONS metric, RLS+MV interop derived from the
rlsFilters query-option marker, cascade-prevention guard using
TableConfig.isMaterializedView() from upstream PR #18564)
- pinot-broker: BrokerResourceOnlineOfflineStateModelFactory
invalidates/refreshes MV cache on ONLINE↔OFFLINE/DROPPED transitions
- pinot-broker: BaseBrokerStarter loads the handler via the SPI
loadHandler factory and wires the
MATERIALIZED_VIEW_CACHE_ENTRY_COUNT gauge
- pinot-common: BrokerResponse.materializedViewQueried response field +
BrokerResponseNative wiring, BrokerMeter QUERY_REWRITE_EXCEPTIONS /
RLS_FILTERS_APPLIED meters, RlsUtils.isRlsAppliedForTable helper
- pinot-core: BrokerReduceService handling for materializedViewQueried
(suppresses the "nested query without gapfill" safety-net when the
MATERIALIZED_VIEW_REWRITE marker is present)
- Integration tests: MaterializedViewClusterIntegrationTest end-to-end
+ targeted unit tests for the broker handler, rewrite engine,
subsumption strategies, equivalence registry, metadata cache lifecycle,
context accessors, and RlsUtils
The split MV branch filters on the user-provided base time column and the
MV's TIMESTAMP column with the watermark literal converted via
sourceTimeFormat (for base) and raw millis (for MV). Aggregation
intermediates are merged at the broker reduce phase.
Co-Authored-By: Hongkun Xu <xu...@163.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* [feature] SSE MV — generic hasMaterializedViewTaskWithDefinedSql
Scan every task config for a non-empty `definedSQL` key instead of only the
built-in MaterializedViewTask task type. This keeps the
`isMaterializedView=true` invariant compatible with downstream MV-style task
variants (e.g. StarTree's MseMaterializedViewTask for multi-stage MVs) without
OSS having to hard-code each variant's task-type label.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
---------
Co-authored-by: Xiang Fu <xi...@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <no...@anthropic.com>
---
pinot-broker/pom.xml | 4 +
.../broker/broker/helix/BaseBrokerStarter.java | 60 +-
...okerResourceOnlineOfflineStateModelFactory.java | 37 +
.../BaseSingleStageBrokerRequestHandler.java | 621 ++++++++++-
.../requesthandler/GrpcBrokerRequestHandler.java | 18 +-
.../SingleConnectionBrokerRequestHandler.java | 205 +++-
.../BaseSingleStageBrokerRequestHandlerTest.java | 924 +++++++++++++++-
.../LiteralOnlyBrokerRequestTest.java | 6 +-
...gleConnectionBrokerRequestHandlerMergeTest.java | 166 +++
.../apache/pinot/common/metrics/BrokerGauge.java | 8 +-
.../apache/pinot/common/metrics/BrokerMeter.java | 4 +
.../pinot/common/response/BrokerResponse.java | 12 +
.../response/broker/BrokerResponseNative.java | 21 +-
.../pinot/sql/parsers/rewriter/RlsUtils.java | 9 +
.../response/broker/BrokerResponseNativeTest.java | 23 +-
.../pinot/sql/parsers/rewriter/RlsUtilsTest.java | 71 ++
.../core/query/reduce/BrokerReduceService.java | 30 +-
.../core/query/reduce/BrokerReduceServiceTest.java | 84 ++
.../MaterializedViewClusterIntegrationTest.java | 1167 ++++++++++++++++++++
.../materializedview/analysis/package-info.java | 36 +-
.../materializedview/consistency/package-info.java | 34 +-
.../context/MaterializedViewContext.java | 118 ++
.../materializedview/context/package-info.java | 35 +
.../materializedview/executor/package-info.java | 33 +-
.../handler/DefaultMaterializedViewHandler.java | 275 +++++
.../handler/MaterializedViewCompileContext.java | 61 +
.../handler/MaterializedViewHandler.java | 149 +++
.../handler/MaterializedViewSplitDispatcher.java | 58 +
.../MaterializedViewSplitExecutionContext.java | 132 +++
.../materializedview/handler/package-info.java | 45 +
.../MaterializedViewDefinitionMetadata.java | 18 +-
.../materializedview/metadata/package-info.java | 34 +
.../pinot/materializedview/package-info.java | 70 ++
.../materializedview/rewrite/ExecutionMode.java | 31 +-
.../pinot/materializedview/rewrite/MatchType.java | 29 +-
.../rewrite/MaterializedViewMatchUtils.java | 385 +++++++
.../rewrite/MaterializedViewMetadataCache.java | 714 ++++++++++++
.../MaterializedViewQueryRewriteEngine.java | 413 +++++++
.../MaterializedViewQueryRewriteEngineFactory.java | 54 +
.../rewrite/MaterializedViewQueryShape.java | 111 ++
.../rewrite/MaterializedViewRewritePlan.java | 131 +++
.../equivalence/AggregationEquivalence.java | 112 ++
.../AggregationEquivalenceRegistry.java | 97 ++
.../equivalence/PassthroughEquivalence.java | 94 ++
.../equivalence/SketchMergeEquivalence.java | 94 ++
.../rewrite/equivalence/package-info.java | 35 +
.../materializedview/rewrite/package-info.java | 43 +
.../strategy/AbstractSubsumptionStrategy.java | 216 ++++
.../strategy/AggregationSubsumptionStrategy.java | 428 +++++++
.../rewrite/strategy/ExactSubsumptionStrategy.java | 214 ++++
.../strategy/MaterializedViewMatchStrategy.java | 47 +
.../rewrite/strategy/ScanSubsumptionStrategy.java | 139 +++
.../rewrite/strategy/package-info.java | 37 +
.../scheduler/MaterializedViewTaskScheduler.java | 28 +-
.../materializedview/scheduler/package-info.java | 37 +-
.../analysis/MaterializedViewAnalyzerTest.java | 22 +
.../context/MaterializedViewContextTest.java | 91 ++
.../DefaultMaterializedViewHandlerTest.java | 138 +++
.../handler/MaterializedViewHandlerLoadTest.java | 104 ++
.../metadata/MaterializedViewMetadataTest.java | 4 +-
.../AggregationSubsumptionStrategyTest.java | 897 +++++++++++++++
.../rewrite/ExactSubsumptionStrategyTest.java | 332 ++++++
...MaterializedViewMetadataCacheLifecycleTest.java | 242 ++++
.../MaterializedViewQueryRewriteEngineTest.java | 454 ++++++++
.../rewrite/ScanSubsumptionStrategyTest.java | 386 +++++++
.../AggregationEquivalenceRegistryTest.java | 136 +++
.../apache/pinot/spi/config/table/TableConfig.java | 27 +-
.../apache/pinot/spi/utils/CommonConstants.java | 23 +
68 files changed, 10711 insertions(+), 202 deletions(-)
diff --git a/pinot-broker/pom.xml b/pinot-broker/pom.xml
index b7965157403..a165f33db17 100644
--- a/pinot-broker/pom.xml
+++ b/pinot-broker/pom.xml
@@ -33,6 +33,10 @@
<pinot.root>${basedir}/..</pinot.root>
</properties>
<dependencies>
+ <dependency>
+ <groupId>org.apache.pinot</groupId>
+ <artifactId>pinot-materialized-view</artifactId>
+ </dependency>
<dependency>
<groupId>org.apache.pinot</groupId>
<artifactId>pinot-query-runtime</artifactId>
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BaseBrokerStarter.java b/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BaseBrokerStarter.java
index 737571dada9..7fdd3154b4e 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BaseBrokerStarter.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BaseBrokerStarter.java
@@ -30,6 +30,7 @@ import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ThreadLocalRandom;
import java.util.concurrent.TimeUnit;
+import javax.annotation.Nullable;
import javax.net.ssl.SSLContext;
import nl.altindag.ssl.SSLFactory;
import org.apache.commons.lang3.StringUtils;
@@ -98,6 +99,7 @@ import org.apache.pinot.core.transport.NettyInspector;
import org.apache.pinot.core.transport.server.routing.stats.ServerRoutingStatsManager;
import org.apache.pinot.core.util.ListenerConfigUtil;
import org.apache.pinot.core.util.trace.ContinuousJfrStarter;
+import org.apache.pinot.materializedview.handler.MaterializedViewHandler;
import org.apache.pinot.query.routing.WorkerManager;
import org.apache.pinot.query.runtime.operator.factory.DefaultQueryOperatorFactoryProvider;
import org.apache.pinot.query.runtime.operator.factory.QueryOperatorFactoryProvider;
@@ -266,7 +268,7 @@ public abstract class BaseBrokerStarter implements ServiceStartable {
/**
* Override to supply a custom {@link SingleConnectionBrokerRequestHandler} subclass (e.g. one
* that overrides {@code onQueryCompletion(RequestContext, BrokerResponse)} for async query
- * logging). The default implementation returns a plain {@link SingleConnectionBrokerRequestHandler}.
+ * logging). Pass {@code null} for {@code materializedViewHandler} to skip MV rewrite.
*/
protected SingleConnectionBrokerRequestHandler createSingleStageBrokerRequestHandler(
PinotConfiguration config, String brokerId, BrokerRequestIdGenerator requestIdGenerator,
@@ -274,24 +276,27 @@ public abstract class BaseBrokerStarter implements ServiceStartable {
QueryQuotaManager queryQuotaManager, TableCache tableCache, NettyConfig nettyConfig,
TlsConfig tlsConfig, ServerRoutingStatsManager serverRoutingStatsManager,
FailureDetector failureDetector, ThreadAccountant threadAccountant,
- MultiClusterRoutingContext multiClusterRoutingContext) {
+ MultiClusterRoutingContext multiClusterRoutingContext,
+ @Nullable MaterializedViewHandler materializedViewHandler) {
return new SingleConnectionBrokerRequestHandler(config, brokerId, requestIdGenerator, routingManager,
accessControlFactory, queryQuotaManager, tableCache, nettyConfig, tlsConfig,
- serverRoutingStatsManager, failureDetector, threadAccountant, multiClusterRoutingContext);
+ serverRoutingStatsManager, failureDetector, threadAccountant, multiClusterRoutingContext,
+ materializedViewHandler);
}
/**
- * Override to supply a custom {@link GrpcBrokerRequestHandler} subclass.
- * The default implementation returns a plain {@link GrpcBrokerRequestHandler}.
+ * Override to supply a custom {@link GrpcBrokerRequestHandler} subclass. Pass {@code null} for
+ * {@code materializedViewHandler} to skip MV rewrite.
*/
protected GrpcBrokerRequestHandler createGrpcBrokerRequestHandler(
PinotConfiguration config, String brokerId, BrokerRequestIdGenerator requestIdGenerator,
RoutingManager routingManager, AccessControlFactory accessControlFactory,
QueryQuotaManager queryQuotaManager, TableCache tableCache, FailureDetector failureDetector,
- ThreadAccountant threadAccountant, MultiClusterRoutingContext multiClusterRoutingContext) {
+ ThreadAccountant threadAccountant, MultiClusterRoutingContext multiClusterRoutingContext,
+ @Nullable MaterializedViewHandler materializedViewHandler) {
return new GrpcBrokerRequestHandler(config, brokerId, requestIdGenerator, routingManager,
accessControlFactory, queryQuotaManager, tableCache, failureDetector, threadAccountant,
- multiClusterRoutingContext);
+ multiClusterRoutingContext, materializedViewHandler);
}
/**
@@ -475,12 +480,36 @@ public abstract class BaseBrokerStarter implements ServiceStartable {
BrokerRequestIdGenerator requestIdGenerator = new BrokerRequestIdGenerator();
String brokerRequestHandlerType =
_brokerConf.getProperty(Broker.BROKER_REQUEST_HANDLER_TYPE, Broker.DEFAULT_BROKER_REQUEST_HANDLER_TYPE);
+ boolean mvRewriteEnabled = _brokerConf.getProperty(
+ Broker.CONFIG_OF_BROKER_QUERY_ENABLE_MATERIALIZED_VIEW_REWRITE,
+ Broker.DEFAULT_BROKER_QUERY_ENABLE_MATERIALIZED_VIEW_REWRITE);
+ boolean isGrpcBroker = brokerRequestHandlerType.equalsIgnoreCase(Broker.GRPC_BROKER_REQUEST_HANDLER_TYPE);
+ MaterializedViewHandler materializedViewHandler = null;
+ if (mvRewriteEnabled) {
+ /// The handler class is configurable via
+ /// `pinot.broker.materialized.view.handler.class` (default: `DefaultMaterializedViewHandler`).
+ /// gRPC streaming reduce cannot merge dual scatter-gather, so the gRPC broker variant passes
+ /// `supportsSplitRewrite=false`; the configured handler must honor that signal so split-rewrite
+ /// plans are suppressed at compile time on gRPC brokers.
+ PinotConfiguration mvHandlerConf = _brokerConf.subset(Broker.MATERIALIZED_VIEW_HANDLER_CONFIG_PREFIX);
+ materializedViewHandler =
+ MaterializedViewHandler.loadHandler(mvHandlerConf, _propertyStore, !isGrpcBroker);
+ /// Expose the MV metadata cache size as a global gauge so operators can monitor
+ /// unbounded growth — a cluster with K MVs should plateau near K; sustained growth would
+ /// indicate a leak in the ZK listener / drop path. Handlers that don't track a cache
+ /// return -1 from `getCacheEntryCount()` and we skip the gauge for those.
+ final MaterializedViewHandler handlerForGauge = materializedViewHandler;
+ if (handlerForGauge.getCacheEntryCount() >= 0) {
+ _brokerMetrics.setOrUpdateGlobalGauge(BrokerGauge.MATERIALIZED_VIEW_CACHE_ENTRY_COUNT,
+ () -> (long) handlerForGauge.getCacheEntryCount());
+ }
+ }
BaseSingleStageBrokerRequestHandler singleStageBrokerRequestHandler;
- if (brokerRequestHandlerType.equalsIgnoreCase(Broker.GRPC_BROKER_REQUEST_HANDLER_TYPE)) {
+ if (isGrpcBroker) {
singleStageBrokerRequestHandler =
createGrpcBrokerRequestHandler(_brokerConf, brokerId, requestIdGenerator, _routingManager,
_accessControlFactory, _queryQuotaManager, _tableCache, _failureDetector, _threadAccountant,
- multiClusterRoutingContext);
+ multiClusterRoutingContext, materializedViewHandler);
} else {
// Default request handler type, i.e. netty
NettyConfig nettyDefaults = NettyConfig.extractNettyConfig(_brokerConf, Broker.BROKER_NETTY_PREFIX);
@@ -505,8 +534,10 @@ public abstract class BaseBrokerStarter implements ServiceStartable {
singleStageBrokerRequestHandler =
createSingleStageBrokerRequestHandler(_brokerConf, brokerId, requestIdGenerator, _routingManager,
_accessControlFactory, _queryQuotaManager, _tableCache, nettyDefaults, tlsDefaults,
- _serverRoutingStatsManager, _failureDetector, _threadAccountant, multiClusterRoutingContext);
+ _serverRoutingStatsManager, _failureDetector, _threadAccountant, multiClusterRoutingContext,
+ materializedViewHandler);
}
+
MultiStageBrokerRequestHandler multiStageBrokerRequestHandler = null;
if (_brokerConf.getProperty(Helix.CONFIG_OF_MULTI_STAGE_ENGINE_ENABLED, Helix.DEFAULT_MULTI_STAGE_ENGINE_ENABLED)) {
_multiStageQueryThrottler = new MultiStageQueryThrottler(_brokerConf);
@@ -632,7 +663,7 @@ public abstract class BaseBrokerStarter implements ServiceStartable {
_participantHelixManager.getStateMachineEngine()
.registerStateModelFactory(BrokerResourceOnlineOfflineStateModelFactory.getStateModelDef(),
new BrokerResourceOnlineOfflineStateModelFactory(_propertyStore, _helixDataAccessor, _routingManager,
- _queryQuotaManager));
+ _queryQuotaManager, materializedViewHandler));
// Register user-define message handler factory
_participantHelixManager.getMessagingService()
.registerMessageHandlerFactory(Message.MessageType.USER_DEFINE_MSG.toString(),
@@ -920,6 +951,13 @@ public abstract class BaseBrokerStarter implements ServiceStartable {
LOGGER.info("Shutting down request handler and broker admin application");
_brokerRequestHandler.shutDown();
+ /// Deregister the MV-cache-size gauge after the handler shut down (and called
+ /// `MaterializedViewHandler.close()`). Without removal, the gauge supplier remains in the
+ /// metrics registry and the reporter keeps polling — the closed handler's `getCacheEntryCount`
+ /// returns 0 (the entry map is cleared on close), which silently masks the shutdown state
+ /// rather than removing the gauge. Removing here also avoids a stale-supplier conflict on
+ /// hot reload / re-init in tests.
+ _brokerMetrics.removeGauge(BrokerGauge.MATERIALIZED_VIEW_CACHE_ENTRY_COUNT.getGaugeName());
_threadAccountant.stopWatcherTask();
_brokerAdminApplication.stop();
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BrokerResourceOnlineOfflineStateModelFactory.java b/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BrokerResourceOnlineOfflineStateModelFactory.java
index a819254c29a..dfe3bc3c1e3 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BrokerResourceOnlineOfflineStateModelFactory.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BrokerResourceOnlineOfflineStateModelFactory.java
@@ -18,6 +18,7 @@
*/
package org.apache.pinot.broker.broker.helix;
+import javax.annotation.Nullable;
import org.apache.helix.HelixDataAccessor;
import org.apache.helix.NotificationContext;
import org.apache.helix.model.Message;
@@ -31,7 +32,9 @@ import org.apache.pinot.broker.queryquota.HelixExternalViewBasedQueryQuotaManage
import org.apache.pinot.broker.routing.manager.BrokerRoutingManager;
import org.apache.pinot.common.metadata.ZKMetadataProvider;
import org.apache.pinot.common.utils.DatabaseUtils;
+import org.apache.pinot.materializedview.handler.MaterializedViewHandler;
import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -51,14 +54,24 @@ public class BrokerResourceOnlineOfflineStateModelFactory extends StateModelFact
private final HelixDataAccessor _helixDataAccessor;
private final BrokerRoutingManager _routingManager;
private final HelixExternalViewBasedQueryQuotaManager _queryQuotaManager;
+ @Nullable
+ private final MaterializedViewHandler _materializedViewHandler;
public BrokerResourceOnlineOfflineStateModelFactory(ZkHelixPropertyStore<ZNRecord> propertyStore,
HelixDataAccessor helixDataAccessor, BrokerRoutingManager routingManager,
HelixExternalViewBasedQueryQuotaManager queryQuotaManager) {
+ this(propertyStore, helixDataAccessor, routingManager, queryQuotaManager, null);
+ }
+
+ public BrokerResourceOnlineOfflineStateModelFactory(ZkHelixPropertyStore<ZNRecord> propertyStore,
+ HelixDataAccessor helixDataAccessor, BrokerRoutingManager routingManager,
+ HelixExternalViewBasedQueryQuotaManager queryQuotaManager,
+ @Nullable MaterializedViewHandler materializedViewHandler) {
_helixDataAccessor = helixDataAccessor;
_propertyStore = propertyStore;
_routingManager = routingManager;
_queryQuotaManager = queryQuotaManager;
+ _materializedViewHandler = materializedViewHandler;
}
public static String getStateModelDef() {
@@ -88,6 +101,11 @@ public class BrokerResourceOnlineOfflineStateModelFactory extends StateModelFact
_helixDataAccessor.getProperty(_helixDataAccessor.keyBuilder().externalView(BROKER_RESOURCE_INSTANCE)));
_queryQuotaManager.createDatabaseRateLimiter(
DatabaseUtils.extractDatabaseFromFullyQualifiedTableName(physicalOrLogicalTable));
+ /// Rebuild the MV cache entry in case its broker resource was previously cycled
+ /// OFFLINE without the definition znode being deleted (e.g. an operator-driven
+ /// OFFLINE/ONLINE toggle for maintenance). Symmetric counterpart to the
+ /// invalidate-on-OFFLINE call below.
+ refreshMaterializedViewCacheForTable(physicalOrLogicalTable);
}
} catch (Exception e) {
LOGGER.error("Caught exception while processing transition from OFFLINE to ONLINE for table: {}",
@@ -105,6 +123,11 @@ public class BrokerResourceOnlineOfflineStateModelFactory extends StateModelFact
_routingManager.removeRoutingForLogicalTable(physicalOrLogicalTable);
_queryQuotaManager.dropTableQueryQuota(physicalOrLogicalTable);
} else {
+ /// Invalidate MV cache BEFORE routing so an in-flight query that has not yet
+ /// passed compile cannot pick up a stale MV candidate after routing has gone
+ /// away — the worst-case window then yields a successful non-MV plan rather
+ /// than a "no servers found" failure on a now-routeless MV table.
+ invalidateMaterializedViewCacheForTable(physicalOrLogicalTable);
_routingManager.removeRouting(physicalOrLogicalTable);
_queryQuotaManager.dropTableQueryQuota(physicalOrLogicalTable);
}
@@ -129,6 +152,8 @@ public class BrokerResourceOnlineOfflineStateModelFactory extends StateModelFact
_routingManager.removeRoutingForLogicalTable(physicalOrLogicalTable);
_queryQuotaManager.dropTableQueryQuota(physicalOrLogicalTable);
} else {
+ /// See `onBecomeOfflineFromOnline` — invalidate MV cache BEFORE routing.
+ invalidateMaterializedViewCacheForTable(physicalOrLogicalTable);
_routingManager.removeRouting(physicalOrLogicalTable);
_queryQuotaManager.dropTableQueryQuota(physicalOrLogicalTable);
}
@@ -139,6 +164,18 @@ public class BrokerResourceOnlineOfflineStateModelFactory extends StateModelFact
}
}
+ private void invalidateMaterializedViewCacheForTable(String tableNameWithType) {
+ if (_materializedViewHandler != null) {
+ _materializedViewHandler.invalidateBaseTable(TableNameBuilder.extractRawTableName(tableNameWithType));
+ }
+ }
+
+ private void refreshMaterializedViewCacheForTable(String tableNameWithType) {
+ if (_materializedViewHandler != null) {
+ _materializedViewHandler.refreshTable(TableNameBuilder.extractRawTableName(tableNameWithType));
+ }
+ }
+
@Transition(from = "ERROR", to = "OFFLINE")
public void onBecomeOfflineFromError(Message message, NotificationContext context) {
LOGGER.info("Transition from ERROR to OFFLINE is no-op for table: {}", message.getPartitionName());
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java
index b50ddbc7428..e343f89a49c 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandler.java
@@ -90,6 +90,7 @@ import org.apache.pinot.core.query.reduce.BaseGapfillProcessor;
import org.apache.pinot.core.query.reduce.GapfillProcessorFactory;
import org.apache.pinot.core.query.request.context.QueryContext;
import org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils;
+import org.apache.pinot.core.routing.ImplicitHybridTableRouteInfo;
import org.apache.pinot.core.routing.ImplicitHybridTableRouteProvider;
import org.apache.pinot.core.routing.LogicalTableRouteProvider;
import org.apache.pinot.core.routing.MultiClusterRoutingContext;
@@ -99,6 +100,12 @@ import org.apache.pinot.core.routing.TableRouteProvider;
import org.apache.pinot.core.routing.timeboundary.TimeBoundaryInfo;
import org.apache.pinot.core.transport.ServerInstance;
import org.apache.pinot.core.util.GapfillUtils;
+import org.apache.pinot.materializedview.context.MaterializedViewContext;
+import org.apache.pinot.materializedview.handler.MaterializedViewCompileContext;
+import org.apache.pinot.materializedview.handler.MaterializedViewHandler;
+import org.apache.pinot.materializedview.handler.MaterializedViewSplitDispatcher;
+import org.apache.pinot.materializedview.handler.MaterializedViewSplitExecutionContext;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewRewritePlan;
import org.apache.pinot.query.parser.utils.ParserUtils;
import org.apache.pinot.spi.accounting.ThreadAccountant;
import org.apache.pinot.spi.auth.AuthorizationResult;
@@ -145,6 +152,8 @@ public abstract class BaseSingleStageBrokerRequestHandler extends BaseBrokerRequ
private static final int MAX_UNAVAILABLE_SEGMENTS_TO_PRINT_IN_QUERY_EXCEPTION = 10;
protected final QueryOptimizer _queryOptimizer = new QueryOptimizer();
+ @Nullable
+ protected final MaterializedViewHandler _materializedViewHandler;
protected final boolean _disableGroovy;
protected final boolean _useApproximateFunction;
protected final int _defaultHllLog2m;
@@ -163,12 +172,26 @@ public abstract class BaseSingleStageBrokerRequestHandler extends BaseBrokerRequ
protected ImplicitHybridTableRouteProvider _implicitHybridTableRouteProvider;
protected LogicalTableRouteProvider _logicalTableRouteProvider;
+ /// Legacy constructor without an MV handler — kept so out-of-tree subclasses compiled against
+ /// the pre-MV signature continue to link. Delegates to the new constructor with
+ /// `materializedViewHandler = null`, which short-circuits every MV gate in this class and
+ /// makes the request flow byte-for-byte equivalent to the pre-MV codebase.
public BaseSingleStageBrokerRequestHandler(PinotConfiguration config, String brokerId,
BrokerRequestIdGenerator requestIdGenerator, RoutingManager routingManager,
AccessControlFactory accessControlFactory, QueryQuotaManager queryQuotaManager, TableCache tableCache,
ThreadAccountant threadAccountant, MultiClusterRoutingContext multiClusterRoutingContext) {
+ this(config, brokerId, requestIdGenerator, routingManager, accessControlFactory, queryQuotaManager, tableCache,
+ threadAccountant, multiClusterRoutingContext, null);
+ }
+
+ public BaseSingleStageBrokerRequestHandler(PinotConfiguration config, String brokerId,
+ BrokerRequestIdGenerator requestIdGenerator, RoutingManager routingManager,
+ AccessControlFactory accessControlFactory, QueryQuotaManager queryQuotaManager, TableCache tableCache,
+ ThreadAccountant threadAccountant, MultiClusterRoutingContext multiClusterRoutingContext,
+ @Nullable MaterializedViewHandler materializedViewHandler) {
super(config, brokerId, requestIdGenerator, routingManager, accessControlFactory, queryQuotaManager, tableCache,
threadAccountant, multiClusterRoutingContext);
+ _materializedViewHandler = materializedViewHandler;
_disableGroovy = _config.getProperty(Broker.DISABLE_GROOVY, Broker.DEFAULT_DISABLE_GROOVY);
_useApproximateFunction = _config.getProperty(Broker.USE_APPROXIMATE_FUNCTION, false);
_defaultHllLog2m = _config.getProperty(CommonConstants.Helix.DEFAULT_HYPERLOGLOG_LOG2M_KEY,
@@ -239,6 +262,9 @@ public abstract class BaseSingleStageBrokerRequestHandler extends BaseBrokerRequ
if (_enableMultistageMigrationMetric) {
_multistageCompileExecutor.shutdownNow();
}
+ if (_materializedViewHandler != null) {
+ _materializedViewHandler.close();
+ }
}
@VisibleForTesting
@@ -390,6 +416,16 @@ public abstract class BaseSingleStageBrokerRequestHandler extends BaseBrokerRequ
JsonNode request, @Nullable RequesterIdentity requesterIdentity, RequestContext requestContext,
@Nullable HttpHeaders httpHeaders, AccessControl accessControl, boolean queryWasLogged)
throws Exception {
+ /// Strip any user-supplied `MATERIALIZED_VIEW_REWRITE` query option so the broker-internal
+ /// marker stamped during a committed `FULL_REWRITE` swap is the only path that can produce
+ /// it. Without this guard a hostile client could spoof the marker and bypass
+ /// `BrokerReduceService`'s "Nested query is not supported without gapfill" safety net on
+ /// any `brokerRequest != serverBrokerRequest` path (current or future). Runs unconditionally
+ /// — the marker is defined globally and `BrokerReduceService` reads it regardless of whether
+ /// this broker has an MV handler wired in.
+ if (sqlNodeAndOptions.getOptions() != null) {
+ sqlNodeAndOptions.getOptions().remove(QueryOptionKey.MATERIALIZED_VIEW_REWRITE);
+ }
// Compile the request into PinotQuery
long compilationStartTimeNs = System.nanoTime();
CompileResult compileResult =
@@ -477,19 +513,23 @@ public abstract class BaseSingleStageBrokerRequestHandler extends BaseBrokerRequ
if (_enableRowColumnLevelAuth) {
TableRowColAccessResult rlsFilters = accessControl.getRowColFilters(requesterIdentity, tableName);
-
- //rewrite query
- Map<String, String> queryOptions =
- pinotQuery.getQueryOptions() == null ? new HashMap<>() : pinotQuery.getQueryOptions();
-
- rlsFilters.getRLSFilters().ifPresent(rowFilters -> {
+ List<String> rowFilters = rlsFilters.getRLSFilters().orElse(null);
+ if (rowFilters != null && !rowFilters.isEmpty()) {
+ /// Apply RLS to `serverPinotQuery` (the query the server actually executes), not to
+ /// `pinotQuery` as the pre-MV codebase did — for gapfill queries `serverPinotQuery` is
+ /// the inner stripped query and `pinotQuery` is the outer gapfill wrapper, so the pre-MV
+ /// behavior left the server-side scan unfiltered. Rewriting `serverPinotQuery` keeps
+ /// the RLS predicate on the table the server sees in both gapfill and non-gapfill paths;
+ /// for non-gapfill queries `serverPinotQuery == pinotQuery` and the behavior matches the
+ /// pre-MV codebase byte-for-byte.
+ Map<String, String> queryOptions =
+ serverPinotQuery.getQueryOptions() == null ? new HashMap<>() : serverPinotQuery.getQueryOptions();
String combinedFilters =
rowFilters.stream().map(filter -> "( " + filter + " )").collect(Collectors.joining(" AND "));
- String rowFiltersKey = RlsUtils.buildRlsFilterKey(rawTableName);
- queryOptions.put(rowFiltersKey, combinedFilters);
- pinotQuery.setQueryOptions(queryOptions);
+ queryOptions.put(RlsUtils.buildRlsFilterKey(rawTableName), combinedFilters);
+ serverPinotQuery.setQueryOptions(queryOptions);
try {
- CalciteSqlParser.queryRewrite(pinotQuery, RlsFiltersRewriter.class);
+ CalciteSqlParser.queryRewrite(serverPinotQuery, RlsFiltersRewriter.class);
rlsFiltersApplied.set(true);
_brokerMetrics.addMeteredTableValue(rawTableName, BrokerMeter.RLS_FILTERS_APPLIED, 1);
} catch (Exception e) {
@@ -497,7 +537,7 @@ public abstract class BaseSingleStageBrokerRequestHandler extends BaseBrokerRequ
"Unable to apply RLS filter: {}. Row-level security filtering will be disabled for this query.",
RlsFiltersRewriter.class.getName(), e);
}
- });
+ }
}
// Validate QPS quota
@@ -520,6 +560,35 @@ public abstract class BaseSingleStageBrokerRequestHandler extends BaseBrokerRequ
routeProvider = _implicitHybridTableRouteProvider;
}
+ /// Materialized-view rewrite (gated on `_materializedViewHandler != null`). Runs AFTER
+ /// authorization / RLS / quota so the auth and quota checks naturally see the user-facing
+ /// base table — there is no need to thread a pre-rewrite query/table through the auth and
+ /// quota call sites. On FULL_REWRITE the helper swaps `serverPinotQuery`, `tableName`,
+ /// `rawTableName`, and `schema` to the MV's variants so the rest of this method (route
+ /// build, scatter-gather) targets the MV. `materializedViewContext` carries the swap
+ /// outcome for downstream consumers: SPLIT path, response annotation, and the
+ /// `tablesQueried` response field which must tag the user's base table, not the MV.
+ MaterializedViewContext materializedViewContext = MaterializedViewContext.empty();
+ /// Capture the user-facing raw table name BEFORE the FULL_REWRITE swap below so the
+ /// response's `tablesQueried` field tags the user's base table — not the MV the broker
+ /// routed through. Equals `rawTableName` when no MV swap happens.
+ String userRawTableName = rawTableName;
+ if (_materializedViewHandler != null) {
+ MaterializedViewCompileOutcome outcome = applyMaterializedViewRewriteAtCompile(
+ requestId, serverPinotQuery, tableName, rawTableName, schema, _tableCache.isIgnoreCase());
+ materializedViewContext = outcome._materializedViewContext;
+ if (outcome._serverPinotQuery != serverPinotQuery) {
+ /// FULL_REWRITE — pick up the swapped state and recompute `serverBrokerRequest`.
+ /// `userRawTableName` is already pinned to the pre-swap value above, so the response
+ /// field continues to tag the base table.
+ serverPinotQuery = outcome._serverPinotQuery;
+ tableName = outcome._tableName;
+ rawTableName = outcome._rawTableName;
+ schema = outcome._schema;
+ serverBrokerRequest = CalciteSqlCompiler.convertToBrokerRequest(serverPinotQuery);
+ }
+ }
+
// Get the appropriate routing manager based on query options
RoutingManager selectedRoutingManager = _multiClusterRoutingContext != null && logicalTableConfig != null
? _multiClusterRoutingContext.getRoutingManager(pinotQuery.getQueryOptions())
@@ -798,6 +867,51 @@ public abstract class BaseSingleStageBrokerRequestHandler extends BaseBrokerRequ
// Execute the query
// TODO: Replace ServerStats with ServerRoutingStatsEntry.
ServerStats serverStats = new ServerStats();
+
+ /// MV Split: attempt parallel base + materialized view queries; on success return early, on
+ /// failure fall through to the standard non-split path with the route restored.
+ ///
+ /// Skip when `pinotQuery.isExplain()`. Split execution dispatches actual scatter-gather to
+ /// both base and MV servers and merges live `DataTable`s — semantics that violate the
+ /// explain-only contract. Falling through to the standard explain path below produces the
+ /// planner-visible output (base-table route, no live data read).
+ if (materializedViewContext.isSplitRewrite() && !pinotQuery.isExplain()) {
+ BrokerResponseNative viewSplitResponse = tryExecuteMaterializedViewSplit(requestId, query, sqlNodeAndOptions,
+ requesterIdentity, requestContext, brokerRequest, serverPinotQuery, schema, materializedViewContext,
+ routeInfo, remainingTimeMs, errorMsgs, numPrunedSegmentsTotal, database, tableName, userRawTableName,
+ rlsFiltersApplied.get(), serverStats, selectedRoutingManager, routeProvider, queryWasLogged);
+ if (viewSplitResponse != null) {
+ return viewSplitResponse;
+ }
+ /// Fallback: the MV split dispatcher mutated `routeInfo` in place — it stored SPLIT-side
+ /// broker requests (carrying `ts >= boundary` filters) via
+ /// `setOfflineBrokerRequest`/`setRealtimeBrokerRequest` and overwrote its routing tables.
+ /// To safely fall through to the standard non-split path below, we MUST restore the route
+ /// to its pre-split state. Easiest: re-acquire a fresh route and recompute its routing
+ /// tables using the original (unmutated) base-table broker requests. Without this,
+ /// `processBrokerRequest` would dispatch the user's "base table" query with
+ /// `ts >= boundary` baked in, silently dropping the historical half of the timeline.
+ materializedViewContext = MaterializedViewContext.empty();
+ routeInfo = routeProvider.getTableRouteInfo(tableName, _tableCache, selectedRoutingManager);
+ if (routeInfo instanceof ImplicitHybridTableRouteInfo) {
+ ((ImplicitHybridTableRouteInfo) routeInfo).setOfflineBrokerRequest(offlineBrokerRequest);
+ ((ImplicitHybridTableRouteInfo) routeInfo).setRealtimeBrokerRequest(realtimeBrokerRequest);
+ }
+ routeProvider.calculateRoutes(routeInfo, selectedRoutingManager, offlineBrokerRequest,
+ realtimeBrokerRequest, requestId);
+ /// After the fallback recomputes the route, refresh the execution-server and
+ /// pruned-segment locals captured pre-split. Without this, the cancel hook (`QueryServers`
+ /// built below) would track the pre-split server set — a cancel during fallback would
+ /// target the wrong instances if rebalancing changed the route between compile and the
+ /// post-split refresh. Per-table pool tags and pruned-segment counts would also reflect
+ /// the stale snapshot. The unavailable-segments error message in `errorMsgs` is left
+ /// as-is — its metric was already recorded at pre-split time.
+ offlineExecutionServers = routeInfo.getOfflineExecutionServers();
+ realtimeExecutionServers = routeInfo.getRealtimeExecutionServers();
+ unavailableSegments = routeInfo.getUnavailableSegments();
+ numPrunedSegmentsTotal = routeInfo.getNumPrunedSegmentsTotal();
+ }
+
// TODO: Handle broker specific operations for explain plan queries such as:
// - Alias handling
// - Compile time function invocation
@@ -837,7 +951,10 @@ public abstract class BaseSingleStageBrokerRequestHandler extends BaseBrokerRequ
brokerResponse = processBrokerRequest(requestId, brokerRequest, serverBrokerRequest, routeInfo,
remainingTimeMs, serverStats, requestContext);
}
- brokerResponse.setTablesQueried(Set.of(rawTableName));
+ brokerResponse.setTablesQueried(Set.of(userRawTableName));
+ if (_materializedViewHandler != null) {
+ _materializedViewHandler.annotateResponse(brokerResponse, materializedViewContext);
+ }
brokerResponse.setPools(Stream.concat(
offlineExecutionServers != null ? offlineExecutionServers.stream() : Stream.empty(),
realtimeExecutionServers != null ? realtimeExecutionServers.stream() : Stream.empty())
@@ -1058,6 +1175,181 @@ public abstract class BaseSingleStageBrokerRequestHandler extends BaseBrokerRequ
return new CompileResult(pinotQuery, serverPinotQuery, schema, tableName, rawTableName);
}
+ /// Mutable holder returned from [#applyMaterializedViewRewriteAtCompile] — Java has no out
+ /// parameters, so the helper returns the (possibly swapped) compile state plus the resulting
+ /// [MaterializedViewContext] in a small bundle.
+ private static final class MaterializedViewCompileOutcome {
+ final PinotQuery _serverPinotQuery;
+ final String _tableName;
+ final String _rawTableName;
+ final Schema _schema;
+ final MaterializedViewContext _materializedViewContext;
+
+ MaterializedViewCompileOutcome(PinotQuery serverPinotQuery, String tableName, String rawTableName,
+ Schema schema, MaterializedViewContext materializedViewContext) {
+ _serverPinotQuery = serverPinotQuery;
+ _tableName = tableName;
+ _rawTableName = rawTableName;
+ _schema = schema;
+ _materializedViewContext = materializedViewContext;
+ }
+ }
+
+ /// MV rewrite is per-MV-table opt-in (controlled by `rewriteEnabled` on the MV definition).
+ /// The broker always runs the handler when configured; the handler/cache filters out MVs that
+ /// are not eligible for rewrite (no watermark yet, rewrite disabled, staleness SLO exceeded).
+ ///
+ /// Returns the (possibly FULL_REWRITE-swapped) compile state and the resulting MV context.
+ /// Any exception inside the rewrite engine is caught here so a broken strategy never kills the
+ /// query — fall back to the base-table path and surface the failure via
+ /// `QUERY_REWRITE_EXCEPTIONS`.
+ private MaterializedViewCompileOutcome applyMaterializedViewRewriteAtCompile(long requestId,
+ PinotQuery serverPinotQuery, String tableName, String rawTableName, Schema schema, boolean ignoreCase) {
+ /// Skip rewrite when the user enabled multi-cluster routing — the broker's
+ /// `validatePhysicalTablesWithMultiClusterRouting` check (in `doHandleRequest`) inspects the
+ /// post-rewrite table name and hard-rejects MV tables that aren't registered in the
+ /// multi-cluster routing map. Without this early-exit, a FULL_REWRITE swap would turn a
+ /// valid base-table query into a hard error with no fallback path. MV+multi-cluster
+ /// composition is out of scope for this PR. Null-guard the option map — `PinotQuery` is
+ /// Thrift-generated and a hand-constructed query (or future code path) may not pre-allocate
+ /// query options; `isMultiClusterRoutingEnabled` would otherwise NPE on `.get`.
+ Map<String, String> serverQueryOptionsForMcRouting = serverPinotQuery.getQueryOptions();
+ if (serverQueryOptionsForMcRouting != null
+ && QueryOptionsUtils.isMultiClusterRoutingEnabled(serverQueryOptionsForMcRouting, false)) {
+ return new MaterializedViewCompileOutcome(
+ serverPinotQuery, tableName, rawTableName, schema, MaterializedViewContext.empty());
+ }
+
+ /// Skip rewrite when the user's query already targets an MV table directly
+ /// ([TableConfig#isMaterializedView()] is the single source of truth for MV identity since
+ /// upstream PR #18564 added the `isMaterializedView` flag on `TableConfig`). Cascading
+ /// MV-to-MV rewrites are not supported — the handler/cache only indexes the base-table →
+ /// MV direction, so this would be a no-op anyway, but the explicit guard makes the intent
+ /// visible and avoids a wasted cache lookup per query. When `tableName` already carries a
+ /// type suffix (the user typed `mv_t_OFFLINE` or `mv_t_REALTIME` explicitly), look up that
+ /// exact variant; otherwise fall back to the OFFLINE variant since MVs are always OFFLINE
+ /// per `validateMaterializedViewInvariants`.
+ String mvLookupTableNameWithType = TableNameBuilder.getTableTypeFromTableName(tableName) != null
+ ? tableName : TableNameBuilder.OFFLINE.tableNameWithType(rawTableName);
+ TableConfig userTableConfig = _tableCache.getTableConfig(mvLookupTableNameWithType);
+ if (userTableConfig != null && userTableConfig.isMaterializedView()) {
+ return new MaterializedViewCompileOutcome(
+ serverPinotQuery, tableName, rawTableName, schema, MaterializedViewContext.empty());
+ }
+
+ MaterializedViewContext materializedViewContext;
+ try {
+ materializedViewContext = _materializedViewHandler.compile(new MaterializedViewCompileContext(
+ serverPinotQuery, tableName, rawTableName, _tableCache));
+ } catch (Exception e) {
+ LOGGER.error("Materialized view rewrite failed for request {} on table {}; "
+ + "falling back to base-table query path", requestId, rawTableName, e);
+ _brokerMetrics.addMeteredTableValue(rawTableName, BrokerMeter.QUERY_REWRITE_EXCEPTIONS, 1);
+ return new MaterializedViewCompileOutcome(
+ serverPinotQuery, tableName, rawTableName, schema, MaterializedViewContext.empty());
+ }
+
+ if (!materializedViewContext.isFullRewrite()) {
+ return new MaterializedViewCompileOutcome(
+ serverPinotQuery, tableName, rawTableName, schema, materializedViewContext);
+ }
+
+ /// `FULL_REWRITE` replaces the base-table query with the MV-OFFLINE query. If the base
+ /// table is hybrid (has both OFFLINE and REALTIME variants), the MV covers only the offline
+ /// half and the swap would silently drop every row ingested through the realtime stream
+ /// since the MV last refreshed. Reject the rewrite at this broker-handler layer (the
+ /// rewrite engine does not have access to the table cache) so the broker emits the matching
+ /// observability signal and continues against the base table unchanged.
+ String baseRawTableName = TableNameBuilder.extractRawTableName(tableName);
+ if (_tableCache.getTableConfig(TableNameBuilder.REALTIME.tableNameWithType(baseRawTableName)) != null) {
+ LOGGER.warn("FULL_REWRITE skipped for request {} on hybrid base table {}: MV would drop "
+ + "realtime data; falling back to base-table query path", requestId, baseRawTableName);
+ _brokerMetrics.addMeteredTableValue(baseRawTableName, BrokerMeter.QUERY_REWRITE_EXCEPTIONS, 1);
+ return new MaterializedViewCompileOutcome(
+ serverPinotQuery, tableName, rawTableName, schema, MaterializedViewContext.empty());
+ }
+
+ /// Swap server-side query/table/schema to the MV. The pre-rewrite query/table is preserved
+ /// inside `materializedViewContext` so ACL/quota/RLS authorize against the base table.
+ /// `pinotQuery` (user-facing) stays pointing at the original so `fillEmptyResponseSchema`
+ /// and query logging use the base table schema/name rather than the materialized view's.
+ MaterializedViewRewritePlan plan = Preconditions.checkNotNull(materializedViewContext.getPlan(),
+ "FULL_REWRITE context must carry a plan");
+ PinotQuery rewrittenServerQuery = plan.getMaterializedViewQuery();
+ /// Mark the server query so `BrokerReduceService` can distinguish MV-rewritten queries from
+ /// gapfill / future federated paths via an explicit signal instead of a structural heuristic.
+ /// Stamped on the swap rather than at plan-construction time so the marker is present iff
+ /// the rewrite was committed to.
+ Map<String, String> serverQueryOptions = rewrittenServerQuery.getQueryOptions();
+ if (serverQueryOptions == null) {
+ serverQueryOptions = new HashMap<>();
+ rewrittenServerQuery.setQueryOptions(serverQueryOptions);
+ }
+ serverQueryOptions.put(QueryOptionKey.MATERIALIZED_VIEW_REWRITE, "true");
+ String rewrittenTableName = plan.getMaterializedViewTableNameWithType();
+ String rewrittenRawTableName = TableNameBuilder.extractRawTableName(rewrittenTableName);
+ Schema rewrittenSchema = _tableCache.getSchema(rewrittenRawTableName);
+ /// The handler's `compile()` already null-checks the MV schema before returning a plan, but
+ /// the broker re-reads the schema here on a fresh `TableCache` query — between those two
+ /// reads a concurrent MV drop or zk listener-driven eviction can leave the second lookup
+ /// null. Fall back to the base-table path with the same observability signal used for the
+ /// case-different-column failure below. Without this null-guard, `outcome._schema` would
+ /// flow as null into `_queryOptimizer.optimize` and `fillEmptyResponseSchema`, neither of
+ /// which contractually accepts null.
+ if (rewrittenSchema == null) {
+ LOGGER.warn("FULL_REWRITE skipped for request {}: MV schema for {} not present in table cache "
+ + "(likely dropped between handler compile and broker re-read); falling back to base-table query path",
+ requestId, rewrittenRawTableName);
+ _brokerMetrics.addMeteredTableValue(rewrittenRawTableName, BrokerMeter.QUERY_REWRITE_EXCEPTIONS, 1);
+ return new MaterializedViewCompileOutcome(
+ serverPinotQuery, tableName, rawTableName, schema, MaterializedViewContext.empty());
+ }
+
+ /// Re-canonicalize column identifiers against the MV table's own column name map. The
+ /// earlier `updateColumnNames` call (in `compileRequest`) ran against the base table; the
+ /// rewritten MV query may reference columns whose case differs from the MV schema's
+ /// canonical case. Wrap in try/catch so a case-sensitive cluster with a case-different MV
+ /// column name falls back to the base-table path instead of failing the query.
+ Map<String, String> mvColumnNameMap = _tableCache.getColumnNameMap(rewrittenRawTableName);
+ if (mvColumnNameMap != null) {
+ try {
+ updateColumnNames(rewrittenRawTableName, rewrittenServerQuery, ignoreCase, mvColumnNameMap);
+ } catch (Exception e) {
+ LOGGER.warn("FULL_REWRITE column re-canonicalization failed for request {} on MV {}; "
+ + "reverting to base-table query path", requestId, rewrittenRawTableName, e);
+ _brokerMetrics.addMeteredTableValue(rewrittenRawTableName, BrokerMeter.QUERY_REWRITE_EXCEPTIONS, 1);
+ return new MaterializedViewCompileOutcome(
+ serverPinotQuery, tableName, rawTableName, schema, MaterializedViewContext.empty());
+ }
+ }
+
+ return new MaterializedViewCompileOutcome(
+ rewrittenServerQuery, rewrittenTableName, rewrittenRawTableName, rewrittenSchema, materializedViewContext);
+ }
+
+ /// Apply row-level security filters to `serverPinotQuery`. When the accessor returns RLS rules
+ /// for `rawTableName`, the rewriter appends them to the query's WHERE clause and stamps a
+ /// `rlsFilters-<rawTableName>` entry in `queryOptions` — the marker the caller later reads to
+ /// stamp `rlsFiltersApplied` on the broker response.
+ private void applyRlsFilters(long requestId, String query, PinotQuery serverPinotQuery, String rawTableName,
+ @Nullable RequesterIdentity requesterIdentity, AccessControl accessControl) {
+ TableRowColAccessResult rlsFilters = accessControl.getRowColFilters(requesterIdentity, rawTableName);
+ List<String> rowFilters = rlsFilters.getRLSFilters().orElse(null);
+ if (CollectionUtils.isEmpty(rowFilters)) {
+ return;
+ }
+
+ Map<String, String> queryOptions =
+ serverPinotQuery.getQueryOptions() == null ? new HashMap<>() : serverPinotQuery.getQueryOptions();
+ String combinedFilters =
+ rowFilters.stream().map(filter -> "( " + filter + " )").collect(Collectors.joining(" AND "));
+ queryOptions.put(RlsUtils.buildRlsFilterKey(rawTableName), combinedFilters);
+ serverPinotQuery.setQueryOptions(queryOptions);
+ CalciteSqlParser.queryRewrite(serverPinotQuery, RlsFiltersRewriter.class);
+ _brokerMetrics.addMeteredTableValue(rawTableName, BrokerMeter.RLS_FILTERS_APPLIED, 1);
+ LOGGER.debug("Applied RLS filters for request {} on table {}: {}", requestId, rawTableName, query);
+ }
+
private void throwAccessDeniedError(long requestId, String query, RequestContext requestContext, String tableName,
AuthorizationResult authorizationResult) {
_brokerMetrics.addMeteredTableValue(tableName, BrokerMeter.REQUEST_DROPPED_DUE_TO_ACCESS_ERROR, 1);
@@ -1123,14 +1415,29 @@ public abstract class BaseSingleStageBrokerRequestHandler extends BaseBrokerRequ
QueryContext serverQueryContext = QueryContextConverterUtils.getQueryContext(serverPinotQuery);
ResultTable resultTable = EmptyResponseUtils.buildEmptyResultTable(serverQueryContext);
brokerResponse.setResultTable(resultTable);
+ /// `pinotQuery` and `serverPinotQuery` differ either when the user query is a gapfill
+ /// (parser wrapped it) or when MV rewrite produced an MV-bound `serverPinotQuery`. The
+ /// gapfill case needs the nested-query post-processor. Any OTHER cause of divergence is
+ /// a malformed nested query: the pre-MV codebase rejected this with a
+ /// `BadQueryRequestException("Nested query is not supported without gapfill")` and we keep
+ /// the same safety net here — mirrors the symmetric check in `BrokerReduceService`. The
+ /// MV-rewrite case is recognized via the broker-internal marker stamped on the swap;
+ /// see `BrokerReduceService#isMaterializedViewRewrite` for the matching predicate.
if (pinotQuery != serverPinotQuery) {
QueryContext queryContext = QueryContextConverterUtils.getQueryContext(pinotQuery);
GapfillUtils.GapfillType gapfillType = GapfillUtils.getGapfillType(queryContext);
- if (gapfillType == null) {
- throw new BadQueryRequestException("Nested query is not supported without gapfill");
+ if (gapfillType != null) {
+ BaseGapfillProcessor gapfillProcessor =
+ GapfillProcessorFactory.getGapfillProcessor(queryContext, gapfillType);
+ gapfillProcessor.process(brokerResponse);
+ } else {
+ Map<String, String> serverQueryOptions = serverPinotQuery.getQueryOptions();
+ boolean isMaterializedViewRewrite = serverQueryOptions != null
+ && Boolean.parseBoolean(serverQueryOptions.get(QueryOptionKey.MATERIALIZED_VIEW_REWRITE));
+ if (!isMaterializedViewRewrite) {
+ throw new BadQueryRequestException("Nested query is not supported without gapfill");
+ }
}
- BaseGapfillProcessor gapfillProcessor = GapfillProcessorFactory.getGapfillProcessor(queryContext, gapfillType);
- gapfillProcessor.process(brokerResponse);
}
fillEmptyResponseSchema(pinotQuery, brokerResponse, schema, database, query);
} catch (Exception e) {
@@ -1190,12 +1497,12 @@ public abstract class BaseSingleStageBrokerRequestHandler extends BaseBrokerRequ
.forEach(operand -> setTimestampIndexExpressionOverrideHints(operand, timestampIndexColumns, pinotQuery));
}
- /** Given a {@link PinotQuery}, check if the WHERE clause will always evaluate to false. */
+ /// Given a [PinotQuery], check if the WHERE clause will always evaluate to false.
private boolean isFilterAlwaysFalse(PinotQuery pinotQuery) {
return FALSE.equals(pinotQuery.getFilterExpression());
}
- /** Given a {@link PinotQuery}, check if the WHERE clause will always evaluate to true. */
+ /// Given a [PinotQuery], check if the WHERE clause will always evaluate to true.
private boolean isFilterAlwaysTrue(PinotQuery pinotQuery) {
return TRUE.equals(pinotQuery.getFilterExpression());
}
@@ -2034,9 +2341,253 @@ public abstract class BaseSingleStageBrokerRequestHandler extends BaseBrokerRequ
}
}
+ /// Broker-side helper invoked by the MV handler's split-dispatcher callback. Owns the generic
+ /// Execute a SPLIT_REWRITE plan via the configured MV handler. Returns a finalized
+ /// `BrokerResponseNative` on success, or `null` to signal the caller should fall through to
+ /// the non-split base-table path. Execute-time MV failures (route gone mid-query, malformed
+ /// time-format from a pre-V2 rolling-upgrade znode, NPE inside a strategy plan) must never
+ /// surface as HTTP 500 — they bump `QUERY_REWRITE_EXCEPTIONS` and return null.
+ /// `BROKER_RESOURCE_MISSING` (MV dropped mid-query) is demoted to WARN so the operator's
+ /// error-log signal isn't drowned in routine MV churn.
+ @Nullable
+ private BrokerResponseNative tryExecuteMaterializedViewSplit(long requestId, String query,
+ SqlNodeAndOptions sqlNodeAndOptions, @Nullable RequesterIdentity requesterIdentity,
+ RequestContext requestContext, BrokerRequest brokerRequest, PinotQuery serverPinotQuery, Schema schema,
+ MaterializedViewContext materializedViewContext, TableRouteInfo routeInfo, long remainingTimeMs,
+ List<QueryProcessingException> errorMsgs, int numPrunedSegmentsTotal, String database,
+ String tableName, String userRawTableName, boolean rlsFiltersApplied, ServerStats serverStats,
+ RoutingManager selectedRoutingManager, TableRouteProvider routeProvider, boolean queryWasLogged) {
+ String clientRequestId = extractClientRequestId(sqlNodeAndOptions);
+ MaterializedViewSplitDispatcher dispatcher =
+ (originalReq, baseQ, baseRoute, baseSch, viewQ, viewTable, viewSch, timeoutMs) ->
+ dispatchMaterializedViewSplit(requestId, originalReq, baseQ, baseRoute, baseSch, viewQ, viewTable,
+ viewSch, timeoutMs, serverStats, requestContext, selectedRoutingManager, routeProvider,
+ clientRequestId, query);
+ MaterializedViewSplitExecutionContext splitCtx = MaterializedViewSplitExecutionContext.builder()
+ .originalBrokerRequest(brokerRequest)
+ .baseServerPinotQuery(serverPinotQuery)
+ .baseSchema(schema)
+ .materializedViewContext(materializedViewContext)
+ .baseRouteInfo(routeInfo)
+ .remainingTimeMs(remainingTimeMs)
+ .dispatcher(dispatcher)
+ .build();
+ BrokerResponseNative viewSplitResponse;
+ try {
+ viewSplitResponse = _materializedViewHandler.executeSplit(splitCtx);
+ } catch (QueryException qe) {
+ LOGGER.warn("Materialized view split execution skipped for request {} on table {}: {}; "
+ + "falling back to base-table query path", requestId, userRawTableName, qe.getMessage());
+ _brokerMetrics.addMeteredTableValue(userRawTableName, BrokerMeter.QUERY_REWRITE_EXCEPTIONS, 1);
+ return null;
+ } catch (Exception e) {
+ LOGGER.error("Materialized view split execution failed for request {} on table {}; "
+ + "falling back to base-table query path", requestId, userRawTableName, e);
+ _brokerMetrics.addMeteredTableValue(userRawTableName, BrokerMeter.QUERY_REWRITE_EXCEPTIONS, 1);
+ return null;
+ }
+ viewSplitResponse.setTablesQueried(Set.of(userRawTableName));
+ _materializedViewHandler.annotateResponse(viewSplitResponse, materializedViewContext);
+ for (QueryProcessingException errorMsg : errorMsgs) {
+ viewSplitResponse.addException(errorMsg);
+ }
+ viewSplitResponse.setNumSegmentsPrunedByBroker(numPrunedSegmentsTotal);
+ fillEmptyResponseSchema(brokerRequest.getPinotQuery(), viewSplitResponse, schema, database, query);
+ long totalTimeMs = System.currentTimeMillis() - requestContext.getRequestArrivalTimeMillis();
+ viewSplitResponse.setTimeUsedMs(totalTimeMs);
+ augmentStatistics(requestContext, viewSplitResponse);
+ viewSplitResponse.setRLSFiltersApplied(rlsFiltersApplied);
+ _queryLogger.logQueryCompleted(
+ new QueryLogger.QueryLogParams(requestContext, tableName, viewSplitResponse,
+ QueryLogger.QueryLogParams.QueryEngine.SINGLE_STAGE, requesterIdentity, serverStats),
+ queryWasLogged);
+ return viewSplitResponse;
+ }
+
+ /// route-build + scatter-gather + reduce work (hybrid offline/realtime split, always-false-filter
+ /// pruning, MV-side route, deep-copy of the reduce request to strip `SERVER_RETURN_FINAL_RESULT`).
+ /// MV-specific concerns (time-boundary computation and per-branch filter attachment) live in the
+ /// MV handler implementation, not here.
+ private BrokerResponseNative dispatchMaterializedViewSplit(long requestId,
+ BrokerRequest originalBrokerRequest, PinotQuery baseQueryWithTimeFilter, TableRouteInfo baseRouteInfo,
+ Schema baseSchema, PinotQuery viewQueryWithTimeFilter, String viewTableNameWithType, Schema viewSchema,
+ long timeoutMs, ServerStats serverStats, RequestContext requestContext, RoutingManager routingManager,
+ TableRouteProvider routeProvider, @Nullable String clientRequestId, String query) throws Exception {
+ /// 1. Optimize and route the base-table side (handles hybrid offline/realtime split).
+ _queryOptimizer.optimize(baseQueryWithTimeFilter, baseSchema);
+ BrokerRequest baseBrokerRequest = CalciteSqlCompiler.convertToBrokerRequest(baseQueryWithTimeFilter);
+ ImplicitHybridTableRouteInfo hybridBaseRoute =
+ prepareBaseTableHybridRoute(baseBrokerRequest, baseRouteInfo, baseSchema);
+
+ BrokerRequest baseOfflineBrokerRequest = hybridBaseRoute.getOfflineBrokerRequest();
+ BrokerRequest baseRealtimeBrokerRequest = hybridBaseRoute.getRealtimeBrokerRequest();
+ if (baseOfflineBrokerRequest != null && isFilterAlwaysFalse(baseOfflineBrokerRequest.getPinotQuery())) {
+ baseOfflineBrokerRequest = null;
+ }
+ if (baseRealtimeBrokerRequest != null && isFilterAlwaysFalse(baseRealtimeBrokerRequest.getPinotQuery())) {
+ baseRealtimeBrokerRequest = null;
+ }
+ hybridBaseRoute.setOfflineBrokerRequest(baseOfflineBrokerRequest);
+ hybridBaseRoute.setRealtimeBrokerRequest(baseRealtimeBrokerRequest);
+ routeProvider.calculateRoutes(hybridBaseRoute, routingManager, baseOfflineBrokerRequest, baseRealtimeBrokerRequest,
+ requestId);
+
+ /// 2. Optimize and route the MV-table side (always offline).
+ /// Apply MV-table expression overrides and `$ts$DAY` timestamp-index hints before
+ /// optimization, mirroring the non-split path. Without this, overrides configured on the
+ /// MV table would be silently dropped on the SPLIT path.
+ handleExpressionOverride(viewQueryWithTimeFilter, _tableCache.getExpressionOverrideMap(viewTableNameWithType));
+ handleTimestampIndexOverride(viewQueryWithTimeFilter, _tableCache.getTableConfig(viewTableNameWithType));
+ _queryOptimizer.optimize(viewQueryWithTimeFilter, viewSchema);
+ BrokerRequest viewBrokerRequest = CalciteSqlCompiler.convertToBrokerRequest(viewQueryWithTimeFilter);
+ TableRouteInfo viewRouteInfo = routeProvider.getTableRouteInfo(viewTableNameWithType, _tableCache, routingManager);
+ /// The MV table can disappear between compile (broker MV cache) and execute (route lookup)
+ /// — operator dropped it, controller mid-state-transition, etc. Treat missing-route the
+ /// same way the base path does (`BROKER_RESOURCE_MISSING` with a graceful message); the
+ /// caller's try/catch wrapper then falls back to the base table.
+ if (!viewRouteInfo.isExists()) {
+ throw new QueryException(QueryErrorCode.BROKER_RESOURCE_MISSING,
+ "Materialized view " + viewTableNameWithType
+ + " is no longer routable (table may have been dropped after rewrite compile)");
+ }
+ /// `MaterializedViewTaskScheduler.generateTasks` only generates for OFFLINE tables, so the
+ /// MV route MUST be OFFLINE-only. Fail loud rather than silently misroute against a
+ /// HYBRID/REALTIME table that operators may have created by mistake.
+ if (!viewRouteInfo.isOffline()) {
+ /// Throw a typed `QueryException` (caught at WARN by `tryExecuteMaterializedViewSplit`)
+ /// rather than `IllegalStateException` (which would log at ERROR with stack), since a
+ /// HYBRID/REALTIME MV route is an operator misconfiguration the broker can recover from
+ /// by falling back to the base-table path — same recovery model as the missing-route
+ /// case above.
+ throw new QueryException(QueryErrorCode.BROKER_RESOURCE_MISSING,
+ "MV split routing requires an OFFLINE materialized-view table, got route type for "
+ + viewTableNameWithType + ": hybrid=" + viewRouteInfo.isHybrid()
+ + ", offline=" + viewRouteInfo.isOffline());
+ }
+ routeProvider.calculateRoutes(viewRouteInfo, routingManager, viewBrokerRequest, null, requestId);
+
+ /// 3. Build the reduce-time broker request (strip `SERVER_RETURN_FINAL_RESULT`).
+ ///
+ /// The `originalBrokerRequest` may carry `SERVER_RETURN_FINAL_RESULT=true` set by the outer
+ /// `doHandleRequest` flow (when `numServers == 1`). Since
+ /// `processMaterializedViewSplitBrokerRequest` passes this request to `BrokerReduceService`
+ /// as the `serverBrokerRequest`, the reducer would build a `QueryContext` with
+ /// `isServerReturnFinalResult() == true` and attempt to cast intermediate objects (e.g.
+ /// `HyperLogLog`) to `Comparable`, causing a `ClassCastException`. Deep-copy so the
+ /// mutation does not leak back to callers that hold the original reference.
+ BrokerRequest reduceBrokerRequest = originalBrokerRequest.deepCopy();
+ Map<String, String> reduceQueryOptions = reduceBrokerRequest.getPinotQuery().getQueryOptions();
+ if (reduceQueryOptions != null) {
+ reduceQueryOptions.remove(QueryOptionKey.SERVER_RETURN_FINAL_RESULT);
+ reduceQueryOptions.remove(QueryOptionKey.SERVER_RETURN_FINAL_RESULT_KEY_UNPARTITIONED);
+ }
+
+ long materializedViewRequestId = _requestIdGenerator.get();
+ if (isQueryCancellationEnabled()) {
+ /// Track the union of base + MV server sets against the user's `requestId`. Cancel will
+ /// be sent under the user's global query id; the MV-side servers may already have
+ /// finished by the time cancel fires, but they would not be reached by global-id cancel
+ /// anyway (they ran under `materializedViewRequestId`). Improving per-sub-request cancel
+ /// is out of scope for this PR — see the TODO in `handleCancel` for the eventual fix.
+ Set<ServerInstance> offlineUnion =
+ unionServers(hybridBaseRoute.getOfflineExecutionServers(), viewRouteInfo.getOfflineExecutionServers());
+ Set<ServerInstance> realtimeUnion =
+ unionServers(hybridBaseRoute.getRealtimeExecutionServers(), viewRouteInfo.getRealtimeExecutionServers());
+ onQueryStart(requestId, clientRequestId, query, new QueryServers(query, offlineUnion, realtimeUnion));
+ try {
+ return processMaterializedViewSplitBrokerRequest(requestId, materializedViewRequestId, reduceBrokerRequest,
+ hybridBaseRoute, viewRouteInfo, timeoutMs, serverStats, requestContext);
+ } finally {
+ onQueryFinish(requestId);
+ LOGGER.debug("Remove track of running MV split query: {}", requestId);
+ }
+ }
+
+ return processMaterializedViewSplitBrokerRequest(requestId, materializedViewRequestId, reduceBrokerRequest,
+ hybridBaseRoute, viewRouteInfo, timeoutMs, serverStats, requestContext);
+ }
+
+ @Nullable
+ private static Set<ServerInstance> unionServers(@Nullable Set<ServerInstance> a, @Nullable Set<ServerInstance> b) {
+ if (a == null) {
+ return b;
+ }
+ if (b == null) {
+ return a;
+ }
+ Set<ServerInstance> union = new HashSet<>(a.size() + b.size());
+ union.addAll(a);
+ union.addAll(b);
+ return union;
+ }
+
/**
- * Helper method to attach the time boundary to the given PinotQuery.
+ * Prepares the offline/realtime broker requests for the base table route based on
+ * whether the base table is hybrid, offline-only, or realtime-only.
+ * This mirrors the logic in the main doHandleRequest flow but operates on a pre-existing
+ * routeInfo for the base table. Per-branch `handleExpressionOverride` /
+ * `handleTimestampIndexOverride` are applied so the SPLIT base-table branches receive the
+ * same expression-override and `$ts$DAY` timestamp-index hints as the non-split path —
+ * without these, table-level overrides configured on the base table would be silently
+ * dropped on the SPLIT path, producing divergent results vs. the base fallback path.
*/
+ private ImplicitHybridTableRouteInfo prepareBaseTableHybridRoute(BrokerRequest baseBrokerRequest,
+ TableRouteInfo baseRouteInfo, Schema schema) {
+ Preconditions.checkState(baseRouteInfo instanceof ImplicitHybridTableRouteInfo,
+ "MV split execution requires ImplicitHybridTableRouteInfo but got: %s",
+ baseRouteInfo.getClass().getSimpleName());
+ ImplicitHybridTableRouteInfo hybridRoute = (ImplicitHybridTableRouteInfo) baseRouteInfo;
+
+ String offlineTableName = baseRouteInfo.getOfflineTableName();
+ String realtimeTableName = baseRouteInfo.getRealtimeTableName();
+ TableConfig offlineTableConfig = baseRouteInfo.getOfflineTableConfig();
+ TableConfig realtimeTableConfig = baseRouteInfo.getRealtimeTableConfig();
+ TimeBoundaryInfo timeBoundaryInfo = baseRouteInfo.getTimeBoundaryInfo();
+
+ if (baseRouteInfo.isHybrid()) {
+ PinotQuery basePinotQuery = baseBrokerRequest.getPinotQuery();
+
+ PinotQuery offlinePinotQuery = basePinotQuery.deepCopy();
+ offlinePinotQuery.getDataSource().setTableName(offlineTableName);
+ if (timeBoundaryInfo != null) {
+ attachTimeBoundary(offlinePinotQuery, timeBoundaryInfo, true);
+ }
+ handleExpressionOverride(offlinePinotQuery, _tableCache.getExpressionOverrideMap(offlineTableName));
+ handleTimestampIndexOverride(offlinePinotQuery, offlineTableConfig);
+ _queryOptimizer.optimize(offlinePinotQuery, schema);
+ BrokerRequest offlineBrokerRequest = CalciteSqlCompiler.convertToBrokerRequest(offlinePinotQuery);
+
+ PinotQuery realtimePinotQuery = basePinotQuery.deepCopy();
+ realtimePinotQuery.getDataSource().setTableName(realtimeTableName);
+ if (timeBoundaryInfo != null) {
+ attachTimeBoundary(realtimePinotQuery, timeBoundaryInfo, false);
+ }
+ handleExpressionOverride(realtimePinotQuery, _tableCache.getExpressionOverrideMap(realtimeTableName));
+ handleTimestampIndexOverride(realtimePinotQuery, realtimeTableConfig);
+ _queryOptimizer.optimize(realtimePinotQuery, schema);
+ BrokerRequest realtimeBrokerRequest = CalciteSqlCompiler.convertToBrokerRequest(realtimePinotQuery);
+
+ hybridRoute.setOfflineBrokerRequest(offlineBrokerRequest);
+ hybridRoute.setRealtimeBrokerRequest(realtimeBrokerRequest);
+ } else if (baseRouteInfo.isOffline()) {
+ setTableName(baseBrokerRequest, offlineTableName);
+ handleExpressionOverride(baseBrokerRequest.getPinotQuery(),
+ _tableCache.getExpressionOverrideMap(offlineTableName));
+ handleTimestampIndexOverride(baseBrokerRequest.getPinotQuery(), offlineTableConfig);
+ hybridRoute.setOfflineBrokerRequest(baseBrokerRequest);
+ } else {
+ setTableName(baseBrokerRequest, realtimeTableName);
+ handleExpressionOverride(baseBrokerRequest.getPinotQuery(),
+ _tableCache.getExpressionOverrideMap(realtimeTableName));
+ handleTimestampIndexOverride(baseBrokerRequest.getPinotQuery(), realtimeTableConfig);
+ hybridRoute.setRealtimeBrokerRequest(baseBrokerRequest);
+ }
+ return hybridRoute;
+ }
+
+ /// Helper method to attach the time boundary to the given PinotQuery.
+ /// Used for standard hybrid table offline/realtime split where offline has
+ /// `ts <= boundary` and realtime has `ts > boundary`.
private static void attachTimeBoundary(PinotQuery pinotQuery, TimeBoundaryInfo timeBoundaryInfo,
boolean isOfflineRequest) {
String functionName = isOfflineRequest ? FilterKind.LESS_THAN_OR_EQUAL.name() : FilterKind.GREATER_THAN.name();
@@ -2055,6 +2606,7 @@ public abstract class BaseSingleStageBrokerRequestHandler extends BaseBrokerRequ
}
}
+
/**
* Processes the optimized broker requests for both OFFLINE and REALTIME table.
* TODO: Directly take PinotQuery
@@ -2064,6 +2616,37 @@ public abstract class BaseSingleStageBrokerRequestHandler extends BaseBrokerRequ
ServerStats serverStats, RequestContext requestContext)
throws Exception;
+ /// Processes an MV-split query by issuing two independent scatter-gather requests - one to the
+ /// base table (for recent data beyond the MV boundary) and one to the materialized view table (for historical
+ /// data up to the boundary) - then merging all returned `DataTable`s into a single
+ /// `dataTableMap` and reducing with the original user query's `BrokerRequest`.
+ ///
+ /// Subclasses must implement this to perform the actual network I/O. The default
+ /// [SingleConnectionBrokerRequestHandler] sends both requests via the `QueryRouter`.
+ ///
+ /// @param requestId unique request identifier
+ /// @param originalBrokerRequest the user's original query (used as the reduce key)
+ /// @param baseRoute routing info for the base table query (`ts > boundary`)
+ /// @param materializedViewRoute routing info for the materialized view table query
+ /// @param timeoutMs remaining timeout in milliseconds
+ /// @param serverStats collector for server-side stats
+ /// @param requestContext request-level context for metrics and tracing
+ /// Default implementation throws — keeps the method concrete so out-of-tree subclasses of
+ /// `BaseSingleStageBrokerRequestHandler` that don't opt into MV split execution continue to
+ /// compile. With `materializedViewHandler == null` (the legacy ctor path) the SPLIT branch
+ /// in `doHandleRequest` is gated off entirely, so this method is unreachable; subclasses that
+ /// DO enable MV split must override (see `SingleConnectionBrokerRequestHandler`). The
+ /// `UnsupportedOperationException` body matches the gRPC broker's behavior: a handler wired
+ /// onto gRPC must report `supportsSplitRewrite() == false` so SPLIT_REWRITE is suppressed at
+ /// compile time, and reaching this method indicates a misconfiguration.
+ protected BrokerResponseNative processMaterializedViewSplitBrokerRequest(long requestId,
+ long materializedViewRequestId, BrokerRequest originalBrokerRequest, TableRouteInfo baseRoute,
+ TableRouteInfo materializedViewRoute, long timeoutMs, ServerStats serverStats, RequestContext requestContext)
+ throws Exception {
+ throw new UnsupportedOperationException(
+ "processMaterializedViewSplitBrokerRequest must be overridden by subclasses that opt into MV split execution");
+ }
+
private String getGlobalQueryId(long requestId) {
return _brokerId + "_" + requestId;
}
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/GrpcBrokerRequestHandler.java b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/GrpcBrokerRequestHandler.java
index 913cb331831..1f3a6d378b1 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/GrpcBrokerRequestHandler.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/GrpcBrokerRequestHandler.java
@@ -25,6 +25,7 @@ import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.TimeUnit;
+import javax.annotation.Nullable;
import javax.annotation.concurrent.ThreadSafe;
import org.apache.pinot.broker.broker.AccessControlFactory;
import org.apache.pinot.broker.queryquota.QueryQuotaManager;
@@ -43,6 +44,7 @@ import org.apache.pinot.core.routing.SegmentsToQuery;
import org.apache.pinot.core.routing.TableRouteInfo;
import org.apache.pinot.core.transport.ServerInstance;
import org.apache.pinot.core.transport.ServerRoutingInstance;
+import org.apache.pinot.materializedview.handler.MaterializedViewHandler;
import org.apache.pinot.spi.accounting.ThreadAccountant;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.env.PinotConfiguration;
@@ -63,14 +65,26 @@ public class GrpcBrokerRequestHandler extends BaseSingleStageBrokerRequestHandle
protected final PinotServerStreamingQueryClient _streamingQueryClient;
protected final FailureDetector _failureDetector;
- // TODO: Support TLS
+ /// Legacy constructor without an MV handler — see [BaseSingleStageBrokerRequestHandler]'s
+ /// legacy ctor for the rationale. Delegates with `materializedViewHandler = null`.
public GrpcBrokerRequestHandler(PinotConfiguration config, String brokerId,
BrokerRequestIdGenerator requestIdGenerator, RoutingManager routingManager,
AccessControlFactory accessControlFactory, QueryQuotaManager queryQuotaManager, TableCache tableCache,
FailureDetector failureDetector, ThreadAccountant threadAccountant,
MultiClusterRoutingContext multiClusterRoutingContext) {
+ this(config, brokerId, requestIdGenerator, routingManager, accessControlFactory, queryQuotaManager, tableCache,
+ failureDetector, threadAccountant, multiClusterRoutingContext, null);
+ }
+
+ /// TODO: Support TLS
+ public GrpcBrokerRequestHandler(PinotConfiguration config, String brokerId,
+ BrokerRequestIdGenerator requestIdGenerator, RoutingManager routingManager,
+ AccessControlFactory accessControlFactory, QueryQuotaManager queryQuotaManager, TableCache tableCache,
+ FailureDetector failureDetector, ThreadAccountant threadAccountant,
+ MultiClusterRoutingContext multiClusterRoutingContext,
+ @Nullable MaterializedViewHandler materializedViewHandler) {
super(config, brokerId, requestIdGenerator, routingManager, accessControlFactory, queryQuotaManager, tableCache,
- threadAccountant, multiClusterRoutingContext);
+ threadAccountant, multiClusterRoutingContext, materializedViewHandler);
_streamingReduceService = new StreamingReduceService(config);
_streamingQueryClient = new PinotServerStreamingQueryClient(GrpcConfig.buildGrpcQueryConfig(config));
_failureDetector = failureDetector;
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/SingleConnectionBrokerRequestHandler.java b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/SingleConnectionBrokerRequestHandler.java
index c7870a3bb59..6e2e21f5043 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/SingleConnectionBrokerRequestHandler.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/SingleConnectionBrokerRequestHandler.java
@@ -18,11 +18,14 @@
*/
package org.apache.pinot.broker.requesthandler;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.Maps;
import java.util.ArrayList;
+import java.util.IdentityHashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;
+import javax.annotation.Nullable;
import javax.annotation.concurrent.ThreadSafe;
import org.apache.pinot.broker.broker.AccessControlFactory;
import org.apache.pinot.broker.queryquota.QueryQuotaManager;
@@ -48,9 +51,11 @@ import org.apache.pinot.core.transport.ServerInstance;
import org.apache.pinot.core.transport.ServerResponse;
import org.apache.pinot.core.transport.ServerRoutingInstance;
import org.apache.pinot.core.transport.server.routing.stats.ServerRoutingStatsManager;
+import org.apache.pinot.materializedview.handler.MaterializedViewHandler;
import org.apache.pinot.spi.accounting.ThreadAccountant;
import org.apache.pinot.spi.env.PinotConfiguration;
import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.exception.QueryException;
import org.apache.pinot.spi.trace.RequestContext;
import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
@@ -70,14 +75,28 @@ public class SingleConnectionBrokerRequestHandler extends BaseSingleStageBrokerR
protected final QueryRouter _queryRouter;
protected final FailureDetector _failureDetector;
+ /// Legacy constructor without an MV handler — see [BaseSingleStageBrokerRequestHandler]'s
+ /// legacy ctor for the rationale. Delegates with `materializedViewHandler = null`.
public SingleConnectionBrokerRequestHandler(PinotConfiguration config, String brokerId,
BrokerRequestIdGenerator requestIdGenerator, RoutingManager routingManager,
AccessControlFactory accessControlFactory, QueryQuotaManager queryQuotaManager, TableCache tableCache,
NettyConfig nettyConfig, TlsConfig tlsConfig, ServerRoutingStatsManager serverRoutingStatsManager,
FailureDetector failureDetector, ThreadAccountant threadAccountant,
MultiClusterRoutingContext multiClusterRoutingContext) {
+ this(config, brokerId, requestIdGenerator, routingManager, accessControlFactory, queryQuotaManager, tableCache,
+ nettyConfig, tlsConfig, serverRoutingStatsManager, failureDetector, threadAccountant,
+ multiClusterRoutingContext, null);
+ }
+
+ public SingleConnectionBrokerRequestHandler(PinotConfiguration config, String brokerId,
+ BrokerRequestIdGenerator requestIdGenerator, RoutingManager routingManager,
+ AccessControlFactory accessControlFactory, QueryQuotaManager queryQuotaManager, TableCache tableCache,
+ NettyConfig nettyConfig, TlsConfig tlsConfig, ServerRoutingStatsManager serverRoutingStatsManager,
+ FailureDetector failureDetector, ThreadAccountant threadAccountant,
+ MultiClusterRoutingContext multiClusterRoutingContext,
+ @Nullable MaterializedViewHandler materializedViewHandler) {
super(config, brokerId, requestIdGenerator, routingManager, accessControlFactory, queryQuotaManager, tableCache,
- threadAccountant, multiClusterRoutingContext);
+ threadAccountant, multiClusterRoutingContext, materializedViewHandler);
_brokerReduceService = new BrokerReduceService(_config);
_queryRouter = new QueryRouter(_brokerId, nettyConfig, tlsConfig, serverRoutingStatsManager, threadAccountant);
_failureDetector = failureDetector;
@@ -198,6 +217,135 @@ public class SingleConnectionBrokerRequestHandler extends BaseSingleStageBrokerR
return brokerResponse;
}
+ @Override
+ protected BrokerResponseNative processMaterializedViewSplitBrokerRequest(long requestId,
+ long materializedViewRequestId, BrokerRequest originalBrokerRequest, TableRouteInfo baseRoute,
+ TableRouteInfo materializedViewRoute, long timeoutMs, ServerStats serverStats, RequestContext requestContext)
+ throws Exception {
+ String rawTableName =
+ TableNameBuilder.extractRawTableName(originalBrokerRequest.getQuerySource().getTableName());
+
+ /// Capture a single wall-clock deadline up front and derive every downstream timeout from it.
+ /// The split path submits two scatter-gathers AND a reduce; passing `timeoutMs` to each of
+ /// them would let two sub-queries individually consume the full budget, leaving the reduce
+ /// with a negative remaining and producing silently-truncated results.
+ long deadlineNs = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(timeoutMs);
+ long scatterGatherStartTimeNs = System.nanoTime();
+
+ /// Submit base-table and materialized-view queries in parallel through the `QueryRouter`.
+ /// Each route may fan out to multiple servers (especially if the base table is hybrid).
+ /// The MV sub-query uses its own request id so it cannot collide with the base sub-query on
+ /// servers that receive both requests.
+ long submitTimeoutMs = Math.max(1L, TimeUnit.NANOSECONDS.toMillis(deadlineNs - System.nanoTime()));
+ AsyncQueryResponse baseAsyncResponse =
+ _queryRouter.submitQuery(requestId, rawTableName, baseRoute, submitTimeoutMs);
+ AsyncQueryResponse materializedViewAsyncResponse =
+ _queryRouter.submitQuery(materializedViewRequestId, rawTableName, materializedViewRoute, submitTimeoutMs);
+
+ /// Collect responses from both queries.
+ Map<ServerRoutingInstance, ServerResponse> baseFinalResponses = baseAsyncResponse.getFinalResponses();
+ Map<ServerRoutingInstance, ServerResponse> viewFinalResponses = materializedViewAsyncResponse.getFinalResponses();
+
+ if (baseAsyncResponse.getStatus() == QueryResponse.Status.TIMED_OUT
+ || materializedViewAsyncResponse.getStatus() == QueryResponse.Status.TIMED_OUT) {
+ _brokerMetrics.addMeteredTableValue(rawTableName, BrokerMeter.BROKER_RESPONSES_WITH_TIMEOUTS, 1);
+ }
+
+ /// Mark failed servers as unhealthy.
+ ServerRoutingInstance baseFailedServer = baseAsyncResponse.getFailedServer();
+ if (baseFailedServer != null) {
+ _failureDetector.markServerUnhealthy(baseFailedServer.getInstanceId(), baseFailedServer.getHostname());
+ }
+ ServerRoutingInstance viewFailedServer = materializedViewAsyncResponse.getFailedServer();
+ if (viewFailedServer != null) {
+ _failureDetector.markServerUnhealthy(viewFailedServer.getInstanceId(), viewFailedServer.getHostname());
+ }
+
+ _brokerMetrics.addPhaseTiming(rawTableName, BrokerQueryPhase.SCATTER_GATHER,
+ System.nanoTime() - scatterGatherStartTimeNs);
+
+ /// Merge `DataTable`s from both base and MV responses into a single map using identity
+ /// equality so that `ServerRoutingInstance` objects from different sub-queries never collide.
+ /// `ServerRoutingInstance.equals()` keyed on (hostname, port, tableType) can produce the
+ /// same hash for base and MV rows on a shared server, causing silent overwrites with a
+ /// regular `HashMap`.
+ int totalServersQueried = baseFinalResponses.size() + viewFinalResponses.size();
+ List<ServerRoutingInstance> serversNotResponded = new ArrayList<>();
+ long[] totalResponseSizeHolder = {0L};
+ Map<ServerRoutingInstance, DataTable> dataTableMap =
+ mergeDataTablesByIdentity(baseFinalResponses, viewFinalResponses, serversNotResponded,
+ totalResponseSizeHolder);
+ long totalResponseSize = totalResponseSizeHolder[0];
+ int numServersResponded = dataTableMap.size();
+
+ /// On a SPLIT query, base and MV scatter-gathers cover DISJOINT halves of the timeline
+ /// (base covers `ts < boundary`, MV covers `ts >= boundary`). Partial failure on either
+ /// side therefore leaves whole time-ranges missing — semantically distinct from a non-split
+ /// hybrid query where partial failure scatters random rows. Refuse to return a
+ /// partially-missing-by-time-range result: throw here so the outer try/catch in
+ /// `BaseSingleStageBrokerRequestHandler` bumps `QUERY_REWRITE_EXCEPTIONS` and falls back to
+ /// the unsplit base-table query path, which covers the full timeline (and will report any
+ /// server failure through the standard non-split error-reporting path).
+ int viewSuccessful = countSuccessfulDataTables(viewFinalResponses);
+ if (!viewFinalResponses.isEmpty() && viewSuccessful < viewFinalResponses.size()) {
+ throw new QueryException(QueryErrorCode.SERVER_NOT_RESPONDING,
+ "Materialized view split: " + (viewFinalResponses.size() - viewSuccessful) + " of "
+ + viewFinalResponses.size() + " MV server(s) failed to return a DataTable; refusing to "
+ + "return a result with missing time ranges");
+ }
+ int baseSuccessful = countSuccessfulDataTables(baseFinalResponses);
+ if (!baseFinalResponses.isEmpty() && baseSuccessful < baseFinalResponses.size()) {
+ throw new QueryException(QueryErrorCode.SERVER_NOT_RESPONDING,
+ "Materialized view split: " + (baseFinalResponses.size() - baseSuccessful) + " of "
+ + baseFinalResponses.size() + " base-table server(s) failed to return a DataTable; refusing "
+ + "to return a result with missing time ranges");
+ }
+
+ /// Reduce using the original user query so that the correct reducer (selection,
+ /// aggregation, group-by) is selected and intermediate results are merged properly.
+ long reduceStartTimeNs = System.nanoTime();
+ long reduceTimeoutMs = TimeUnit.NANOSECONDS.toMillis(deadlineNs - reduceStartTimeNs);
+ if (reduceTimeoutMs <= 0) {
+ throw new QueryException(QueryErrorCode.BROKER_TIMEOUT,
+ "Broker timeout exceeded after MV split scatter-gather; no time remaining for reduce");
+ }
+ BrokerResponseNative brokerResponse =
+ _brokerReduceService.reduceOnDataTable(originalBrokerRequest, originalBrokerRequest, dataTableMap,
+ reduceTimeoutMs, _brokerMetrics);
+ long reduceTimeNanos = System.nanoTime() - reduceStartTimeNs;
+ _brokerMetrics.addPhaseTiming(rawTableName, BrokerQueryPhase.REDUCE, reduceTimeNanos);
+
+ brokerResponse.setNumServersQueried(totalServersQueried);
+ brokerResponse.setNumServersResponded(numServersResponded);
+ brokerResponse.setBrokerReduceTimeMs(TimeUnit.NANOSECONDS.toMillis(reduceTimeNanos));
+
+ /// Propagate send exceptions from both queries.
+ Exception baseSendException = baseAsyncResponse.getException();
+ if (baseSendException != null) {
+ brokerResponse.addException(
+ new QueryProcessingException(QueryErrorCode.BROKER_REQUEST_SEND, baseSendException.getMessage()));
+ }
+ Exception materializedViewSendException = materializedViewAsyncResponse.getException();
+ if (materializedViewSendException != null) {
+ brokerResponse.addException(
+ new QueryProcessingException(QueryErrorCode.BROKER_REQUEST_SEND, materializedViewSendException.getMessage()));
+ }
+
+ int numServersNotResponded = serversNotResponded.size();
+ if (numServersNotResponded != 0) {
+ brokerResponse.addException(new QueryProcessingException(QueryErrorCode.SERVER_NOT_RESPONDING,
+ String.format("%d servers %s not responded", numServersNotResponded, serversNotResponded)));
+ _brokerMetrics.addMeteredTableValue(rawTableName,
+ BrokerMeter.BROKER_RESPONSES_WITH_PARTIAL_SERVERS_RESPONDED, 1);
+ }
+ if (brokerResponse.getExceptionsSize() > 0) {
+ _brokerMetrics.addMeteredTableValue(rawTableName, BrokerMeter.BROKER_RESPONSES_WITH_PROCESSING_EXCEPTIONS, 1);
+ }
+ _brokerMetrics.addMeteredTableValue(rawTableName, BrokerMeter.TOTAL_SERVER_RESPONSE_SIZE, totalResponseSize);
+
+ return brokerResponse;
+ }
+
/**
* Snapshot of server-side scatter statistics. Passed to {@link ScatterResult} so that server
* counts are always derived from the live scatter, not from a data table map that may have been
@@ -305,4 +453,59 @@ public class SingleConnectionBrokerRequestHandler extends BaseSingleStageBrokerR
return FailureDetector.ServerState.UNHEALTHY;
}
}
+
+ /// Counts responses that successfully returned a DataTable. Production callers use this in the
+ /// MV-split path to detect "all of one side's servers failed" — if `viewFinalResponses` is
+ /// non-empty AND the count is zero, the split would silently undercount the historical half
+ /// (and symmetrically for the base side), so the caller throws to trigger the outer fallback.
+ /// Package-private for direct unit-test coverage of the guard's boolean.
+ @VisibleForTesting
+ static int countSuccessfulDataTables(Map<ServerRoutingInstance, ServerResponse> responses) {
+ int count = 0;
+ for (ServerResponse r : responses.values()) {
+ if (r.getDataTable() != null) {
+ count++;
+ }
+ }
+ return count;
+ }
+
+ /// Merges base and MV server responses into one IdentityHashMap whose entries are accumulated
+ /// by reference equality rather than by `ServerRoutingInstance.equals()`. The MV split path
+ /// can route both sub-queries to the same physical server (same hostname+port+tableType, so
+ /// `equals()` matches) but with distinct `ServerRoutingInstance` instances — a regular HashMap
+ /// would silently overwrite one DataTable with the other and produce under-counted results.
+ ///
+ /// Package-private so the test suite can pin this contract without spinning up a broker.
+ @VisibleForTesting
+ static Map<ServerRoutingInstance, DataTable> mergeDataTablesByIdentity(
+ Map<ServerRoutingInstance, ServerResponse> baseResponses,
+ Map<ServerRoutingInstance, ServerResponse> viewResponses,
+ List<ServerRoutingInstance> serversNotResponded, long[] totalResponseSizeHolder) {
+ int totalServers = baseResponses.size() + viewResponses.size();
+ Map<ServerRoutingInstance, DataTable> dataTableMap = new IdentityHashMap<>(totalServers);
+ long totalResponseSize = 0;
+ for (Map.Entry<ServerRoutingInstance, ServerResponse> entry : baseResponses.entrySet()) {
+ ServerResponse serverResponse = entry.getValue();
+ DataTable dataTable = serverResponse.getDataTable();
+ if (dataTable != null) {
+ dataTableMap.put(entry.getKey(), dataTable);
+ totalResponseSize += serverResponse.getResponseSize();
+ } else {
+ serversNotResponded.add(entry.getKey());
+ }
+ }
+ for (Map.Entry<ServerRoutingInstance, ServerResponse> entry : viewResponses.entrySet()) {
+ ServerResponse serverResponse = entry.getValue();
+ DataTable dataTable = serverResponse.getDataTable();
+ if (dataTable != null) {
+ dataTableMap.put(entry.getKey(), dataTable);
+ totalResponseSize += serverResponse.getResponseSize();
+ } else {
+ serversNotResponded.add(entry.getKey());
+ }
+ }
+ totalResponseSizeHolder[0] = totalResponseSize;
+ return dataTableMap;
+ }
}
diff --git a/pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandlerTest.java b/pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandlerTest.java
index aa31a46f82e..a791dd0f1a6 100644
--- a/pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandlerTest.java
+++ b/pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/BaseSingleStageBrokerRequestHandlerTest.java
@@ -26,6 +26,8 @@ import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicReference;
import org.apache.helix.model.InstanceConfig;
+import org.apache.pinot.broker.api.AccessControl;
+import org.apache.pinot.broker.broker.AccessControlFactory;
import org.apache.pinot.broker.broker.AllowAllAccessControlFactory;
import org.apache.pinot.broker.queryquota.QueryQuotaManager;
import org.apache.pinot.broker.routing.manager.BrokerRoutingManager;
@@ -42,7 +44,18 @@ import org.apache.pinot.core.routing.SegmentsToQuery;
import org.apache.pinot.core.routing.TableRouteInfo;
import org.apache.pinot.core.routing.timeboundary.TimeBoundaryInfo;
import org.apache.pinot.core.transport.ServerInstance;
+import org.apache.pinot.materializedview.context.MaterializedViewContext;
+import org.apache.pinot.materializedview.handler.DefaultMaterializedViewHandler;
+import org.apache.pinot.materializedview.handler.MaterializedViewCompileContext;
+import org.apache.pinot.materializedview.handler.MaterializedViewHandler;
+import org.apache.pinot.materializedview.handler.MaterializedViewSplitExecutionContext;
+import org.apache.pinot.materializedview.rewrite.ExecutionMode;
+import org.apache.pinot.materializedview.rewrite.MatchType;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewQueryRewriteEngine;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewRewritePlan;
import org.apache.pinot.spi.accounting.ThreadAccountantUtils;
+import org.apache.pinot.spi.auth.TableAuthorizationResult;
+import org.apache.pinot.spi.auth.TableRowColAccessResultImpl;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TenantConfig;
import org.apache.pinot.spi.data.FieldSpec.DataType;
@@ -53,7 +66,9 @@ import org.apache.pinot.spi.exception.BadQueryRequestException;
import org.apache.pinot.spi.trace.LoggerConstants;
import org.apache.pinot.spi.trace.RequestContext;
import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants.Broker;
import org.apache.pinot.spi.utils.CommonConstants.Query.Range;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.apache.pinot.sql.FilterKind;
import org.apache.pinot.sql.parsers.CalciteSqlParser;
import org.apache.pinot.util.TestUtils;
@@ -114,7 +129,7 @@ public class BaseSingleStageBrokerRequestHandlerTest {
BaseSingleStageBrokerRequestHandler handler =
new BaseSingleStageBrokerRequestHandler(config, "testBrokerId", new BrokerRequestIdGenerator(),
mock(org.apache.pinot.core.routing.RoutingManager.class), new AllowAllAccessControlFactory(),
- queryQuotaManager, tableCache, ThreadAccountantUtils.getNoOpAccountant(), null) {
+ queryQuotaManager, tableCache, ThreadAccountantUtils.getNoOpAccountant(), null, null) {
@Override
public void start() {
}
@@ -130,6 +145,14 @@ public class BaseSingleStageBrokerRequestHandlerTest {
return new BrokerResponseNative();
}
+ @Override
+ protected BrokerResponseNative processMaterializedViewSplitBrokerRequest(long requestId,
+ long materializedViewRequestId, BrokerRequest originalBrokerRequest, TableRouteInfo baseRoute,
+ TableRouteInfo materializedViewRoute, long timeoutMs, ServerStats serverStats,
+ RequestContext requestContext) {
+ return new BrokerResponseNative();
+ }
+
@Override
protected void onQueryCompletion(RequestContext requestContext, BrokerResponse brokerResponse) {
capturedResponse.set(brokerResponse);
@@ -145,6 +168,40 @@ public class BaseSingleStageBrokerRequestHandlerTest {
"onQueryCompletion hook must be called with the BrokerResponse from handleRequest");
}
+ /// Pins the legacy (pre-MV) constructor signature so out-of-tree subclasses compiled against
+ /// it continue to link. Invokes the 9-arg constructor (no trailing `MaterializedViewHandler`
+ /// parameter) and asserts it instantiates cleanly — and that
+ /// `processMaterializedViewSplitBrokerRequest` is a concrete method on the base class (would
+ /// fail to compile this anonymous subclass if it were abstract).
+ @Test
+ public void testLegacyConstructorIsAvailableForCustomSubclasses() {
+ PinotConfiguration config = new PinotConfiguration();
+ QueryQuotaManager queryQuotaManager = mock(QueryQuotaManager.class);
+ TableCache tableCache = mock(TableCache.class);
+
+ BaseSingleStageBrokerRequestHandler handler =
+ new BaseSingleStageBrokerRequestHandler(config, "testBrokerId", new BrokerRequestIdGenerator(),
+ mock(org.apache.pinot.core.routing.RoutingManager.class), new AllowAllAccessControlFactory(),
+ queryQuotaManager, tableCache, ThreadAccountantUtils.getNoOpAccountant(), null) {
+ @Override
+ public void start() {
+ }
+
+ @Override
+ public void shutDown() {
+ }
+
+ @Override
+ protected BrokerResponseNative processBrokerRequest(long requestId, BrokerRequest originalBrokerRequest,
+ BrokerRequest serverBrokerRequest, TableRouteInfo route, long timeoutMs, ServerStats serverStats,
+ RequestContext requestContext) {
+ return new BrokerResponseNative();
+ }
+ };
+
+ Assert.assertNotNull(handler);
+ }
+
@Test
public void testGetActualColumnNameCaseSensitive() {
Map<String, String> columnNameMap = new HashMap<>();
@@ -248,7 +305,7 @@ public class BaseSingleStageBrokerRequestHandlerTest {
BaseSingleStageBrokerRequestHandler requestHandler =
new BaseSingleStageBrokerRequestHandler(config, "testBrokerId", new BrokerRequestIdGenerator(), routingManager,
new AllowAllAccessControlFactory(), queryQuotaManager, tableCache,
- ThreadAccountantUtils.getNoOpAccountant(), null) {
+ ThreadAccountantUtils.getNoOpAccountant(), null, null) {
@Override
public void start() {
}
@@ -266,6 +323,15 @@ public class BaseSingleStageBrokerRequestHandlerTest {
latch.await();
return null;
}
+
+ @Override
+ protected BrokerResponseNative processMaterializedViewSplitBrokerRequest(long requestId,
+ long materializedViewRequestId, BrokerRequest originalBrokerRequest, TableRouteInfo baseRoute,
+ TableRouteInfo materializedViewRoute, long timeoutMs, ServerStats serverStats,
+ RequestContext requestContext)
+ throws Exception {
+ throw new UnsupportedOperationException("Not implemented in test");
+ }
};
CompletableFuture.runAsync(() -> {
try {
@@ -378,7 +444,7 @@ public class BaseSingleStageBrokerRequestHandlerTest {
return new BaseSingleStageBrokerRequestHandler(config, "testBrokerId", new BrokerRequestIdGenerator(),
routingManager, new AllowAllAccessControlFactory(), queryQuotaManager, tableCache,
- ThreadAccountantUtils.getNoOpAccountant(), null) {
+ ThreadAccountantUtils.getNoOpAccountant(), null, null) {
@Override
public void start() {
}
@@ -394,6 +460,14 @@ public class BaseSingleStageBrokerRequestHandlerTest {
capturedRouteInfo.set(route);
return BrokerResponseNative.empty();
}
+
+ @Override
+ protected BrokerResponseNative processMaterializedViewSplitBrokerRequest(long requestId,
+ long materializedViewRequestId, BrokerRequest originalBrokerRequest, TableRouteInfo baseRoute,
+ TableRouteInfo materializedViewRoute, long timeoutMs, ServerStats serverStats, RequestContext requestContext)
+ throws Exception {
+ throw new UnsupportedOperationException("Not implemented in test");
+ }
};
}
@@ -498,4 +572,848 @@ public class BaseSingleStageBrokerRequestHandlerTest {
assertRangeFilter(routeInfo.getRealtimeBrokerRequest(), "created_15min",
"(1772109900000" + Range.DELIMITER + "1772113500000]", "Realtime mixed");
}
+
+ /**
+ * Bug: FULL_REWRITE overwrites tableName to the materialized view table name, so
+ * _queryQuotaManager.acquire(tableName) charges quota against the MV
+ * instead of the base table. A throttled base table is effectively
+ * bypassed when its quota allows no traffic but the MV has no quota entry.
+ *
+ * <p>Before fix: acquire("baseTable_OFFLINE") is never called; the MV
+ * table passes because the mock only denies the base table name.
+ * <p>After fix: the base table name is used for quota accounting and the
+ * request is correctly rate-limited.
+ */
+ @Test
+ public void testMaterializedViewFullRewriteQuotaAccountedAgainstBaseTable()
+ throws Exception {
+ String baseOfflineTable = "baseTable_OFFLINE";
+ String materializedViewOfflineTable = "mv_baseTable_OFFLINE";
+ String baseRawTable = "baseTable";
+ String materializedViewRawTable = "mv_baseTable";
+
+ /// materialized view query: exact same columns but issued against the MV
+ String userSql = "SELECT ts, SUM(revenue) FROM baseTable GROUP BY ts LIMIT 100 "
+ + "";
+ PinotQuery materializedViewQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT ts, SUM(revenue) FROM mv_baseTable_OFFLINE GROUP BY ts LIMIT 100");
+
+ MaterializedViewRewritePlan plan = new MaterializedViewRewritePlan(
+ materializedViewOfflineTable, MatchType.EXACT, ExecutionMode.FULL_REWRITE, materializedViewQuery, 1.0);
+
+ AtomicReference<PinotQuery> querySeenByMaterializedViewRewrite = new AtomicReference<>();
+ MaterializedViewQueryRewriteEngine materializedViewEngine = mock(MaterializedViewQueryRewriteEngine.class);
+ when(materializedViewEngine.tryRewrite(any(PinotQuery.class), anyString())).thenAnswer(invocation -> {
+ querySeenByMaterializedViewRewrite.set(((PinotQuery) invocation.getArgument(0)).deepCopy());
+ return plan;
+ });
+ MaterializedViewHandler materializedViewHandler = new DefaultMaterializedViewHandler(materializedViewEngine);
+
+ Schema baseSchema = new Schema.SchemaBuilder()
+ .setSchemaName(baseRawTable)
+ .addSingleValueDimension("ts", DataType.STRING)
+ .addMetric("revenue", DataType.DOUBLE)
+ .build();
+ Schema materializedViewSchema = new Schema.SchemaBuilder()
+ .setSchemaName(materializedViewRawTable)
+ .addSingleValueDimension("ts", DataType.STRING)
+ .addMetric("revenue", DataType.DOUBLE)
+ .build();
+
+ TableCache tableCache = mock(TableCache.class);
+ when(tableCache.getActualTableName(baseRawTable)).thenReturn(baseRawTable);
+ when(tableCache.getSchema(baseRawTable)).thenReturn(baseSchema);
+ when(tableCache.getSchema(materializedViewRawTable)).thenReturn(materializedViewSchema);
+ when(tableCache.getColumnNameMap(anyString())).thenReturn(Map.of("ts", "ts", "revenue", "revenue"));
+ TableConfig tableCfg = mock(TableConfig.class);
+ TenantConfig tenant = new TenantConfig("t_BROKER", "t_SERVER", null);
+ when(tableCfg.getTenantConfig()).thenReturn(tenant);
+ when(tableCache.getTableConfig(baseOfflineTable)).thenReturn(tableCfg);
+ when(tableCache.getTableConfig(materializedViewOfflineTable)).thenReturn(tableCfg);
+
+ BrokerRoutingManager routingManager = mock(BrokerRoutingManager.class);
+ when(routingManager.routingExists(baseOfflineTable)).thenReturn(true);
+ when(routingManager.routingExists(materializedViewOfflineTable)).thenReturn(true);
+ when(routingManager.getQueryTimeoutMs(anyString())).thenReturn(10000L);
+ RoutingTable rt = mock(RoutingTable.class);
+ when(rt.getServerInstanceToSegmentsMap()).thenReturn(
+ Map.of(new ServerInstance(new InstanceConfig("server01_9000")),
+ new SegmentsToQuery(List.of("seg01"), List.of())));
+ when(routingManager.getRoutingTable(any(), Mockito.anyLong())).thenReturn(rt);
+
+ /// Only deny the base table; the MV has no quota entry (returns true).
+ QueryQuotaManager quotaManager = mock(QueryQuotaManager.class);
+ when(quotaManager.acquireDatabase(anyString())).thenReturn(true);
+ when(quotaManager.acquireApplication(anyString())).thenReturn(true);
+ /// Base table is over quota; MV is not throttled.
+ when(quotaManager.acquire(baseOfflineTable)).thenReturn(false);
+ when(quotaManager.acquire(materializedViewOfflineTable)).thenReturn(true);
+
+ BrokerMetrics.register(mock(BrokerMetrics.class));
+ PinotConfiguration config = new PinotConfiguration();
+ BrokerQueryEventListenerFactory.init(config);
+
+ BaseSingleStageBrokerRequestHandler handler =
+ new BaseSingleStageBrokerRequestHandler(config, "broker1", new BrokerRequestIdGenerator(),
+ routingManager, new AllowAllAccessControlFactory(), quotaManager, tableCache,
+ ThreadAccountantUtils.getNoOpAccountant(), null, materializedViewHandler) {
+ @Override
+ public void start() {
+ }
+
+ @Override
+ public void shutDown() {
+ }
+
+ @Override
+ protected BrokerResponseNative processBrokerRequest(long requestId,
+ BrokerRequest originalBrokerRequest, BrokerRequest serverBrokerRequest,
+ TableRouteInfo route, long timeoutMs, ServerStats serverStats,
+ RequestContext requestContext) {
+ /// Should not reach here — quota must reject before routing
+ Assert.fail("processBrokerRequest should not be called when base table is over quota");
+ return null;
+ }
+
+ @Override
+ protected BrokerResponseNative processMaterializedViewSplitBrokerRequest(long requestId,
+ long materializedViewRequestId, BrokerRequest originalBrokerRequest, TableRouteInfo baseRoute,
+ TableRouteInfo materializedViewRoute, long timeoutMs, ServerStats serverStats,
+ RequestContext requestContext) {
+ Assert.fail("processMaterializedViewSplitBrokerRequest should not be called when base table is over quota");
+ return null;
+ }
+ };
+
+ BrokerResponseNative response = (BrokerResponseNative) handler.handleRequest(userSql);
+ Assert.assertNotNull(response);
+ /// The request must be rejected with TOO_MANY_REQUESTS because the base table is over quota
+ Assert.assertEquals(response.getExceptionsSize(), 1,
+ "Expected quota rejection exception but got: " + response.getExceptions());
+ Assert.assertEquals(response.getExceptions().get(0).getErrorCode(),
+ org.apache.pinot.spi.exception.QueryErrorCode.TOO_MANY_REQUESTS.getId(),
+ "Expected TOO_MANY_REQUESTS error code");
+ }
+
+ /**
+ * Bug: FULL_REWRITE overwrites tableName to the materialized view table name, so
+ * accessControl.getRowColFilters(requesterIdentity, tableName) fetches RLS
+ * policy for the materialized view table instead of the base table.
+ *
+ * <p>Before fix: getRowColFilters is called with the materialized view table name
+ * "mv_baseTable_OFFLINE".
+ * <p>After fix: getRowColFilters is called with the original base table
+ * name "baseTable_OFFLINE".
+ */
+ @Test
+ public void testMaterializedViewFullRewriteRlsLookupUsesBaseTable()
+ throws Exception {
+ String baseOfflineTable = "baseTable_OFFLINE";
+ String materializedViewOfflineTable = "mv_baseTable_OFFLINE";
+ String baseRawTable = "baseTable";
+ String materializedViewRawTable = "mv_baseTable";
+
+ String userSql = "SELECT ts, SUM(revenue) FROM baseTable GROUP BY ts LIMIT 100 "
+ + "";
+ PinotQuery materializedViewQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT ts, SUM(revenue) FROM mv_baseTable_OFFLINE GROUP BY ts LIMIT 100");
+
+ MaterializedViewRewritePlan plan = new MaterializedViewRewritePlan(
+ materializedViewOfflineTable, MatchType.EXACT, ExecutionMode.FULL_REWRITE, materializedViewQuery, 1.0);
+
+ AtomicReference<PinotQuery> querySeenByMaterializedViewRewrite = new AtomicReference<>();
+ MaterializedViewQueryRewriteEngine materializedViewEngine = mock(MaterializedViewQueryRewriteEngine.class);
+ when(materializedViewEngine.tryRewrite(any(PinotQuery.class), anyString())).thenAnswer(invocation -> {
+ querySeenByMaterializedViewRewrite.set(((PinotQuery) invocation.getArgument(0)).deepCopy());
+ return plan;
+ });
+ MaterializedViewHandler materializedViewHandler = new DefaultMaterializedViewHandler(materializedViewEngine);
+
+ Schema baseSchema = new Schema.SchemaBuilder()
+ .setSchemaName(baseRawTable)
+ .addSingleValueDimension("ts", DataType.STRING)
+ .addMetric("revenue", DataType.DOUBLE)
+ .build();
+ Schema materializedViewSchema = new Schema.SchemaBuilder()
+ .setSchemaName(materializedViewRawTable)
+ .addSingleValueDimension("ts", DataType.STRING)
+ .addMetric("revenue", DataType.DOUBLE)
+ .build();
+
+ TableCache tableCache = mock(TableCache.class);
+ when(tableCache.getActualTableName(baseRawTable)).thenReturn(baseRawTable);
+ when(tableCache.getSchema(baseRawTable)).thenReturn(baseSchema);
+ when(tableCache.getSchema(materializedViewRawTable)).thenReturn(materializedViewSchema);
+ when(tableCache.getColumnNameMap(anyString())).thenReturn(Map.of("ts", "ts", "revenue", "revenue"));
+ TableConfig tableCfg = mock(TableConfig.class);
+ TenantConfig tenant = new TenantConfig("t_BROKER", "t_SERVER", null);
+ when(tableCfg.getTenantConfig()).thenReturn(tenant);
+ when(tableCache.getTableConfig(baseOfflineTable)).thenReturn(tableCfg);
+ when(tableCache.getTableConfig(materializedViewOfflineTable)).thenReturn(tableCfg);
+
+ BrokerRoutingManager routingManager = mock(BrokerRoutingManager.class);
+ when(routingManager.routingExists(baseOfflineTable)).thenReturn(true);
+ when(routingManager.routingExists(materializedViewOfflineTable)).thenReturn(true);
+ when(routingManager.getQueryTimeoutMs(anyString())).thenReturn(10000L);
+ RoutingTable rt = mock(RoutingTable.class);
+ when(rt.getServerInstanceToSegmentsMap()).thenReturn(
+ Map.of(new ServerInstance(new InstanceConfig("server01_9000")),
+ new SegmentsToQuery(List.of("seg01"), List.of())));
+ when(routingManager.getRoutingTable(any(), Mockito.anyLong())).thenReturn(rt);
+
+ QueryQuotaManager quotaManager = mock(QueryQuotaManager.class);
+ when(quotaManager.acquire(anyString())).thenReturn(true);
+ when(quotaManager.acquireDatabase(anyString())).thenReturn(true);
+ when(quotaManager.acquireApplication(anyString())).thenReturn(true);
+
+ /// Use a concrete AccessControl that allows everything but records the table passed to getRowColFilters.
+ /// A Mockito mock of an interface cannot reliably stub default interface methods, so we use an
+ /// anonymous implementation to avoid NPEs from un-stubbed default-method paths.
+ List<String> capturedRlsTables = new java.util.ArrayList<>();
+ AccessControl accessControl = new AccessControl() {
+ @Override
+ public org.apache.pinot.spi.auth.AuthorizationResult authorize(
+ org.apache.pinot.spi.auth.broker.RequesterIdentity identity, BrokerRequest request) {
+ return TableAuthorizationResult.success();
+ }
+
+ @Override
+ public TableAuthorizationResult authorize(
+ org.apache.pinot.spi.auth.broker.RequesterIdentity identity, Set<String> tables) {
+ return TableAuthorizationResult.success();
+ }
+
+ @Override
+ public org.apache.pinot.spi.auth.TableRowColAccessResult getRowColFilters(
+ org.apache.pinot.spi.auth.broker.RequesterIdentity identity, String tableWithType) {
+ capturedRlsTables.add(tableWithType);
+ return new TableRowColAccessResultImpl(List.of("ts = 'allowed'"));
+ }
+ };
+
+ AccessControlFactory accessControlFactory = mock(AccessControlFactory.class);
+ when(accessControlFactory.create()).thenReturn(accessControl);
+
+ BrokerMetrics.register(mock(BrokerMetrics.class));
+ /// Enable row/column-level auth so the RLS path is exercised
+ PinotConfiguration config = new PinotConfiguration(
+ Map.of(Broker.CONFIG_OF_BROKER_ENABLE_ROW_COLUMN_LEVEL_AUTH, "true"));
+ BrokerQueryEventListenerFactory.init(config);
+
+ BaseSingleStageBrokerRequestHandler handler =
+ new BaseSingleStageBrokerRequestHandler(config, "broker1", new BrokerRequestIdGenerator(),
+ routingManager, accessControlFactory, quotaManager, tableCache,
+ ThreadAccountantUtils.getNoOpAccountant(), null, materializedViewHandler) {
+ @Override
+ public void start() {
+ }
+
+ @Override
+ public void shutDown() {
+ }
+
+ @Override
+ protected BrokerResponseNative processBrokerRequest(long requestId,
+ BrokerRequest originalBrokerRequest, BrokerRequest serverBrokerRequest,
+ TableRouteInfo route, long timeoutMs, ServerStats serverStats,
+ RequestContext requestContext) {
+ return BrokerResponseNative.empty();
+ }
+
+ @Override
+ protected BrokerResponseNative processMaterializedViewSplitBrokerRequest(long requestId,
+ long materializedViewRequestId, BrokerRequest originalBrokerRequest, TableRouteInfo baseRoute,
+ TableRouteInfo materializedViewRoute, long timeoutMs, ServerStats serverStats,
+ RequestContext requestContext) {
+ return BrokerResponseNative.empty();
+ }
+ };
+
+ BrokerResponseNative response = (BrokerResponseNative) handler.handleRequest(userSql);
+
+ /// getRowColFilters must have been called with the base table name, not the materialized view table
+ Assert.assertFalse(capturedRlsTables.isEmpty(),
+ "getRowColFilters should have been called");
+ String rlsTable = capturedRlsTables.get(0);
+ Assert.assertNotEquals(rlsTable, materializedViewOfflineTable,
+ "RLS filter lookup must NOT use materialized view table name but got: " + rlsTable);
+ /// The RLS lookup must use the original base table identity, not the materialized view table.
+ /// The table name stored in preRewriteTableName is the raw name from compileSingleStageBrokerRequest
+ /// (before type resolution), so we assert on baseRawTable, not baseOfflineTable.
+ Assert.assertEquals(rlsTable, baseRawTable,
+ "RLS filter lookup must use base table '" + baseRawTable
+ + "' not materialized view table, but got: " + rlsTable);
+ Assert.assertTrue(response.getRLSFiltersApplied(), "response should report that RLS filters were applied");
+ PinotQuery rewriteInput = querySeenByMaterializedViewRewrite.get();
+ Assert.assertNotNull(rewriteInput, "MV rewrite should see the server query after RLS rewrite");
+ Assert.assertNotNull(rewriteInput.getFilterExpression(), "MV rewrite input must carry the base-table RLS filter");
+ Assert.assertTrue(rewriteInput.getFilterExpression().toString().contains("allowed"),
+ "MV rewrite input must include the RLS predicate, got: " + rewriteInput.getFilterExpression());
+ }
+
+ /// The split-mode time-boundary filter attach helpers moved to
+ /// org.apache.pinot.materializedview.handler.DefaultMaterializedViewHandler#attachFilter; their
+ /// regression coverage lives in DefaultMaterializedViewHandlerTest in pinot-materialized-view.
+
+ /**
+ * Pins the security-style defense that a user-supplied `materializedViewRewrite=true` query
+ * option (e.g. via `SET materializedViewRewrite='true'`) is stripped at the broker entry
+ * before any compile work. Without the strip, a hostile client could stamp the
+ * broker-internal marker themselves and bypass `BrokerReduceService`'s "Nested query is not
+ * supported without gapfill" safety net on any path where `brokerRequest != serverBrokerRequest`.
+ */
+ @Test
+ public void testMaterializedViewMarkerStrippedFromUserSuppliedOptions()
+ throws Exception {
+ String baseOfflineTable = "baseTable_OFFLINE";
+ String baseRawTable = "baseTable";
+
+ /// User attempts to set the internal MV-rewrite marker via the SQL SET-options syntax.
+ /// The strip in handleRequest must remove it before the marker can reach the server query.
+ String userSql = "SET materializedViewRewrite='true';"
+ + "SELECT ts, SUM(revenue) FROM baseTable GROUP BY ts LIMIT 100";
+
+ Schema baseSchema = new Schema.SchemaBuilder()
+ .setSchemaName(baseRawTable)
+ .addSingleValueDimension("ts", DataType.STRING)
+ .addMetric("revenue", DataType.DOUBLE)
+ .build();
+
+ TableCache tableCache = mock(TableCache.class);
+ when(tableCache.getActualTableName(baseRawTable)).thenReturn(baseRawTable);
+ when(tableCache.getSchema(baseRawTable)).thenReturn(baseSchema);
+ when(tableCache.getColumnNameMap(anyString())).thenReturn(Map.of("ts", "ts", "revenue", "revenue"));
+ TableConfig tableCfg = mock(TableConfig.class);
+ TenantConfig tenant = new TenantConfig("t_BROKER", "t_SERVER", null);
+ when(tableCfg.getTenantConfig()).thenReturn(tenant);
+ when(tableCache.getTableConfig(baseOfflineTable)).thenReturn(tableCfg);
+
+ BrokerRoutingManager routingManager = mock(BrokerRoutingManager.class);
+ when(routingManager.routingExists(baseOfflineTable)).thenReturn(true);
+ when(routingManager.getQueryTimeoutMs(anyString())).thenReturn(10000L);
+ RoutingTable rt = mock(RoutingTable.class);
+ when(rt.getServerInstanceToSegmentsMap()).thenReturn(
+ Map.of(new ServerInstance(new InstanceConfig("server01_9000")),
+ new SegmentsToQuery(List.of("seg01"), List.of())));
+ when(routingManager.getRoutingTable(any(), Mockito.anyLong())).thenReturn(rt);
+
+ QueryQuotaManager quotaManager = mock(QueryQuotaManager.class);
+ when(quotaManager.acquireDatabase(anyString())).thenReturn(true);
+ when(quotaManager.acquireApplication(anyString())).thenReturn(true);
+ when(quotaManager.acquire(anyString())).thenReturn(true);
+
+ BrokerMetrics.register(mock(BrokerMetrics.class));
+ PinotConfiguration config = new PinotConfiguration();
+ BrokerQueryEventListenerFactory.init(config);
+
+ AtomicReference<Map<String, String>> capturedServerOptions = new AtomicReference<>();
+ BaseSingleStageBrokerRequestHandler handler =
+ new BaseSingleStageBrokerRequestHandler(config, "broker1", new BrokerRequestIdGenerator(),
+ routingManager, new AllowAllAccessControlFactory(), quotaManager, tableCache,
+ ThreadAccountantUtils.getNoOpAccountant(), null, /*materializedViewHandler*/ null) {
+ @Override
+ public void start() {
+ }
+
+ @Override
+ public void shutDown() {
+ }
+
+ @Override
+ protected BrokerResponseNative processBrokerRequest(long requestId,
+ BrokerRequest originalBrokerRequest, BrokerRequest serverBrokerRequest,
+ TableRouteInfo route, long timeoutMs, ServerStats serverStats,
+ RequestContext requestContext) {
+ Map<String, String> options = serverBrokerRequest.getPinotQuery().getQueryOptions();
+ capturedServerOptions.set(options == null ? Map.of() : Map.copyOf(options));
+ return BrokerResponseNative.empty();
+ }
+
+ @Override
+ protected BrokerResponseNative processMaterializedViewSplitBrokerRequest(long requestId,
+ long materializedViewRequestId, BrokerRequest originalBrokerRequest, TableRouteInfo baseRoute,
+ TableRouteInfo materializedViewRoute, long timeoutMs, ServerStats serverStats,
+ RequestContext requestContext) {
+ return BrokerResponseNative.empty();
+ }
+ };
+
+ handler.handleRequest(userSql);
+ Map<String, String> serverOptions = capturedServerOptions.get();
+ Assert.assertNotNull(serverOptions, "processBrokerRequest should have been reached and captured server options");
+ Assert.assertFalse(serverOptions.containsKey(
+ CommonConstants.Broker.Request.QueryOptionKey.MATERIALIZED_VIEW_REWRITE),
+ "User-supplied materializedViewRewrite option must be stripped before compile but options were: "
+ + serverOptions);
+ }
+
+ /**
+ * Pins the C1 fix: FULL_REWRITE is skipped at the broker layer when the base table has a
+ * REALTIME sibling, because a batch MV cannot cover newly-streamed rows. Without this guard
+ * the MV swap would silently drop all rows ingested via the realtime stream since the MV last
+ * refreshed — an invisible data-loss path.
+ */
+ @Test
+ public void testMaterializedViewFullRewriteSkippedForHybridBaseTable()
+ throws Exception {
+ String baseOfflineTable = "baseTable_OFFLINE";
+ String baseRealtimeTable = "baseTable_REALTIME";
+ String materializedViewOfflineTable = "mv_baseTable_OFFLINE";
+ String baseRawTable = "baseTable";
+ String materializedViewRawTable = "mv_baseTable";
+
+ String userSql = "SELECT ts, SUM(revenue) FROM baseTable GROUP BY ts LIMIT 100";
+ PinotQuery materializedViewQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT ts, SUM(revenue) FROM mv_baseTable_OFFLINE GROUP BY ts LIMIT 100");
+
+ MaterializedViewRewritePlan plan = new MaterializedViewRewritePlan(
+ materializedViewOfflineTable, MatchType.EXACT, ExecutionMode.FULL_REWRITE, materializedViewQuery, 1.0);
+
+ MaterializedViewQueryRewriteEngine materializedViewEngine = mock(MaterializedViewQueryRewriteEngine.class);
+ when(materializedViewEngine.tryRewrite(any(PinotQuery.class), anyString())).thenReturn(plan);
+ MaterializedViewHandler materializedViewHandler = new DefaultMaterializedViewHandler(materializedViewEngine);
+
+ Schema baseSchema = new Schema.SchemaBuilder()
+ .setSchemaName(baseRawTable)
+ .addSingleValueDimension("ts", DataType.STRING)
+ .addMetric("revenue", DataType.DOUBLE)
+ .build();
+ Schema materializedViewSchema = new Schema.SchemaBuilder()
+ .setSchemaName(materializedViewRawTable)
+ .addSingleValueDimension("ts", DataType.STRING)
+ .addMetric("revenue", DataType.DOUBLE)
+ .build();
+
+ TableCache tableCache = mock(TableCache.class);
+ when(tableCache.getActualTableName(baseRawTable)).thenReturn(baseRawTable);
+ when(tableCache.getSchema(baseRawTable)).thenReturn(baseSchema);
+ when(tableCache.getSchema(materializedViewRawTable)).thenReturn(materializedViewSchema);
+ when(tableCache.getColumnNameMap(anyString())).thenReturn(Map.of("ts", "ts", "revenue", "revenue"));
+ TableConfig tableCfg = mock(TableConfig.class);
+ TenantConfig tenant = new TenantConfig("t_BROKER", "t_SERVER", null);
+ when(tableCfg.getTenantConfig()).thenReturn(tenant);
+ when(tableCache.getTableConfig(baseOfflineTable)).thenReturn(tableCfg);
+ when(tableCache.getTableConfig(materializedViewOfflineTable)).thenReturn(tableCfg);
+ /// Critical to this test: the base table is HYBRID. The realtime sibling must exist in the cache
+ /// so the FULL_REWRITE hybrid-guard at BaseSingleStageBrokerRequestHandler trips.
+ when(tableCache.getTableConfig(baseRealtimeTable)).thenReturn(tableCfg);
+
+ BrokerRoutingManager routingManager = mock(BrokerRoutingManager.class);
+ when(routingManager.routingExists(baseOfflineTable)).thenReturn(true);
+ when(routingManager.routingExists(baseRealtimeTable)).thenReturn(true);
+ when(routingManager.getQueryTimeoutMs(anyString())).thenReturn(10000L);
+ RoutingTable rt = mock(RoutingTable.class);
+ when(rt.getServerInstanceToSegmentsMap()).thenReturn(
+ Map.of(new ServerInstance(new InstanceConfig("server01_9000")),
+ new SegmentsToQuery(List.of("seg01"), List.of())));
+ when(routingManager.getRoutingTable(any(), Mockito.anyLong())).thenReturn(rt);
+
+ QueryQuotaManager quotaManager = mock(QueryQuotaManager.class);
+ when(quotaManager.acquireDatabase(anyString())).thenReturn(true);
+ when(quotaManager.acquireApplication(anyString())).thenReturn(true);
+ when(quotaManager.acquire(anyString())).thenReturn(true);
+
+ BrokerMetrics.register(mock(BrokerMetrics.class));
+ PinotConfiguration config = new PinotConfiguration();
+ BrokerQueryEventListenerFactory.init(config);
+
+ AtomicReference<BrokerRequest> capturedServerBrokerRequest = new AtomicReference<>();
+ BaseSingleStageBrokerRequestHandler handler =
+ new BaseSingleStageBrokerRequestHandler(config, "broker1", new BrokerRequestIdGenerator(),
+ routingManager, new AllowAllAccessControlFactory(), quotaManager, tableCache,
+ ThreadAccountantUtils.getNoOpAccountant(), null, materializedViewHandler) {
+ @Override
+ public void start() {
+ }
+
+ @Override
+ public void shutDown() {
+ }
+
+ @Override
+ protected BrokerResponseNative processBrokerRequest(long requestId,
+ BrokerRequest originalBrokerRequest, BrokerRequest serverBrokerRequest,
+ TableRouteInfo route, long timeoutMs, ServerStats serverStats,
+ RequestContext requestContext) {
+ capturedServerBrokerRequest.set(serverBrokerRequest);
+ return BrokerResponseNative.empty();
+ }
+
+ @Override
+ protected BrokerResponseNative processMaterializedViewSplitBrokerRequest(long requestId,
+ long materializedViewRequestId, BrokerRequest originalBrokerRequest, TableRouteInfo baseRoute,
+ TableRouteInfo materializedViewRoute, long timeoutMs, ServerStats serverStats,
+ RequestContext requestContext) {
+ Assert.fail("Split path must not be entered when FULL_REWRITE is skipped");
+ return null;
+ }
+ };
+
+ BrokerResponseNative response = (BrokerResponseNative) handler.handleRequest(userSql);
+ BrokerRequest serverBrokerRequest = capturedServerBrokerRequest.get();
+ Assert.assertNotNull(serverBrokerRequest,
+ "processBrokerRequest must have been reached after the FULL_REWRITE skip; exceptions: "
+ + response.getExceptions());
+ /// Server-side query MUST target a base-table variant (OFFLINE or REALTIME) — never the MV
+ /// table — because the hybrid guard rejected the FULL_REWRITE swap. Hybrid tables dispatch
+ /// to both offline + realtime broker requests, so the test accepts either base-side name.
+ String serverTableName = serverBrokerRequest.getPinotQuery().getDataSource().getTableName();
+ Assert.assertTrue(
+ serverTableName.equals(baseOfflineTable) || serverTableName.equals(baseRealtimeTable),
+ "FULL_REWRITE must be skipped on hybrid base tables to avoid silently dropping realtime data; "
+ + "expected server query against a base-table variant but got: " + serverTableName);
+ Assert.assertNotEquals(serverTableName, materializedViewOfflineTable,
+ "Server query must not target the MV table when the FULL_REWRITE was skipped");
+ /// And the MV-rewrite marker must NOT have been stamped on the server query.
+ Map<String, String> serverOptions = serverBrokerRequest.getPinotQuery().getQueryOptions();
+ Assert.assertTrue(serverOptions == null
+ || !serverOptions.containsKey(CommonConstants.Broker.Request.QueryOptionKey.MATERIALIZED_VIEW_REWRITE),
+ "FULL_REWRITE marker must not be stamped when rewrite is skipped; options: " + serverOptions);
+ }
+
+ /**
+ * Pins the F1 fix: when the rewrite engine returns a FULL_REWRITE plan and no skip condition
+ * trips (base table is OFFLINE-only, schema present, etc.), the server-side query MUST actually
+ * be swapped to target the MV — not the base table. The earlier `watermarkMs <= 0` guard in
+ * `DefaultMaterializedViewHandler.compile` was over-broad and silently dropped every
+ * FULL_REWRITE attempt while `annotateResponse` continued to stamp `materializedViewQueried`,
+ * producing a false-positive on every operator-facing metric. This test asserts the actual
+ * dataSource table name on the server query equals the MV name AND the response's
+ * `materializedViewQueried` matches.
+ */
+ @Test
+ public void testMaterializedViewFullRewriteActuallySwapsServerQuery()
+ throws Exception {
+ String baseOfflineTable = "baseTable_OFFLINE";
+ String materializedViewOfflineTable = "mv_baseTable_OFFLINE";
+ String baseRawTable = "baseTable";
+ String materializedViewRawTable = "mv_baseTable";
+
+ String userSql = "SELECT ts, SUM(revenue) FROM baseTable GROUP BY ts LIMIT 100";
+ PinotQuery materializedViewQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT ts, SUM(revenue) FROM mv_baseTable_OFFLINE GROUP BY ts LIMIT 100");
+
+ /// Engine returns FULL_REWRITE with watermarkMs=0 (the actual production code path for batch
+ /// MVs). Pre-F1, this was silently downgraded to "no rewrite" by the handler's `<= 0` guard
+ /// while the response still claimed the MV was used.
+ MaterializedViewRewritePlan plan = new MaterializedViewRewritePlan(
+ materializedViewOfflineTable, MatchType.EXACT, ExecutionMode.FULL_REWRITE, materializedViewQuery, 1.0);
+
+ MaterializedViewQueryRewriteEngine materializedViewEngine = mock(MaterializedViewQueryRewriteEngine.class);
+ when(materializedViewEngine.tryRewrite(any(PinotQuery.class), anyString())).thenReturn(plan);
+ MaterializedViewHandler materializedViewHandler = new DefaultMaterializedViewHandler(materializedViewEngine);
+
+ Schema baseSchema = new Schema.SchemaBuilder()
+ .setSchemaName(baseRawTable)
+ .addSingleValueDimension("ts", DataType.STRING)
+ .addMetric("revenue", DataType.DOUBLE)
+ .build();
+ Schema materializedViewSchema = new Schema.SchemaBuilder()
+ .setSchemaName(materializedViewRawTable)
+ .addSingleValueDimension("ts", DataType.STRING)
+ .addMetric("revenue", DataType.DOUBLE)
+ .build();
+
+ TableCache tableCache = mock(TableCache.class);
+ when(tableCache.getActualTableName(baseRawTable)).thenReturn(baseRawTable);
+ when(tableCache.getSchema(baseRawTable)).thenReturn(baseSchema);
+ when(tableCache.getSchema(materializedViewRawTable)).thenReturn(materializedViewSchema);
+ when(tableCache.getColumnNameMap(anyString())).thenReturn(Map.of("ts", "ts", "revenue", "revenue"));
+ TableConfig tableCfg = mock(TableConfig.class);
+ TenantConfig tenant = new TenantConfig("t_BROKER", "t_SERVER", null);
+ when(tableCfg.getTenantConfig()).thenReturn(tenant);
+ when(tableCache.getTableConfig(baseOfflineTable)).thenReturn(tableCfg);
+ when(tableCache.getTableConfig(materializedViewOfflineTable)).thenReturn(tableCfg);
+ /// OFFLINE-only base table — no realtime sibling, so the hybrid-guard does NOT trip.
+ when(tableCache.getTableConfig(TableNameBuilder.REALTIME.tableNameWithType(baseRawTable))).thenReturn(null);
+
+ BrokerRoutingManager routingManager = mock(BrokerRoutingManager.class);
+ when(routingManager.routingExists(baseOfflineTable)).thenReturn(true);
+ when(routingManager.routingExists(materializedViewOfflineTable)).thenReturn(true);
+ when(routingManager.getQueryTimeoutMs(anyString())).thenReturn(10000L);
+ RoutingTable rt = mock(RoutingTable.class);
+ when(rt.getServerInstanceToSegmentsMap()).thenReturn(
+ Map.of(new ServerInstance(new InstanceConfig("server01_9000")),
+ new SegmentsToQuery(List.of("seg01"), List.of())));
+ when(routingManager.getRoutingTable(any(), Mockito.anyLong())).thenReturn(rt);
+
+ QueryQuotaManager quotaManager = mock(QueryQuotaManager.class);
+ when(quotaManager.acquireDatabase(anyString())).thenReturn(true);
+ when(quotaManager.acquireApplication(anyString())).thenReturn(true);
+ when(quotaManager.acquire(anyString())).thenReturn(true);
+
+ BrokerMetrics.register(mock(BrokerMetrics.class));
+ PinotConfiguration config = new PinotConfiguration();
+ BrokerQueryEventListenerFactory.init(config);
+
+ AtomicReference<BrokerRequest> capturedServerBrokerRequest = new AtomicReference<>();
+ BaseSingleStageBrokerRequestHandler handler =
+ new BaseSingleStageBrokerRequestHandler(config, "broker1", new BrokerRequestIdGenerator(),
+ routingManager, new AllowAllAccessControlFactory(), quotaManager, tableCache,
+ ThreadAccountantUtils.getNoOpAccountant(), null, materializedViewHandler) {
+ @Override
+ public void start() {
+ }
+
+ @Override
+ public void shutDown() {
+ }
+
+ @Override
+ protected BrokerResponseNative processBrokerRequest(long requestId,
+ BrokerRequest originalBrokerRequest, BrokerRequest serverBrokerRequest,
+ TableRouteInfo route, long timeoutMs, ServerStats serverStats,
+ RequestContext requestContext) {
+ capturedServerBrokerRequest.set(serverBrokerRequest);
+ return BrokerResponseNative.empty();
+ }
+
+ @Override
+ protected BrokerResponseNative processMaterializedViewSplitBrokerRequest(long requestId,
+ long materializedViewRequestId, BrokerRequest originalBrokerRequest, TableRouteInfo baseRoute,
+ TableRouteInfo materializedViewRoute, long timeoutMs, ServerStats serverStats,
+ RequestContext requestContext) {
+ return BrokerResponseNative.empty();
+ }
+ };
+
+ BrokerResponseNative response = (BrokerResponseNative) handler.handleRequest(userSql);
+ BrokerRequest serverBrokerRequest = capturedServerBrokerRequest.get();
+ Assert.assertNotNull(serverBrokerRequest,
+ "processBrokerRequest must have been reached after the FULL_REWRITE swap; exceptions: "
+ + response.getExceptions());
+ /// The actual swap MUST have happened — server-side query targets the MV, not the base table.
+ String serverTableName = serverBrokerRequest.getPinotQuery().getDataSource().getTableName();
+ Assert.assertEquals(serverTableName, materializedViewOfflineTable,
+ "FULL_REWRITE must swap the server query to target the MV table but got: " + serverTableName);
+ /// The MV-rewrite marker MUST be stamped on the server query.
+ Map<String, String> serverOptions = serverBrokerRequest.getPinotQuery().getQueryOptions();
+ Assert.assertNotNull(serverOptions, "Server query options must contain the MV-rewrite marker");
+ Assert.assertEquals(
+ serverOptions.get(CommonConstants.Broker.Request.QueryOptionKey.MATERIALIZED_VIEW_REWRITE), "true",
+ "FULL_REWRITE marker must be stamped on the committed swap; options: " + serverOptions);
+ /// And the response must report the MV name (no false positive — the swap really did happen).
+ Assert.assertEquals(response.getMaterializedViewQueried(), materializedViewOfflineTable,
+ "Response must report the MV name when the swap was committed");
+ }
+
+ /**
+ * Pins the cascade-prevention guard: when the user's query already targets an MV table
+ * directly ({@code TableConfig.isMaterializedView()} is {@code true}), the broker must skip
+ * MV rewrite entirely. Cascading MV-to-MV rewrites are not supported, and the explicit
+ * guard uses the new flag from PR #18564 as the single source of truth for MV identity.
+ */
+ @Test
+ public void testMaterializedViewRewriteSkippedWhenUserQueryTargetsMaterializedView()
+ throws Exception {
+ String materializedViewOfflineTable = "mv_baseTable_OFFLINE";
+ String materializedViewRawTable = "mv_baseTable";
+
+ String userSql = "SELECT ts, SUM(revenue) FROM mv_baseTable GROUP BY ts LIMIT 100";
+
+ /// The engine should NEVER be invoked — the broker's cascade guard fires first. Use a strict
+ /// mock that fails if `tryRewrite` is called.
+ MaterializedViewQueryRewriteEngine materializedViewEngine = mock(MaterializedViewQueryRewriteEngine.class);
+ when(materializedViewEngine.tryRewrite(any(PinotQuery.class), anyString()))
+ .thenThrow(new AssertionError("MV engine must not be invoked when the user query already targets an MV"));
+ MaterializedViewHandler materializedViewHandler = new DefaultMaterializedViewHandler(materializedViewEngine);
+
+ Schema materializedViewSchema = new Schema.SchemaBuilder()
+ .setSchemaName(materializedViewRawTable)
+ .addSingleValueDimension("ts", DataType.STRING)
+ .addMetric("revenue", DataType.DOUBLE)
+ .build();
+
+ TableCache tableCache = mock(TableCache.class);
+ when(tableCache.getActualTableName(materializedViewRawTable)).thenReturn(materializedViewRawTable);
+ when(tableCache.getSchema(materializedViewRawTable)).thenReturn(materializedViewSchema);
+ when(tableCache.getColumnNameMap(anyString())).thenReturn(Map.of("ts", "ts", "revenue", "revenue"));
+ TableConfig mvTableCfg = mock(TableConfig.class);
+ TenantConfig tenant = new TenantConfig("t_BROKER", "t_SERVER", null);
+ when(mvTableCfg.getTenantConfig()).thenReturn(tenant);
+ /// The MV's TableConfig declares MV identity via the new isMaterializedView flag.
+ when(mvTableCfg.isMaterializedView()).thenReturn(true);
+ when(tableCache.getTableConfig(materializedViewOfflineTable)).thenReturn(mvTableCfg);
+
+ BrokerRoutingManager routingManager = mock(BrokerRoutingManager.class);
+ when(routingManager.routingExists(materializedViewOfflineTable)).thenReturn(true);
+ when(routingManager.getQueryTimeoutMs(anyString())).thenReturn(10000L);
+ RoutingTable rt = mock(RoutingTable.class);
+ when(rt.getServerInstanceToSegmentsMap()).thenReturn(
+ Map.of(new ServerInstance(new InstanceConfig("server01_9000")),
+ new SegmentsToQuery(List.of("seg01"), List.of())));
+ when(routingManager.getRoutingTable(any(), Mockito.anyLong())).thenReturn(rt);
+
+ QueryQuotaManager quotaManager = mock(QueryQuotaManager.class);
+ when(quotaManager.acquireDatabase(anyString())).thenReturn(true);
+ when(quotaManager.acquireApplication(anyString())).thenReturn(true);
+ when(quotaManager.acquire(anyString())).thenReturn(true);
+
+ BrokerMetrics.register(mock(BrokerMetrics.class));
+ PinotConfiguration config = new PinotConfiguration();
+ BrokerQueryEventListenerFactory.init(config);
+
+ AtomicReference<BrokerRequest> capturedServerBrokerRequest = new AtomicReference<>();
+ BaseSingleStageBrokerRequestHandler handler =
+ new BaseSingleStageBrokerRequestHandler(config, "broker1", new BrokerRequestIdGenerator(),
+ routingManager, new AllowAllAccessControlFactory(), quotaManager, tableCache,
+ ThreadAccountantUtils.getNoOpAccountant(), null, materializedViewHandler) {
+ @Override
+ public void start() {
+ }
+
+ @Override
+ public void shutDown() {
+ }
+
+ @Override
+ protected BrokerResponseNative processBrokerRequest(long requestId,
+ BrokerRequest originalBrokerRequest, BrokerRequest serverBrokerRequest,
+ TableRouteInfo route, long timeoutMs, ServerStats serverStats,
+ RequestContext requestContext) {
+ capturedServerBrokerRequest.set(serverBrokerRequest);
+ return BrokerResponseNative.empty();
+ }
+
+ @Override
+ protected BrokerResponseNative processMaterializedViewSplitBrokerRequest(long requestId,
+ long materializedViewRequestId, BrokerRequest originalBrokerRequest, TableRouteInfo baseRoute,
+ TableRouteInfo materializedViewRoute, long timeoutMs, ServerStats serverStats,
+ RequestContext requestContext) {
+ Assert.fail("Split path must not be entered when the cascade guard fires");
+ return null;
+ }
+ };
+
+ BrokerResponseNative response = (BrokerResponseNative) handler.handleRequest(userSql);
+ BrokerRequest serverBrokerRequest = capturedServerBrokerRequest.get();
+ Assert.assertNotNull(serverBrokerRequest,
+ "processBrokerRequest must have been reached without invoking MV rewrite; exceptions: "
+ + response.getExceptions());
+ /// The server query MUST still target the MV table the user explicitly named — the rewrite is
+ /// skipped, but the user's choice of table stands.
+ String serverTableName = serverBrokerRequest.getPinotQuery().getDataSource().getTableName();
+ Assert.assertEquals(serverTableName, materializedViewOfflineTable,
+ "User-issued MV query must run as-is when the cascade guard fires; got: " + serverTableName);
+ /// The MV-rewrite marker must NOT have been stamped (no rewrite happened).
+ Map<String, String> serverOptions = serverBrokerRequest.getPinotQuery().getQueryOptions();
+ Assert.assertTrue(serverOptions == null
+ || !serverOptions.containsKey(CommonConstants.Broker.Request.QueryOptionKey.MATERIALIZED_VIEW_REWRITE),
+ "MV-rewrite marker must not be stamped when the cascade guard skipped rewrite; options: " + serverOptions);
+ /// And the response must NOT report a materializedViewQueried — no MV was selected.
+ Assert.assertNull(response.getMaterializedViewQueried(),
+ "Response must not report a materializedViewQueried when the cascade guard fired");
+ }
+
+ /// Pins the fix for the reviewer-flagged regression: `EXPLAIN PLAN FOR <query>` against a
+ /// SPLIT-eligible MV must NOT enter `tryExecuteMaterializedViewSplit` (which would dispatch
+ /// dual scatter-gather to base+MV and merge live `DataTable`s). The broker must fall
+ /// through to the standard explain handling instead.
+ @Test
+ public void testMaterializedViewSplitSkippedForExplainQuery()
+ throws Exception {
+ String baseOfflineTable = "baseTable_OFFLINE";
+ String materializedViewOfflineTable = "mv_baseTable_OFFLINE";
+ String baseRawTable = "baseTable";
+ String materializedViewRawTable = "mv_baseTable";
+
+ String userSql = "EXPLAIN PLAN FOR SELECT ts, SUM(revenue) FROM baseTable GROUP BY ts";
+ PinotQuery materializedViewServerQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT ts, SUM(revenue) FROM mv_baseTable_OFFLINE GROUP BY ts");
+
+ MaterializedViewRewritePlan plan = new MaterializedViewRewritePlan(
+ materializedViewOfflineTable, MatchType.EXACT, ExecutionMode.SPLIT_REWRITE,
+ materializedViewServerQuery, 1.0);
+
+ Schema baseSchema = new Schema.SchemaBuilder()
+ .setSchemaName(baseRawTable)
+ .addSingleValueDimension("ts", DataType.STRING)
+ .addMetric("revenue", DataType.DOUBLE)
+ .build();
+ Schema materializedViewSchema = new Schema.SchemaBuilder()
+ .setSchemaName(materializedViewRawTable)
+ .addSingleValueDimension("ts", DataType.STRING)
+ .addMetric("revenue", DataType.DOUBLE)
+ .build();
+
+ /// Mock the handler directly so we can return a SPLIT context from `compile()` and assert
+ /// `executeSplit()` is NEVER called. Using the concrete `DefaultMaterializedViewHandler`
+ /// would force us to thread a full split plan + dispatcher through the engine mock; the
+ /// direct handler mock is the minimal scaffolding needed to pin the explain-skip behavior.
+ MaterializedViewHandler materializedViewHandler = mock(MaterializedViewHandler.class);
+ when(materializedViewHandler.compile(any(MaterializedViewCompileContext.class)))
+ .thenReturn(MaterializedViewContext.forSplitRewrite(
+ plan, materializedViewServerQuery, materializedViewOfflineTable, materializedViewSchema));
+ when(materializedViewHandler.executeSplit(any(MaterializedViewSplitExecutionContext.class)))
+ .thenThrow(new AssertionError("executeSplit must not run for EXPLAIN queries"));
+
+ TableCache tableCache = mock(TableCache.class);
+ when(tableCache.getActualTableName(baseRawTable)).thenReturn(baseRawTable);
+ when(tableCache.getSchema(baseRawTable)).thenReturn(baseSchema);
+ when(tableCache.getSchema(materializedViewRawTable)).thenReturn(materializedViewSchema);
+ when(tableCache.getColumnNameMap(anyString())).thenReturn(Map.of("ts", "ts", "revenue", "revenue"));
+ TableConfig tableCfg = mock(TableConfig.class);
+ TenantConfig tenant = new TenantConfig("t_BROKER", "t_SERVER", null);
+ when(tableCfg.getTenantConfig()).thenReturn(tenant);
+ when(tableCache.getTableConfig(baseOfflineTable)).thenReturn(tableCfg);
+
+ BrokerRoutingManager routingManager = mock(BrokerRoutingManager.class);
+ when(routingManager.routingExists(baseOfflineTable)).thenReturn(true);
+ when(routingManager.getQueryTimeoutMs(anyString())).thenReturn(10000L);
+ RoutingTable rt = mock(RoutingTable.class);
+ when(rt.getServerInstanceToSegmentsMap()).thenReturn(
+ Map.of(new ServerInstance(new InstanceConfig("server01_9000")),
+ new SegmentsToQuery(List.of("seg01"), List.of())));
+ when(routingManager.getRoutingTable(any(), Mockito.anyLong())).thenReturn(rt);
+
+ QueryQuotaManager quotaManager = mock(QueryQuotaManager.class);
+ when(quotaManager.acquireDatabase(anyString())).thenReturn(true);
+ when(quotaManager.acquireApplication(anyString())).thenReturn(true);
+ when(quotaManager.acquire(anyString())).thenReturn(true);
+
+ BrokerMetrics.register(mock(BrokerMetrics.class));
+ PinotConfiguration config = new PinotConfiguration();
+ BrokerQueryEventListenerFactory.init(config);
+
+ AtomicReference<BrokerRequest> capturedServerBrokerRequest = new AtomicReference<>();
+ BaseSingleStageBrokerRequestHandler handler =
+ new BaseSingleStageBrokerRequestHandler(config, "broker1", new BrokerRequestIdGenerator(),
+ routingManager, new AllowAllAccessControlFactory(), quotaManager, tableCache,
+ ThreadAccountantUtils.getNoOpAccountant(), null, materializedViewHandler) {
+ @Override
+ public void start() {
+ }
+
+ @Override
+ public void shutDown() {
+ }
+
+ @Override
+ protected BrokerResponseNative processBrokerRequest(long requestId,
+ BrokerRequest originalBrokerRequest, BrokerRequest serverBrokerRequest,
+ TableRouteInfo route, long timeoutMs, ServerStats serverStats,
+ RequestContext requestContext) {
+ capturedServerBrokerRequest.set(serverBrokerRequest);
+ return BrokerResponseNative.empty();
+ }
+
+ @Override
+ protected BrokerResponseNative processMaterializedViewSplitBrokerRequest(long requestId,
+ long materializedViewRequestId, BrokerRequest originalBrokerRequest, TableRouteInfo baseRoute,
+ TableRouteInfo materializedViewRoute, long timeoutMs, ServerStats serverStats,
+ RequestContext requestContext) {
+ Assert.fail("processMaterializedViewSplitBrokerRequest must not run for EXPLAIN queries");
+ return null;
+ }
+ };
+
+ handler.handleRequest(userSql);
+ BrokerRequest serverBrokerRequest = capturedServerBrokerRequest.get();
+ Assert.assertNotNull(serverBrokerRequest,
+ "EXPLAIN must fall through to the standard explain handling — processBrokerRequest captures the request");
+ /// The standard explain path uses the BASE table, not the MV. The split branch was skipped.
+ String serverTableName = serverBrokerRequest.getPinotQuery().getDataSource().getTableName();
+ Assert.assertEquals(serverTableName, baseOfflineTable,
+ "EXPLAIN must route to the base table, not the MV; SPLIT must not have swapped routing");
+ }
}
diff --git a/pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/LiteralOnlyBrokerRequestTest.java b/pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/LiteralOnlyBrokerRequestTest.java
index f0038143cd5..a40a1e9cf81 100644
--- a/pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/LiteralOnlyBrokerRequestTest.java
+++ b/pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/LiteralOnlyBrokerRequestTest.java
@@ -174,7 +174,7 @@ public class LiteralOnlyBrokerRequestTest {
new SingleConnectionBrokerRequestHandler(new PinotConfiguration(), "testBrokerId",
new BrokerRequestIdGenerator(), null, ACCESS_CONTROL_FACTORY, null, null, null, null,
mock(ServerRoutingStatsManager.class), mock(FailureDetector.class),
- ThreadAccountantUtils.getNoOpAccountant(), null);
+ ThreadAccountantUtils.getNoOpAccountant(), null, null);
long randNum = RANDOM.nextLong();
byte[] randBytes = new byte[12];
@@ -200,7 +200,7 @@ public class LiteralOnlyBrokerRequestTest {
new SingleConnectionBrokerRequestHandler(new PinotConfiguration(), "testBrokerId",
new BrokerRequestIdGenerator(), null, ACCESS_CONTROL_FACTORY, null, null, null, null,
mock(ServerRoutingStatsManager.class), mock(FailureDetector.class),
- ThreadAccountantUtils.getNoOpAccountant(), null);
+ ThreadAccountantUtils.getNoOpAccountant(), null, null);
long currentTsMin = System.currentTimeMillis();
BrokerResponse brokerResponse = requestHandler.handleRequest(
"SELECT now() AS currentTs, fromDateTime('2020-01-01 UTC', 'yyyy-MM-dd z') AS firstDayOf2020");
@@ -356,7 +356,7 @@ public class LiteralOnlyBrokerRequestTest {
new SingleConnectionBrokerRequestHandler(new PinotConfiguration(), "testBrokerId",
new BrokerRequestIdGenerator(), null, ACCESS_CONTROL_FACTORY, null, null, null, null,
mock(ServerRoutingStatsManager.class), mock(FailureDetector.class),
- ThreadAccountantUtils.getNoOpAccountant(), null);
+ ThreadAccountantUtils.getNoOpAccountant(), null, null);
// Test 1: select constant
BrokerResponse brokerResponse = requestHandler.handleRequest("EXPLAIN PLAN FOR SELECT 1.5, 'test'");
diff --git a/pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/SingleConnectionBrokerRequestHandlerMergeTest.java b/pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/SingleConnectionBrokerRequestHandlerMergeTest.java
new file mode 100644
index 00000000000..f871effbcc7
--- /dev/null
+++ b/pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/SingleConnectionBrokerRequestHandlerMergeTest.java
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.requesthandler;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.core.transport.ServerResponse;
+import org.apache.pinot.core.transport.ServerRoutingInstance;
+import org.apache.pinot.spi.config.table.TableType;
+import org.testng.annotations.Test;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotSame;
+import static org.testng.Assert.assertTrue;
+
+
+/// Regression test for the MV-split DataTable merge contract in
+/// {@link SingleConnectionBrokerRequestHandler#mergeDataTablesByIdentity}.
+///
+/// <p>Pins the invariant that the same physical server (matching hostname/port/tableType) appearing
+/// on BOTH the base sub-query and the MV sub-query contributes two distinct entries to the merged
+/// map. Using a regular HashMap would collapse the two entries (since
+/// {@link ServerRoutingInstance#equals} keys on hostname/port/tableType), silently undercounting
+/// results during the broker reduce.
+public class SingleConnectionBrokerRequestHandlerMergeTest {
+
+ @Test
+ public void testSameServerOnBothBranchesProducesTwoEntries() {
+ /// Two distinct ServerRoutingInstance objects with identical hostname/port/tableType — i.e.
+ /// hash/equals identical, references different. This mirrors the production case where the
+ /// base sub-query and MV sub-query each hit an OFFLINE server hosted on the same host:port.
+ ServerRoutingInstance baseInstance = new ServerRoutingInstance("server-1", 9000, TableType.OFFLINE);
+ ServerRoutingInstance viewInstance = new ServerRoutingInstance("server-1", 9000, TableType.OFFLINE);
+ assertEquals(baseInstance, viewInstance, "Test fixture: instances must be equal()");
+ assertEquals(baseInstance.hashCode(), viewInstance.hashCode(),
+ "Test fixture: instances must hash equally");
+ assertNotSame(baseInstance, viewInstance, "Test fixture: instances must be distinct references");
+
+ DataTable baseDataTable = mock(DataTable.class);
+ DataTable viewDataTable = mock(DataTable.class);
+
+ ServerResponse baseResponse = mock(ServerResponse.class);
+ when(baseResponse.getDataTable()).thenReturn(baseDataTable);
+ when(baseResponse.getResponseSize()).thenReturn(100);
+ ServerResponse viewResponse = mock(ServerResponse.class);
+ when(viewResponse.getDataTable()).thenReturn(viewDataTable);
+ when(viewResponse.getResponseSize()).thenReturn(50);
+
+ Map<ServerRoutingInstance, ServerResponse> baseResponses = new HashMap<>();
+ baseResponses.put(baseInstance, baseResponse);
+ Map<ServerRoutingInstance, ServerResponse> viewResponses = new HashMap<>();
+ viewResponses.put(viewInstance, viewResponse);
+
+ List<ServerRoutingInstance> serversNotResponded = new ArrayList<>();
+ long[] totalResponseSizeHolder = {0L};
+ Map<ServerRoutingInstance, DataTable> merged = SingleConnectionBrokerRequestHandler
+ .mergeDataTablesByIdentity(baseResponses, viewResponses, serversNotResponded,
+ totalResponseSizeHolder);
+
+ assertEquals(merged.size(), 2,
+ "IdentityHashMap must hold both DataTables when the two sub-queries hit the same physical server. "
+ + "A regular HashMap collapses them via ServerRoutingInstance.equals(), silently dropping one. "
+ + "Got: " + merged);
+ assertTrue(merged.values().contains(baseDataTable), "Base DataTable must be retained");
+ assertTrue(merged.values().contains(viewDataTable), "View DataTable must be retained");
+ assertEquals(totalResponseSizeHolder[0], 150L);
+ assertEquals(serversNotResponded.size(), 0);
+ }
+
+ /// Pins the production guard logic in `processMaterializedViewSplitBrokerRequest`:
+ ///
+ /// if (!viewFinalResponses.isEmpty() && countSuccessfulDataTables(viewFinalResponses) == 0) throw …
+ ///
+ /// The guard catches the case where every server on one branch of the MV split returned no
+ /// DataTable. Without this regression test, flipping the comparison (e.g. `!= 0`) or removing
+ /// the call would silently undercount the historical half of every MV-split query.
+ @Test
+ public void testCountSuccessfulDataTablesAllNullReturnsZero() {
+ ServerRoutingInstance s1 = new ServerRoutingInstance("server-1", 9000, TableType.OFFLINE);
+ ServerRoutingInstance s2 = new ServerRoutingInstance("server-2", 9000, TableType.OFFLINE);
+ ServerResponse r1 = mock(ServerResponse.class);
+ when(r1.getDataTable()).thenReturn(null);
+ ServerResponse r2 = mock(ServerResponse.class);
+ when(r2.getDataTable()).thenReturn(null);
+ Map<ServerRoutingInstance, ServerResponse> responses = new HashMap<>();
+ responses.put(s1, r1);
+ responses.put(s2, r2);
+
+ int count = SingleConnectionBrokerRequestHandler.countSuccessfulDataTables(responses);
+ assertEquals(count, 0,
+ "All-null DataTable responses must yield zero — guard depends on this to detect total split-side failure");
+ }
+
+ @Test
+ public void testCountSuccessfulDataTablesMixedReturnsCorrectCount() {
+ ServerRoutingInstance s1 = new ServerRoutingInstance("server-1", 9000, TableType.OFFLINE);
+ ServerRoutingInstance s2 = new ServerRoutingInstance("server-2", 9000, TableType.OFFLINE);
+ ServerRoutingInstance s3 = new ServerRoutingInstance("server-3", 9000, TableType.OFFLINE);
+ ServerResponse ok1 = mock(ServerResponse.class);
+ when(ok1.getDataTable()).thenReturn(mock(DataTable.class));
+ ServerResponse failed = mock(ServerResponse.class);
+ when(failed.getDataTable()).thenReturn(null);
+ ServerResponse ok2 = mock(ServerResponse.class);
+ when(ok2.getDataTable()).thenReturn(mock(DataTable.class));
+ Map<ServerRoutingInstance, ServerResponse> responses = new HashMap<>();
+ responses.put(s1, ok1);
+ responses.put(s2, failed);
+ responses.put(s3, ok2);
+
+ int count = SingleConnectionBrokerRequestHandler.countSuccessfulDataTables(responses);
+ assertEquals(count, 2,
+ "countSuccessfulDataTables returns the count of responses carrying a non-null DataTable "
+ + "(2 of 3); the production guard then compares against responses.size() to detect partial "
+ + "failure on a SPLIT side — see testSplitGuardTripsOnAnyPartialFailureComment for the "
+ + "downstream behavior anchored against this count");
+ }
+
+ @Test
+ public void testCountSuccessfulDataTablesEmptyMapReturnsZero() {
+ /// Empty map is the "this side was not dispatched" case — the production guard skips it with
+ /// `!viewFinalResponses.isEmpty() && ...`. Confirming the count is zero anchors the
+ /// documented short-circuit so future refactors don't accidentally split the two checks.
+ assertEquals(SingleConnectionBrokerRequestHandler.countSuccessfulDataTables(new HashMap<>()), 0);
+ }
+
+ @Test
+ public void testNullDataTableRecordedAsServerNotResponded() {
+ ServerRoutingInstance instance = new ServerRoutingInstance("server-1", 9000, TableType.OFFLINE);
+ ServerResponse response = mock(ServerResponse.class);
+ when(response.getDataTable()).thenReturn(null);
+ when(response.getResponseSize()).thenReturn(0);
+
+ Map<ServerRoutingInstance, ServerResponse> baseResponses = new HashMap<>();
+ baseResponses.put(instance, response);
+ List<ServerRoutingInstance> serversNotResponded = new ArrayList<>();
+ long[] totalResponseSizeHolder = {0L};
+ Map<ServerRoutingInstance, DataTable> merged = SingleConnectionBrokerRequestHandler
+ .mergeDataTablesByIdentity(baseResponses, new HashMap<>(), serversNotResponded,
+ totalResponseSizeHolder);
+
+ assertEquals(merged.size(), 0);
+ assertEquals(serversNotResponded.size(), 1);
+ assertEquals(serversNotResponded.get(0), instance);
+ }
+}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerGauge.java b/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerGauge.java
index 7402c3c85d7..3d5b2ae9366 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerGauge.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerGauge.java
@@ -113,7 +113,13 @@ public enum BrokerGauge implements AbstractMetrics.Gauge {
GRPC_TOTAL_MAX_DIRECT_MEMORY("bytes", true),
/// Exports the total amount of direct memory allocated by the shaded Netty code used by gRPC
/// It is basically an adaptor for io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent.usedDirectMemory()
- GRPC_TOTAL_USED_DIRECT_MEMORY("bytes", true);
+ GRPC_TOTAL_USED_DIRECT_MEMORY("bytes", true),
+
+ /// Number of MV definition+runtime entries the broker currently holds in
+ /// `MaterializedViewMetadataCache`. Operators monitor this to confirm the cache is not
+ /// growing unboundedly: a cluster with K MVs should plateau near K; sustained growth
+ /// signals a leak in the ZK listener / drop path.
+ MATERIALIZED_VIEW_CACHE_ENTRY_COUNT("materializedViewCacheEntries", true);
private final String _brokerGaugeName;
private final String _unit;
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMeter.java b/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMeter.java
index 5e8387519fa..3e06f532728 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMeter.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMeter.java
@@ -119,6 +119,10 @@ public class BrokerMeter implements AbstractMetrics.Meter {
"QUERY_VALIDATION_EXCEPTIONS", "exceptions", false);
// Query validation phase.
public static final BrokerMeter UNKNOWN_COLUMN_EXCEPTIONS = create("UNKNOWN_COLUMN_EXCEPTIONS", "exceptions", false);
+ /// Materialized-view rewrite path: strategy bug or contract violation triggered the
+ /// defense-in-depth fallback to the base-table query path. Non-zero values indicate a
+ /// strategy regression that must be investigated; the query itself still succeeded.
+ public static final BrokerMeter QUERY_REWRITE_EXCEPTIONS = create("QUERY_REWRITE_EXCEPTIONS", "exceptions", false);
// Queries preempted by accountant
public static final BrokerMeter QUERIES_KILLED = create("QUERIES_KILLED", "query", true);
public static final BrokerMeter QUERIES_THROTTLED = create("QUERIES_THROTTLED", "query", true);
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/response/BrokerResponse.java b/pinot-common/src/main/java/org/apache/pinot/common/response/BrokerResponse.java
index 3206935029c..061c2fed570 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/response/BrokerResponse.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/response/BrokerResponse.java
@@ -437,4 +437,16 @@ public interface BrokerResponse {
* @return true if RLS filters were applied, false otherwise
*/
boolean getRLSFiltersApplied();
+
+ /// Get the materialized view table name that was hit (used) for this query, or `null`
+ /// if no materialized view was used. The default returns `null` so impls that do not track MV
+ /// rewrite (e.g. MSE response paths) need no explicit override; the matching *setter* is
+ /// deliberately NOT defaulted here — callers that want to record an MV-queried name must obtain
+ /// a concrete `BrokerResponseNative` and invoke its setter directly. Without that asymmetry, a
+ /// future caller could silently drop the MV-queried name on any impl that forgot to override
+ /// the setter, producing an observability hole indistinguishable from "no MV was used".
+ @Nullable
+ default String getMaterializedViewQueried() {
+ return null;
+ }
}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java b/pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java
index 6db0c1f609c..513dfdfce69 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java
@@ -46,7 +46,8 @@ import org.apache.pinot.spi.utils.JsonUtils;
"resultTable", "numRowsResultSet", "partialResult", "exceptions", "numGroupsLimitReached",
"numGroupsWarningLimitReached", "maxRowsInDistinctReached", "maxRowsWithoutChangeInDistinctReached",
"maxExecutionTimeInDistinctReached", "timeUsedMs",
- "requestId", "clientRequestId", "brokerId", "numDocsScanned", "totalDocs", "numEntriesScannedInFilter",
+ "requestId", "clientRequestId", "brokerId", "numDocsScanned", "totalDocs",
+ "numEntriesScannedInFilter",
"numEntriesScannedPostFilter", "numServersQueried", "numServersResponded", "numSegmentsQueried",
"numSegmentsProcessed", "numSegmentsMatched", "numConsumingSegmentsQueried", "numConsumingSegmentsProcessed",
"numConsumingSegmentsMatched", "minConsumingFreshnessTimeMs", "numSegmentsPrunedByBroker",
@@ -57,7 +58,7 @@ import org.apache.pinot.spi.utils.JsonUtils;
"explainPlanNumEmptyFilterSegments", "explainPlanNumMatchAllFilterSegments", "traceInfo", "tablesQueried",
"offlineThreadMemAllocatedBytes", "realtimeThreadMemAllocatedBytes", "offlineResponseSerMemAllocatedBytes",
"realtimeResponseSerMemAllocatedBytes", "offlineTotalMemAllocatedBytes", "realtimeTotalMemAllocatedBytes",
- "pools", "rlsFiltersApplied", "groupsTrimmed"
+ "pools", "rlsFiltersApplied", "groupsTrimmed", "materializedViewQueried"
})
@JsonIgnoreProperties(ignoreUnknown = true)
public class BrokerResponseNative implements BrokerResponse {
@@ -122,6 +123,9 @@ public class BrokerResponseNative implements BrokerResponse {
private Set<Integer> _pools = Set.of();
private boolean _rlsFiltersApplied = false;
+ @Nullable
+ private String _materializedViewQueried;
+
public BrokerResponseNative() {
}
@@ -632,4 +636,17 @@ public class BrokerResponseNative implements BrokerResponse {
public boolean getRLSFiltersApplied() {
return _rlsFiltersApplied;
}
+
+ @JsonProperty("materializedViewQueried")
+ public void setMaterializedViewQueried(@Nullable String materializedViewQueried) {
+ _materializedViewQueried = materializedViewQueried;
+ }
+
+ @Nullable
+ @JsonProperty("materializedViewQueried")
+ @JsonInclude(JsonInclude.Include.NON_NULL)
+ @Override
+ public String getMaterializedViewQueried() {
+ return _materializedViewQueried;
+ }
}
diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java
index 7d7524b28ab..df8dea28cca 100644
--- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java
+++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java
@@ -20,6 +20,7 @@ package org.apache.pinot.sql.parsers.rewriter;
import java.util.Map;
import java.util.stream.Collectors;
+import org.apache.logging.log4j.util.Strings;
import org.apache.pinot.spi.utils.CommonConstants;
@@ -41,4 +42,12 @@ public class RlsUtils {
public static String getRlsFilterForTable(Map<String, String> queryOptions, String tableName) {
return queryOptions.get(buildRlsFilterKey(tableName));
}
+
+ /// `true` iff a non-empty RLS filter has been stamped for the given table in `queryOptions`.
+ /// Matches the truthiness check inside [RlsFiltersRewriter#rewrite(PinotQuery)] so callers
+ /// stamp `BrokerResponse.setRLSFiltersApplied(...)` iff the rewriter actually attached a
+ /// predicate. Safe against a `null` `queryOptions` map.
+ public static boolean isRlsAppliedForTable(Map<String, String> queryOptions, String tableName) {
+ return queryOptions != null && !Strings.isEmpty(getRlsFilterForTable(queryOptions, tableName));
+ }
}
diff --git a/pinot-common/src/test/java/org/apache/pinot/common/response/broker/BrokerResponseNativeTest.java b/pinot-common/src/test/java/org/apache/pinot/common/response/broker/BrokerResponseNativeTest.java
index 71c7eb00fb9..e25008636bf 100644
--- a/pinot-common/src/test/java/org/apache/pinot/common/response/broker/BrokerResponseNativeTest.java
+++ b/pinot-common/src/test/java/org/apache/pinot/common/response/broker/BrokerResponseNativeTest.java
@@ -25,7 +25,6 @@ import org.testng.annotations.Test;
public class BrokerResponseNativeTest {
-
@Test
public void testEmptyResponse()
throws IOException {
@@ -64,4 +63,26 @@ public class BrokerResponseNativeTest {
Assert.assertEquals(newBrokerResponse.getExceptions().get(1).getErrorCode(), 400);
Assert.assertEquals(newBrokerResponse.getExceptions().get(1).getMessage(), errorMsgStr);
}
+
+ @Test
+ public void testMaterializedViewResponseFields()
+ throws IOException {
+ BrokerResponseNative expected = new BrokerResponseNative();
+ expected.setMaterializedViewQueried("orders_by_day_OFFLINE");
+
+ String brokerString = expected.toJsonString();
+ Assert.assertTrue(brokerString.contains("\"materializedViewQueried\""));
+
+ BrokerResponseNative actual = BrokerResponseNative.fromJsonString(brokerString);
+ Assert.assertEquals(actual.getMaterializedViewQueried(), expected.getMaterializedViewQueried());
+ }
+
+ @Test
+ public void testMaterializedViewQueriedAbsentWhenNull()
+ throws IOException {
+ BrokerResponseNative response = new BrokerResponseNative();
+ String json = response.toJsonString();
+ Assert.assertFalse(json.contains("materializedViewQueried"),
+ "Null materializedViewQueried should be suppressed by @JsonInclude(NON_NULL)");
+ }
}
diff --git a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/rewriter/RlsUtilsTest.java b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/rewriter/RlsUtilsTest.java
new file mode 100644
index 00000000000..d44192a9189
--- /dev/null
+++ b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/rewriter/RlsUtilsTest.java
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.sql.parsers.rewriter;
+
+import java.util.HashMap;
+import java.util.Map;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+/// Pins the `RlsUtils.isRlsAppliedForTable` contract used by the broker to stamp
+/// `BrokerResponse.setRLSFiltersApplied(...)`. Future refactors that break any of the
+/// short-circuits would silently flip the response field for every query.
+public class RlsUtilsTest {
+
+ @Test
+ public void testIsRlsAppliedForTableNullMap() {
+ Assert.assertFalse(RlsUtils.isRlsAppliedForTable(null, "t"));
+ }
+
+ @Test
+ public void testIsRlsAppliedForTableEmptyMap() {
+ Assert.assertFalse(RlsUtils.isRlsAppliedForTable(new HashMap<>(), "t"));
+ }
+
+ @Test
+ public void testIsRlsAppliedForTableMissingKey() {
+ Map<String, String> opts = new HashMap<>();
+ opts.put("rlsFilters-other", "( x = 1 )");
+ Assert.assertFalse(RlsUtils.isRlsAppliedForTable(opts, "t"));
+ }
+
+ @Test
+ public void testIsRlsAppliedForTableEmptyValue() {
+ Map<String, String> opts = new HashMap<>();
+ opts.put(RlsUtils.buildRlsFilterKey("t"), "");
+ Assert.assertFalse(RlsUtils.isRlsAppliedForTable(opts, "t"),
+ "Empty filter value must read as not-applied — matches RlsFiltersRewriter's own short-circuit");
+ }
+
+ @Test
+ public void testIsRlsAppliedForTableNonEmptyValue() {
+ Map<String, String> opts = new HashMap<>();
+ opts.put(RlsUtils.buildRlsFilterKey("t"), "( x = 1 )");
+ Assert.assertTrue(RlsUtils.isRlsAppliedForTable(opts, "t"));
+ }
+
+ @Test
+ public void testIsRlsAppliedForTableDifferentTable() {
+ Map<String, String> opts = new HashMap<>();
+ opts.put(RlsUtils.buildRlsFilterKey("t"), "( x = 1 )");
+ Assert.assertFalse(RlsUtils.isRlsAppliedForTable(opts, "other"),
+ "Marker for table `t` must not register as applied for table `other`");
+ }
+}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BrokerReduceService.java b/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BrokerReduceService.java
index d938deee1ec..07dd8300ee0 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BrokerReduceService.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BrokerReduceService.java
@@ -42,6 +42,7 @@ import org.apache.pinot.spi.exception.QueryErrorCode;
import org.apache.pinot.spi.exception.QueryException;
import org.apache.pinot.spi.query.QueryThreadContext;
import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants.Broker.Request.QueryOptionKey;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -188,13 +189,22 @@ public class BrokerReduceService extends BaseReduceService {
if (brokerRequest == serverBrokerRequest) {
queryContext = serverQueryContext;
} else {
+ /// `brokerRequest` and `serverBrokerRequest` differ when the query is either (a) a gapfill
+ /// (parser wrapped the user's SELECT into an outer SELECT) or (b) an MV-rewritten query
+ /// (broker rewrote `serverBrokerRequest` to target the MV table while the user's
+ /// `brokerRequest` stays at the base table for result-shape derivations). Only (a) needs
+ /// the gapfill post-processor; (b) just uses the user's `queryContext` directly. Any other
+ /// mismatched `brokerRequest != serverBrokerRequest` caller is rejected to preserve the
+ /// original `BadQueryRequestException` invariant ("Nested query is not supported without
+ /// gapfill") that pre-dated MV rewrite.
queryContext = QueryContextConverterUtils.getQueryContext(brokerRequest.getPinotQuery());
GapfillUtils.GapfillType gapfillType = GapfillUtils.getGapfillType(queryContext);
- if (gapfillType == null) {
+ if (gapfillType != null) {
+ BaseGapfillProcessor gapfillProcessor = GapfillProcessorFactory.getGapfillProcessor(queryContext, gapfillType);
+ gapfillProcessor.process(brokerResponseNative);
+ } else if (!isMaterializedViewRewrite(serverBrokerRequest)) {
throw new BadQueryRequestException("Nested query is not supported without gapfill");
}
- BaseGapfillProcessor gapfillProcessor = GapfillProcessorFactory.getGapfillProcessor(queryContext, gapfillType);
- gapfillProcessor.process(brokerResponseNative);
}
if (!serverQueryContext.isExplain()) {
@@ -203,6 +213,20 @@ public class BrokerReduceService extends BaseReduceService {
return brokerResponseNative;
}
+ /// MV rewrite signal: when the broker applies a FULL_REWRITE, it stamps the rewritten
+ /// server-side query with the [QueryOptionKey#MATERIALIZED_VIEW_REWRITE] marker. Reading
+ /// the marker here keeps the "Nested query is not supported without gapfill" safety net
+ /// active for any other path that produces `brokerRequest != serverBrokerRequest` (federated
+ /// query, JOIN rewrite, logical-table swap) — only the MV rewrite path explicitly opts out.
+ private static boolean isMaterializedViewRewrite(BrokerRequest serverBrokerRequest) {
+ if (serverBrokerRequest.getPinotQuery() == null
+ || serverBrokerRequest.getPinotQuery().getQueryOptions() == null) {
+ return false;
+ }
+ return Boolean.parseBoolean(
+ serverBrokerRequest.getPinotQuery().getQueryOptions().get(QueryOptionKey.MATERIALIZED_VIEW_REWRITE));
+ }
+
public void shutDown() {
_reduceExecutorService.shutdownNow();
}
diff --git a/pinot-core/src/test/java/org/apache/pinot/core/query/reduce/BrokerReduceServiceTest.java b/pinot-core/src/test/java/org/apache/pinot/core/query/reduce/BrokerReduceServiceTest.java
index da2e6e3e164..8f3b54e4a43 100644
--- a/pinot-core/src/test/java/org/apache/pinot/core/query/reduce/BrokerReduceServiceTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/query/reduce/BrokerReduceServiceTest.java
@@ -34,6 +34,7 @@ import org.apache.pinot.core.common.datatable.DataTableBuilderFactory;
import org.apache.pinot.core.transport.ServerRoutingInstance;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
import org.apache.pinot.spi.exception.QueryErrorCode;
import org.apache.pinot.spi.query.QueryThreadContext;
import org.apache.pinot.spi.utils.CommonConstants;
@@ -43,6 +44,8 @@ import org.testng.annotations.Test;
import static org.mockito.Mockito.mock;
import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertThrows;
+import static org.testng.Assert.assertTrue;
public class BrokerReduceServiceTest {
@@ -121,6 +124,87 @@ public class BrokerReduceServiceTest {
brokerReduceService.shutDown();
}
+ /**
+ * Pins the safety net: when `brokerRequest != serverBrokerRequest` (i.e. a path that rewrote
+ * the server-side query), no gapfill is requested, AND no MV-rewrite marker is present, the
+ * reducer MUST throw `BadQueryRequestException("Nested query is not supported without
+ * gapfill")`. This is the guardrail that prevents any future federated / JOIN / nested-query
+ * rewrite path from silently degrading to a meaningless reduce.
+ */
+ @Test
+ public void testNestedQueryRejectedWhenNoMaterializedViewMarker()
+ throws IOException {
+ BrokerReduceService service =
+ new BrokerReduceService(new PinotConfiguration(Map.of(Broker.CONFIG_OF_MAX_REDUCE_THREADS_PER_QUERY, 2)));
+ try {
+ BrokerRequest brokerRequest = CalciteSqlCompiler.compileToBrokerRequest("SELECT COUNT(*) FROM userTable");
+ BrokerRequest serverBrokerRequest =
+ CalciteSqlCompiler.compileToBrokerRequest("SELECT COUNT(*) FROM mv_userTable_OFFLINE");
+ /// No QUERY_OPTION_MATERIALIZED_VIEW_REWRITE marker — this simulates an unknown rewrite
+ /// path the broker may grow in the future. The reducer must refuse the request.
+ DataTableBuilder dataTableBuilder = DataTableBuilderFactory.getDataTableBuilder(
+ new DataSchema(new String[]{"count(*)"}, new ColumnDataType[]{ColumnDataType.LONG}));
+ dataTableBuilder.startRow();
+ dataTableBuilder.setColumn(0, 7L);
+ dataTableBuilder.finishRow();
+ DataTable dataTable = dataTableBuilder.build();
+ Map<ServerRoutingInstance, DataTable> dataTableMap = new HashMap<>();
+ dataTableMap.put(new ServerRoutingInstance("localhost", 9000, TableType.OFFLINE), dataTable);
+ try (QueryThreadContext ignore = QueryThreadContext.openForSseTest()) {
+ assertThrows(BadQueryRequestException.class,
+ () -> service.reduceOnDataTable(brokerRequest, serverBrokerRequest, dataTableMap, 10_000L,
+ mock(BrokerMetrics.class)));
+ }
+ } finally {
+ service.shutDown();
+ }
+ }
+
+ /**
+ * Pins the MV-rewrite opt-out: when the server-side query carries the broker-internal
+ * `MATERIALIZED_VIEW_REWRITE=true` marker, the reducer skips the "Nested query is not
+ * supported without gapfill" safety net and proceeds to reduce normally.
+ *
+ * <p>The marker is broker-internal — `BaseSingleStageBrokerRequestHandler.handleRequest` strips
+ * any user-supplied copy of this option at request entry, so the only way a server query can
+ * carry it is if the broker itself stamped it during a committed FULL_REWRITE.
+ */
+ @Test
+ public void testMaterializedViewMarkerOptsOutOfNestedQueryGuard()
+ throws IOException {
+ BrokerReduceService service =
+ new BrokerReduceService(new PinotConfiguration(Map.of(Broker.CONFIG_OF_MAX_REDUCE_THREADS_PER_QUERY, 2)));
+ try {
+ BrokerRequest brokerRequest = CalciteSqlCompiler.compileToBrokerRequest("SELECT COUNT(*) FROM userTable");
+ BrokerRequest serverBrokerRequest =
+ CalciteSqlCompiler.compileToBrokerRequest("SELECT COUNT(*) FROM mv_userTable_OFFLINE");
+ serverBrokerRequest.getPinotQuery().putToQueryOptions(
+ CommonConstants.Broker.Request.QueryOptionKey.MATERIALIZED_VIEW_REWRITE, "true");
+
+ DataTableBuilder dataTableBuilder = DataTableBuilderFactory.getDataTableBuilder(
+ new DataSchema(new String[]{"count(*)"}, new ColumnDataType[]{ColumnDataType.LONG}));
+ dataTableBuilder.startRow();
+ dataTableBuilder.setColumn(0, 42L);
+ dataTableBuilder.finishRow();
+ DataTable dataTable = dataTableBuilder.build();
+ Map<ServerRoutingInstance, DataTable> dataTableMap = new HashMap<>();
+ dataTableMap.put(new ServerRoutingInstance("localhost", 9000, TableType.OFFLINE), dataTable);
+
+ BrokerResponseNative response;
+ try (QueryThreadContext ignore = QueryThreadContext.openForSseTest()) {
+ response = service.reduceOnDataTable(brokerRequest, serverBrokerRequest, dataTableMap, 10_000L,
+ mock(BrokerMetrics.class));
+ }
+ /// No exception, and the count rolled up correctly. The "Nested query is not supported"
+ /// safety net was correctly opted out by the marker.
+ assertTrue(response.getExceptions().stream()
+ .noneMatch(e -> e.getErrorCode() == QueryErrorCode.QUERY_VALIDATION.getId()),
+ "MV-marked nested query must not raise QUERY_VALIDATION; exceptions: " + response.getExceptions());
+ } finally {
+ service.shutDown();
+ }
+ }
+
private BrokerResponseNative reduce(BrokerReduceService brokerReduceService, BrokerRequest brokerRequest,
Map<ServerRoutingInstance, DataTable> dataTableMap, long reduceTimeoutMs) {
try (QueryThreadContext ignore = QueryThreadContext.openForSseTest()) {
diff --git a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MaterializedViewClusterIntegrationTest.java b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MaterializedViewClusterIntegrationTest.java
new file mode 100644
index 00000000000..fbf7a0b7a88
--- /dev/null
+++ b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MaterializedViewClusterIntegrationTest.java
@@ -0,0 +1,1167 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.integration.tests;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.io.File;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.helix.store.HelixPropertyStore;
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.pinot.common.utils.TarCompressionUtils;
+import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
+import org.apache.pinot.materializedview.metadata.MaterializedViewDefinitionMetadata;
+import org.apache.pinot.materializedview.metadata.MaterializedViewDefinitionMetadata.MaterializedViewSplitSpec;
+import org.apache.pinot.materializedview.metadata.MaterializedViewDefinitionMetadataUtils;
+import org.apache.pinot.materializedview.metadata.MaterializedViewRuntimeMetadata;
+import org.apache.pinot.materializedview.metadata.MaterializedViewRuntimeMetadataUtils;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.readers.GenericRowRecordReader;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.CommonConstants.Broker;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.util.TestUtils;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.*;
+
+
+/// End-to-end integration tests for the Materialized View (MV) query rewrite pipeline.
+///
+/// <p>These tests bypass the MV task generator/executor and directly set up the MV
+/// infrastructure (MV table, segments, ZK metadata) to validate the broker's query
+/// rewrite engine in a real cluster environment.
+///
+/// <p>Phase 1 covers core rewrite scenarios:
+/// <ol>
+/// <li>{@link #testFullRewriteWithExactMatch()} — MV covers full data range,
+/// {@code ExactSubsumptionStrategy} rewrites the query to the MV table.</li>
+/// <li>{@link #testSplitRewriteWithAggReagg()} — MV covers a historical range,
+/// {@code AggregationSubsumptionStrategy} merges MV + base table in split mode.</li>
+/// <li>{@link #testColdStartSkip()} — MV exists but {@code watermarkMs == 0},
+/// verifying the broker skips the MV during cold-start.</li>
+/// </ol>
+///
+/// <p>Phase 2 covers incremental updates and consistency:
+/// <ol>
+/// <li>{@link #testIncrementalAppend()} — advance split MV coverage via new segments
+/// and ZK metadata update; verify the expanded coverage is respected.</li>
+/// <li>{@link #testFreshnessGating()} — tighten the per-MV {@code stalenessThresholdMs} SLO
+/// on the definition → broker skips the MV; relax it → broker hits it again.</li>
+/// <li>{@link #testMaterializedViewDefinitionDeletion()} — delete MV definition from ZK → broker
+/// no longer considers the MV as a candidate.</li>
+/// </ol>
+///
+/// <p>Phase 3 covers edge cases and advanced matching:
+/// <ol>
+/// <li>{@link #testScanSubsumptionWithResidualFilter()} — SCAN-shaped MV with a
+/// user WHERE clause that becomes a residual filter on the MV table.</li>
+/// <li>{@link #testQueryWithLimitOffsetOrderBy()} — split mode query with
+/// ORDER BY + LIMIT + OFFSET; verify correct pagination and ordering.</li>
+/// <li>{@link #testMultipleMaterializedViewCostSelection()} — two MVs match the same query at
+/// different costs; verify the lower-cost MV is selected.</li>
+/// </ol>
+///
+/// <p><b>Why this extends {@link BaseClusterIntegrationTest} directly instead of
+/// {@link org.apache.pinot.integration.tests.custom.CustomDataQueryClusterIntegrationTest}:</b>
+/// {@code CustomDataQueryClusterIntegrationTest} runs all sub-tests against a single
+/// suite-level shared cluster that is started once in {@code @BeforeSuite}. That shared
+/// cluster is created before any sub-test has a chance to call {@link #overrideBrokerConf},
+/// so {@code CONFIG_OF_BROKER_QUERY_ENABLE_MATERIALIZED_VIEW_REWRITE} cannot be applied to
+/// it. MV rewrite is disabled by default; without a broker started with that flag the entire
+/// test suite would be a no-op. A dedicated cluster started in {@code @BeforeClass} is
+/// therefore required until the shared cluster supports per-test broker-config overrides.
+public class MaterializedViewClusterIntegrationTest extends BaseClusterIntegrationTest {
+
+ private static final String SOURCE_TABLE_NAME = "materializedViewSourceTable";
+ private static final String MATERIALIZED_VIEW_FULL_TABLE_NAME = "materializedViewFullTable";
+ private static final String MATERIALIZED_VIEW_SPLIT_TABLE_NAME = "materializedViewSplitTable";
+ private static final String MATERIALIZED_VIEW_COLD_TABLE_NAME = "materializedViewColdTable";
+ private static final String MATERIALIZED_VIEW_SCAN_TABLE_NAME = "materializedViewScanTable";
+ private static final String MATERIALIZED_VIEW_COST_TABLE_NAME = "materializedViewCostTable";
+
+ private static final String MATERIALIZED_VIEW_FULL_TABLE_OFFLINE = MATERIALIZED_VIEW_FULL_TABLE_NAME + "_OFFLINE";
+ private static final String MATERIALIZED_VIEW_SPLIT_TABLE_OFFLINE = MATERIALIZED_VIEW_SPLIT_TABLE_NAME + "_OFFLINE";
+ private static final String MATERIALIZED_VIEW_COLD_TABLE_OFFLINE = MATERIALIZED_VIEW_COLD_TABLE_NAME + "_OFFLINE";
+ private static final String MATERIALIZED_VIEW_SCAN_TABLE_OFFLINE = MATERIALIZED_VIEW_SCAN_TABLE_NAME + "_OFFLINE";
+ private static final String MATERIALIZED_VIEW_COST_TABLE_OFFLINE = MATERIALIZED_VIEW_COST_TABLE_NAME + "_OFFLINE";
+
+ private static final String TIME_COLUMN = "DaysSinceEpoch";
+
+ /// Airline dataset spans DaysSinceEpoch ~16071–16101 (days since epoch)
+ /// 16071 * 86400000 = 1_388_534_400_000 (Jan 1, 2014)
+ /// 16102 * 86400000 = 1_391_212_800_000 (Feb 1, 2014)
+ private static final long DATA_MIN_TIME_MS = 16071L * 86_400_000L;
+ private static final long DATA_MAX_TIME_MS = 16102L * 86_400_000L;
+
+ /// Split boundary: MV covers first ~15 days, base table covers the rest
+ private static final long SPLIT_BOUNDARY_MS = 16086L * 86_400_000L;
+
+ private static final String[] CARRIERS = {"AA", "DL", "UA", "WN", "US", "B6", "OO", "EV", "MQ", "NK"};
+
+ private PinotHelixResourceManager _helixResourceManager;
+ private HelixPropertyStore<ZNRecord> _propertyStore;
+
+ @Override
+ protected String getTableName() {
+ return SOURCE_TABLE_NAME;
+ }
+
+ @Override
+ @Nullable
+ protected String getSortedColumn() {
+ return null;
+ }
+
+ @Override
+ @Nullable
+ protected List<String> getInvertedIndexColumns() {
+ return null;
+ }
+
+ @Override
+ @Nullable
+ protected List<String> getNoDictionaryColumns() {
+ return null;
+ }
+
+ @Override
+ @Nullable
+ protected List<String> getRangeIndexColumns() {
+ return null;
+ }
+
+ @Override
+ @Nullable
+ protected List<String> getBloomFilterColumns() {
+ return null;
+ }
+
+ @Override
+ protected void overrideBrokerConf(PinotConfiguration brokerConf) {
+ brokerConf.setProperty(Broker.CONFIG_OF_BROKER_QUERY_ENABLE_MATERIALIZED_VIEW_REWRITE, true);
+ }
+
+ @BeforeClass
+ public void setUp()
+ throws Exception {
+ TestUtils.ensureDirectoriesExistAndEmpty(_tempDir, _segmentDir, _tarDir);
+
+ startZk();
+ startController();
+ startBroker();
+ startServer();
+
+ _helixResourceManager = _controllerStarter.getHelixResourceManager();
+ _propertyStore = _helixResourceManager.getPropertyStore();
+
+ /// --- Source table: load airline data ---
+ Schema sourceSchema = createSchema();
+ sourceSchema.setSchemaName(SOURCE_TABLE_NAME);
+ addSchema(sourceSchema);
+
+ TableConfig sourceTableConfig = new TableConfigBuilder(TableType.OFFLINE)
+ .setTableName(SOURCE_TABLE_NAME)
+ .setTimeColumnName(TIME_COLUMN)
+ .setNumReplicas(1)
+ .build();
+ addTableConfig(sourceTableConfig);
+
+ List<File> avroFiles = unpackAvroData(_tempDir);
+ ClusterIntegrationTestUtils.buildSegmentsFromAvro(avroFiles, sourceTableConfig, sourceSchema,
+ 0, _segmentDir, _tarDir);
+ uploadSegments(SOURCE_TABLE_NAME, _tarDir);
+ waitForAllDocsLoaded(600_000L);
+
+ /// --- MV tables: build with synthetic pre-aggregated data ---
+ setupFullRewriteMv();
+ setupSplitRewriteMv();
+ setupColdStartMv();
+ setupScanMv();
+ setupCostCompetitorMv();
+
+ /// Wait for the broker's MaterializedViewMetadataCache to register every newly-published MV
+ /// by polling a sentinel query that should be served by the full-rewrite MV. Polling on a
+ /// real query (rather than `Thread.sleep`) lets the test progress as soon as the ZK watchers
+ /// catch up — fast machines no longer pay 5s of wall clock and slow CI machines no longer
+ /// race the timeout.
+ waitForMaterializedViewRegistered(MATERIALIZED_VIEW_FULL_TABLE_OFFLINE,
+ "SELECT Carrier, SUM(ArrDelayMinutes) FROM " + SOURCE_TABLE_NAME + " GROUP BY Carrier");
+ }
+
+ /// Polls a query that is known to be rewritable to the given MV until the broker's metadata
+ /// cache has caught up. Used in place of `Thread.sleep` after publishing MV znodes from a
+ /// test setup step.
+ private void waitForMaterializedViewRegistered(String expectedMaterializedViewOfflineTable, String sentinelQuery) {
+ TestUtils.waitForCondition(() -> {
+ try {
+ JsonNode response = postQuery(sentinelQuery);
+ String materializedViewQueried = getMaterializedViewQueried(response);
+ return expectedMaterializedViewOfflineTable.equals(materializedViewQueried);
+ } catch (Exception e) {
+ return false;
+ }
+ }, 100L, 30_000L,
+ "MV " + expectedMaterializedViewOfflineTable + " not registered with the broker cache",
+ Duration.ofSeconds(6));
+ }
+
+ @AfterClass
+ public void tearDown()
+ throws Exception {
+ /// Drop MV metadata FIRST so the controller's MV-dependency check on the base table allows
+ /// the source-table drop below. The controller refuses to delete a base table while MV
+ /// definition znodes still reference it.
+ cleanupMaterializedViewMetadata(MATERIALIZED_VIEW_FULL_TABLE_OFFLINE);
+ cleanupMaterializedViewMetadata(MATERIALIZED_VIEW_SPLIT_TABLE_OFFLINE);
+ cleanupMaterializedViewMetadata(MATERIALIZED_VIEW_COLD_TABLE_OFFLINE);
+ cleanupMaterializedViewMetadata(MATERIALIZED_VIEW_SCAN_TABLE_OFFLINE);
+ cleanupMaterializedViewMetadata(MATERIALIZED_VIEW_COST_TABLE_OFFLINE);
+
+ /// Drop MV tables before the base table so the controller's referential-integrity check
+ /// (introduced in pinot-controller to prevent orphaned MVs) does not block the base drop.
+ dropOfflineTable(MATERIALIZED_VIEW_FULL_TABLE_NAME);
+ dropOfflineTable(MATERIALIZED_VIEW_SPLIT_TABLE_NAME);
+ dropOfflineTable(MATERIALIZED_VIEW_COLD_TABLE_NAME);
+ dropOfflineTable(MATERIALIZED_VIEW_SCAN_TABLE_NAME);
+ dropOfflineTable(MATERIALIZED_VIEW_COST_TABLE_NAME);
+ dropOfflineTable(SOURCE_TABLE_NAME);
+
+ stopServer();
+ stopBroker();
+ stopController();
+ stopZk();
+ FileUtils.deleteDirectory(_tempDir);
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Phase 1, Test 1: Full rewrite with ExactSubsumptionStrategy
+ /// -----------------------------------------------------------------------
+
+ /// Verifies that when an MV fully covers the source data and the query matches
+ /// the MV definition exactly, the broker rewrites the query to hit the MV table
+ /// via {@code FULL_REWRITE} execution mode.
+ @Test
+ public void testFullRewriteWithExactMatch()
+ throws Exception {
+ String query = ""
+ + "SELECT Carrier, SUM(ArrDelayMinutes) AS sum_ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME + " "
+ + "GROUP BY Carrier";
+ JsonNode response = postQuery(query);
+ assertNoExceptions(response);
+
+ String materializedViewQueried = getMaterializedViewQueried(response);
+ assertNotNull(materializedViewQueried, "Expected MV to be hit for exact-match query. Response: " + response);
+ assertEquals(materializedViewQueried, MATERIALIZED_VIEW_FULL_TABLE_OFFLINE,
+ "Expected materializedViewQueried to be the full-rewrite MV table");
+
+ JsonNode resultTable = response.get("resultTable");
+ assertNotNull(resultTable, "resultTable should not be null");
+ assertTrue(resultTable.get("rows").size() > 0, "Result should have rows");
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Phase 1, Test 2: Split rewrite with AggregationSubsumptionStrategy
+ /// -----------------------------------------------------------------------
+
+ /// Verifies split-mode execution with re-aggregation. The MV covers only
+ /// historical data (up to {@code SPLIT_BOUNDARY_MS}). The broker merges
+ /// MV results with recent base table data.
+ ///
+ /// <p>Uses a query with {@code Origin} in GROUP BY, which matches the split
+ /// MV definition but not the full MV definition, ensuring this test targets
+ /// the split MV specifically.
+ @Test
+ public void testSplitRewriteWithAggReagg()
+ throws Exception {
+ /// This query matches the split MV (which groups by DaysSinceEpoch, Carrier, Origin)
+ /// via AggregationSubsumptionStrategy, but does NOT match the full MV (which only
+ /// groups by Carrier). The user query groups by (Carrier, Origin) which is a subset
+ /// of the split MV's (DaysSinceEpoch, Carrier, Origin), enabling re-aggregation.
+ String query = ""
+ + "SELECT Carrier, Origin, SUM(ArrDelayMinutes) AS sum_ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME + " "
+ + "GROUP BY Carrier, Origin";
+ JsonNode materializedViewResponse = postQuery(query);
+ assertNoExceptions(materializedViewResponse);
+
+ String materializedViewQueried = getMaterializedViewQueried(materializedViewResponse);
+ assertNotNull(materializedViewQueried,
+ "Expected materialized view to be hit for split-mode query. Response: " + materializedViewResponse);
+ assertEquals(materializedViewQueried, MATERIALIZED_VIEW_SPLIT_TABLE_OFFLINE,
+ "Expected materializedViewQueried to be the split-rewrite MV table");
+
+ JsonNode resultTable = materializedViewResponse.get("resultTable");
+ assertNotNull(resultTable, "resultTable should not be null");
+ assertTrue(resultTable.get("rows").size() > 0, "Result should have rows");
+
+ /// Verify result count matches direct query
+ String directQuery = "SELECT Carrier, Origin, SUM(ArrDelayMinutes) AS sum_ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME + " GROUP BY Carrier, Origin";
+ JsonNode directResponse = postQuery(directQuery);
+ assertNoExceptions(directResponse);
+
+ int directRowCount = directResponse.get("resultTable").get("rows").size();
+ int materializedViewRowCount = resultTable.get("rows").size();
+ assertEquals(materializedViewRowCount, directRowCount,
+ "Split MV query should return same number of groups as direct query");
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Phase 1, Test 3: Cold-start skip
+ /// -----------------------------------------------------------------------
+
+ /// Verifies that the broker skips an MV with `watermarkMs == 0`,
+ /// simulating a cold-start where the metadata exists but no APPEND has completed.
+ ///
+ /// The cold-start MV has the same definition as the split MV (grouping by
+ /// DaysSinceEpoch, Carrier, Origin). We use a query with `Carrier, Origin`
+ /// in GROUP BY, which would match both the split MV and the cold-start MV via
+ /// [AggregationSubsumptionStrategy]. The cold-start MV should be skipped
+ /// because its `watermarkMs == 0`, and the split MV should be hit instead.
+ @Test
+ public void testColdStartSkip()
+ throws Exception {
+ /// This query matches both the split MV and the cold-start MV structurally.
+ /// Only the split MV should be hit; the cold-start MV should be skipped.
+ String query = ""
+ + "SELECT Carrier, Origin, SUM(ArrDelayMinutes) AS sum_ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME + " "
+ + "GROUP BY Carrier, Origin";
+ JsonNode response = postQuery(query);
+ assertNoExceptions(response);
+
+ String materializedViewQueried = getMaterializedViewQueried(response);
+ assertNotNull(materializedViewQueried, "Expected an MV to be hit. Response: " + response);
+ assertNotEquals(materializedViewQueried, MATERIALIZED_VIEW_COLD_TABLE_OFFLINE,
+ "Cold-start MV should NOT be hit (watermarkMs == 0)");
+ assertEquals(materializedViewQueried, MATERIALIZED_VIEW_SPLIT_TABLE_OFFLINE,
+ "Expected the split MV to be hit instead of the cold-start MV");
+
+ JsonNode resultTable = response.get("resultTable");
+ assertNotNull(resultTable, "resultTable should not be null");
+ assertTrue(resultTable.get("rows").size() > 0,
+ "Query should still return results via the split MV");
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Phase 2, Test 4: Incremental append — coverage boundary advances
+ /// -----------------------------------------------------------------------
+
+ /// Simulates an incremental APPEND by uploading additional MV segments that cover
+ /// days 16086–16095 (previously only 16071–16085 were materialized), then advancing
+ /// `watermarkMs` in ZK. Verifies that the broker respects the expanded
+ /// coverage boundary and still produces correct split-mode results.
+ ///
+ /// This test depends on the split MV state established by [#setUp()].
+ @Test(dependsOnMethods = "testSplitRewriteWithAggReagg")
+ public void testIncrementalAppend()
+ throws Exception {
+ long extendedBoundaryMs = 16096L * 86_400_000L;
+
+ /// Upload additional MV segments for the newly materialized time window
+ Schema materializedViewSchema = new Schema.SchemaBuilder()
+ .setSchemaName(MATERIALIZED_VIEW_SPLIT_TABLE_NAME)
+ .addDateTime(TIME_COLUMN, FieldSpec.DataType.INT, "1:DAYS:EPOCH", "1:DAYS")
+ .addSingleValueDimension("Carrier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("Origin", FieldSpec.DataType.STRING)
+ .addMetric("sum_ArrDelayMinutes", FieldSpec.DataType.DOUBLE)
+ .build();
+
+ TableConfig materializedViewTableConfig = new TableConfigBuilder(TableType.OFFLINE)
+ .setTableName(MATERIALIZED_VIEW_SPLIT_TABLE_NAME)
+ .setTimeColumnName(TIME_COLUMN)
+ .setNumReplicas(1)
+ .build();
+
+ String[] origins = {"SFO", "LAX", "JFK", "ORD", "ATL"};
+ List<GenericRow> rows = new ArrayList<>();
+ for (int day = 16086; day < 16096; day++) {
+ for (String carrier : CARRIERS) {
+ for (String origin : origins) {
+ GenericRow row = new GenericRow();
+ row.putValue(TIME_COLUMN, day);
+ row.putValue("Carrier", carrier);
+ row.putValue("Origin", origin);
+ row.putValue("sum_ArrDelayMinutes",
+ 100.0 + (day + carrier.hashCode() + origin.hashCode()) % 200);
+ rows.add(row);
+ }
+ }
+ }
+ buildAndUploadSegment(materializedViewTableConfig, materializedViewSchema, rows,
+ MATERIALIZED_VIEW_SPLIT_TABLE_NAME, "materializedViewSplitSeg2");
+
+ long previousCount = getCurrentCountStarResult(MATERIALIZED_VIEW_SPLIT_TABLE_NAME);
+ TestUtils.waitForCondition(
+ () -> getCurrentCountStarResult(MATERIALIZED_VIEW_SPLIT_TABLE_NAME) > previousCount,
+ 100L, 60_000L, "New MV split segments not loaded", Duration.ofSeconds(6));
+
+ /// Advance watermarkMs in ZK to reflect the newly materialized window
+ MaterializedViewRuntimeMetadata updatedRuntime = new MaterializedViewRuntimeMetadata(
+
+ MATERIALIZED_VIEW_SPLIT_TABLE_OFFLINE, extendedBoundaryMs, new HashMap<>());
+ MaterializedViewRuntimeMetadataUtils.persist(_propertyStore, updatedRuntime, -1);
+
+ /// Wait for ZK watcher to propagate the watermark update to the broker cache. Poll a
+ /// sentinel query that should be served by the split MV; the watermark advance is
+ /// observable indirectly because the rewrite path keeps choosing the split MV (which
+ /// depends on the updated watermark for the split-boundary filter).
+ waitForMaterializedViewRegistered(MATERIALIZED_VIEW_SPLIT_TABLE_OFFLINE,
+ "SELECT Carrier, Origin, SUM(ArrDelayMinutes) FROM " + SOURCE_TABLE_NAME + " GROUP BY Carrier, Origin");
+
+ /// Query and verify: the split MV should still be hit with the advanced boundary
+ String query = ""
+ + "SELECT Carrier, Origin, SUM(ArrDelayMinutes) AS sum_ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME + " "
+ + "GROUP BY Carrier, Origin";
+ JsonNode response = postQuery(query);
+ assertNoExceptions(response);
+
+ String materializedViewQueried = getMaterializedViewQueried(response);
+ assertNotNull(materializedViewQueried, "MV should be hit after incremental append. Response: " + response);
+ assertEquals(materializedViewQueried, MATERIALIZED_VIEW_SPLIT_TABLE_OFFLINE);
+
+ JsonNode resultTable = response.get("resultTable");
+ assertNotNull(resultTable);
+ assertTrue(resultTable.get("rows").size() > 0);
+
+ /// Compare with direct query to verify correctness
+ String directQuery = "SELECT Carrier, Origin, SUM(ArrDelayMinutes) AS sum_ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME + " GROUP BY Carrier, Origin";
+ JsonNode directResponse = postQuery(directQuery);
+ assertNoExceptions(directResponse);
+ assertEquals(
+ resultTable.get("rows").size(),
+ directResponse.get("resultTable").get("rows").size(),
+ "Row count should match direct query after incremental append");
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Correctness: non-atomic endSegmentReplace <-> watermarkMs publish window
+ /// -----------------------------------------------------------------------
+
+ /// Reproduces the non-atomic publish window between the minion executor's
+ /// `endSegmentReplace` (new MV segments become queryable on servers) and
+ /// `postProcess` (ZK `watermarkMs` advances). During that window
+ /// the broker observes a stale `watermarkMs = W_old` while the MV
+ /// table physically contains already-materialized rows in `[W_old, W_new)`.
+ ///
+ /// Before the `materializedViewTime < watermarkMs` filter was attached on the MV branch:
+ ///
+ /// - MV branch: reads rows `[0, W_new)` (no upper bound) — includes the new,
+ /// not-yet-published range.
+ /// - Base branch: reads rows `[W_old, +inf)` — also includes the new range.
+ ///
+ /// The overlap `[W_old, W_new)` is double-counted in the merged result.
+ ///
+ /// This test simulates that exact window by uploading MV segments covering
+ /// a range beyond the current `watermarkMs` without advancing it,
+ /// and asserts that the MV rewrite still produces the same sums as a baseline
+ /// query taken before the overlap segments existed — i.e., the MV branch
+ /// correctly excludes the not-yet-published range.
+ ///
+ /// Depends on [#testIncrementalAppend()] which leaves the split MV
+ /// covering days 16071–16095 and `watermarkMs = 16096 * day`.
+ @Test(dependsOnMethods = "testIncrementalAppend")
+ public void testSplitCorrectnessDuringNonAtomicPublishWindow()
+ throws Exception {
+ /// NOTE: explicit ORDER BY on grouping keys + large LIMIT is required here.
+ /// Without them we inherit the default broker LIMIT (10) with no ORDER BY, so the
+ /// "top 10" groups are picked by server-side iteration order. Adding new MV segments
+ /// perturbs segment iteration / group-merge order even when the materializedViewTime < watermarkMs
+ /// filter correctly excludes every row in those segments, which would make this test
+ /// flaky (keySet mismatch) for reasons unrelated to the CRITICAL #1 invariant we are
+ /// actually verifying. Sorting by the grouping keys and asking for more rows than the
+ /// real cardinality makes the result deterministic regardless of segment layout.
+ String query = ""
+ + "SELECT Carrier, Origin, SUM(ArrDelayMinutes) AS sum_ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME + " "
+ + "GROUP BY Carrier, Origin "
+ + "ORDER BY Carrier, Origin "
+ + "LIMIT 10000";
+
+ /// Step 1: baseline — split query before the simulated overlap.
+ /// MV has days 16071..16095, watermarkMs = 16096 * day.
+ JsonNode baseline = postQuery(query);
+ assertNoExceptions(baseline);
+ /// Verify the split MV is actually being hit — if testIncrementalAppend did not run (e.g.
+ /// test was invoked in isolation), this will produce a clear skip-style failure here rather
+ /// than a misleading assertion failure deeper in the test.
+ assertEquals(getMaterializedViewQueried(baseline), MATERIALIZED_VIEW_SPLIT_TABLE_OFFLINE,
+ "Precondition failed: split MV must be hit. "
+ + "Ensure testIncrementalAppend ran first (dependsOnMethods contract).");
+ Map<String, Double> baselineSums = collectSumByCarrierOrigin(baseline);
+ assertFalse(baselineSums.isEmpty(), "Baseline must contain grouped rows");
+
+ /// Step 2: simulate endSegmentReplace without postProcess — upload MV segments
+ /// for days 16096..16100 (ALREADY physically queryable on servers), but leave
+ /// watermarkMs at 16096 in ZK (postProcess has not yet run).
+ Schema materializedViewSchema = new Schema.SchemaBuilder()
+ .setSchemaName(MATERIALIZED_VIEW_SPLIT_TABLE_NAME)
+ .addDateTime(TIME_COLUMN, FieldSpec.DataType.INT, "1:DAYS:EPOCH", "1:DAYS")
+ .addSingleValueDimension("Carrier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("Origin", FieldSpec.DataType.STRING)
+ .addMetric("sum_ArrDelayMinutes", FieldSpec.DataType.DOUBLE)
+ .build();
+ TableConfig materializedViewTableConfig = new TableConfigBuilder(TableType.OFFLINE)
+ .setTableName(MATERIALIZED_VIEW_SPLIT_TABLE_NAME)
+ .setTimeColumnName(TIME_COLUMN)
+ .setNumReplicas(1)
+ .build();
+
+ String[] origins = {"SFO", "LAX", "JFK", "ORD", "ATL"};
+ List<GenericRow> rows = new ArrayList<>();
+ for (int day = 16096; day <= 16100; day++) {
+ for (String carrier : CARRIERS) {
+ for (String origin : origins) {
+ GenericRow row = new GenericRow();
+ row.putValue(TIME_COLUMN, day);
+ row.putValue("Carrier", carrier);
+ row.putValue("Origin", origin);
+ /// Use large, distinctive sums so any leak into the merged result is
+ /// trivially detectable — not a rounding-level perturbation.
+ row.putValue("sum_ArrDelayMinutes", 1_000_000.0);
+ rows.add(row);
+ }
+ }
+ }
+ long beforeCount = getCurrentCountStarResult(MATERIALIZED_VIEW_SPLIT_TABLE_NAME);
+ buildAndUploadSegment(materializedViewTableConfig, materializedViewSchema, rows,
+ MATERIALIZED_VIEW_SPLIT_TABLE_NAME, "materializedViewSplitOverlap");
+ TestUtils.waitForCondition(
+ () -> getCurrentCountStarResult(MATERIALIZED_VIEW_SPLIT_TABLE_NAME) > beforeCount,
+ 100L, 60_000L, "Overlap MV segments not visible on servers", Duration.ofSeconds(6));
+
+ /// Intentionally NO MaterializedViewRuntimeMetadataUtils.persist here — this is the whole
+ /// point of the test: segments are live, ZK watermark is stale.
+
+ /// Step 3: rerun the split query. With the materializedViewTime < watermarkMs filter
+ /// on the MV branch, the newly uploaded "leak" segments are excluded and
+ /// sums must equal the baseline. Without the fix, each group's sum would be
+ /// inflated by 1_000_000 * 5 days = 5_000_000 relative to the baseline.
+ JsonNode after = postQuery(query);
+ assertNoExceptions(after);
+ assertEquals(getMaterializedViewQueried(after), MATERIALIZED_VIEW_SPLIT_TABLE_OFFLINE,
+ "Split MV must still be hit after overlap segments are published");
+
+ Map<String, Double> afterSums = collectSumByCarrierOrigin(after);
+ assertEquals(afterSums.keySet(), baselineSums.keySet(),
+ "Group keys must not change when new MV segments are published beyond coverage");
+ for (Map.Entry<String, Double> e : baselineSums.entrySet()) {
+ double expected = e.getValue();
+ double actual = afterSums.get(e.getKey());
+ /// Allow tiny floating-point drift; any leak is at least 1_000_000 per affected group.
+ assertEquals(actual, expected, 1e-6,
+ "MV branch must exclude segments beyond watermarkMs (non-atomic publish window). "
+ + "Group=" + e.getKey() + ", baseline=" + expected + ", after=" + actual
+ + ". A large delta here means the MV branch read rows in [watermarkMs, newMax), "
+ + "double-counting with the base branch.");
+ }
+ }
+
+ /// Extracts (Carrier|Origin -> sum_ArrDelayMinutes) from a grouped response.
+ private static Map<String, Double> collectSumByCarrierOrigin(JsonNode response) {
+ Map<String, Double> result = new HashMap<>();
+ JsonNode rowsNode = response.get("resultTable").get("rows");
+ for (int i = 0; i < rowsNode.size(); i++) {
+ JsonNode row = rowsNode.get(i);
+ String key = row.get(0).asText() + "|" + row.get(1).asText();
+ result.put(key, row.get(2).asDouble());
+ }
+ return result;
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Phase 2, Test 5: Staleness SLO gating — broker skips MV when SLO trips
+ /// -----------------------------------------------------------------------
+
+ /// Verifies that the broker skips an MV when its staleness SLO is violated
+ /// (`now - watermarkMs > stalenessThresholdMs`), and resumes using it once
+ /// the SLO is disabled or relaxed.
+ ///
+ /// Under V2, freshness is not a persistent enum on the runtime ZNode — it is derived
+ /// on read from `stalenessThresholdMs` on the MV definition. This test exercises
+ /// the SLO branch of [MaterializedViewQueryRewriteEngine#isEligible]. The full
+ /// MV's watermark is fixed at `DATA_MAX_TIME_MS` (an ancient timestamp), so any
+ /// positive SLO of reasonable size will be exceeded.
+ @Test(dependsOnMethods = "testFullRewriteWithExactMatch")
+ public void testFreshnessGating()
+ throws Exception {
+ String query = ""
+ + "SELECT Carrier, SUM(ArrDelayMinutes) AS sum_ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME + " GROUP BY Carrier";
+ String definedSql = "SELECT Carrier, SUM(ArrDelayMinutes) AS sum_ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME + " GROUP BY Carrier";
+
+ /// Step 1: Verify MV is currently hit (precondition; SLO disabled).
+ JsonNode preResponse = postQuery(query);
+ assertNoExceptions(preResponse);
+ assertEquals(getMaterializedViewQueried(preResponse), MATERIALIZED_VIEW_FULL_TABLE_OFFLINE,
+ "Precondition: full MV should be hit before SLO is tightened");
+
+ /// Step 2: Tighten the SLO on the MV definition. watermarkMs = DATA_MAX_TIME_MS is a
+ /// 2014-era epoch, so any small positive threshold trips `(now - watermarkMs) > threshold`.
+ MaterializedViewDefinitionMetadata staleDefinition = new MaterializedViewDefinitionMetadata(
+ MATERIALIZED_VIEW_FULL_TABLE_OFFLINE,
+ Collections.singletonList(SOURCE_TABLE_NAME),
+ definedSql,
+ Collections.emptyMap(),
+ null,
+ 1L,
+ true);
+ MaterializedViewDefinitionMetadataUtils.persist(_propertyStore, staleDefinition, -1);
+
+ /// Wait for the broker cache to observe the tightened SLO by polling until the MV stops
+ /// being chosen for the query.
+ TestUtils.waitForCondition(() -> {
+ try {
+ String mv = getMaterializedViewQueried(postQuery(query));
+ return mv == null || !MATERIALIZED_VIEW_FULL_TABLE_OFFLINE.equals(mv);
+ } catch (Exception e) {
+ return false;
+ }
+ }, 100L, 30_000L, "Broker did not pick up the tightened staleness SLO", Duration.ofSeconds(6));
+
+ /// Step 3: Query again — the MV should NOT be hit (SLO trips eligibility gate).
+ JsonNode staleResponse = postQuery(query);
+ assertNoExceptions(staleResponse);
+ String staleMaterializedViewQueried = getMaterializedViewQueried(staleResponse);
+ if (staleMaterializedViewQueried != null) {
+ assertNotEquals(staleMaterializedViewQueried, MATERIALIZED_VIEW_FULL_TABLE_OFFLINE,
+ "MV with violated staleness SLO should NOT be hit");
+ }
+
+ /// Query should still return results (broker falls back to base table).
+ JsonNode staleResultTable = staleResponse.get("resultTable");
+ assertNotNull(staleResultTable);
+ assertTrue(staleResultTable.get("rows").size() > 0,
+ "Query should still return results when MV is excluded by SLO");
+
+ /// Step 4: Disable the SLO again by setting stalenessThresholdMs back to 0.
+ MaterializedViewDefinitionMetadata freshDefinition = new MaterializedViewDefinitionMetadata(
+ MATERIALIZED_VIEW_FULL_TABLE_OFFLINE,
+ Collections.singletonList(SOURCE_TABLE_NAME),
+ definedSql,
+ Collections.emptyMap(),
+ null,
+ 0L,
+ true);
+ MaterializedViewDefinitionMetadataUtils.persist(_propertyStore, freshDefinition, -1);
+
+ /// Wait for the broker cache to observe the relaxed SLO; poll until the MV is chosen
+ /// again rather than fixed-sleeping.
+ waitForMaterializedViewRegistered(MATERIALIZED_VIEW_FULL_TABLE_OFFLINE, query);
+
+ /// Step 5: Query again — the MV should be hit again.
+ JsonNode freshResponse = postQuery(query);
+ assertNoExceptions(freshResponse);
+ assertEquals(getMaterializedViewQueried(freshResponse), MATERIALIZED_VIEW_FULL_TABLE_OFFLINE,
+ "MV should be hit again after SLO is disabled");
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Phase 2, Test 6: MV definition deletion — broker stops rewriting
+ /// -----------------------------------------------------------------------
+
+ /// Verifies that when an MV's definition is deleted from ZooKeeper, the broker
+ /// removes it from the cache and other MVs continue to serve queries.
+ @Test(dependsOnMethods = "testColdStartSkip")
+ public void testMaterializedViewDefinitionDeletion()
+ throws Exception {
+ String query = ""
+ + "SELECT Carrier, Origin, SUM(ArrDelayMinutes) AS sum_ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME + " GROUP BY Carrier, Origin";
+
+ /// Precondition: split MV is currently the chosen MV for this query.
+ JsonNode preResponse = postQuery(query);
+ assertNoExceptions(preResponse);
+ assertEquals(getMaterializedViewQueried(preResponse), MATERIALIZED_VIEW_SPLIT_TABLE_OFFLINE,
+ "Precondition: split MV should be queried for this aggregation");
+
+ /// Delete the cold MV's definition + runtime from ZK.
+ MaterializedViewDefinitionMetadataUtils.delete(_propertyStore, MATERIALIZED_VIEW_COLD_TABLE_OFFLINE);
+ MaterializedViewRuntimeMetadataUtils.delete(_propertyStore, MATERIALIZED_VIEW_COLD_TABLE_OFFLINE);
+
+ /// Wait for the broker cache to observe the deletion by polling until the cold MV is no
+ /// longer chosen for a query that uniquely matches it (no-op here because split MV is the
+ /// only candidate for the test query, but the poll exits as soon as the listener fires and
+ /// any subsequent query against COLD's distinctive shape returns null).
+ waitForMaterializedViewRegistered(MATERIALIZED_VIEW_SPLIT_TABLE_OFFLINE, query);
+
+ /// After deletion the split MV should still serve the query unaffected.
+ JsonNode postResponse = postQuery(query);
+ assertNoExceptions(postResponse);
+ String materializedViewQueried = getMaterializedViewQueried(postResponse);
+ assertNotNull(materializedViewQueried, "Split MV should still be available. Response: " + postResponse);
+ assertEquals(materializedViewQueried, MATERIALIZED_VIEW_SPLIT_TABLE_OFFLINE);
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Phase 3, Test 7: Scan subsumption with residual WHERE filter
+ /// -----------------------------------------------------------------------
+
+ /// Verifies that a SCAN-shaped MV (no GROUP BY, no aggregation) is matched by
+ /// {@code ScanSubsumptionStrategy} when the user query selects a subset of the
+ /// MV's columns and adds a WHERE filter that references only MV columns.
+ ///
+ /// <p>The scan MV stores {@code Carrier, Origin, Dest, ArrDelayMinutes} from the
+ /// source table. The user query selects {@code Carrier, ArrDelayMinutes} with
+ /// {@code WHERE Origin = 'SFO'} — the WHERE clause becomes a residual filter
+ /// on the MV table.
+ @Test
+ public void testScanSubsumptionWithResidualFilter()
+ throws Exception {
+ /// Query with WHERE filter — should hit the scan MV via ScanSubsumptionStrategy
+ String query = ""
+ + "SELECT Carrier, ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME + " "
+ + "WHERE Origin = 'SFO'";
+ JsonNode materializedViewResponse = postQuery(query);
+ assertNoExceptions(materializedViewResponse);
+
+ String materializedViewQueried = getMaterializedViewQueried(materializedViewResponse);
+ assertNotNull(materializedViewQueried, "Expected scan MV to be hit. Response: " + materializedViewResponse);
+ assertEquals(materializedViewQueried, MATERIALIZED_VIEW_SCAN_TABLE_OFFLINE,
+ "Expected materializedViewQueried to be the scan MV table");
+
+ JsonNode resultTable = materializedViewResponse.get("resultTable");
+ assertNotNull(resultTable, "resultTable should not be null");
+ assertTrue(resultTable.get("rows").size() > 0, "Result should have rows");
+
+ /// Verify result count matches direct query (without MV rewrite)
+ String directQuery = "SELECT Carrier, ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME + " WHERE Origin = 'SFO'";
+ JsonNode directResponse = postQuery(directQuery);
+ assertNoExceptions(directResponse);
+
+ assertEquals(
+ resultTable.get("rows").size(),
+ directResponse.get("resultTable").get("rows").size(),
+ "Scan MV query should return same row count as direct query");
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Phase 3, Test 8: Split mode with ORDER BY + LIMIT + OFFSET
+ /// -----------------------------------------------------------------------
+
+ /// Verifies that a split-mode query with ORDER BY, LIMIT, and OFFSET produces
+ /// correct results. The broker must collect enough rows from both MV and base
+ /// table sides to satisfy the OFFSET + LIMIT requirement, then apply final
+ /// ordering and pagination during the merge phase.
+ @Test
+ public void testQueryWithLimitOffsetOrderBy()
+ throws Exception {
+ String materializedViewQuery = ""
+ + "SELECT Carrier, Origin, SUM(ArrDelayMinutes) AS sum_ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME + " "
+ + "GROUP BY Carrier, Origin "
+ + "ORDER BY sum_ArrDelayMinutes DESC "
+ + "LIMIT 5 OFFSET 3";
+ JsonNode materializedViewResponse = postQuery(materializedViewQuery);
+ assertNoExceptions(materializedViewResponse);
+
+ String materializedViewQueried = getMaterializedViewQueried(materializedViewResponse);
+ assertNotNull(materializedViewQueried,
+ "Expected materialized view to be hit for LIMIT/OFFSET query. Response: " + materializedViewResponse);
+ assertEquals(materializedViewQueried, MATERIALIZED_VIEW_SPLIT_TABLE_OFFLINE);
+
+ JsonNode materializedViewRows = materializedViewResponse.get("resultTable").get("rows");
+ assertNotNull(materializedViewRows);
+ assertEquals(materializedViewRows.size(), 5, "LIMIT 5 should return exactly 5 rows");
+
+ for (int i = 1; i < materializedViewRows.size(); i++) {
+ double prev = materializedViewRows.get(i - 1).get(2).asDouble();
+ double curr = materializedViewRows.get(i).get(2).asDouble();
+ assertTrue(prev >= curr,
+ "Results should be ordered DESC by sum_ArrDelayMinutes: row " + (i - 1)
+ + " (" + prev + ") < row " + i + " (" + curr + ")");
+ }
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Phase 3, Test 9: Multiple MVs — cost-based selection
+ /// -----------------------------------------------------------------------
+
+ /// Verifies that when two MVs match the same user query, the one with the lower
+ /// rewrite cost is selected.
+ ///
+ /// <p>The full MV matches {@code GROUP BY Carrier} via {@code ExactSubsumptionStrategy}
+ /// (cost 0.0). The cost-competitor MV (GROUP BY {@code DaysSinceEpoch, Carrier},
+ /// full coverage, no split spec) matches the same query via
+ /// {@code AggregationSubsumptionStrategy} (cost 6.0). The full MV should win.
+ @Test
+ public void testMultipleMaterializedViewCostSelection()
+ throws Exception {
+ String query = ""
+ + "SELECT Carrier, SUM(ArrDelayMinutes) AS sum_ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME + " GROUP BY Carrier";
+ JsonNode response = postQuery(query);
+ assertNoExceptions(response);
+
+ /// The full MV (EXACT, cost 0.0) should win over the cost competitor (AGG_REAGG, cost 6.0).
+ /// V2 only surfaces the chosen MV via materializedViewQueried; candidate set is not exposed.
+ String materializedViewQueried = getMaterializedViewQueried(response);
+ assertNotNull(materializedViewQueried, "Expected an MV to be hit. Response: " + response);
+ assertEquals(materializedViewQueried, MATERIALIZED_VIEW_FULL_TABLE_OFFLINE,
+ "Full MV (EXACT, cost 0.0) should be selected over cost competitor (AGG_REAGG, cost 6.0)");
+ }
+
+ /// -----------------------------------------------------------------------
+ /// MV table setup
+ /// -----------------------------------------------------------------------
+
+ /// Full-rewrite MV: no split spec, covers all data.
+ /// Definition: {@code SELECT Carrier, SUM(ArrDelayMinutes) AS sum_ArrDelayMinutes
+ /// FROM materializedViewSourceTable GROUP BY Carrier}
+ private void setupFullRewriteMv()
+ throws Exception {
+ Schema materializedViewSchema = new Schema.SchemaBuilder()
+ .setSchemaName(MATERIALIZED_VIEW_FULL_TABLE_NAME)
+ .addSingleValueDimension("Carrier", FieldSpec.DataType.STRING)
+ .addMetric("sum_ArrDelayMinutes", FieldSpec.DataType.DOUBLE)
+ .build();
+ addSchema(materializedViewSchema);
+
+ TableConfig materializedViewTableConfig = new TableConfigBuilder(TableType.OFFLINE)
+ .setTableName(MATERIALIZED_VIEW_FULL_TABLE_NAME)
+ .setNumReplicas(1)
+ .build();
+ addTableConfig(materializedViewTableConfig);
+
+ /// Build synthetic pre-aggregated rows: one row per carrier
+ List<GenericRow> rows = new ArrayList<>();
+ for (String carrier : CARRIERS) {
+ GenericRow row = new GenericRow();
+ row.putValue("Carrier", carrier);
+ row.putValue("sum_ArrDelayMinutes", 1000.0 + carrier.hashCode() % 500);
+ rows.add(row);
+ }
+ buildAndUploadSegment(materializedViewTableConfig, materializedViewSchema, rows,
+ MATERIALIZED_VIEW_FULL_TABLE_NAME, "materializedViewFullSeg");
+
+ waitForAnyDocLoaded(MATERIALIZED_VIEW_FULL_TABLE_NAME, 60_000L);
+
+ String definedSql = "SELECT Carrier, SUM(ArrDelayMinutes) AS sum_ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME + " GROUP BY Carrier";
+ MaterializedViewDefinitionMetadata definition = new MaterializedViewDefinitionMetadata(
+ MATERIALIZED_VIEW_FULL_TABLE_OFFLINE,
+ Collections.singletonList(SOURCE_TABLE_NAME),
+ definedSql,
+ Collections.emptyMap(),
+ null);
+ MaterializedViewDefinitionMetadataUtils.persist(_propertyStore, definition, -1);
+
+ MaterializedViewRuntimeMetadata runtime = new MaterializedViewRuntimeMetadata(
+ MATERIALIZED_VIEW_FULL_TABLE_OFFLINE, DATA_MAX_TIME_MS, new HashMap<>());
+ MaterializedViewRuntimeMetadataUtils.persist(_propertyStore, runtime, -1);
+ }
+
+ /// Split-rewrite MV: has split spec, covers historical data up to SPLIT_BOUNDARY_MS.
+ /// Definition: {@code SELECT DaysSinceEpoch, Carrier, Origin,
+ /// SUM(ArrDelayMinutes) AS sum_ArrDelayMinutes
+ /// FROM materializedViewSourceTable GROUP BY DaysSinceEpoch, Carrier, Origin}
+ ///
+ /// <p>Includes {@code Origin} to differentiate from the full MV, ensuring test 2's
+ /// query (with {@code GROUP BY Carrier, Origin}) hits this MV via
+ /// {@code AggregationSubsumptionStrategy} and not the full MV.
+ private void setupSplitRewriteMv()
+ throws Exception {
+ Schema materializedViewSchema = new Schema.SchemaBuilder()
+ .setSchemaName(MATERIALIZED_VIEW_SPLIT_TABLE_NAME)
+ .addDateTime(TIME_COLUMN, FieldSpec.DataType.INT, "1:DAYS:EPOCH", "1:DAYS")
+ .addSingleValueDimension("Carrier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("Origin", FieldSpec.DataType.STRING)
+ .addMetric("sum_ArrDelayMinutes", FieldSpec.DataType.DOUBLE)
+ .build();
+ addSchema(materializedViewSchema);
+
+ TableConfig materializedViewTableConfig = new TableConfigBuilder(TableType.OFFLINE)
+ .setTableName(MATERIALIZED_VIEW_SPLIT_TABLE_NAME)
+ .setTimeColumnName(TIME_COLUMN)
+ .setNumReplicas(1)
+ .build();
+ addTableConfig(materializedViewTableConfig);
+
+ /// Build synthetic rows: one per (day, carrier, origin) combination
+ String[] origins = {"SFO", "LAX", "JFK", "ORD", "ATL"};
+ List<GenericRow> rows = new ArrayList<>();
+ for (int day = 16071; day < 16086; day++) {
+ for (String carrier : CARRIERS) {
+ for (String origin : origins) {
+ GenericRow row = new GenericRow();
+ row.putValue(TIME_COLUMN, day);
+ row.putValue("Carrier", carrier);
+ row.putValue("Origin", origin);
+ row.putValue("sum_ArrDelayMinutes",
+ 100.0 + (day + carrier.hashCode() + origin.hashCode()) % 200);
+ rows.add(row);
+ }
+ }
+ }
+ buildAndUploadSegment(materializedViewTableConfig, materializedViewSchema, rows,
+ MATERIALIZED_VIEW_SPLIT_TABLE_NAME, "materializedViewSplitSeg");
+
+ waitForAnyDocLoaded(MATERIALIZED_VIEW_SPLIT_TABLE_NAME, 60_000L);
+
+ String definedSql = "SELECT " + TIME_COLUMN + ", Carrier, Origin, "
+ + "SUM(ArrDelayMinutes) AS sum_ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME + " "
+ + "GROUP BY " + TIME_COLUMN + ", Carrier, Origin";
+ MaterializedViewSplitSpec splitSpec =
+ new MaterializedViewSplitSpec(TIME_COLUMN, "1:DAYS:EPOCH", TIME_COLUMN, "1:DAYS:EPOCH", 86_400_000L);
+ MaterializedViewDefinitionMetadata definition = new MaterializedViewDefinitionMetadata(
+ MATERIALIZED_VIEW_SPLIT_TABLE_OFFLINE,
+ Collections.singletonList(SOURCE_TABLE_NAME),
+ definedSql,
+ Collections.emptyMap(),
+ splitSpec);
+ MaterializedViewDefinitionMetadataUtils.persist(_propertyStore, definition, -1);
+
+ MaterializedViewRuntimeMetadata runtime = new MaterializedViewRuntimeMetadata(
+ MATERIALIZED_VIEW_SPLIT_TABLE_OFFLINE, SPLIT_BOUNDARY_MS, new HashMap<>());
+ MaterializedViewRuntimeMetadataUtils.persist(_propertyStore, runtime, -1);
+ }
+
+ /// Cold-start MV: definition with split spec exists, but no APPEND has completed yet
+ /// (watermarkMs is the initial cold-start placeholder and the partitions map is empty).
+ /// The cold-start check in `resolvePlan` only triggers for incremental MVs (with
+ /// a split spec), so a split spec is required here to exercise the cold-start skip path.
+ private void setupColdStartMv()
+ throws Exception {
+ Schema materializedViewSchema = new Schema.SchemaBuilder()
+ .setSchemaName(MATERIALIZED_VIEW_COLD_TABLE_NAME)
+ .addDateTime(TIME_COLUMN, FieldSpec.DataType.INT, "1:DAYS:EPOCH", "1:DAYS")
+ .addSingleValueDimension("Carrier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("Origin", FieldSpec.DataType.STRING)
+ .addMetric("sum_ArrDelayMinutes", FieldSpec.DataType.DOUBLE)
+ .build();
+ addSchema(materializedViewSchema);
+
+ TableConfig materializedViewTableConfig = new TableConfigBuilder(TableType.OFFLINE)
+ .setTableName(MATERIALIZED_VIEW_COLD_TABLE_NAME)
+ .setTimeColumnName(TIME_COLUMN)
+ .setNumReplicas(1)
+ .build();
+ addTableConfig(materializedViewTableConfig);
+ /// No segments uploaded — empty MV table (pre-APPEND state)
+
+ /// Same definition as the split MV; cold-start state is conveyed via a 0/initial watermark
+ /// and empty partitions map on the runtime znode below.
+ String definedSql = "SELECT " + TIME_COLUMN + ", Carrier, Origin, "
+ + "SUM(ArrDelayMinutes) AS sum_ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME + " "
+ + "GROUP BY " + TIME_COLUMN + ", Carrier, Origin";
+ MaterializedViewSplitSpec splitSpec =
+ new MaterializedViewSplitSpec(TIME_COLUMN, "1:DAYS:EPOCH", TIME_COLUMN, "1:DAYS:EPOCH", 86_400_000L);
+ MaterializedViewDefinitionMetadata definition = new MaterializedViewDefinitionMetadata(
+ MATERIALIZED_VIEW_COLD_TABLE_OFFLINE,
+ Collections.singletonList(SOURCE_TABLE_NAME),
+ definedSql,
+ Collections.emptyMap(),
+ splitSpec);
+ MaterializedViewDefinitionMetadataUtils.persist(_propertyStore, definition, -1);
+
+ /// Cold-start placeholder: watermark seeded at the source-table minimum, partitions map empty,
+ /// so the broker has no confirmed coverage to query against.
+ MaterializedViewRuntimeMetadata runtime = new MaterializedViewRuntimeMetadata(
+
+ MATERIALIZED_VIEW_COLD_TABLE_OFFLINE, DATA_MIN_TIME_MS, new HashMap<>());
+ MaterializedViewRuntimeMetadataUtils.persist(_propertyStore, runtime, -1);
+ }
+
+ /// Scan MV: no GROUP BY, no aggregation. Stores a projection of the source table
+ /// columns for scan-subsumption matching.
+ /// Definition: {@code SELECT Carrier, Origin, Dest, ArrDelayMinutes FROM materializedViewSourceTable}
+ private void setupScanMv()
+ throws Exception {
+ Schema materializedViewSchema = new Schema.SchemaBuilder()
+ .setSchemaName(MATERIALIZED_VIEW_SCAN_TABLE_NAME)
+ .addSingleValueDimension("Carrier", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("Origin", FieldSpec.DataType.STRING)
+ .addSingleValueDimension("Dest", FieldSpec.DataType.STRING)
+ .addMetric("ArrDelayMinutes", FieldSpec.DataType.DOUBLE)
+ .build();
+ addSchema(materializedViewSchema);
+
+ TableConfig materializedViewTableConfig = new TableConfigBuilder(TableType.OFFLINE)
+ .setTableName(MATERIALIZED_VIEW_SCAN_TABLE_NAME)
+ .setNumReplicas(1)
+ .build();
+ addTableConfig(materializedViewTableConfig);
+
+ /// Build synthetic scan rows with representative data
+ String[] origins = {"SFO", "LAX", "JFK", "ORD", "ATL"};
+ String[] dests = {"LAX", "SFO", "ORD", "JFK", "DFW"};
+ List<GenericRow> rows = new ArrayList<>();
+ for (String carrier : CARRIERS) {
+ for (int i = 0; i < origins.length; i++) {
+ GenericRow row = new GenericRow();
+ row.putValue("Carrier", carrier);
+ row.putValue("Origin", origins[i]);
+ row.putValue("Dest", dests[i]);
+ row.putValue("ArrDelayMinutes", 10.0 + (carrier.hashCode() + i) % 50);
+ rows.add(row);
+ }
+ }
+ buildAndUploadSegment(materializedViewTableConfig, materializedViewSchema, rows,
+ MATERIALIZED_VIEW_SCAN_TABLE_NAME, "materializedViewScanSeg");
+
+ waitForAnyDocLoaded(MATERIALIZED_VIEW_SCAN_TABLE_NAME, 60_000L);
+
+ String definedSql = "SELECT Carrier, Origin, Dest, ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME;
+ MaterializedViewDefinitionMetadata definition = new MaterializedViewDefinitionMetadata(
+ MATERIALIZED_VIEW_SCAN_TABLE_OFFLINE,
+ Collections.singletonList(SOURCE_TABLE_NAME),
+ definedSql,
+ Collections.emptyMap(),
+ null);
+ MaterializedViewDefinitionMetadataUtils.persist(_propertyStore, definition, -1);
+
+ MaterializedViewRuntimeMetadata runtime = new MaterializedViewRuntimeMetadata(
+ MATERIALIZED_VIEW_SCAN_TABLE_OFFLINE, DATA_MAX_TIME_MS, new HashMap<>());
+ MaterializedViewRuntimeMetadataUtils.persist(_propertyStore, runtime, -1);
+ }
+
+ /// Cost-competitor MV: groups by {@code DaysSinceEpoch, Carrier} with full coverage
+ /// and no split spec. This MV matches {@code GROUP BY Carrier} queries via
+ /// {@code AggregationSubsumptionStrategy} (cost 6.0), competing with the full MV
+ /// which matches via {@code ExactSubsumptionStrategy} (cost 0.0).
+ private void setupCostCompetitorMv()
+ throws Exception {
+ Schema materializedViewSchema = new Schema.SchemaBuilder()
+ .setSchemaName(MATERIALIZED_VIEW_COST_TABLE_NAME)
+ .addDateTime(TIME_COLUMN, FieldSpec.DataType.INT, "1:DAYS:EPOCH", "1:DAYS")
+ .addSingleValueDimension("Carrier", FieldSpec.DataType.STRING)
+ .addMetric("sum_ArrDelayMinutes", FieldSpec.DataType.DOUBLE)
+ .build();
+ addSchema(materializedViewSchema);
+
+ TableConfig materializedViewTableConfig = new TableConfigBuilder(TableType.OFFLINE)
+ .setTableName(MATERIALIZED_VIEW_COST_TABLE_NAME)
+ .setTimeColumnName(TIME_COLUMN)
+ .setNumReplicas(1)
+ .build();
+ addTableConfig(materializedViewTableConfig);
+
+ /// Build synthetic rows: one per (day, carrier) combination
+ List<GenericRow> rows = new ArrayList<>();
+ for (int day = 16071; day <= 16101; day++) {
+ for (String carrier : CARRIERS) {
+ GenericRow row = new GenericRow();
+ row.putValue(TIME_COLUMN, day);
+ row.putValue("Carrier", carrier);
+ row.putValue("sum_ArrDelayMinutes", 50.0 + (day + carrier.hashCode()) % 100);
+ rows.add(row);
+ }
+ }
+ buildAndUploadSegment(materializedViewTableConfig, materializedViewSchema, rows,
+ MATERIALIZED_VIEW_COST_TABLE_NAME, "materializedViewCostSeg");
+
+ waitForAnyDocLoaded(MATERIALIZED_VIEW_COST_TABLE_NAME, 60_000L);
+
+ String definedSql = "SELECT " + TIME_COLUMN + ", Carrier, "
+ + "SUM(ArrDelayMinutes) AS sum_ArrDelayMinutes "
+ + "FROM " + SOURCE_TABLE_NAME + " "
+ + "GROUP BY " + TIME_COLUMN + ", Carrier";
+ MaterializedViewDefinitionMetadata definition = new MaterializedViewDefinitionMetadata(
+ MATERIALIZED_VIEW_COST_TABLE_OFFLINE,
+ Collections.singletonList(SOURCE_TABLE_NAME),
+ definedSql,
+ Collections.emptyMap(),
+ null);
+ MaterializedViewDefinitionMetadataUtils.persist(_propertyStore, definition, -1);
+
+ MaterializedViewRuntimeMetadata runtime = new MaterializedViewRuntimeMetadata(
+ MATERIALIZED_VIEW_COST_TABLE_OFFLINE, DATA_MAX_TIME_MS, new HashMap<>());
+ MaterializedViewRuntimeMetadataUtils.persist(_propertyStore, runtime, -1);
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Segment building helpers
+ /// -----------------------------------------------------------------------
+
+ private void buildAndUploadSegment(TableConfig tableConfig, Schema schema,
+ List<GenericRow> rows, String tableName, String segmentName)
+ throws Exception {
+ File segOutputDir = new File(_tempDir, segmentName + "_output");
+ File tarDir = new File(_tempDir, segmentName + "_tar");
+ TestUtils.ensureDirectoriesExistAndEmpty(segOutputDir, tarDir);
+
+ SegmentGeneratorConfig config = new SegmentGeneratorConfig(tableConfig, schema);
+ config.setTableName(tableConfig.getTableName());
+ config.setOutDir(segOutputDir.getAbsolutePath());
+ config.setSegmentName(segmentName);
+
+ SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl();
+ driver.init(config, new GenericRowRecordReader(rows));
+ driver.build();
+
+ File segmentDir = new File(segOutputDir, segmentName);
+ File tarFile = new File(tarDir, segmentName + TarCompressionUtils.TAR_GZ_FILE_EXTENSION);
+ TarCompressionUtils.createCompressedTarFile(segmentDir, tarFile);
+
+ uploadSegments(tableName, tarDir);
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Utility helpers
+ /// -----------------------------------------------------------------------
+
+ @Nullable
+ private static String getMaterializedViewQueried(JsonNode response) {
+ return response.has("materializedViewQueried") && !response.get("materializedViewQueried").isNull()
+ ? response.get("materializedViewQueried").asText() : null;
+ }
+
+ private static void assertNoExceptions(JsonNode response) {
+ JsonNode exceptions = response.get("exceptions");
+ assertTrue(exceptions == null || exceptions.isEmpty(),
+ "Query returned exceptions: " + exceptions);
+ }
+
+ private void cleanupMaterializedViewMetadata(String materializedViewTableNameWithType) {
+ try {
+ MaterializedViewDefinitionMetadataUtils.delete(_propertyStore, materializedViewTableNameWithType);
+ } catch (Exception e) {
+ /// Ignore cleanup failures
+ }
+ try {
+ MaterializedViewRuntimeMetadataUtils.delete(_propertyStore, materializedViewTableNameWithType);
+ } catch (Exception e) {
+ /// Ignore cleanup failures
+ }
+ }
+}
diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/analysis/package-info.java
similarity index 50%
copy from pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java
copy to pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/analysis/package-info.java
index 7d7524b28ab..42bc8bc77e0 100644
--- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/analysis/package-info.java
@@ -16,29 +16,15 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.sql.parsers.rewriter;
-import java.util.Map;
-import java.util.stream.Collectors;
-import org.apache.pinot.spi.utils.CommonConstants;
-
-
-public class RlsUtils {
-
- private RlsUtils() {
- }
-
- public static String buildRlsFilterKey(String tableName) {
- return String.format("%s-%s", CommonConstants.RLS_FILTERS, tableName);
- }
-
- public static Map<String, String> extractRlsFilters(Map<String, String> requestMetadata) {
- return requestMetadata.entrySet().stream()
- .filter(e -> e.getKey().startsWith(CommonConstants.RLS_FILTERS))
- .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
- }
-
- public static String getRlsFilterForTable(Map<String, String> queryOptions, String tableName) {
- return queryOptions.get(buildRlsFilterKey(tableName));
- }
-}
+/// **Definition — MV SQL validation.**
+///
+/// Parses and validates the `definedSQL` field on the MV's `TableConfig` at create time:
+/// legal aggregation functions, group-by / dimension column coverage, supported time-bucket
+/// expressions. Runs in the controller before the MV znode is persisted, so a malformed
+/// definition is rejected with a clear error rather than producing broken segments later.
+///
+/// See also [org.apache.pinot.materializedview.metadata] for the persisted definition
+/// schema and [org.apache.pinot.materializedview.scheduler] for the consumer that turns a
+/// validated definition into minion tasks.
+package org.apache.pinot.materializedview.analysis;
diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/consistency/package-info.java
similarity index 50%
copy from pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java
copy to pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/consistency/package-info.java
index 7d7524b28ab..e9b621e97fb 100644
--- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/consistency/package-info.java
@@ -16,29 +16,13 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.sql.parsers.rewriter;
-import java.util.Map;
-import java.util.stream.Collectors;
-import org.apache.pinot.spi.utils.CommonConstants;
-
-
-public class RlsUtils {
-
- private RlsUtils() {
- }
-
- public static String buildRlsFilterKey(String tableName) {
- return String.format("%s-%s", CommonConstants.RLS_FILTERS, tableName);
- }
-
- public static Map<String, String> extractRlsFilters(Map<String, String> requestMetadata) {
- return requestMetadata.entrySet().stream()
- .filter(e -> e.getKey().startsWith(CommonConstants.RLS_FILTERS))
- .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
- }
-
- public static String getRlsFilterForTable(Map<String, String> queryOptions, String tableName) {
- return queryOptions.get(buildRlsFilterKey(tableName));
- }
-}
+/// **Definition — controller-side reconciler.**
+///
+/// Cross-checks materialized-view state across the base-table TableConfig, the MV's own
+/// TableConfig, the MV definition znode in [org.apache.pinot.materializedview.metadata],
+/// and any in-flight minion tasks. Resolves drift caused by base-table drops, renames,
+/// ingestion-config changes, or partial controller failure during MV creation.
+///
+/// Runs on the controller leader; the broker and minion are unaware of this package.
+package org.apache.pinot.materializedview.consistency;
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/context/MaterializedViewContext.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/context/MaterializedViewContext.java
new file mode 100644
index 00000000000..0a9d0aa8693
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/context/MaterializedViewContext.java
@@ -0,0 +1,118 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.context;
+
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewRewritePlan;
+import org.apache.pinot.spi.data.Schema;
+
+
+/// Broker-facing materialized-view state produced during query compilation.
+///
+/// Carries the rewrite plan and, when split execution is selected, the MV-side
+/// query/table/schema needed by the SPLIT dispatcher. This class is immutable
+/// and thread-safe when the contained query/schema objects are not mutated
+/// after construction.
+public final class MaterializedViewContext {
+ private static final MaterializedViewContext EMPTY = new MaterializedViewContext(null, null, false);
+
+ /// Non-null exactly when a swap was committed (either `forFullRewrite` or `forSplitRewrite`).
+ /// `empty()` carries a null plan — `annotateResponse` keys on the matching
+ /// [#isFullRewrite()] / [#isSplitRewrite()] flag rather than plan-presence so the response
+ /// field is stamped only when the broker actually swapped to an MV.
+ @Nullable
+ private final MaterializedViewRewritePlan _plan;
+ @Nullable
+ private final SplitRewriteContext _splitRewriteContext;
+ private final boolean _fullRewrite;
+
+ private MaterializedViewContext(@Nullable MaterializedViewRewritePlan plan,
+ @Nullable SplitRewriteContext splitRewriteContext, boolean fullRewrite) {
+ _plan = plan;
+ _splitRewriteContext = splitRewriteContext;
+ _fullRewrite = fullRewrite;
+ }
+
+ public static MaterializedViewContext empty() {
+ return EMPTY;
+ }
+
+ public static MaterializedViewContext forSplitRewrite(MaterializedViewRewritePlan plan,
+ PinotQuery viewServerPinotQuery, String viewTableNameWithType, Schema viewSchema) {
+ SplitRewriteContext splitRewriteContext =
+ new SplitRewriteContext(viewServerPinotQuery, viewTableNameWithType, viewSchema);
+ return new MaterializedViewContext(plan, splitRewriteContext, false);
+ }
+
+ public static MaterializedViewContext forFullRewrite(MaterializedViewRewritePlan plan) {
+ return new MaterializedViewContext(plan, null, true);
+ }
+
+ public boolean isSplitRewrite() {
+ return _splitRewriteContext != null;
+ }
+
+ public boolean isFullRewrite() {
+ return _fullRewrite;
+ }
+
+ @Nullable
+ public String getMaterializedViewQueriedName() {
+ return _plan != null ? _plan.getMaterializedViewTableNameWithType() : null;
+ }
+
+ @Nullable
+ public MaterializedViewRewritePlan getPlan() {
+ return _plan;
+ }
+
+ @Nullable
+ public SplitRewriteContext getSplitRewriteContext() {
+ return _splitRewriteContext;
+ }
+
+ /// MV branch state for split execution. Callers that need the raw table name derive it
+ /// via [TableNameBuilder#extractRawTableName(String)] from
+ /// [#getMaterializedViewTableNameWithType()]; storing a separate raw-name field would just
+ /// duplicate that single string operation.
+ public static final class SplitRewriteContext {
+ private final PinotQuery _materializedViewServerPinotQuery;
+ private final String _materializedViewTableNameWithType;
+ private final Schema _materializedViewSchema;
+
+ private SplitRewriteContext(PinotQuery viewServerPinotQuery, String viewTableNameWithType, Schema viewSchema) {
+ _materializedViewServerPinotQuery = viewServerPinotQuery;
+ _materializedViewTableNameWithType = viewTableNameWithType;
+ _materializedViewSchema = viewSchema;
+ }
+
+ public PinotQuery getMaterializedViewServerPinotQuery() {
+ return _materializedViewServerPinotQuery;
+ }
+
+ public String getMaterializedViewTableNameWithType() {
+ return _materializedViewTableNameWithType;
+ }
+
+ public Schema getMaterializedViewSchema() {
+ return _materializedViewSchema;
+ }
+ }
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/context/package-info.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/context/package-info.java
new file mode 100644
index 00000000000..3b9c98e44cb
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/context/package-info.java
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/// **Query rewrite — compile-time carrier.**
+///
+/// [MaterializedViewContext][org.apache.pinot.materializedview.context.MaterializedViewContext]
+/// is the immutable handoff object the broker's
+/// [MaterializedViewHandler][org.apache.pinot.materializedview.handler.MaterializedViewHandler]
+/// returns from `compile()`. It tells the broker which mode applies:
+///
+/// - `isFullRewrite()` — swap the server query to target the MV; route + scatter-gather
+/// run against the MV table.
+/// - `isSplitRewrite()` — dispatch dual scatter-gather (base + MV) and merge results.
+/// - neither — base-table path unchanged.
+///
+/// For SPLIT mode the context also carries the MV-side query, table name, and schema that
+/// the split dispatcher needs to build the second scatter-gather; the broker keeps its own
+/// pre-rewrite locals for the base side.
+package org.apache.pinot.materializedview.context;
diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/executor/package-info.java
similarity index 50%
copy from pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java
copy to pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/executor/package-info.java
index 7d7524b28ab..6c7a5f6aed0 100644
--- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/executor/package-info.java
@@ -16,29 +16,12 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.sql.parsers.rewriter;
-import java.util.Map;
-import java.util.stream.Collectors;
-import org.apache.pinot.spi.utils.CommonConstants;
-
-
-public class RlsUtils {
-
- private RlsUtils() {
- }
-
- public static String buildRlsFilterKey(String tableName) {
- return String.format("%s-%s", CommonConstants.RLS_FILTERS, tableName);
- }
-
- public static Map<String, String> extractRlsFilters(Map<String, String> requestMetadata) {
- return requestMetadata.entrySet().stream()
- .filter(e -> e.getKey().startsWith(CommonConstants.RLS_FILTERS))
- .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
- }
-
- public static String getRlsFilterForTable(Map<String, String> queryOptions, String tableName) {
- return queryOptions.get(buildRlsFilterKey(tableName));
- }
-}
+/// **View generation — minion task executor.**
+///
+/// Minion-side worker that runs the SQL specified by the MV's `definedSQL`, builds the
+/// resulting Pinot segments, and uploads them to the MV's offline table. Invoked
+/// per-`PinotTaskConfig` produced by the scheduler in
+/// [org.apache.pinot.materializedview.scheduler], and on success the matching runtime
+/// znode (see [org.apache.pinot.materializedview.metadata]) is advanced.
+package org.apache.pinot.materializedview.executor;
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/handler/DefaultMaterializedViewHandler.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/handler/DefaultMaterializedViewHandler.java
new file mode 100644
index 00000000000..207f0c93117
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/handler/DefaultMaterializedViewHandler.java
@@ -0,0 +1,275 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.handler;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.core.routing.timeboundary.TimeBoundaryInfo;
+import org.apache.pinot.materializedview.context.MaterializedViewContext;
+import org.apache.pinot.materializedview.context.MaterializedViewContext.SplitRewriteContext;
+import org.apache.pinot.materializedview.metadata.MaterializedViewDefinitionMetadata.MaterializedViewSplitSpec;
+import org.apache.pinot.materializedview.rewrite.ExecutionMode;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewQueryRewriteEngine;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewQueryRewriteEngineFactory;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewRewritePlan;
+import org.apache.pinot.spi.data.DateTimeFormatSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.CommonConstants.Broker.Request.QueryOptionKey;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.FilterKind;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Default [MaterializedViewHandler] backed by a [MaterializedViewQueryRewriteEngine].
+///
+/// Owns all MV-specific compile-time decisions and split-execution time-boundary attachment, so
+/// the broker request handler only needs to invoke the SPI hooks and provide a dispatcher
+/// callback for the generic route-build + scatter-gather + reduce work.
+///
+/// Thread-safe: shares the underlying rewrite engine (which is itself thread-safe) across all
+/// request threads.
+public class DefaultMaterializedViewHandler implements MaterializedViewHandler {
+ private static final Logger LOGGER = LoggerFactory.getLogger(DefaultMaterializedViewHandler.class);
+
+ private MaterializedViewQueryRewriteEngine _rewriteEngine;
+ private boolean _supportsSplitRewrite;
+
+ /// Public no-arg constructor required by [MaterializedViewHandler#loadHandler]. State is
+ /// populated by [#init] or by the test-only convenience constructors below.
+ public DefaultMaterializedViewHandler() {
+ }
+
+ /// Test-only: pre-built rewrite engine + explicit split-rewrite capability.
+ @VisibleForTesting
+ public DefaultMaterializedViewHandler(MaterializedViewQueryRewriteEngine rewriteEngine,
+ boolean supportsSplitRewrite) {
+ _rewriteEngine = rewriteEngine;
+ _supportsSplitRewrite = supportsSplitRewrite;
+ }
+
+ /// Test-only: pre-built rewrite engine; defaults split-rewrite support to `true`.
+ @VisibleForTesting
+ public DefaultMaterializedViewHandler(MaterializedViewQueryRewriteEngine rewriteEngine) {
+ this(rewriteEngine, true);
+ }
+
+ @Override
+ public void init(PinotConfiguration configuration, ZkHelixPropertyStore<ZNRecord> propertyStore,
+ boolean supportsSplitRewrite) {
+ _rewriteEngine = MaterializedViewQueryRewriteEngineFactory.createDefault(propertyStore);
+ _supportsSplitRewrite = supportsSplitRewrite;
+ }
+
+ @Override
+ public boolean supportsSplitRewrite() {
+ return _supportsSplitRewrite;
+ }
+
+ @Override
+ public MaterializedViewContext compile(MaterializedViewCompileContext ctx) {
+ MaterializedViewRewritePlan plan =
+ _rewriteEngine.tryRewrite(ctx.getServerPinotQuery(), ctx.getRawTableName());
+ if (plan == null) {
+ return MaterializedViewContext.empty();
+ }
+
+ String mvTableNameWithType = plan.getMaterializedViewTableNameWithType();
+ String mvRawTableName = TableNameBuilder.extractRawTableName(mvTableNameWithType);
+ Schema mvSchema = ctx.getTableCache().getSchema(mvRawTableName);
+ if (mvSchema == null) {
+ LOGGER.warn("MV schema not found for {}; skipping rewrite", mvTableNameWithType);
+ return MaterializedViewContext.empty();
+ }
+
+ /// Cold-start defense-in-depth: an MV with `watermarkMs <= 0` has no committed coverage yet
+ /// (no APPEND has completed). The rewrite engine's `resolvePlan` already filters cold-start
+ /// candidates for split-spec MVs (which carry `watermarkMs` on the plan) — this guard
+ /// re-checks them in case a future engine change loses that filter.
+ ///
+ /// ONLY applies to `SPLIT_REWRITE`: `FULL_REWRITE` plans intentionally carry `watermarkMs=0`
+ /// (no boundary literal is attached at execute time, so the watermark is irrelevant), and a
+ /// blanket `<= 0` check would wrongly drop every `FULL_REWRITE` attempt.
+ if (plan.getExecMode() == ExecutionMode.SPLIT_REWRITE && plan.getWatermarkMs() <= 0) {
+ LOGGER.debug("MV {} has watermarkMs={} <= 0; skipping SPLIT_REWRITE for request",
+ mvTableNameWithType, plan.getWatermarkMs());
+ return MaterializedViewContext.empty();
+ }
+
+ if (plan.getExecMode() == ExecutionMode.SPLIT_REWRITE) {
+ if (!_supportsSplitRewrite) {
+ LOGGER.debug("Handler does not support SPLIT_REWRITE; skipping for MV {}", mvTableNameWithType);
+ return MaterializedViewContext.empty();
+ }
+ return MaterializedViewContext.forSplitRewrite(plan, plan.getMaterializedViewQuery(),
+ mvTableNameWithType, mvSchema);
+ }
+
+ /// `FULL_REWRITE`: the broker performs the actual swap after this method returns; it
+ /// already holds the pre-rewrite query/table in its own locals (the swap is gated on
+ /// `[#isFullRewrite()]` and runs after ACL/quota/RLS so those checks naturally target the
+ /// base table). The returned context carries the plan plus the FULL_REWRITE flag so
+ /// `annotateResponse` can stamp the response field on a committed swap.
+ return MaterializedViewContext.forFullRewrite(plan);
+ }
+
+ @Override
+ public BrokerResponseNative executeSplit(MaterializedViewSplitExecutionContext ctx) throws Exception {
+ MaterializedViewContext mvContext = ctx.getMaterializedViewContext();
+ SplitRewriteContext splitRewriteContext = Preconditions.checkNotNull(mvContext.getSplitRewriteContext(),
+ "MV split context must be set for split execution");
+ MaterializedViewRewritePlan plan = Preconditions.checkNotNull(mvContext.getPlan(),
+ "MV rewrite plan must be set for split execution");
+ long boundaryTimeMs = plan.getWatermarkMs();
+
+ MaterializedViewSplitSpec splitSpec = plan.getSplitSpec();
+ Preconditions.checkNotNull(splitSpec,
+ "MaterializedViewSplitSpec must be set when execMode == SPLIT_REWRITE for MV: %s",
+ plan.getMaterializedViewTableNameWithType());
+
+ /// Both base and MV columns may use any DateTimeFieldSpec format (INT days, TIMESTAMP millis,
+ /// SIMPLE_DATE_FORMAT, etc.), so each boundary literal goes through its own
+ /// `*TimeFormat.fromMillisToFormat` to land in the column's native unit. Without this, the
+ /// raw millis literal applied to an INT-days column would produce "Cannot convert value to
+ /// type: INT" at scan time.
+ String baseTimeColumn = splitSpec.getSourceTimeColumn();
+ String viewTimeColumn = splitSpec.getMaterializedViewTimeColumn();
+ String sourceTimeFormat = splitSpec.getSourceTimeFormat();
+ String viewTimeFormat = splitSpec.getMaterializedViewTimeFormat();
+ Preconditions.checkState(sourceTimeFormat != null && !sourceTimeFormat.isEmpty(),
+ "Internal error: sourceTimeFormat must be non-null/non-empty at SPLIT_REWRITE execution for MV: %s",
+ plan.getMaterializedViewTableNameWithType());
+ /// Rolling-upgrade safety: definition znodes written by a pre-V2 controller persisted only
+ /// sourceTimeFormat. When the broker upgrades ahead of the controller, an existing MV's
+ /// splitMaterializedViewTimeFormat is null. Inferring it from sourceTimeFormat would
+ /// silently double-count rows when the two columns use different formats (e.g. base in
+ /// millis-epoch, MV in days-epoch — the day-valued column would always satisfy
+ /// `materializedViewTime < <millis-literal>`). Refuse to dispatch and surface a clear
+ /// QueryException so the broker's outer try/catch falls back to the base-table path with
+ /// QUERY_REWRITE_EXCEPTIONS bumped; the operator's signal is that the MV must be re-created
+ /// under the upgraded controller.
+ Preconditions.checkState(viewTimeFormat != null && !viewTimeFormat.isEmpty(),
+ "Materialized view %s was created by a pre-V2 controller and is missing "
+ + "splitMaterializedViewTimeFormat. Re-create the MV under the upgraded controller, "
+ + "or query the base table directly. Broker will fall back to the base-table query path.",
+ plan.getMaterializedViewTableNameWithType());
+ String baseBoundaryLiteral =
+ new DateTimeFormatSpec(sourceTimeFormat).fromMillisToFormat(boundaryTimeMs);
+ String viewBoundaryLiteral =
+ new DateTimeFormatSpec(viewTimeFormat).fromMillisToFormat(boundaryTimeMs);
+ TimeBoundaryInfo baseBoundary = new TimeBoundaryInfo(baseTimeColumn, baseBoundaryLiteral);
+ TimeBoundaryInfo viewBoundary = new TimeBoundaryInfo(viewTimeColumn, viewBoundaryLiteral);
+
+ /// --- Build base-side query with `ts >= boundary` filter ---
+ /// MV split requires servers to return intermediate results so the broker can merge DataTables
+ /// from both the base table and the MV correctly. LIMIT/OFFSET semantics: both sub-queries
+ /// (base + MV) inherit the user's original LIMIT and OFFSET unchanged. The final reduce uses
+ /// the original BrokerRequest (which retains those values), so OFFSET is correctly applied
+ /// during the merge phase, not at the sub-query level.
+ PinotQuery basePinotQuery = ctx.getBaseServerPinotQuery().deepCopy();
+ /// CalciteSqlParser normally populates queryOptions; defend against tests / SDK paths that
+ /// construct a PinotQuery with null options to avoid an NPE that would surface as a generic
+ /// Exception in the broker's outer catch and silently bump QUERY_REWRITE_EXCEPTIONS.
+ if (basePinotQuery.getQueryOptions() != null) {
+ basePinotQuery.getQueryOptions().remove(QueryOptionKey.SERVER_RETURN_FINAL_RESULT);
+ basePinotQuery.getQueryOptions().remove(QueryOptionKey.SERVER_RETURN_FINAL_RESULT_KEY_UNPARTITIONED);
+ }
+ /// GREATER_THAN_OR_EQUAL (>=) covers ts == boundaryTimeMs since the MV materializes
+ /// ts < boundaryTimeMs (exclusive upper bound). Pairing >= on base with < on MV makes the
+ /// two halves of the timeline disjoint and exhaustive.
+ attachFilter(basePinotQuery, baseBoundary, FilterKind.GREATER_THAN_OR_EQUAL);
+
+ /// --- Build MV-side query with `materializedViewTime < boundary` filter ---
+ /// Without this filter, when the broker observes a stale watermarkMs (during the
+ /// executor's non-atomic endSegmentReplace -> ZK publish window) rows in
+ /// [W_observed, W_actual) would be visible on both branches and double-counted.
+ PinotQuery viewPinotQuery = splitRewriteContext.getMaterializedViewServerPinotQuery().deepCopy();
+ if (viewPinotQuery.getQueryOptions() != null) {
+ viewPinotQuery.getQueryOptions().remove(QueryOptionKey.SERVER_RETURN_FINAL_RESULT);
+ viewPinotQuery.getQueryOptions().remove(QueryOptionKey.SERVER_RETURN_FINAL_RESULT_KEY_UNPARTITIONED);
+ }
+ attachFilter(viewPinotQuery, viewBoundary, FilterKind.LESS_THAN);
+
+ LOGGER.info("MV split execution: baseTable={}, materializedViewTable={}, "
+ + "boundaryTimeMs={}, baseColumn={}, viewColumn={}",
+ ctx.getOriginalBrokerRequest().getQuerySource().getTableName(),
+ splitRewriteContext.getMaterializedViewTableNameWithType(), boundaryTimeMs,
+ baseTimeColumn, viewTimeColumn);
+
+ /// --- 4. Hand off to the broker for hybrid route prep + dual scatter-gather + reduce ---
+ return ctx.getDispatcher().dispatch(ctx.getOriginalBrokerRequest(), basePinotQuery,
+ ctx.getBaseRouteInfo(), ctx.getBaseSchema(), viewPinotQuery,
+ splitRewriteContext.getMaterializedViewTableNameWithType(),
+ splitRewriteContext.getMaterializedViewSchema(), ctx.getRemainingTimeMs());
+ }
+
+ @Override
+ public void annotateResponse(BrokerResponseNative response, MaterializedViewContext mvContext) {
+ /// The response field is the operator's signal that an MV actually served the query.
+ /// Gate on the committed execution mode so a `fromRewriteResult` path (rewrite matched
+ /// structurally but the handler skipped the swap — e.g. `SPLIT_REWRITE` on a cold-start MV
+ /// or an MV-schema miss) does not produce a false-positive annotation. Without this guard
+ /// the broker would report `materializedViewQueried=mv_orders_OFFLINE` even when the server
+ /// query stayed against the base table.
+ if (mvContext == null || (!mvContext.isFullRewrite() && !mvContext.isSplitRewrite())) {
+ return;
+ }
+ response.setMaterializedViewQueried(mvContext.getMaterializedViewQueriedName());
+ }
+
+ @Override
+ public void invalidateBaseTable(String rawTableName) {
+ _rewriteEngine.invalidateBaseTable(rawTableName);
+ }
+
+ @Override
+ public void refreshTable(String rawTableName) {
+ _rewriteEngine.refreshTable(rawTableName);
+ }
+
+ @Override
+ public int getCacheEntryCount() {
+ return _rewriteEngine.getCacheEntryCount();
+ }
+
+ @Override
+ public void close() {
+ _rewriteEngine.close();
+ }
+
+ /// Attaches a `col <op> value` filter to `pinotQuery`, AND-ing it onto any existing filter
+ /// expression. Single helper used for both the base-side `>=` and the MV-side `<` boundary
+ /// filters.
+ @VisibleForTesting
+ static void attachFilter(PinotQuery pinotQuery, TimeBoundaryInfo boundary, FilterKind op) {
+ Expression timeFilter = RequestUtils.getFunctionExpression(op.name(),
+ RequestUtils.getIdentifierExpression(boundary.getTimeColumn()),
+ RequestUtils.getLiteralExpression(boundary.getTimeValue()));
+ Expression existing = pinotQuery.getFilterExpression();
+ pinotQuery.setFilterExpression(existing == null ? timeFilter
+ : RequestUtils.getFunctionExpression(FilterKind.AND.name(), existing, timeFilter));
+ }
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/handler/MaterializedViewCompileContext.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/handler/MaterializedViewCompileContext.java
new file mode 100644
index 00000000000..b0903b0575f
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/handler/MaterializedViewCompileContext.java
@@ -0,0 +1,61 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.handler;
+
+import org.apache.pinot.common.config.provider.TableCache;
+import org.apache.pinot.common.request.PinotQuery;
+
+
+/// Inputs the broker passes to [MaterializedViewHandler#compile] at query compile time.
+///
+/// Carries the post-base-rewrite server query (after standard broker rewrites such as HLL log2m
+/// and approximate-function override have run) plus the table names and table cache the handler
+/// needs to (a) decide whether an MV applies and (b) construct any rewritten server query /
+/// split context. The MV-side schema is fetched on demand via [#getTableCache()]; the base
+/// table schema is reachable the same way if a custom handler ever needs it.
+public final class MaterializedViewCompileContext {
+ private final PinotQuery _serverPinotQuery;
+ private final String _tableNameWithType;
+ private final String _rawTableName;
+ private final TableCache _tableCache;
+
+ public MaterializedViewCompileContext(PinotQuery serverPinotQuery, String tableNameWithType,
+ String rawTableName, TableCache tableCache) {
+ _serverPinotQuery = serverPinotQuery;
+ _tableNameWithType = tableNameWithType;
+ _rawTableName = rawTableName;
+ _tableCache = tableCache;
+ }
+
+ public PinotQuery getServerPinotQuery() {
+ return _serverPinotQuery;
+ }
+
+ public String getTableNameWithType() {
+ return _tableNameWithType;
+ }
+
+ public String getRawTableName() {
+ return _rawTableName;
+ }
+
+ public TableCache getTableCache() {
+ return _tableCache;
+ }
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/handler/MaterializedViewHandler.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/handler/MaterializedViewHandler.java
new file mode 100644
index 00000000000..49a7dff0c11
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/handler/MaterializedViewHandler.java
@@ -0,0 +1,149 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.handler;
+
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.materializedview.context.MaterializedViewContext;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Broker-facing materialized-view SPI. The single-stage broker request handler delegates all MV
+/// concerns through this interface so the bulk of MV logic stays in `pinot-materialized-view` and
+/// broker code remains unaware of the details.
+///
+/// Lifecycle hooks the broker calls:
+///
+/// 1. [#compile(MaterializedViewCompileContext)] — at compile time, returns a
+/// [MaterializedViewContext] the broker uses to decide between passthrough, full rewrite, or
+/// split execution.
+/// 2. [#executeSplit(MaterializedViewSplitExecutionContext)] — when the compile decision is
+/// split-rewrite, the handler owns building dual routes, attaching time boundaries, dual
+/// scatter-gather, and merging the results.
+/// 3. [#annotateResponse(BrokerResponseNative, MaterializedViewContext)] — adds MV metadata
+/// (candidates evaluated, MV used) onto the broker response.
+/// 4. [#invalidateBaseTable(String)] — called from the broker's resource state model when a
+/// base table goes OFFLINE / DROPPED.
+/// 5. [#supportsSplitRewrite()] — capability flag the broker queries to suppress split-rewrite
+/// plans on broker variants that cannot merge dual scatter-gather (e.g. gRPC streaming).
+///
+/// Implementations must be thread-safe; the broker invokes hooks concurrently from multiple
+/// request threads.
+public interface MaterializedViewHandler {
+
+ /// Lifecycle hook invoked once by the broker after the implementation is instantiated via
+ /// reflection. Implementations should subscribe to ZK paths, build their metadata cache, and
+ /// read any handler-specific configuration here.
+ ///
+ /// @param configuration broker-side configuration scoped to the MV-handler prefix
+ /// @param propertyStore Helix property store for ZK-backed metadata access
+ /// @param supportsSplitRewrite whether the broker variant this handler is wired into can merge
+ /// dual scatter-gather (gRPC-streaming brokers should be wired with `false` so split-rewrite
+ /// plans are suppressed at compile time)
+ default void init(PinotConfiguration configuration, ZkHelixPropertyStore<ZNRecord> propertyStore,
+ boolean supportsSplitRewrite) {
+ }
+
+ /// Compiles a user query against the configured MV catalog. Returns a [MaterializedViewContext]
+ /// describing the rewrite decision. Implementations must not mutate the input context's query.
+ MaterializedViewContext compile(MaterializedViewCompileContext ctx);
+
+ /// Executes a split-rewrite plan. Called by the broker when the compile decision was
+ /// [MaterializedViewContext#isSplitRewrite()]. The handler owns the entire dual scatter-gather
+ /// + reduce flow; it dispatches actual server I/O via the dispatcher callback supplied by the
+ /// broker subclass on [MaterializedViewSplitExecutionContext#getDispatcher()].
+ BrokerResponseNative executeSplit(MaterializedViewSplitExecutionContext ctx) throws Exception;
+
+ /// Annotates the broker response with MV-tracking metadata (candidate names, MV queried). Called
+ /// by the broker after both passthrough and rewrite paths so MV miss/hit counts surface uniformly.
+ void annotateResponse(BrokerResponseNative response, MaterializedViewContext mvContext);
+
+ /// Invalidates any cached state for the given base table. Called from the broker's resource state
+ /// model on ONLINE→OFFLINE / ONLINE→DROPPED transitions.
+ ///
+ /// @param rawTableName raw base-table name without type suffix
+ void invalidateBaseTable(String rawTableName);
+
+ /// Refreshes any cached state for the given table. Called from the broker's resource state
+ /// model on OFFLINE→ONLINE transitions so a previously-evicted MV cache entry is rebuilt
+ /// even when the underlying definition znode was not deleted (operator toggled the broker
+ /// resource OFFLINE/ONLINE for maintenance). Default no-op preserves backward compatibility
+ /// for implementations that don't track per-table cache state.
+ default void refreshTable(String rawTableName) {
+ }
+
+ /// Number of MV entries currently held by this handler's metadata cache, or `-1` if the
+ /// implementation does not track a cache (in which case the broker will not surface a
+ /// gauge). The broker polls this on a fixed interval to expose
+ /// `BrokerGauge.MATERIALIZED_VIEW_CACHE_ENTRY_COUNT` so operators can monitor for unbounded
+ /// growth that would otherwise indicate a leak in the ZK listener / drop path.
+ default int getCacheEntryCount() {
+ return -1;
+ }
+
+ /// Release any resources held by this handler (ZK listener subscriptions, executor pools,
+ /// open connections). Called from the broker's `shutDown()` so a hot-reload or test
+ /// teardown does not leak listener slots. Default no-op for handlers that hold no
+ /// reclaimable state.
+ default void close() {
+ }
+
+ /// Whether this handler supports split-rewrite execution (dual scatter-gather to base + MV with
+ /// a merged reduce). Broker variants that cannot perform the merge (e.g. gRPC streaming) should
+ /// be configured with a handler that returns `false`; the broker will then suppress split-rewrite
+ /// plans at compile time and either fall back to full-rewrite (if eligible) or execute the user
+ /// query against the base table unchanged.
+ boolean supportsSplitRewrite();
+
+ /// Config key (relative to `CommonConstants.Broker.MATERIALIZED_VIEW_HANDLER_CONFIG_PREFIX`)
+ /// holding the fully-qualified class name of a [MaterializedViewHandler] implementation.
+ /// Default is [DefaultMaterializedViewHandler]. Operators wiring a custom handler set this to
+ /// their own class name; their implementation receives the same config subset at [#init] time
+ /// so they can pass through any handler-specific options.
+ String HANDLER_CLASS_CONFIG_KEY = "class";
+
+ /// Loads, instantiates, and initializes the [MaterializedViewHandler] class named by the
+ /// [#HANDLER_CLASS_CONFIG_KEY] property of `configuration`. The implementation must have a
+ /// public no-arg constructor.
+ ///
+ /// @param configuration broker-side configuration already scoped to the MV-handler prefix (i.e.
+ /// after `_brokerConf.subset(MATERIALIZED_VIEW_HANDLER_CONFIG_PREFIX)`)
+ /// @param propertyStore Helix property store passed through to [#init]
+ /// @param supportsSplitRewrite see [#init]'s parameter of the same name
+ static MaterializedViewHandler loadHandler(PinotConfiguration configuration,
+ ZkHelixPropertyStore<ZNRecord> propertyStore, boolean supportsSplitRewrite) {
+ String className = configuration.getProperty(HANDLER_CLASS_CONFIG_KEY,
+ DefaultMaterializedViewHandler.class.getName());
+ Logger logger = LoggerFactory.getLogger(MaterializedViewHandler.class);
+ try {
+ logger.info("Instantiating MaterializedViewHandler class {}", className);
+ MaterializedViewHandler handler = (MaterializedViewHandler) Class.forName(className)
+ .getDeclaredConstructor().newInstance();
+ logger.info("Initializing MaterializedViewHandler class {} (supportsSplitRewrite={})",
+ className, supportsSplitRewrite);
+ handler.init(configuration, propertyStore, supportsSplitRewrite);
+ return handler;
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to load MaterializedViewHandler class: " + className, e);
+ }
+ }
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/handler/MaterializedViewSplitDispatcher.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/handler/MaterializedViewSplitDispatcher.java
new file mode 100644
index 00000000000..2bfc9391987
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/handler/MaterializedViewSplitDispatcher.java
@@ -0,0 +1,58 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.handler;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.core.routing.TableRouteInfo;
+import org.apache.pinot.spi.data.Schema;
+
+
+/// Callback supplied by the broker so the MV handler can perform the dual route-build +
+/// scatter-gather + reduce without depending on broker-internal types (request id, server stats,
+/// request context, table-cache helpers — all captured in the closure of the lambda the broker
+/// registers per-query).
+///
+/// The MV handler's job is to compute time boundaries and attach the boundary filters to the
+/// per-branch queries; the broker owns the rest of the route-and-dispatch dance because it
+/// touches generic broker internals (hybrid offline/realtime split, optimizer invocation,
+/// scatter-gather via `QueryRouter`, dual reduce via `BrokerReduceService`).
+@FunctionalInterface
+public interface MaterializedViewSplitDispatcher {
+
+ /// Builds offline/realtime broker requests for the base table (handling hybrid time-boundary +
+ /// always-false-filter pruning), builds the MV-side broker request and route, then issues two
+ /// scatter-gather requests and reduces them with `originalBrokerRequest` as the merge key.
+ ///
+ /// @param originalBrokerRequest the user's original query (used as the reduce-time merge key
+ /// so the user's LIMIT/OFFSET semantics are preserved).
+ /// @param baseQueryWithTimeFilter base-side server query with the `ts >= boundary` filter
+ /// already attached.
+ /// @param baseRouteInfo routing info pre-computed for the base table.
+ /// @param baseSchema base-table schema (for query optimization).
+ /// @param viewQueryWithTimeFilter MV-side server query with the
+ /// `materializedViewTime < boundary` filter already attached.
+ /// @param viewTableNameWithType fully-qualified MV table name.
+ /// @param viewSchema MV-table schema (for query optimization).
+ /// @param timeoutMs remaining timeout in milliseconds.
+ BrokerResponseNative dispatch(BrokerRequest originalBrokerRequest, PinotQuery baseQueryWithTimeFilter,
+ TableRouteInfo baseRouteInfo, Schema baseSchema, PinotQuery viewQueryWithTimeFilter,
+ String viewTableNameWithType, Schema viewSchema, long timeoutMs) throws Exception;
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/handler/MaterializedViewSplitExecutionContext.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/handler/MaterializedViewSplitExecutionContext.java
new file mode 100644
index 00000000000..9715da03ee8
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/handler/MaterializedViewSplitExecutionContext.java
@@ -0,0 +1,132 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.handler;
+
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.routing.TableRouteInfo;
+import org.apache.pinot.materializedview.context.MaterializedViewContext;
+import org.apache.pinot.spi.data.Schema;
+
+
+/// Inputs the broker passes to [MaterializedViewHandler#executeSplit] for a split-rewrite
+/// execution. The MV handler's job is to attach the per-branch time-boundary filters and then
+/// call back via the [MaterializedViewSplitDispatcher] for the actual route-build +
+/// scatter-gather + reduce, so this context only carries what the handler needs to compute and
+/// attach those filters.
+public final class MaterializedViewSplitExecutionContext {
+ private final BrokerRequest _originalBrokerRequest;
+ private final PinotQuery _baseServerPinotQuery;
+ private final Schema _baseSchema;
+ private final MaterializedViewContext _materializedViewContext;
+ private final TableRouteInfo _baseRouteInfo;
+ private final long _remainingTimeMs;
+ private final MaterializedViewSplitDispatcher _dispatcher;
+
+ private MaterializedViewSplitExecutionContext(Builder b) {
+ _originalBrokerRequest = b._originalBrokerRequest;
+ _baseServerPinotQuery = b._baseServerPinotQuery;
+ _baseSchema = b._baseSchema;
+ _materializedViewContext = b._materializedViewContext;
+ _baseRouteInfo = b._baseRouteInfo;
+ _remainingTimeMs = b._remainingTimeMs;
+ _dispatcher = b._dispatcher;
+ }
+
+ public BrokerRequest getOriginalBrokerRequest() {
+ return _originalBrokerRequest;
+ }
+
+ public PinotQuery getBaseServerPinotQuery() {
+ return _baseServerPinotQuery;
+ }
+
+ public Schema getBaseSchema() {
+ return _baseSchema;
+ }
+
+ public MaterializedViewContext getMaterializedViewContext() {
+ return _materializedViewContext;
+ }
+
+ public TableRouteInfo getBaseRouteInfo() {
+ return _baseRouteInfo;
+ }
+
+ public long getRemainingTimeMs() {
+ return _remainingTimeMs;
+ }
+
+ public MaterializedViewSplitDispatcher getDispatcher() {
+ return _dispatcher;
+ }
+
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ public static final class Builder {
+ private BrokerRequest _originalBrokerRequest;
+ private PinotQuery _baseServerPinotQuery;
+ private Schema _baseSchema;
+ private MaterializedViewContext _materializedViewContext;
+ private TableRouteInfo _baseRouteInfo;
+ private long _remainingTimeMs;
+ private MaterializedViewSplitDispatcher _dispatcher;
+
+ public Builder originalBrokerRequest(BrokerRequest originalBrokerRequest) {
+ _originalBrokerRequest = originalBrokerRequest;
+ return this;
+ }
+
+ public Builder baseServerPinotQuery(PinotQuery baseServerPinotQuery) {
+ _baseServerPinotQuery = baseServerPinotQuery;
+ return this;
+ }
+
+ public Builder baseSchema(Schema baseSchema) {
+ _baseSchema = baseSchema;
+ return this;
+ }
+
+ public Builder materializedViewContext(MaterializedViewContext materializedViewContext) {
+ _materializedViewContext = materializedViewContext;
+ return this;
+ }
+
+ public Builder baseRouteInfo(TableRouteInfo baseRouteInfo) {
+ _baseRouteInfo = baseRouteInfo;
+ return this;
+ }
+
+ public Builder remainingTimeMs(long remainingTimeMs) {
+ _remainingTimeMs = remainingTimeMs;
+ return this;
+ }
+
+ public Builder dispatcher(MaterializedViewSplitDispatcher dispatcher) {
+ _dispatcher = dispatcher;
+ return this;
+ }
+
+ public MaterializedViewSplitExecutionContext build() {
+ return new MaterializedViewSplitExecutionContext(this);
+ }
+ }
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/handler/package-info.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/handler/package-info.java
new file mode 100644
index 00000000000..9f7b6775631
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/handler/package-info.java
@@ -0,0 +1,45 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/// **Query rewrite — broker SPI surface.**
+///
+/// The broker delegates every MV concern through
+/// [MaterializedViewHandler][org.apache.pinot.materializedview.handler.MaterializedViewHandler]
+/// so the bulk of MV-aware logic stays out of `pinot-broker`. Hooks:
+///
+/// - [#compile][org.apache.pinot.materializedview.handler.MaterializedViewHandler#compile] —
+/// match the user query against the MV catalog; return a
+/// [MaterializedViewContext][org.apache.pinot.materializedview.context.MaterializedViewContext]
+/// describing the rewrite decision (no rewrite / full rewrite / split rewrite).
+/// - [#executeSplit][org.apache.pinot.materializedview.handler.MaterializedViewHandler#executeSplit] —
+/// own the dual scatter-gather + merge flow when the compile decision was SPLIT_REWRITE.
+/// The broker supplies the dispatcher callback that performs the actual server I/O.
+/// - [#annotateResponse][org.apache.pinot.materializedview.handler.MaterializedViewHandler#annotateResponse] —
+/// stamp the MV-used metadata onto the broker response.
+/// - [#invalidateBaseTable][org.apache.pinot.materializedview.handler.MaterializedViewHandler#invalidateBaseTable]
+/// / [#refreshTable][org.apache.pinot.materializedview.handler.MaterializedViewHandler#refreshTable] —
+/// called from the broker resource state model when a base table or MV table cycles
+/// ONLINE↔OFFLINE/DROPPED so the handler's cache stays consistent.
+/// - [#close][org.apache.pinot.materializedview.handler.MaterializedViewHandler#close] —
+/// release ZK listener slots / executor pools on broker shutdown.
+///
+/// [DefaultMaterializedViewHandler][org.apache.pinot.materializedview.handler.DefaultMaterializedViewHandler]
+/// is the in-tree implementation; operators can wire a custom handler via the
+/// `pinot.broker.materialized.view.handler.class` config.
+package org.apache.pinot.materializedview.handler;
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/metadata/MaterializedViewDefinitionMetadata.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/metadata/MaterializedViewDefinitionMetadata.java
index 0057f1219a3..28d851ce76f 100644
--- a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/metadata/MaterializedViewDefinitionMetadata.java
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/metadata/MaterializedViewDefinitionMetadata.java
@@ -46,6 +46,7 @@ public class MaterializedViewDefinitionMetadata {
private static final String SPLIT_SOURCE_TIME_COLUMN_KEY = "splitSourceTimeColumn";
private static final String SPLIT_SOURCE_TIME_FORMAT_KEY = "splitSourceTimeFormat";
private static final String SPLIT_MATERIALIZED_VIEW_TIME_COLUMN_KEY = "splitMaterializedViewTimeColumn";
+ private static final String SPLIT_MATERIALIZED_VIEW_TIME_FORMAT_KEY = "splitMaterializedViewTimeFormat";
private static final String SPLIT_BUCKET_MS_KEY = "splitBucketMs";
private static final String STALENESS_THRESHOLD_MS_KEY = "stalenessThresholdMs";
private static final String REWRITE_ENABLED_KEY = "rewriteEnabled";
@@ -143,6 +144,10 @@ public class MaterializedViewDefinitionMetadata {
znRecord.setSimpleField(SPLIT_SOURCE_TIME_COLUMN_KEY, _splitSpec.getSourceTimeColumn());
znRecord.setSimpleField(SPLIT_SOURCE_TIME_FORMAT_KEY, _splitSpec.getSourceTimeFormat());
znRecord.setSimpleField(SPLIT_MATERIALIZED_VIEW_TIME_COLUMN_KEY, _splitSpec.getMaterializedViewTimeColumn());
+ String viewFormat = _splitSpec.getMaterializedViewTimeFormat();
+ if (viewFormat != null) {
+ znRecord.setSimpleField(SPLIT_MATERIALIZED_VIEW_TIME_FORMAT_KEY, viewFormat);
+ }
znRecord.setLongField(SPLIT_BUCKET_MS_KEY, _splitSpec.getBucketMs());
}
@@ -176,9 +181,10 @@ public class MaterializedViewDefinitionMetadata {
if (sourceTimeColumn != null) {
String sourceTimeFormat = znRecord.getSimpleField(SPLIT_SOURCE_TIME_FORMAT_KEY);
String viewTimeColumn = znRecord.getSimpleField(SPLIT_MATERIALIZED_VIEW_TIME_COLUMN_KEY);
+ String viewTimeFormat = znRecord.getSimpleField(SPLIT_MATERIALIZED_VIEW_TIME_FORMAT_KEY);
long bucketMs = znRecord.getLongField(SPLIT_BUCKET_MS_KEY, 0L);
- splitSpec =
- new MaterializedViewSplitSpec(sourceTimeColumn, sourceTimeFormat, viewTimeColumn, bucketMs);
+ splitSpec = new MaterializedViewSplitSpec(sourceTimeColumn, sourceTimeFormat, viewTimeColumn,
+ viewTimeFormat, bucketMs);
}
long stalenessThresholdMs = znRecord.getLongField(STALENESS_THRESHOLD_MS_KEY, 0L);
@@ -205,13 +211,15 @@ public class MaterializedViewDefinitionMetadata {
private final String _sourceTimeColumn;
private final String _sourceTimeFormat;
private final String _materializedViewTimeColumn;
+ private final String _materializedViewTimeFormat;
private final long _bucketMs;
public MaterializedViewSplitSpec(String sourceTimeColumn, String sourceTimeFormat,
- String viewTimeColumn, long bucketMs) {
+ String viewTimeColumn, String viewTimeFormat, long bucketMs) {
_sourceTimeColumn = sourceTimeColumn;
_sourceTimeFormat = sourceTimeFormat;
_materializedViewTimeColumn = viewTimeColumn;
+ _materializedViewTimeFormat = viewTimeFormat;
_bucketMs = bucketMs;
}
@@ -227,6 +235,10 @@ public class MaterializedViewDefinitionMetadata {
return _materializedViewTimeColumn;
}
+ public String getMaterializedViewTimeFormat() {
+ return _materializedViewTimeFormat;
+ }
+
public long getBucketMs() {
return _bucketMs;
}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/metadata/package-info.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/metadata/package-info.java
new file mode 100644
index 00000000000..40d187085d9
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/metadata/package-info.java
@@ -0,0 +1,34 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/// **Definition — ZK znode schemas.**
+///
+/// The materialized-view subsystem persists two znodes per MV table:
+///
+/// - **Definition znode** — `MaterializedViewDefinitionMetadata`. Slowly-changing:
+/// `definedSQL`, base table list, rewrite-enable flag, optional split spec. Written by
+/// the controller at create time; read by the broker's MV metadata cache.
+/// - **Runtime znode** — `MaterializedViewRuntimeMetadata`. Rapidly-changing: watermark,
+/// per-partition state. Written by the minion task on each successful refresh; read by
+/// the broker's MV metadata cache.
+///
+/// Both znodes live under `PROPERTYSTORE/MATERIALIZED_VIEW/{DEFINITION,RUNTIME}` paths,
+/// the broker's cache subscribes to child-change + data-change events on both parent
+/// paths so updates propagate to the rewrite engine without polling.
+package org.apache.pinot.materializedview.metadata;
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/package-info.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/package-info.java
new file mode 100644
index 00000000000..3498e91379f
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/package-info.java
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/// Apache Pinot materialized view (MV) subsystem. An MV is an OFFLINE Pinot table whose rows
+/// are produced by a saved SQL query against one or more base Pinot tables. The MV is
+/// refreshed periodically by a minion task and the broker transparently rewrites user queries
+/// to read from the MV when its definition subsumes the query.
+///
+/// The subsystem is divided into three concerns, each living in its own sub-package family.
+/// New contributors should start with the role they're modifying:
+///
+/// **1. Definition** — how an MV is *declared* and stored.
+/// - [analysis][org.apache.pinot.materializedview.analysis] — parses and validates the MV's
+/// `definedSQL` at create time (legal aggregations, group-by/dimension columns,
+/// supported time expressions).
+/// - [metadata][org.apache.pinot.materializedview.metadata] — ZK znode schemas for the
+/// MV definition record and per-partition runtime state.
+/// - [consistency][org.apache.pinot.materializedview.consistency] — controller-side
+/// reconciler that keeps base-table / MV-table state aligned across drops, renames, and
+/// ingestion-config drift.
+///
+/// **2. View generation** — how the MV's *data* is produced.
+/// - [scheduler][org.apache.pinot.materializedview.scheduler] — controller-side minion
+/// task scheduler. Decides which time windows of the MV need to be (re)materialized
+/// and generates `PinotTaskConfig`s.
+/// - [executor][org.apache.pinot.materializedview.executor] — minion-side task executor
+/// that runs the MV's SQL, builds the resulting segments, and uploads them.
+///
+/// **3. Query rewrite** — how the broker *uses* an existing MV at query time.
+/// - [context][org.apache.pinot.materializedview.context] — broker-side compile-time
+/// carrier holding the rewrite plan and any per-mode (FULL_REWRITE / SPLIT_REWRITE)
+/// state the broker request handler consumes.
+/// - [handler][org.apache.pinot.materializedview.handler] — broker SPI
+/// ([MaterializedViewHandler][org.apache.pinot.materializedview.handler.MaterializedViewHandler])
+/// and default implementation. The broker delegates compile-time match, FULL_REWRITE
+/// swap, and SPLIT_REWRITE dual scatter-gather through this interface so the bulk of
+/// MV-aware logic lives outside `pinot-broker`.
+/// - [rewrite][org.apache.pinot.materializedview.rewrite] — the rewrite engine itself: a
+/// metadata cache that mirrors definition/runtime znodes, plus subsumption strategies
+/// (see [rewrite.strategy][org.apache.pinot.materializedview.rewrite.strategy]) and
+/// aggregation equivalence rules
+/// (see [rewrite.equivalence][org.apache.pinot.materializedview.rewrite.equivalence]).
+///
+/// Cross-package data flow at query time:
+///
+/// user query
+/// -> broker `BaseSingleStageBrokerRequestHandler`
+/// -> `handler.MaterializedViewHandler#compile`
+/// -> `rewrite.MaterializedViewQueryRewriteEngine`
+/// -> `rewrite.strategy.*` match
+/// -> `context.MaterializedViewContext`
+/// -> broker dispatches the rewritten query (FULL_REWRITE) or merges base + MV
+/// scatter-gather (SPLIT_REWRITE)
+package org.apache.pinot.materializedview;
diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/ExecutionMode.java
similarity index 50%
copy from pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java
copy to pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/ExecutionMode.java
index 7d7524b28ab..cbf21b1a41b 100644
--- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/ExecutionMode.java
@@ -16,29 +16,18 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.sql.parsers.rewriter;
+package org.apache.pinot.materializedview.rewrite;
-import java.util.Map;
-import java.util.stream.Collectors;
-import org.apache.pinot.spi.utils.CommonConstants;
+/// Determines how the broker should execute an MV-rewritten query.
+public enum ExecutionMode {
-public class RlsUtils {
+ /// The MV fully replaces the base table — the rewritten query targets the MV
+ /// table and no base-table query is needed.
+ FULL_REWRITE,
- private RlsUtils() {
- }
-
- public static String buildRlsFilterKey(String tableName) {
- return String.format("%s-%s", CommonConstants.RLS_FILTERS, tableName);
- }
-
- public static Map<String, String> extractRlsFilters(Map<String, String> requestMetadata) {
- return requestMetadata.entrySet().stream()
- .filter(e -> e.getKey().startsWith(CommonConstants.RLS_FILTERS))
- .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
- }
-
- public static String getRlsFilterForTable(Map<String, String> queryOptions, String tableName) {
- return queryOptions.get(buildRlsFilterKey(tableName));
- }
+ /// The MV covers only historical data (ts < watermarkMs). The broker
+ /// issues two queries — one against the MV and one against the base table
+ /// (ts >= watermarkMs) — and merges the results.
+ SPLIT_REWRITE
}
diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MatchType.java
similarity index 50%
copy from pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java
copy to pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MatchType.java
index 7d7524b28ab..aa9287112d6 100644
--- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MatchType.java
@@ -16,29 +16,18 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.sql.parsers.rewriter;
+package org.apache.pinot.materializedview.rewrite;
-import java.util.Map;
-import java.util.stream.Collectors;
-import org.apache.pinot.spi.utils.CommonConstants;
+/// Classifies how a user query was matched against a materialized view definition.
+public enum MatchType {
-public class RlsUtils {
+ /// User query is structurally identical to the MV definition.
+ EXACT,
- private RlsUtils() {
- }
+ /// User query is a scan-subset of the MV (projection subset, superset WHERE).
+ SCAN_SUBSUME,
- public static String buildRlsFilterKey(String tableName) {
- return String.format("%s-%s", CommonConstants.RLS_FILTERS, tableName);
- }
-
- public static Map<String, String> extractRlsFilters(Map<String, String> requestMetadata) {
- return requestMetadata.entrySet().stream()
- .filter(e -> e.getKey().startsWith(CommonConstants.RLS_FILTERS))
- .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
- }
-
- public static String getRlsFilterForTable(Map<String, String> queryOptions, String tableName) {
- return queryOptions.get(buildRlsFilterKey(tableName));
- }
+ /// User query requires re-aggregation over MV pre-computed columns.
+ AGG_REAGG
}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewMatchUtils.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewMatchUtils.java
new file mode 100644
index 00000000000..e7cd92297dc
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewMatchUtils.java
@@ -0,0 +1,385 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.rewrite;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.sql.FilterKind;
+
+
+/// Shared utility methods for materialized view subsumption matching strategies.
+///
+/// **Thread-safety:** stateless utility class. All methods are pure functions of their
+/// arguments and do not retain references to or mutate inputs. Safe to invoke concurrently
+/// from the broker hot path. Future contributors adding caching to any method must preserve
+/// this contract (use thread-safe data structures and avoid sharing per-thread state).
+public final class MaterializedViewMatchUtils {
+
+ private MaterializedViewMatchUtils() {
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Alias handling
+ /// -----------------------------------------------------------------------
+
+ /// Strips the top-level `AS` alias from an expression if present.
+ /// For `as(expr, alias)` returns `expr`; otherwise returns the
+ /// expression unchanged.
+ ///
+ /// @param expr the expression to unwrap
+ /// @return the inner expression without the alias wrapper
+ public static Expression stripAlias(Expression expr) {
+ if (expr.getType() == ExpressionType.FUNCTION) {
+ Function func = expr.getFunctionCall();
+ if (func != null && "as".equals(func.getOperator())) {
+ return func.getOperands().get(0);
+ }
+ }
+ return expr;
+ }
+
+ /// Extracts the output column name that a SELECT expression maps to in the
+ /// MV table schema.
+ ///
+ ///
+ /// - `as(expr, alias)` → alias identifier name
+ /// - bare identifier → identifier name
+ /// - anything else → throws [IllegalStateException]
+ ///
+ ///
+ /// @param expr a SELECT-list expression from the MV's compiled query
+ /// @return the MV table column name
+ public static String extractMaterializedViewColumnName(Expression expr) {
+ Function func = expr.getFunctionCall();
+ if (func != null && "as".equals(func.getOperator())) {
+ Expression aliasExpr = func.getOperands().get(1);
+ return aliasExpr.getIdentifier().getName();
+ }
+ if (expr.getType() == ExpressionType.IDENTIFIER) {
+ return expr.getIdentifier().getName();
+ }
+ throw new IllegalStateException(
+ "Cannot extract MV column name from expression: " + RequestUtils.prettyPrint(expr));
+ }
+
+ /// Builds a mapping from alias-stripped expressions to MV table column names
+ /// for all entries in the MV's SELECT list.
+ ///
+/// Example: for `SELECT city, SUM(revenue) AS sum_rev FROM ...`
+/// the returned map is:
+/// ```text
+/// Identifier("city") -> "city"
+/// Function("SUM", [Identifier("revenue")]) -> "sum_rev"
+/// ```
+ ///
+ /// @param viewQuery the MV's compiled PinotQuery
+ /// @return map from stripped expression to MV column name
+ public static Map<Expression, String> buildMaterializedViewProjectionMap(PinotQuery viewQuery) {
+ List<Expression> selectList = viewQuery.getSelectList();
+ if (selectList == null) {
+ return Map.of();
+ }
+ Map<Expression, String> map = new HashMap<>(selectList.size());
+ for (Expression expr : selectList) {
+ Expression stripped = stripAlias(expr);
+ String columnName = extractMaterializedViewColumnName(expr);
+ map.put(stripped, columnName);
+ }
+ return map;
+ }
+
+ /// Extracts the user-specified alias name from an expression, if present.
+ /// Returns `null` for expressions without an alias.
+ ///
+ /// @param expr a SELECT-list expression from the user's query
+ /// @return the alias name, or `null` if the expression has no alias
+ @Nullable
+ public static String extractUserAlias(Expression expr) {
+ Function func = expr.getFunctionCall();
+ if (func != null && "as".equals(func.getOperator())) {
+ return func.getOperands().get(1).getIdentifier().getName();
+ }
+ return null;
+ }
+
+ /// Rewrites the user query's SELECT list by replacing each expression with
+ /// its corresponding MV table column name, while preserving the user's
+ /// original alias if one was specified.
+ ///
+ /// For example, given:
+ /// ```sql
+ /// MV defined: SELECT city, SUM(revenue) AS sum_rev FROM orders GROUP BY city
+ /// User query: SELECT city, SUM(revenue) AS r_sum FROM orders GROUP BY city
+ /// ```
+ /// the rewritten SELECT list will be:
+ /// ```sql
+ /// SELECT city, sum_rev AS r_sum FROM materialized_view_table GROUP BY city
+ /// ```
+ ///
+ /// @param userSelectList the user query's original SELECT expressions
+ /// @param viewProjectionMap alias-stripped expression → MV column name
+ /// @return the rewritten SELECT expression list
+ public static List<Expression> rewriteSelectList(List<Expression> userSelectList,
+ Map<Expression, String> viewProjectionMap) {
+ List<Expression> rewritten = new ArrayList<>(userSelectList.size());
+ for (Expression expr : userSelectList) {
+ Expression stripped = stripAlias(expr);
+ String materializedViewColumnName = viewProjectionMap.get(stripped);
+ Expression materializedViewColExpr = RequestUtils.getIdentifierExpression(materializedViewColumnName);
+
+ String userAlias = extractUserAlias(expr);
+ String resultAlias = userAlias != null ? userAlias
+ : implicitAliasForUserExpression(stripped, materializedViewColumnName);
+ if (resultAlias != null) {
+ materializedViewColExpr = RequestUtils.getFunctionExpression("as", materializedViewColExpr,
+ RequestUtils.getIdentifierExpression(resultAlias));
+ }
+ rewritten.add(materializedViewColExpr);
+ }
+ return rewritten;
+ }
+
+ /// Computes an implicit alias so the rewritten result column carries the same name the user
+ /// would have seen against the base table. Without this, a user query like
+ /// `SELECT SUM(revenue) FROM orders` rewritten against an MV column `sum_rev` would surface
+ /// `sum_rev` to the client (silently breaking any consumer reading results by column name).
+ ///
+ /// Returns `null` (no alias needed) when the user expression is already a bare identifier whose
+ /// name matches the MV column name — in that case the natural result column name already
+ /// matches the user's expectation.
+ @Nullable
+ private static String implicitAliasForUserExpression(Expression strippedUserExpr, String materializedViewColumnName) {
+ if (strippedUserExpr.getType() == ExpressionType.IDENTIFIER
+ && strippedUserExpr.getIdentifier().getName().equals(materializedViewColumnName)) {
+ return null;
+ }
+ return RequestUtils.prettyPrint(strippedUserExpr);
+ }
+
+ /// -----------------------------------------------------------------------
+ /// WHERE / residual filter
+ /// -----------------------------------------------------------------------
+
+ /// Checks whether the user's WHERE filter is an AND-superset of the MV's
+ /// WHERE filter and extracts the residual predicates.
+ ///
+ /// Rules:
+ ///
+ /// - MV has no filter, user has a filter → entire user filter is residual
+ /// - User has no filter, MV has a filter → not a superset, returns null
+ /// - User conjuncts contain all MV conjuncts → difference is residual
+ /// - Otherwise → returns null (no match)
+ ///
+ ///
+ /// **TODO(materialized-view): OR / IN subsumption is currently correctness-safe but conservative.**
+ /// OR and IN nodes are treated as opaque atomic conjuncts and compared via
+ /// `Objects.equals` (structure + operand-order sensitive), not by semantic
+ /// set containment. The following patterns are rejected today even though the MV
+ /// technically covers the user's rows; users fall back to the base table:
+ ///
+ /// - OR-operand reorder — MV `a OR b` vs user `b OR a`
+ /// - IN-operand reorder — MV `x IN (1,2,3)` vs user `x IN (3,1,2)`
+ /// - IN / OR superset — MV `x IN (1,2,3)` (or `a OR b OR c`)
+ /// covers user `x IN (1,2)`, `x = 1`, or `a OR b`
+ /// - IN and chained-OR equivalence — `x IN (1,2)` vs `x = 1 OR x = 2`
+ ///
+ /// Before relaxing any of the above, add regression tests covering the opposite
+ /// direction (MV narrower than user, e.g. MV `x IN (1,2)` vs user
+ /// `x IN (1,2,3)`) so the correctness guardrail is not eroded.
+ ///
+ /// @param userFilter the user query's filter expression (may be null)
+ /// @param materializedViewFilter the MV query's filter expression (may be null)
+ /// @return the residual filter expression, or `null` if the user filter
+ /// is not a valid superset of the MV filter
+ @Nullable
+ public static Expression tryExtractResidualFilter(@Nullable Expression userFilter,
+ @Nullable Expression materializedViewFilter) {
+ if (materializedViewFilter == null && userFilter != null) {
+ return userFilter;
+ }
+ if (userFilter == null) {
+ return null;
+ }
+
+ Set<Expression> userConjuncts = flattenAnd(userFilter);
+ Set<Expression> materializedViewConjuncts = flattenAnd(materializedViewFilter);
+
+ if (!userConjuncts.containsAll(materializedViewConjuncts)) {
+ return null;
+ }
+
+ Set<Expression> residual = new LinkedHashSet<>(userConjuncts);
+ residual.removeAll(materializedViewConjuncts);
+
+ if (residual.isEmpty()) {
+ return null;
+ }
+
+ return buildAndExpression(new ArrayList<>(residual));
+ }
+
+ /// Flattens a filter expression into a set of AND conjuncts. If the top-level
+ /// operator is AND, it is recursively unwrapped; otherwise the entire expression
+ /// is treated as a single conjunct.
+ ///
+ /// Uses [LinkedHashSet] so that residual filter ordering (and any plan-string
+ /// renderings derived from it) is deterministic across rewrites of the same query —
+ /// avoids surprising non-deterministic AND-operand orderings in logs and any
+ /// downstream `equals`-based plan caching.
+ public static Set<Expression> flattenAnd(Expression filter) {
+ Set<Expression> conjuncts = new LinkedHashSet<>();
+ collectAndConjuncts(filter, conjuncts);
+ return conjuncts;
+ }
+
+ private static void collectAndConjuncts(Expression expr, Set<Expression> conjuncts) {
+ if (expr.getType() == ExpressionType.FUNCTION) {
+ Function function = expr.getFunctionCall();
+ if (function != null && FilterKind.AND.name().equals(function.getOperator())) {
+ for (Expression operand : function.getOperands()) {
+ collectAndConjuncts(operand, conjuncts);
+ }
+ return;
+ }
+ }
+ conjuncts.add(expr);
+ }
+
+ /// Builds an AND expression from the given list of operands. Returns the
+ /// single operand directly when the list has exactly one element.
+ public static Expression buildAndExpression(List<Expression> operands) {
+ if (operands.size() == 1) {
+ return operands.get(0);
+ }
+ return RequestUtils.getFunctionExpression(FilterKind.AND.name(), operands);
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Expression remapping
+ /// -----------------------------------------------------------------------
+
+ /// Recursively rewrites an expression tree by replacing any sub-expression
+ /// found in `viewProjectionMap` with a simple identifier referencing the
+ /// corresponding MV column name.
+ ///
+ /// This is used to transform HAVING, ORDER BY, and residual WHERE
+ /// expressions from base-table semantics to MV-table semantics. For example,
+ /// `SUM(revenue) > 1000` becomes `sum_rev > 1000` when the
+ /// projection map contains `SUM(revenue) -> "sum_rev"`.
+ ///
+ /// @param expr the expression to remap
+ /// @param viewProjectionMap alias-stripped expression → MV column name
+ /// @return the remapped expression (may be the same instance if nothing changed)
+ public static Expression remapExpression(Expression expr, Map<Expression, String> viewProjectionMap) {
+ if (viewProjectionMap.containsKey(expr)) {
+ return RequestUtils.getIdentifierExpression(viewProjectionMap.get(expr));
+ }
+
+ if (expr.getType() == ExpressionType.FUNCTION) {
+ Function func = expr.getFunctionCall();
+ if (func != null && func.getOperands() != null) {
+ List<Expression> originalOperands = func.getOperands();
+ List<Expression> remapped = new ArrayList<>(originalOperands.size());
+ boolean changed = false;
+ for (Expression operand : originalOperands) {
+ Expression result = remapExpression(operand, viewProjectionMap);
+ remapped.add(result);
+ if (result != operand) {
+ changed = true;
+ }
+ }
+ if (changed) {
+ return RequestUtils.getFunctionExpression(func.getOperator(), remapped);
+ }
+ }
+ }
+
+ return expr;
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Column reference collection
+ /// -----------------------------------------------------------------------
+
+ /// Collects all column identifiers referenced by the given expression.
+ /// Useful for verifying that a residual filter or ORDER BY clause only
+ /// references columns available in the MV table.
+ ///
+ /// @param expr the expression to scan
+ /// @return set of column name strings
+ public static Set<String> collectReferencedColumns(Expression expr) {
+ Set<String> columns = new HashSet<>();
+ collectColumnsRecursive(expr, columns);
+ return columns;
+ }
+
+ /// Returns whether every column name in `requiredColumns` (user/base-side names) is reachable
+ /// through `viewProjectionMap` — i.e. there exists a key in the map whose form is an
+ /// [ExpressionType#IDENTIFIER] with the matching name. Used by SCAN-style strategies to
+ /// validate that a residual WHERE filter and ORDER BY clause only reference columns the MV
+ /// actually projects.
+ ///
+ /// Without this helper, callers that compare against `viewProjectionMap.values()`
+ /// (MV-side column names) would over-reject MVs whose SELECT list aliases the column —
+ /// e.g. `SELECT base_col AS mv_col` would silently disqualify any user query referencing
+ /// `base_col` in the residual filter even though [#remapExpression] can correctly rewrite
+ /// the reference to `mv_col`.
+ public static boolean coversReferencedColumns(Set<String> requiredColumns,
+ Map<Expression, String> viewProjectionMap) {
+ if (requiredColumns.isEmpty()) {
+ return true;
+ }
+ Set<String> covered = new HashSet<>(viewProjectionMap.size());
+ for (Expression key : viewProjectionMap.keySet()) {
+ if (key.getType() == ExpressionType.IDENTIFIER) {
+ covered.add(key.getIdentifier().getName());
+ }
+ }
+ return covered.containsAll(requiredColumns);
+ }
+
+ private static void collectColumnsRecursive(Expression expr, Set<String> columns) {
+ if (expr.getType() == ExpressionType.IDENTIFIER) {
+ String name = expr.getIdentifier().getName();
+ if (!"*".equals(name)) {
+ columns.add(name);
+ }
+ return;
+ }
+ if (expr.getType() == ExpressionType.FUNCTION) {
+ Function func = expr.getFunctionCall();
+ if (func != null && func.getOperands() != null) {
+ for (Expression operand : func.getOperands()) {
+ collectColumnsRecursive(operand, columns);
+ }
+ }
+ }
+ }
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewMetadataCache.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewMetadataCache.java
new file mode 100644
index 00000000000..08a3a432e7f
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewMetadataCache.java
@@ -0,0 +1,714 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.rewrite;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.annotation.Nullable;
+import org.apache.commons.collections4.CollectionUtils;
+import org.apache.helix.AccessOption;
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.helix.zookeeper.zkclient.IZkChildListener;
+import org.apache.helix.zookeeper.zkclient.IZkDataListener;
+import org.apache.pinot.common.metadata.ZKMetadataProvider;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.materializedview.metadata.MaterializedViewDefinitionMetadata;
+import org.apache.pinot.materializedview.metadata.MaterializedViewDefinitionMetadata.MaterializedViewSplitSpec;
+import org.apache.pinot.materializedview.metadata.MaterializedViewRuntimeMetadata;
+import org.apache.pinot.materializedview.metadata.PartitionInfo;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Broker-side cache that maintains a reverse index from base table names to their
+/// materialized view entries. Subscribes to two ZK paths:
+///
+/// - `/CONFIGS/MATERIALIZED_VIEW/DEFINITION` — low-frequency changes trigger SQL recompilation
+/// - `/CONFIGS/MATERIALIZED_VIEW/RUNTIME` — high-frequency changes only update mutable
+/// runtime state (watermarkMs, partitions map) without any SQL parsing
+///
+///
+/// Thread-safety: all mutations go through synchronized ZK listener callbacks;
+/// reads use a [ConcurrentHashMap] and are lock-free.
+public class MaterializedViewMetadataCache {
+ private static final Logger LOGGER = LoggerFactory.getLogger(MaterializedViewMetadataCache.class);
+
+ private static final String MATERIALIZED_VIEW_DEFINITION_PARENT_PATH =
+ ZKMetadataProvider.getPropertyStorePathForMaterializedViewDefinitionPrefix();
+ private static final String MATERIALIZED_VIEW_DEFINITION_PATH_PREFIX = MATERIALIZED_VIEW_DEFINITION_PARENT_PATH + "/";
+ private static final String MATERIALIZED_VIEW_RUNTIME_PARENT_PATH =
+ ZKMetadataProvider.getPropertyStorePathForMaterializedViewRuntimePrefix();
+ private static final String MATERIALIZED_VIEW_RUNTIME_PATH_PREFIX = MATERIALIZED_VIEW_RUNTIME_PARENT_PATH + "/";
+
+ private final ZkHelixPropertyStore<ZNRecord> _propertyStore;
+ private final ZkDefinitionListener _definitionListener = new ZkDefinitionListener();
+ private final ZkRuntimeListener _runtimeListener = new ZkRuntimeListener();
+ /// Shared lock so that definition and runtime listener callbacks are mutually exclusive,
+ /// preventing a concurrent putDefinitionEntry from overwriting a runtime update that arrived
+ /// between the _materializedViewEntryMap.get and _materializedViewEntryMap.put in putDefinitionEntry.
+ private final Object _cacheLock = new Object();
+
+ private final Map<String, MaterializedViewCacheEntry> _materializedViewEntryMap = new ConcurrentHashMap<>();
+ /// Holds runtime znode payloads that arrive BEFORE the matching definition. ZK does not
+ /// guarantee ordering between sibling child-change notifications, and on broker startup the
+ /// runtime znode listener may fire ahead of the definition znode listener. Dropping the
+ /// runtime update on the floor would leave the MV in a cold-start state until the next
+ /// scheduler tick re-publishes the runtime znode. Pending entries are consumed (and removed)
+ /// when [#putDefinitionEntry] eventually constructs the cache entry.
+ ///
+ /// All reads and writes of this map happen inside `synchronized (_cacheLock)`, so a plain
+ /// HashMap is sufficient — using ConcurrentHashMap here would imply lock-free reads which the
+ /// rest of the code does not rely on.
+ private final Map<String, MaterializedViewRuntimeMetadata> _pendingRuntimeStates = new HashMap<>();
+ private final Map<String, List<MaterializedViewCacheEntry>> _baseTableToMaterializedViewMap =
+ new ConcurrentHashMap<>();
+
+ /// Explicit subscription ledgers for definition and runtime data watchers.
+ ///
+ /// Without these, [#close] would reconstruct the set of subscribed paths from
+ /// `_materializedViewEntryMap` + `_pendingRuntimeStates` — but a watcher subscribed via
+ /// `subscribeDataChanges` BEFORE `_propertyStore.get` (in [#addDefinitions] and
+ /// [#loadRuntimeStates]) survives even when the get returns `null` and no map entry is
+ /// produced. That watcher would then leak on shutdown / hot reload. The ledger makes the
+ /// set of active subscriptions authoritative.
+ ///
+ /// Guarded by `_cacheLock`. Subscribe/unsubscribe goes through the helper methods below so
+ /// the ledger never drifts from the actual ZK subscription state.
+ private final Set<String> _subscribedDefinitionPaths = new HashSet<>();
+ private final Set<String> _subscribedRuntimePaths = new HashSet<>();
+
+ public MaterializedViewMetadataCache(ZkHelixPropertyStore<ZNRecord> propertyStore) {
+ _propertyStore = propertyStore;
+
+ synchronized (_cacheLock) {
+ _propertyStore.subscribeChildChanges(MATERIALIZED_VIEW_DEFINITION_PARENT_PATH, _definitionListener);
+ _propertyStore.subscribeChildChanges(MATERIALIZED_VIEW_RUNTIME_PARENT_PATH, _runtimeListener);
+
+ List<String> defChildren =
+ _propertyStore.getChildNames(MATERIALIZED_VIEW_DEFINITION_PARENT_PATH, AccessOption.PERSISTENT);
+ if (CollectionUtils.isNotEmpty(defChildren)) {
+ List<String> defPaths = new ArrayList<>(defChildren.size());
+ List<String> rtPaths = new ArrayList<>(defChildren.size());
+ for (String viewTableName : defChildren) {
+ defPaths.add(MATERIALIZED_VIEW_DEFINITION_PATH_PREFIX + viewTableName);
+ rtPaths.add(MATERIALIZED_VIEW_RUNTIME_PATH_PREFIX + viewTableName);
+ }
+ addDefinitions(defPaths);
+ loadRuntimeStates(rtPaths);
+ }
+ }
+
+ LOGGER.info("Initialized MaterializedViewMetadataCache with {} materialized view entries",
+ _materializedViewEntryMap.size());
+ }
+
+ @Nullable
+ public List<MaterializedViewCacheEntry> getMaterializedViewEntriesForBaseTable(String rawBaseTableName) {
+ return _baseTableToMaterializedViewMap.get(rawBaseTableName);
+ }
+
+ /// Removes all MV cache entries that reference the given raw base table name. Called when a base
+ /// table is deleted so stale reverse-index entries do not survive to contaminate queries against
+ /// a newly created table with the same name.
+ ///
+ /// Every MV that lists `rawBaseTableName` among its base tables is evicted, regardless of whether
+ /// the MV references a single base table or multiple. The drop+recreate pattern (operator drops
+ /// `tableA`, recreates with a new schema) would otherwise leave the broker holding a stale
+ /// `compiledQuery` that targets a column shape that no longer matches the new `tableA`, leading
+ /// to silent wrong results. The MV's definition ZNode listener will repopulate the cache only
+ /// when an authoritative new definition is published.
+ public void invalidateBaseTable(String rawBaseTableName) {
+ /// Hold _cacheLock across the entire invalidate so that the (unsubscribe + map
+ /// mutation) operation is atomic with respect to concurrent ZK listener callbacks
+ /// (which also synchronize on _cacheLock). The Helix ZkHelixPropertyStore's
+ /// subscribe/unsubscribe methods only mutate internal listener maps under their
+ /// own short critical sections; they do not call back into user code, so there is
+ /// no lock-ordering cycle with _cacheLock. Splitting the work across two critical
+ /// sections (with unsubscribe outside the lock) reintroduces races where a
+ /// concurrent putDefinitionEntry between the unsubscribe and the second lock
+ /// acquisition would leave a fresh entry with no live data subscription.
+ synchronized (_cacheLock) {
+ /// Evict every MV that lists `rawBaseTableName` among its base tables.
+ List<MaterializedViewCacheEntry> entries = _baseTableToMaterializedViewMap.get(rawBaseTableName);
+ if (entries != null) {
+ for (MaterializedViewCacheEntry entry : new ArrayList<>(entries)) {
+ removeDefinitionEntry(
+ MATERIALIZED_VIEW_DEFINITION_PATH_PREFIX + entry.getMaterializedViewTableNameWithType());
+ }
+ }
+ /// Also evict any MV cache entry whose own tableNameWithType starts with `rawBaseTableName_`.
+ /// The broker resource state model fires invalidate{Base,}Table on every table that goes
+ /// ONLINE→OFFLINE/DROPPED, including MV tables themselves (MVs are OFFLINE Pinot tables in
+ /// Helix). Without this, a dropped MV remains in `_materializedViewEntryMap` until the
+ /// definition znode listener fires; during the gap the broker may compile queries against
+ /// the now-offline MV.
+ ///
+ /// `removeDefinitionEntry` unsubscribes BOTH the definition and the paired runtime data
+ /// listeners (see implementation), so a definition znode that hasn't been deleted in ZK
+ /// will not auto-resubscribe. The parent-path child-change listener stays subscribed —
+ /// that's required to detect new MV creations and is correct. Operator guidance: drop
+ /// the MV definition znode before taking the MV table OFFLINE in Helix to avoid the brief
+ /// QUERY_REWRITE_EXCEPTIONS noise between the OFFLINE transition and the znode delete.
+ String offlineMvName = TableNameBuilder.OFFLINE.tableNameWithType(rawBaseTableName);
+ if (_materializedViewEntryMap.containsKey(offlineMvName)) {
+ removeDefinitionEntry(MATERIALIZED_VIEW_DEFINITION_PATH_PREFIX + offlineMvName);
+ }
+ }
+ }
+
+ /// Number of MV entries currently held in the cache. Surfaced so the broker can expose this
+ /// as a gauge for unbounded-growth detection — a cluster with K MVs should plateau near K;
+ /// sustained growth signals a leak in the ZK listener / drop path.
+ public int size() {
+ return _materializedViewEntryMap.size();
+ }
+
+ /// Unsubscribe every ZK listener this cache installed (parent-path child watchers + per-MV
+ /// definition + runtime data watchers). Called from the broker's shutdown path so a hot
+ /// reload or test teardown doesn't leak Helix watcher slots. After close the cache must
+ /// not be reused.
+ public void close() {
+ synchronized (_cacheLock) {
+ _propertyStore.unsubscribeChildChanges(MATERIALIZED_VIEW_DEFINITION_PARENT_PATH, _definitionListener);
+ _propertyStore.unsubscribeChildChanges(MATERIALIZED_VIEW_RUNTIME_PARENT_PATH, _runtimeListener);
+ /// Iterate the subscription ledgers, not the entry maps. A watcher subscribed before a
+ /// failed `_propertyStore.get` (in [#addDefinitions] / [#loadRuntimeStates]) survives even
+ /// when no map entry was ever produced — without the ledger, those watcher slots would
+ /// leak on shutdown / hot reload.
+ for (String path : _subscribedDefinitionPaths) {
+ _propertyStore.unsubscribeDataChanges(path, _definitionListener);
+ }
+ for (String path : _subscribedRuntimePaths) {
+ _propertyStore.unsubscribeDataChanges(path, _runtimeListener);
+ }
+ _subscribedDefinitionPaths.clear();
+ _subscribedRuntimePaths.clear();
+ _materializedViewEntryMap.clear();
+ _baseTableToMaterializedViewMap.clear();
+ _pendingRuntimeStates.clear();
+ }
+ }
+
+ /// Rebuilds cache entries that were evicted during an earlier ONLINE→OFFLINE transition for the
+ /// given table. Called from the broker resource state model on OFFLINE→ONLINE transitions so a
+ /// previous cycle (or a transient broker-resource rebalance) does not leave this broker
+ /// permanently unable to consider an MV until the definition znode is republished.
+ ///
+ /// Two complementary scenarios are handled because the broker resource state model fires for
+ /// BOTH base tables and MV tables — and `invalidateBaseTable` evicts cache entries in both
+ /// cases (an MV evicted because its base table cycled is the more common one):
+ ///
+ /// 1. **MV table cycled** — the transitioning table IS the MV. Look up its own definition
+ /// znode under `<rawTableName>_OFFLINE` and reload it.
+ /// 2. **Base table cycled** — the transitioning table is a base referenced by one or more
+ /// MVs. Walk every MV definition znode currently in ZK, decode its base-table list, and
+ /// reload any whose base tables include either `rawTableName_OFFLINE` or
+ /// `rawTableName_REALTIME` and that are missing from the in-memory cache. Without this
+ /// path, a base-table OFFLINE→ONLINE bounce permanently silences MV rewrite on this broker.
+ ///
+ /// Idempotent: entries already in the cache are skipped; absent definition znodes (genuine
+ /// drop) are also skipped. Subscribes the matching runtime listener + rebuilds runtime state
+ /// alongside the definition.
+ public void refreshTable(String rawTableName) {
+ String offlineMvName = TableNameBuilder.OFFLINE.tableNameWithType(rawTableName);
+ String offlineMvDefPath = MATERIALIZED_VIEW_DEFINITION_PATH_PREFIX + offlineMvName;
+ synchronized (_cacheLock) {
+ /// Case 1: the transitioning table is an MV — direct rehydrate of its own entry.
+ if (!_materializedViewEntryMap.containsKey(offlineMvName)
+ && _propertyStore.exists(offlineMvDefPath, AccessOption.PERSISTENT)) {
+ addDefinitions(List.of(offlineMvDefPath));
+ String offlineMvRuntimePath = MATERIALIZED_VIEW_RUNTIME_PATH_PREFIX + offlineMvName;
+ if (_propertyStore.exists(offlineMvRuntimePath, AccessOption.PERSISTENT)) {
+ loadRuntimeStates(List.of(offlineMvRuntimePath));
+ }
+ }
+
+ /// Case 2: the transitioning table may be a BASE table referenced by other MVs whose
+ /// entries were evicted during the matching `invalidateBaseTable`. Walk every MV
+ /// definition child znode (a low-frequency operation — only on resource state transitions)
+ /// and reload any whose base-table list mentions this table and whose in-memory entry is
+ /// gone.
+ List<String> defChildren =
+ _propertyStore.getChildNames(MATERIALIZED_VIEW_DEFINITION_PARENT_PATH, AccessOption.PERSISTENT);
+ if (CollectionUtils.isEmpty(defChildren)) {
+ return;
+ }
+ /// `baseTables` stores the raw base-table name (no type suffix); compare against the raw
+ /// form of the transitioning table. We also accept typed siblings (`<raw>_OFFLINE` /
+ /// `<raw>_REALTIME`) so a definition created with a typed name (older format or operator
+ /// typo) is still picked up.
+ String rawSibling = TableNameBuilder.extractRawTableName(rawTableName);
+ String offlineSibling = TableNameBuilder.OFFLINE.tableNameWithType(rawSibling);
+ String realtimeSibling = TableNameBuilder.REALTIME.tableNameWithType(rawSibling);
+ List<String> defPathsToReload = new ArrayList<>();
+ List<String> runtimePathsToReload = new ArrayList<>();
+ for (String mvViewTableName : defChildren) {
+ if (_materializedViewEntryMap.containsKey(mvViewTableName)) {
+ continue;
+ }
+ String defPath = MATERIALIZED_VIEW_DEFINITION_PATH_PREFIX + mvViewTableName;
+ ZNRecord znRecord = _propertyStore.get(defPath, null, AccessOption.PERSISTENT);
+ if (znRecord == null) {
+ continue;
+ }
+ MaterializedViewDefinitionMetadata definition = MaterializedViewDefinitionMetadata.fromZNRecord(znRecord);
+ List<String> baseTables = definition.getBaseTables();
+ if (baseTables == null) {
+ continue;
+ }
+ if (baseTables.contains(rawSibling)
+ || baseTables.contains(offlineSibling)
+ || baseTables.contains(realtimeSibling)) {
+ defPathsToReload.add(defPath);
+ runtimePathsToReload.add(MATERIALIZED_VIEW_RUNTIME_PATH_PREFIX + mvViewTableName);
+ }
+ }
+ if (!defPathsToReload.isEmpty()) {
+ addDefinitions(defPathsToReload);
+ /// loadRuntimeStates already tolerates missing znodes (the get returns null and the entry
+ /// stays at cold-start until the runtime listener fires the first update).
+ loadRuntimeStates(runtimePathsToReload);
+ }
+ }
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Definition path handling (low frequency — triggers SQL recompilation)
+ /// -----------------------------------------------------------------------
+
+ private void addDefinitions(List<String> paths) {
+ for (String path : paths) {
+ subscribeDefinitionWatcher(path);
+ }
+ List<ZNRecord> znRecords = _propertyStore.get(paths, null, AccessOption.PERSISTENT);
+ for (ZNRecord znRecord : znRecords) {
+ if (znRecord != null) {
+ try {
+ putDefinitionEntry(znRecord);
+ } catch (Exception e) {
+ LOGGER.error("Failed to add MV definition for ZNRecord: {}", znRecord.getId(), e);
+ }
+ }
+ }
+ }
+
+ private void putDefinitionEntry(ZNRecord znRecord) {
+ MaterializedViewDefinitionMetadata definition = MaterializedViewDefinitionMetadata.fromZNRecord(znRecord);
+ String viewTableNameWithType = definition.getMaterializedViewTableNameWithType();
+
+ PinotQuery compiledQuery = null;
+ String definedSql = definition.getDefinedSql();
+ if (definedSql != null && !definedSql.isEmpty()) {
+ try {
+ compiledQuery = CalciteSqlParser.compileToPinotQuery(definedSql);
+ } catch (Exception e) {
+ /// ERROR (not WARN): a null compiledQuery silently disables MV rewrite for this view —
+ /// every match attempt returns null without logging. Operators must be alerted at
+ /// ingestion time. The cache entry is still stored (with compiledQuery=null) so the
+ /// controller-facing definition listing remains visible; only rewrite is disabled.
+ LOGGER.error("Failed to compile definedSql for MV {}; rewrite disabled until the SQL is "
+ + "fixed and the definition znode is updated. SQL: {}", viewTableNameWithType, definedSql, e);
+ }
+ }
+
+ MaterializedViewCacheEntry existing = _materializedViewEntryMap.get(viewTableNameWithType);
+ MaterializedViewCacheEntry newEntry;
+ if (existing != null) {
+ newEntry = new MaterializedViewCacheEntry(definition, compiledQuery,
+ existing.getWatermarkMs(), existing.getPartitions());
+ } else {
+ /// Honor a runtime znode that arrived ahead of this definition znode (ZK does not
+ /// guarantee sibling-listener ordering). Without this, the new entry would start at
+ /// (watermarkMs=0, partitions=Map.of()) and the broker would treat the MV as cold until
+ /// the next scheduler tick re-publishes the runtime znode.
+ MaterializedViewRuntimeMetadata pending = _pendingRuntimeStates.remove(viewTableNameWithType);
+ if (pending != null) {
+ newEntry = new MaterializedViewCacheEntry(definition, compiledQuery,
+ pending.getWatermarkMs(), pending.getPartitions());
+ } else {
+ newEntry = new MaterializedViewCacheEntry(definition, compiledQuery, 0L, Map.of());
+ }
+ }
+
+ MaterializedViewCacheEntry oldEntry = _materializedViewEntryMap.put(viewTableNameWithType, newEntry);
+ if (oldEntry != null) {
+ removeFromReverseIndex(oldEntry);
+ }
+ addToReverseIndex(newEntry);
+ }
+
+ private void removeDefinitionEntry(String path) {
+ unsubscribeDefinitionWatcher(path);
+ String viewTableNameWithType = path.substring(MATERIALIZED_VIEW_DEFINITION_PATH_PREFIX.length());
+ /// Also unsubscribe the paired runtime listener and drop any pending runtime payload — an MV
+ /// drop leaves no consumer for either, and the runtime znode may linger if the operator only
+ /// deleted the definition. Without this, every dropped MV leaks one Helix watcher slot.
+ unsubscribeRuntimeWatcher(MATERIALIZED_VIEW_RUNTIME_PATH_PREFIX + viewTableNameWithType);
+ _pendingRuntimeStates.remove(viewTableNameWithType);
+ MaterializedViewCacheEntry removed = _materializedViewEntryMap.remove(viewTableNameWithType);
+ if (removed != null) {
+ removeFromReverseIndex(removed);
+ }
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Runtime path handling (high frequency — only updates scalar fields)
+ /// -----------------------------------------------------------------------
+
+ private void loadRuntimeStates(List<String> paths) {
+ for (String path : paths) {
+ subscribeRuntimeWatcher(path);
+ }
+ List<ZNRecord> znRecords = _propertyStore.get(paths, null, AccessOption.PERSISTENT);
+ for (ZNRecord znRecord : znRecords) {
+ if (znRecord != null) {
+ try {
+ updateRuntimeState(znRecord);
+ } catch (Exception e) {
+ LOGGER.error("Failed to load MV runtime for ZNRecord: {}", znRecord.getId(), e);
+ }
+ }
+ }
+ }
+
+ /// Subscribe + ledger. Idempotent — re-subscribing the same path is a no-op at the ZK layer
+ /// (Helix dedupes by `(path, listener)`) but the ledger insertion stays unique.
+ private void subscribeDefinitionWatcher(String path) {
+ _propertyStore.subscribeDataChanges(path, _definitionListener);
+ _subscribedDefinitionPaths.add(path);
+ }
+
+ private void subscribeRuntimeWatcher(String path) {
+ _propertyStore.subscribeDataChanges(path, _runtimeListener);
+ _subscribedRuntimePaths.add(path);
+ }
+
+ /// Unsubscribe + ledger. Idempotent — if the ledger already lost the entry (e.g. a previous
+ /// close), the ZK unsubscribe is still attempted so a stale watcher is cleared anyway.
+ private void unsubscribeDefinitionWatcher(String path) {
+ _propertyStore.unsubscribeDataChanges(path, _definitionListener);
+ _subscribedDefinitionPaths.remove(path);
+ }
+
+ private void unsubscribeRuntimeWatcher(String path) {
+ _propertyStore.unsubscribeDataChanges(path, _runtimeListener);
+ _subscribedRuntimePaths.remove(path);
+ }
+
+ private void updateRuntimeState(ZNRecord znRecord) {
+ String viewTableNameWithType = znRecord.getId();
+ MaterializedViewRuntimeMetadata runtime = MaterializedViewRuntimeMetadata.fromZNRecord(znRecord);
+ MaterializedViewCacheEntry entry = _materializedViewEntryMap.get(viewTableNameWithType);
+ if (entry == null) {
+ /// Definition znode listener may fire after the runtime znode listener; buffer this update
+ /// until the matching definition arrives via [#putDefinitionEntry]. See _pendingRuntimeStates.
+ LOGGER.debug("Buffering runtime update for MV pending definition load: {}", viewTableNameWithType);
+ _pendingRuntimeStates.put(viewTableNameWithType, runtime);
+ return;
+ }
+ entry.setRuntimeState(runtime.getWatermarkMs(), runtime.getPartitions());
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Reverse index management
+ /// -----------------------------------------------------------------------
+
+ private void addToReverseIndex(MaterializedViewCacheEntry entry) {
+ String mvTableName = entry.getDefinition().getMaterializedViewTableNameWithType();
+ for (String baseTable : entry.getDefinition().getBaseTables()) {
+ _baseTableToMaterializedViewMap.compute(baseTable, (key, existing) -> {
+ /// Always return an unmodifiableList — the broker query path iterates without copying, so a
+ /// mutable singleton would race with a concurrent compute() that copies-and-replaces.
+ /// Dedupe on viewTableName so a duplicate add (e.g. cold-start initial-load racing with
+ /// a queued listener callback for the same MV) does not produce a doubled candidate that
+ /// the rewrite engine would evaluate twice per query.
+ if (existing == null) {
+ return List.of(entry);
+ }
+ for (MaterializedViewCacheEntry existingEntry : existing) {
+ if (mvTableName.equals(existingEntry.getDefinition().getMaterializedViewTableNameWithType())) {
+ /// Replace in place so the latest entry wins (covers definition-update flows).
+ List<MaterializedViewCacheEntry> updated = new ArrayList<>(existing);
+ updated.replaceAll(e -> mvTableName.equals(e.getDefinition().getMaterializedViewTableNameWithType())
+ ? entry : e);
+ return Collections.unmodifiableList(updated);
+ }
+ }
+ List<MaterializedViewCacheEntry> updated = new ArrayList<>(existing);
+ updated.add(entry);
+ return Collections.unmodifiableList(updated);
+ });
+ }
+ }
+
+ private void removeFromReverseIndex(MaterializedViewCacheEntry entry) {
+ for (String baseTable : entry.getDefinition().getBaseTables()) {
+ _baseTableToMaterializedViewMap.compute(baseTable, (key, existing) -> {
+ if (existing == null) {
+ return null;
+ }
+ List<MaterializedViewCacheEntry> updated = new ArrayList<>(existing);
+ updated.removeIf(e -> e.getDefinition().getMaterializedViewTableNameWithType()
+ .equals(entry.getDefinition().getMaterializedViewTableNameWithType()));
+ return updated.isEmpty() ? null : Collections.unmodifiableList(updated);
+ });
+ }
+ }
+
+ /// -----------------------------------------------------------------------
+ /// ZK listeners
+ /// -----------------------------------------------------------------------
+
+ private class ZkDefinitionListener implements IZkChildListener, IZkDataListener {
+ @Override
+ public void handleChildChange(String path, List<String> children) {
+ synchronized (_cacheLock) {
+ Set<String> newChildSet = CollectionUtils.isNotEmpty(children)
+ ? new HashSet<>(children) : Collections.emptySet();
+
+ /// Evict entries that are no longer present in ZK (MV table deleted).
+ for (String existing : new ArrayList<>(_materializedViewEntryMap.keySet())) {
+ if (!newChildSet.contains(existing)) {
+ removeDefinitionEntry(MATERIALIZED_VIEW_DEFINITION_PATH_PREFIX + existing);
+ }
+ }
+
+ /// Register newly added entries.
+ List<String> defPathsToAdd = new ArrayList<>();
+ List<String> rtPathsToAdd = new ArrayList<>();
+ for (String viewTableName : newChildSet) {
+ if (!_materializedViewEntryMap.containsKey(viewTableName)) {
+ defPathsToAdd.add(MATERIALIZED_VIEW_DEFINITION_PATH_PREFIX + viewTableName);
+ rtPathsToAdd.add(MATERIALIZED_VIEW_RUNTIME_PATH_PREFIX + viewTableName);
+ }
+ }
+ if (!defPathsToAdd.isEmpty()) {
+ addDefinitions(defPathsToAdd);
+ loadRuntimeStates(rtPathsToAdd);
+ }
+ }
+ }
+
+ @Override
+ public void handleDataChange(String path, Object data) {
+ synchronized (_cacheLock) {
+ if (data != null) {
+ try {
+ putDefinitionEntry((ZNRecord) data);
+ } catch (Exception e) {
+ LOGGER.error("Failed to refresh MV definition for: {}", path, e);
+ }
+ }
+ }
+ }
+
+ @Override
+ public void handleDataDeleted(String path) {
+ synchronized (_cacheLock) {
+ removeDefinitionEntry(path);
+ }
+ }
+ }
+
+ private class ZkRuntimeListener implements IZkChildListener, IZkDataListener {
+ @Override
+ public void handleChildChange(String path, List<String> children) {
+ synchronized (_cacheLock) {
+ Set<String> newChildSet = CollectionUtils.isNotEmpty(children)
+ ? new HashSet<>(children) : Collections.emptySet();
+
+ /// Unsubscribe data listeners for runtime nodes that are no longer present.
+ for (String viewTableName : _materializedViewEntryMap.keySet()) {
+ if (!newChildSet.contains(viewTableName)) {
+ markRuntimeStateMissing(viewTableName);
+ unsubscribeRuntimeWatcher(MATERIALIZED_VIEW_RUNTIME_PATH_PREFIX + viewTableName);
+ }
+ }
+ /// Evict pending runtime payloads whose runtime znode has disappeared without a matching
+ /// definition ever arriving — they would otherwise leak indefinitely. Also unsubscribe
+ /// the data watcher we subscribed when the orphan first arrived — `removeIf` alone only
+ /// drops the in-memory map entry, leaving the ZK watcher slot held.
+ Iterator<String> pendingIter = _pendingRuntimeStates.keySet().iterator();
+ while (pendingIter.hasNext()) {
+ String pendingKey = pendingIter.next();
+ if (!newChildSet.contains(pendingKey)) {
+ pendingIter.remove();
+ unsubscribeRuntimeWatcher(MATERIALIZED_VIEW_RUNTIME_PATH_PREFIX + pendingKey);
+ }
+ }
+
+ /// Subscribe for any newly visible runtime nodes.
+ List<String> rtPathsToAdd = new ArrayList<>();
+ for (String viewTableName : newChildSet) {
+ if (_materializedViewEntryMap.containsKey(viewTableName)) {
+ rtPathsToAdd.add(MATERIALIZED_VIEW_RUNTIME_PATH_PREFIX + viewTableName);
+ }
+ }
+ if (!rtPathsToAdd.isEmpty()) {
+ loadRuntimeStates(rtPathsToAdd);
+ }
+ }
+ }
+
+ @Override
+ public void handleDataChange(String path, Object data) {
+ synchronized (_cacheLock) {
+ if (data != null) {
+ try {
+ updateRuntimeState((ZNRecord) data);
+ } catch (Exception e) {
+ LOGGER.error("Failed to refresh MV runtime for: {}", path, e);
+ }
+ }
+ }
+ }
+
+ @Override
+ public void handleDataDeleted(String path) {
+ synchronized (_cacheLock) {
+ String viewTableNameWithType = path.substring(MATERIALIZED_VIEW_RUNTIME_PATH_PREFIX.length());
+ markRuntimeStateMissing(viewTableNameWithType);
+ /// Drop any pending payload too — an orphan runtime znode (no matching definition ever
+ /// arrived) leaves an entry in _pendingRuntimeStates that would otherwise live forever.
+ _pendingRuntimeStates.remove(viewTableNameWithType);
+ unsubscribeRuntimeWatcher(path);
+ }
+ }
+ }
+
+ private void markRuntimeStateMissing(String viewTableNameWithType) {
+ MaterializedViewCacheEntry entry = _materializedViewEntryMap.get(viewTableNameWithType);
+ if (entry != null) {
+ entry.setRuntimeState(0L, Map.of());
+ }
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Cache entry
+ /// -----------------------------------------------------------------------
+
+ /// A cached entry holding the MV definition (with pre-compiled [PinotQuery]) and a
+ /// volatile atomic reference to runtime state (`watermarkMs` + per-partition map).
+ ///
+ /// Runtime state is published through a single `volatile` field so readers always
+ /// see a consistent `(watermarkMs, partitions)` pair — never a mix of values from
+ /// different ZK updates.
+ ///
+ /// Thread-safety: ZK listener mutations are synchronized on the cache lock. Reads
+ /// of the reverse index and entry map are unsynchronized; a transient window during
+ /// definition updates may show both old and new entries for the same base table, but
+ /// this is safe (worst case: a duplicate candidate is evaluated and discarded).
+ public static class MaterializedViewCacheEntry {
+ private final MaterializedViewDefinitionMetadata _definition;
+ private final PinotQuery _compiledQuery;
+ /// Cached projection map (alias-stripped expr → MV column name) computed once at entry
+ /// construction. Empty map when `_compiledQuery` is null. The map is unmodifiable so it
+ /// can be shared across query threads without copying.
+ private final Map<Expression, String> _viewProjectionMap;
+ /// Cached flattened-AND conjunct set of the MV's WHERE clause. Empty set when the MV
+ /// has no filter or `_compiledQuery` is null. Unmodifiable for safe cross-thread sharing.
+ private final Set<Expression> _viewFilterConjuncts;
+
+ /// Immutable pair published atomically.
+ private volatile RuntimeState _runtimeState;
+
+ public MaterializedViewCacheEntry(MaterializedViewDefinitionMetadata definition,
+ @Nullable PinotQuery compiledQuery, long watermarkMs, Map<Long, PartitionInfo> partitions) {
+ _definition = definition;
+ _compiledQuery = compiledQuery;
+ _runtimeState = new RuntimeState(watermarkMs, partitions);
+ if (compiledQuery == null) {
+ _viewProjectionMap = Map.of();
+ _viewFilterConjuncts = Set.of();
+ } else {
+ _viewProjectionMap =
+ Collections.unmodifiableMap(MaterializedViewMatchUtils.buildMaterializedViewProjectionMap(compiledQuery));
+ Expression viewFilter = compiledQuery.getFilterExpression();
+ _viewFilterConjuncts = viewFilter == null
+ ? Set.of()
+ : Collections.unmodifiableSet(MaterializedViewMatchUtils.flattenAnd(viewFilter));
+ }
+ }
+
+ public MaterializedViewDefinitionMetadata getDefinition() {
+ return _definition;
+ }
+
+ @Nullable
+ public PinotQuery getCompiledQuery() {
+ return _compiledQuery;
+ }
+
+ public Map<Expression, String> getViewProjectionMap() {
+ return _viewProjectionMap;
+ }
+
+ public Set<Expression> getViewFilterConjuncts() {
+ return _viewFilterConjuncts;
+ }
+
+ /// Scheduling watermark. Under Design C the broker uses this as the split point for
+ /// the 2-way SPLIT_REWRITE (MV-side: `time < watermarkMs`; base-side: `time >= watermarkMs`).
+ /// Per-bucket eligibility is checked against `getPartitions()` for finer routing in V2.
+ public long getWatermarkMs() {
+ return _runtimeState._watermarkMs;
+ }
+
+ /// Immutable snapshot of `bucketStart → PartitionInfo`. A bucket's absence means the MV
+ /// does not cover that time range.
+ public Map<Long, PartitionInfo> getPartitions() {
+ return _runtimeState._partitions;
+ }
+
+ /// Atomically replaces both runtime scalars.
+ void setRuntimeState(long watermarkMs, Map<Long, PartitionInfo> partitions) {
+ _runtimeState = new RuntimeState(watermarkMs, partitions);
+ }
+
+ private static final class RuntimeState {
+ final long _watermarkMs;
+ final Map<Long, PartitionInfo> _partitions;
+
+ RuntimeState(long watermarkMs, Map<Long, PartitionInfo> partitions) {
+ _watermarkMs = watermarkMs;
+ _partitions = partitions.isEmpty() ? Map.of() : Map.copyOf(partitions);
+ }
+ }
+
+ @Nullable
+ public MaterializedViewSplitSpec getSplitSpec() {
+ return _definition.getSplitSpec();
+ }
+
+ public String getMaterializedViewTableNameWithType() {
+ return _definition.getMaterializedViewTableNameWithType();
+ }
+ }
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewQueryRewriteEngine.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewQueryRewriteEngine.java
new file mode 100644
index 00000000000..c94f0f07a25
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewQueryRewriteEngine.java
@@ -0,0 +1,413 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.rewrite;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.materializedview.metadata.MaterializedViewDefinitionMetadata.MaterializedViewSplitSpec;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewMetadataCache.MaterializedViewCacheEntry;
+import org.apache.pinot.materializedview.rewrite.strategy.MaterializedViewMatchStrategy;
+import org.apache.pinot.spi.data.DateTimeFormatSpec;
+import org.apache.pinot.sql.FilterKind;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Orchestrates materialized view query rewriting in two phases:
+///
+///
+/// - **Static matching** — strategies produce [MaterializedViewRewritePlan]
+/// fragments (rewritten MV query + match type + cost) without considering
+/// runtime state.
+/// - **Runtime gating and plan resolution** — freshness is checked
+/// before matching; after matching, execution mode is determined and
+/// split-compatibility is verified.
+///
+///
+/// For each candidate MV, strategies are tried in registration order (highest
+/// precision first). Once a strategy produces a plan that also passes runtime
+/// compatibility checks ([#resolvePlan]), no lower-precision strategies
+/// are attempted for the same MV. If a strategy matches structurally but its
+/// plan is rejected by runtime checks (e.g. EXACT match incompatible with
+/// split-mode GROUP BY), the engine falls through to the next strategy.
+/// The overall best plan across all candidates is then selected by lowest cost.
+///
+/// Thread-safety: this class is immutable after construction and safe to share.
+public class MaterializedViewQueryRewriteEngine {
+ private static final Logger LOGGER = LoggerFactory.getLogger(MaterializedViewQueryRewriteEngine.class);
+
+ private final MaterializedViewMetadataCache _materializedViewMetadataCache;
+ private final List<MaterializedViewMatchStrategy> _strategies;
+
+ public MaterializedViewQueryRewriteEngine(MaterializedViewMetadataCache materializedViewMetadataCache,
+ List<MaterializedViewMatchStrategy> strategies) {
+ _materializedViewMetadataCache = materializedViewMetadataCache;
+ _strategies = List.copyOf(strategies);
+ }
+
+ /// Removes all MV cache entries whose base tables include the given raw table name.
+ /// Called when a base table is deleted so stale reverse-index entries do not survive.
+ public void invalidateBaseTable(String rawBaseTableName) {
+ _materializedViewMetadataCache.invalidateBaseTable(rawBaseTableName);
+ }
+
+ /// Rebuilds the MV cache entry for the given table name from ZK if the cache is currently
+ /// missing it. Called from the broker's OFFLINE→ONLINE state transition so an MV whose
+ /// broker resource was cycled (without deleting the definition znode) becomes queryable
+ /// again without requiring a broker restart or a definition-znode republish.
+ public void refreshTable(String rawTableName) {
+ _materializedViewMetadataCache.refreshTable(rawTableName);
+ }
+
+ /// Number of MV entries currently held in the underlying metadata cache. Exposed so the
+ /// broker can wire a gauge on this value.
+ public int getCacheEntryCount() {
+ return _materializedViewMetadataCache.size();
+ }
+
+ /// Release any resources held by this engine. Delegates to the underlying metadata cache so
+ /// its ZK listener subscriptions are unwound.
+ public void close() {
+ _materializedViewMetadataCache.close();
+ }
+
+ /// Attempts to rewrite the given query to use a materialized view.
+ ///
+ /// @param pinotQuery the compiled user query
+ /// @param rawBaseTableName the raw (no type suffix) name of the base table being queried
+ /// @return the best rewrite plan for the user query, or `null` if no candidate MVs exist for
+ /// the base table or none matched
+ @Nullable
+ public MaterializedViewRewritePlan tryRewrite(PinotQuery pinotQuery, String rawBaseTableName) {
+ List<MaterializedViewCacheEntry> candidates =
+ _materializedViewMetadataCache.getMaterializedViewEntriesForBaseTable(rawBaseTableName);
+ if (candidates == null || candidates.isEmpty()) {
+ return null;
+ }
+
+ MaterializedViewRewritePlan bestPlan = null;
+ String bestStrategyName = null;
+
+ for (MaterializedViewCacheEntry candidate : candidates) {
+ if (!isEligible(candidate)) {
+ continue;
+ }
+
+ for (MaterializedViewMatchStrategy strategy : _strategies) {
+ /// Strategies signal "no structural match" by returning null and "fully resolved plan"
+ /// by returning non-null; any thrown exception is a strategy bug or contract violation
+ /// and is surfaced here rather than silently swallowed. Swallowing would mask a bug
+ /// behind a less-precise strategy match (or a base-only fallback) without operator
+ /// visibility on a hot path. The caller (`BaseSingleStageBrokerRequestHandler`) wraps
+ /// the whole rewrite in its own try/catch so a thrown exception still falls back to the
+ /// base query, but surfaces the stack via the standard query-error path.
+ MaterializedViewRewritePlan plan = strategy.match(pinotQuery, candidate);
+ if (plan == null) {
+ continue;
+ }
+
+ plan = resolvePlan(plan, candidate, pinotQuery);
+ if (plan == null) {
+ /// Strategy matched structurally but the plan is incompatible with the required
+ /// execution mode (e.g. EXACT + split + GROUP BY). Try lower-precision strategies for
+ /// this candidate.
+ continue;
+ }
+
+ if (bestPlan == null || plan.compareTo(bestPlan) < 0) {
+ bestPlan = plan;
+ bestStrategyName = strategy.getClass().getSimpleName();
+ }
+ /// A fully resolved plan was found; skip lower-precision strategies.
+ break;
+ }
+ }
+
+ if (bestPlan != null) {
+ LOGGER.info("MV rewrite succeeded for table [{}]: strategy={}, materializedViewTable={}, "
+ + "matchType={}, execMode={}, cost={}, originalQuery=[{}], rewrittenQuery=[{}]",
+ rawBaseTableName, bestStrategyName, bestPlan.getMaterializedViewTableNameWithType(),
+ bestPlan.getMatchType(), bestPlan.getExecMode(), bestPlan.getCost(),
+ pinotQuery, bestPlan.getMaterializedViewQuery());
+ } else if (LOGGER.isDebugEnabled()) {
+ List<String> candidateNames = new ArrayList<>(candidates.size());
+ for (MaterializedViewCacheEntry candidate : candidates) {
+ candidateNames.add(candidate.getMaterializedViewTableNameWithType());
+ }
+ LOGGER.debug("MV rewrite miss for table [{}]: evaluated {} candidate(s)={}, "
+ + "queryShape={}, userQuery=[{}]",
+ rawBaseTableName, candidates.size(), candidateNames,
+ MaterializedViewQueryShape.classify(pinotQuery), pinotQuery);
+ }
+
+ return bestPlan;
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Runtime gating — separated from AST matching
+ /// -----------------------------------------------------------------------
+
+ /// Checks whether a candidate MV is eligible for rewrite based on runtime state.
+ /// Under Design C eligibility is:
+ /// 1. `rewriteEnabled=true` on the MV definition (operator kill switch)
+ /// 2. `watermarkMs > 0` (cold-start guard — the scheduler has committed coverage up to here)
+ /// 3. If `stalenessThresholdMs > 0`: `now - watermarkMs <= stalenessThresholdMs` (SLO)
+ ///
+ /// Per-bucket eligibility is enforced later by the handler / dispatcher. Note: an empty
+ /// partition map is permitted — V1 routing uses `watermarkMs` as the split point, so the
+ /// partition map is only consulted for V2 N-way routing.
+ private static boolean isEligible(MaterializedViewCacheEntry candidate) {
+ if (!candidate.getDefinition().isRewriteEnabled()) {
+ LOGGER.debug("MV skip [{}]: rewriteEnabled=false on definition",
+ candidate.getMaterializedViewTableNameWithType());
+ return false;
+ }
+ if (candidate.getWatermarkMs() <= 0) {
+ LOGGER.debug("MV skip [{}]: watermarkMs={} (cold-start, no data yet)",
+ candidate.getMaterializedViewTableNameWithType(), candidate.getWatermarkMs());
+ return false;
+ }
+ long stalenessThresholdMs = candidate.getDefinition().getStalenessThresholdMs();
+ if (stalenessThresholdMs > 0) {
+ long ageMs = System.currentTimeMillis() - candidate.getWatermarkMs();
+ if (ageMs > stalenessThresholdMs) {
+ LOGGER.debug("MV skip [{}]: staleness {}ms exceeds threshold {}ms",
+ candidate.getMaterializedViewTableNameWithType(), ageMs, stalenessThresholdMs);
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Execution mode resolution — separated from AST matching
+ /// -----------------------------------------------------------------------
+
+ /// Resolves the execution mode for a plan fragment produced by a strategy,
+ /// and checks split-mode compatibility.
+ ///
+ /// The decision is driven by whether the MV definition includes a
+ /// [MaterializedViewSplitSpec]:
+ ///
+ /// - **No split spec** (batch MV): the MV is assumed to cover all base
+ /// table data. Returns [ExecutionMode#FULL_REWRITE].
+ /// - **Has split spec** (incremental MV): the MV covers only historical
+ /// data. `watermarkMs` must be > 0 (i.e. at least one APPEND
+ /// has successfully completed) for the MV to be queryable. If coverage is
+ /// not yet confirmed, returns `null` to skip this MV.
+ ///
+ ///
+ /// In split mode, EXACT match is rejected when the user query has GROUP BY
+ /// because it replaces aggregation functions with plain MV column references,
+ /// producing incompatible DataTable schemas during merge.
+ ///
+ /// @return the resolved plan with execution mode set, or `null` if the
+ /// plan is incompatible with the required execution mode
+ @Nullable
+ private static MaterializedViewRewritePlan resolvePlan(MaterializedViewRewritePlan plan,
+ MaterializedViewCacheEntry candidate, PinotQuery userQuery) {
+ MaterializedViewSplitSpec splitSpec = candidate.getSplitSpec();
+
+ if (splitSpec != null) {
+ long watermarkMs = candidate.getWatermarkMs();
+ if (watermarkMs <= 0) {
+ /// Incremental MV has no confirmed coverage yet (cold-start: generator
+ /// initialized the watermark but no APPEND has completed). Skip this MV
+ /// to avoid split queries against an empty MV table.
+ LOGGER.info("MV skip [{}]: cold-start, no coverage confirmed yet "
+ + "(watermark initialized but no APPEND completed), matchType={}",
+ candidate.getMaterializedViewTableNameWithType(), plan.getMatchType());
+ return null;
+ }
+
+ /// Split mode needs both sides of the boundary filter:
+ /// base branch: sourceTime >= watermarkMs
+ /// MV branch: materializedViewTime < watermarkMs
+ /// Both columns are TIMESTAMP (enforced at create time), so only the column names matter
+ /// here. A missing MV column name indicates malformed metadata or a partial upgrade —
+ /// skip the candidate rather than emit an unguarded split.
+ if (splitSpec.getMaterializedViewTimeColumn() == null || splitSpec.getMaterializedViewTimeColumn().isEmpty()) {
+ LOGGER.warn("MV skip [{}]: split spec missing MV-side time column; "
+ + "cannot attach materializedViewTime < boundary filter. Skipping to avoid double-counting.",
+ candidate.getMaterializedViewTableNameWithType());
+ return null;
+ }
+
+ if (isQueryFullyCoveredByMaterializedView(userQuery, splitSpec, watermarkMs)) {
+ LOGGER.info("MV full rewrite [{}]: user time filter is fully covered by watermarkMs={}, matchType={}",
+ candidate.getMaterializedViewTableNameWithType(), watermarkMs, plan.getMatchType());
+ return plan.withExecMode(ExecutionMode.FULL_REWRITE, null, 0);
+ }
+
+ /// EXACT match replaces aggregation functions (e.g. SUM(col)) with plain
+ /// MV column references (e.g. col_sum), producing physical column types
+ /// on the MV side that are incompatible with the base table's aggregation
+ /// intermediate types during split-mode merge. The schema mismatch applies
+ /// whenever the MV definition has any aggregation (GROUP BY or aggregate
+ /// functions in SELECT) — not only when the user query has GROUP BY. Reject
+ /// and let AggregationSubsumptionStrategy handle the query instead.
+ if (plan.getMatchType() == MatchType.EXACT) {
+ /// EXACT match is only produced by ExactSubsumptionStrategy, which itself
+ /// requires a compiled MV definition to perform the structural match. Reaching
+ /// EXACT with a null compiled query indicates a contract violation in the
+ /// strategy chain — fail loud rather than silently fall through to the
+ /// schema-incompatible split-merge path.
+ PinotQuery viewQuery = candidate.getCompiledQuery();
+ Preconditions.checkState(viewQuery != null,
+ "EXACT match without a compiled MV definition for table: %s",
+ candidate.getMaterializedViewTableNameWithType());
+ if (MaterializedViewQueryShape.classify(viewQuery) == MaterializedViewQueryShape.AGGREGATION) {
+ LOGGER.info("MV skip [{}]: EXACT match rejected in split mode for aggregation MV "
+ + "(schema incompatibility during merge, falling back to AGG_REAGG)",
+ candidate.getMaterializedViewTableNameWithType());
+ return null;
+ }
+ }
+
+ /// AGG_REAGG plans that use non-distributive re-aggregation (e.g. COUNT->SUM) are
+ /// not split-safe: the base side produces COUNT intermediates while the MV side
+ /// produces SUM intermediates, and the broker reducer (using the original COUNT
+ /// function) will misinterpret the MV DataTables. For incremental MVs, FULL_REWRITE
+ /// is also unsafe because watermarkMs only proves historical coverage before the
+ /// boundary; routing only to the MV would silently drop newer base-table rows.
+ if (!plan.isSplitSafe()) {
+ LOGGER.info("MV skip [{}]: AGG_REAGG plan is not split-safe "
+ + "(non-distributive re-aggregation such as COUNT->SUM). "
+ + "Cannot use SPLIT_REWRITE or partial FULL_REWRITE safely.",
+ candidate.getMaterializedViewTableNameWithType());
+ return null;
+ }
+
+ return plan.withExecMode(ExecutionMode.SPLIT_REWRITE, splitSpec, watermarkMs);
+ }
+
+ return plan.withExecMode(ExecutionMode.FULL_REWRITE, null, 0);
+ }
+
+ private static boolean isQueryFullyCoveredByMaterializedView(PinotQuery userQuery,
+ MaterializedViewSplitSpec splitSpec, long watermarkMs) {
+ Expression filter = userQuery.getFilterExpression();
+ if (filter == null) {
+ return false;
+ }
+ /// The user's WHERE filter is expressed in the base column's native unit (which may be
+ /// INT-days, INT-seconds, TIMESTAMP-millis, etc.), so the boundary literal must be converted
+ /// to the same unit before comparison.
+ String sourceTimeFormat = splitSpec.getSourceTimeFormat();
+ if (sourceTimeFormat == null || sourceTimeFormat.isEmpty()) {
+ return false;
+ }
+ BigDecimal coverageBoundary =
+ parseDecimal(new DateTimeFormatSpec(sourceTimeFormat).fromMillisToFormat(watermarkMs));
+ if (coverageBoundary == null) {
+ return false;
+ }
+ String sourceTimeColumn = splitSpec.getSourceTimeColumn();
+ for (Expression conjunct : MaterializedViewMatchUtils.flattenAnd(filter)) {
+ if (isUpperBoundWithinCoverage(conjunct, sourceTimeColumn, coverageBoundary)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private static boolean isUpperBoundWithinCoverage(Expression expr, String sourceTimeColumn,
+ BigDecimal coverageBoundary) {
+ Function function = expr.getFunctionCall();
+ if (function == null || function.getOperandsSize() != 2) {
+ return false;
+ }
+ List<Expression> operands = function.getOperands();
+ Expression left = operands.get(0);
+ Expression right = operands.get(1);
+ String operator = function.getOperator();
+
+ if (isIdentifier(left, sourceTimeColumn)) {
+ BigDecimal literal = getNumericLiteral(right);
+ if (literal == null) {
+ return false;
+ }
+ if (FilterKind.LESS_THAN.name().equalsIgnoreCase(operator)) {
+ return literal.compareTo(coverageBoundary) <= 0;
+ }
+ if (FilterKind.LESS_THAN_OR_EQUAL.name().equalsIgnoreCase(operator)) {
+ return literal.compareTo(coverageBoundary) < 0;
+ }
+ return false;
+ }
+
+ if (isIdentifier(right, sourceTimeColumn)) {
+ BigDecimal literal = getNumericLiteral(left);
+ if (literal == null) {
+ return false;
+ }
+ if (FilterKind.GREATER_THAN.name().equalsIgnoreCase(operator)) {
+ return literal.compareTo(coverageBoundary) <= 0;
+ }
+ if (FilterKind.GREATER_THAN_OR_EQUAL.name().equalsIgnoreCase(operator)) {
+ return literal.compareTo(coverageBoundary) < 0;
+ }
+ }
+ return false;
+ }
+
+ private static boolean isIdentifier(Expression expr, String column) {
+ return expr.getType() == ExpressionType.IDENTIFIER
+ && column.equalsIgnoreCase(expr.getIdentifier().getName());
+ }
+
+ @Nullable
+ private static BigDecimal getNumericLiteral(Expression expr) {
+ if (expr.getType() != ExpressionType.LITERAL || expr.getLiteral() == null) {
+ return null;
+ }
+ Object value = RequestUtils.getLiteralValue(expr.getLiteral());
+ if (value instanceof BigDecimal) {
+ return (BigDecimal) value;
+ }
+ /// For every numeric type go through the value's canonical String form to preserve full
+ /// precision. In particular a Long beyond 2^53 (e.g. modern epoch millis) cannot be
+ /// round-tripped through Double without precision loss, which would mis-classify the user's
+ /// upper bound vs the watermark and silently emit FULL_REWRITE against partial coverage.
+ if (value instanceof Number) {
+ return new BigDecimal(value.toString());
+ }
+ if (value instanceof String) {
+ return parseDecimal((String) value);
+ }
+ return null;
+ }
+
+ @Nullable
+ private static BigDecimal parseDecimal(String value) {
+ try {
+ return new BigDecimal(value);
+ } catch (NumberFormatException e) {
+ return null;
+ }
+ }
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewQueryRewriteEngineFactory.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewQueryRewriteEngineFactory.java
new file mode 100644
index 00000000000..98e30a5df6c
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewQueryRewriteEngineFactory.java
@@ -0,0 +1,54 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.rewrite;
+
+import java.util.List;
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.pinot.materializedview.rewrite.strategy.AggregationSubsumptionStrategy;
+import org.apache.pinot.materializedview.rewrite.strategy.ExactSubsumptionStrategy;
+import org.apache.pinot.materializedview.rewrite.strategy.ScanSubsumptionStrategy;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Factory for the broker's default [MaterializedViewQueryRewriteEngine] wiring.
+///
+/// Default strategy order (highest precision first, per [MaterializedViewQueryRewriteEngine]):
+/// [ExactSubsumptionStrategy], [ScanSubsumptionStrategy],
+/// [AggregationSubsumptionStrategy].
+///
+/// Thread-safety: the returned engine is immutable and safe to share across components.
+public final class MaterializedViewQueryRewriteEngineFactory {
+ private static final Logger LOGGER = LoggerFactory.getLogger(MaterializedViewQueryRewriteEngineFactory.class);
+
+ private MaterializedViewQueryRewriteEngineFactory() {
+ }
+
+ /// Builds a [MaterializedViewQueryRewriteEngine] backed by a new
+ /// [MaterializedViewMetadataCache] on the given Helix property store.
+ public static MaterializedViewQueryRewriteEngine createDefault(ZkHelixPropertyStore<ZNRecord> propertyStore) {
+ LOGGER.info("Initializing MV metadata cache and query rewrite engine");
+ MaterializedViewMetadataCache materializedViewMetadataCache = new MaterializedViewMetadataCache(propertyStore);
+ return new MaterializedViewQueryRewriteEngine(materializedViewMetadataCache, List.of(
+ new ExactSubsumptionStrategy(),
+ new ScanSubsumptionStrategy(),
+ new AggregationSubsumptionStrategy()));
+ }
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewQueryShape.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewQueryShape.java
new file mode 100644
index 00000000000..5cc35baeb94
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewQueryShape.java
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.rewrite;
+
+import java.util.List;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+
+
+/// Classifies a [PinotQuery] into a high-level structural shape so that
+/// subsumption strategies can quickly determine whether they are applicable.
+///
+/// `DISTINCT` is given its own value so strategies can explicitly
+/// reject it rather than misclassifying it as `SCAN`. Any query shape
+/// not matching `DISTINCT`, `AGGREGATION`, or `SCAN` falls
+/// through to `SCAN` as the safe default (no strategy will accept it
+/// unless its `acceptsShape` check passes).
+///
+/// **Thread-safety:** stateless enum. Methods are pure functions of their arguments and do
+/// not retain references to inputs. Safe to invoke concurrently from the broker hot path.
+public enum MaterializedViewQueryShape {
+
+ /// No aggregation functions, no GROUP BY, no DISTINCT.
+ SCAN,
+
+ /// Contains aggregation functions in SELECT and/or a GROUP BY clause.
+ AGGREGATION,
+
+ /// `SELECT DISTINCT ...` query. CalciteSqlParser compiles this as
+ /// `SELECT distinct(col1, col2, ...)` — a single function-call
+ /// expression wrapping the select list.
+ DISTINCT;
+
+ /// Determines the structural shape of the given query.
+ ///
+ /// @param query the compiled PinotQuery to classify
+ /// @return the query shape
+ public static MaterializedViewQueryShape classify(PinotQuery query) {
+ List<Expression> selectList = query.getSelectList();
+
+ /// DISTINCT detection: CalciteSqlParser wraps SELECT DISTINCT as a single
+ /// distinct(...) function call containing all projected columns.
+ if (selectList != null && selectList.size() == 1) {
+ Function func = selectList.get(0).getFunctionCall();
+ if (func != null && "distinct".equals(func.getOperator())) {
+ return DISTINCT;
+ }
+ }
+
+ if (query.isSetGroupByList()) {
+ return AGGREGATION;
+ }
+
+ if (selectList != null) {
+ for (Expression expr : selectList) {
+ if (containsAggregation(expr)) {
+ return AGGREGATION;
+ }
+ }
+ }
+
+ return SCAN;
+ }
+
+ /// Recursively checks whether an expression contains an aggregation function call.
+ /// Aliases (`as(expr, name)`) are transparently unwrapped.
+ private static boolean containsAggregation(Expression expr) {
+ if (expr.getType() != ExpressionType.FUNCTION) {
+ return false;
+ }
+ Function func = expr.getFunctionCall();
+ if (func == null) {
+ return false;
+ }
+ String operator = func.getOperator();
+
+ if ("as".equals(operator)) {
+ return containsAggregation(func.getOperands().get(0));
+ }
+
+ if (AggregationFunctionType.isAggregationFunction(operator)) {
+ return true;
+ }
+
+ for (Expression operand : func.getOperands()) {
+ if (containsAggregation(operand)) {
+ return true;
+ }
+ }
+ return false;
+ }
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewRewritePlan.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewRewritePlan.java
new file mode 100644
index 00000000000..f2c29718af6
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/MaterializedViewRewritePlan.java
@@ -0,0 +1,131 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.rewrite;
+
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.materializedview.metadata.MaterializedViewDefinitionMetadata.MaterializedViewSplitSpec;
+
+
+/// Structured output of the MV rewrite layer, separating static matching
+/// ([MatchType]) from runtime execution decisions ([ExecutionMode]).
+///
+/// Strategies produce plan fragments with `execMode = null`. The
+/// [MaterializedViewQueryRewriteEngine] then resolves execution mode and
+/// populates the split-mode runtime fields via
+/// [#withExecMode(ExecutionMode, MaterializedViewSplitSpec, long)].
+///
+/// Lower cost is better. The rewrite engine picks the plan with the lowest
+/// cost when multiple MVs match a single user query.
+public class MaterializedViewRewritePlan implements Comparable<MaterializedViewRewritePlan> {
+
+ private final String _materializedViewTableNameWithType;
+ private final MatchType _matchType;
+ @Nullable
+ private final ExecutionMode _execMode;
+ private final PinotQuery _materializedViewQuery;
+ private final double _cost;
+
+ /// Split-mode runtime fields — populated by the engine during plan resolution,
+ /// not by strategies. Only meaningful when execMode == SPLIT_REWRITE.
+ @Nullable
+ private final MaterializedViewSplitSpec _splitSpec;
+ private final long _watermarkMs;
+
+ /// Whether this plan can be used in SPLIT_REWRITE mode. False for AGG_REAGG
+ /// plans that use non-distributive re-aggregation rules (e.g. COUNT->SUM)
+ /// because the MV side produces intermediate types incompatible with what the
+ /// base-side reducer expects.
+ private final boolean _splitSafe;
+
+ public MaterializedViewRewritePlan(String viewTableNameWithType, MatchType matchType,
+ @Nullable ExecutionMode execMode, PinotQuery viewQuery, double cost) {
+ this(viewTableNameWithType, matchType, execMode, viewQuery, cost, null, 0, true);
+ }
+
+ public MaterializedViewRewritePlan(String viewTableNameWithType, MatchType matchType,
+ @Nullable ExecutionMode execMode, PinotQuery viewQuery, double cost, boolean splitSafe) {
+ this(viewTableNameWithType, matchType, execMode, viewQuery, cost, null, 0, splitSafe);
+ }
+
+ public MaterializedViewRewritePlan(String viewTableNameWithType, MatchType matchType,
+ @Nullable ExecutionMode execMode, PinotQuery viewQuery, double cost,
+ @Nullable MaterializedViewSplitSpec splitSpec, long watermarkMs, boolean splitSafe) {
+ _materializedViewTableNameWithType = viewTableNameWithType;
+ _matchType = matchType;
+ _execMode = execMode;
+ _materializedViewQuery = viewQuery;
+ _cost = cost;
+ _splitSpec = splitSpec;
+ _watermarkMs = watermarkMs;
+ _splitSafe = splitSafe;
+ }
+
+ /// Returns a new plan with execution mode + split runtime parameters set, preserving all other
+ /// fields from this plan. The broker's split execution path uses
+ /// [#getMaterializedViewQuery()] as the MV-side template and the pre-rewrite
+ /// `compileResult._serverPinotQuery` as the base-side template — no separate base-query copy is
+ /// retained on the plan to avoid an O(query-size) `deepCopy()` per candidate.
+ public MaterializedViewRewritePlan withExecMode(ExecutionMode execMode,
+ @Nullable MaterializedViewSplitSpec splitSpec, long watermarkMs) {
+ return new MaterializedViewRewritePlan(_materializedViewTableNameWithType, _matchType, execMode,
+ _materializedViewQuery, _cost, splitSpec, watermarkMs, _splitSafe);
+ }
+
+ public String getMaterializedViewTableNameWithType() {
+ return _materializedViewTableNameWithType;
+ }
+
+ public MatchType getMatchType() {
+ return _matchType;
+ }
+
+ @Nullable
+ public ExecutionMode getExecMode() {
+ return _execMode;
+ }
+
+ public PinotQuery getMaterializedViewQuery() {
+ return _materializedViewQuery;
+ }
+
+ public double getCost() {
+ return _cost;
+ }
+
+ @Nullable
+ public MaterializedViewSplitSpec getSplitSpec() {
+ return _splitSpec;
+ }
+
+ /// Watermark from the MV's runtime metadata. Under Design C this is the split point for
+ /// SPLIT_REWRITE: base side `time >= watermarkMs`, MV side `time < watermarkMs`.
+ public long getWatermarkMs() {
+ return _watermarkMs;
+ }
+
+ public boolean isSplitSafe() {
+ return _splitSafe;
+ }
+
+ @Override
+ public int compareTo(MaterializedViewRewritePlan other) {
+ return Double.compare(_cost, other._cost);
+ }
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/equivalence/AggregationEquivalence.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/equivalence/AggregationEquivalence.java
new file mode 100644
index 00000000000..b863216ac9d
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/equivalence/AggregationEquivalence.java
@@ -0,0 +1,112 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.rewrite.equivalence;
+
+import java.util.ArrayList;
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+
+
+/// Defines an equivalence transformation for re-aggregating a pre-computed MV
+/// column when the MV has finer granularity than the user query.
+///
+/// Each implementation encapsulates one category of aggregation functions
+/// that share the same re-aggregation pattern. For example, distributive
+/// functions (SUM, MIN, MAX) can be re-aggregated with the same function,
+/// while sketch-based functions (DISTINCTCOUNTHLL) require the MV to store
+/// a raw sketch variant for merging.
+///
+/// Implementations must be stateless and thread-safe.
+///
+/// @see PassthroughEquivalence
+/// @see SketchMergeEquivalence
+/// @see AggregationEquivalenceRegistry
+public interface AggregationEquivalence {
+
+ /// Returns `true` if this equivalence can be used in split-mode execution.
+ ///
+ /// In split mode the base side returns intermediates for the original user
+ /// aggregation function, while the MV side returns intermediates for the
+ /// re-aggregation function. The broker reducer merges all DataTables using the
+ /// original function's reducer. If the two functions produce incompatible
+ /// intermediate types (e.g. COUNT produces LONG intermediates but SUM also
+ /// produces LONG — however their DataTable column types may differ), split mode
+ /// should be rejected.
+ ///
+ /// An equivalence is split-safe when the MV-side re-aggregation returns the
+ /// same intermediate format as the user function. COUNT→SUM is not split-safe.
+ ///
+ /// @return `true` if this equivalence produces split-safe intermediates
+ default boolean isSplitSafe() {
+ return false;
+ }
+
+ /// Returns `true` if this equivalence can handle the given combination
+ /// of user aggregation function and MV aggregation function.
+ ///
+ /// @param userFunctionName the aggregation function name from the user query
+ /// (uppercase, e.g. "SUM", "DISTINCTCOUNTHLL")
+ /// @param materializedViewFunctionName the aggregation function name from the MV definition
+ /// (uppercase, e.g. "SUM", "DISTINCTCOUNTRAWHLL")
+ boolean matches(String userFunctionName, String materializedViewFunctionName);
+
+ /// Returns `true` if this equivalence rule can re-aggregate an MV column produced by
+ /// `materializedViewFunctionName`. Used by the analyzer to validate at create/update time
+ /// that every aggregation in an MV's `definedSQL` has a re-aggregation rule, so the
+ /// operator gets a clear error instead of silently observing zero MV rewrites.
+ boolean supportsMaterializedViewFunction(String materializedViewFunctionName);
+
+ /// Builds a re-aggregation expression that applies the appropriate function
+ /// on the pre-computed MV column.
+ ///
+ /// @param userAggExpression the original aggregation expression from the user
+ /// query (e.g. `SUM(revenue)`)
+ /// @param materializedViewColumnName the MV column name that stores the pre-computed
+ /// value (e.g. `sum_rev`)
+ /// @return the re-aggregation expression (e.g. `SUM(sum_rev)`), or
+ /// `null` if the transformation cannot be applied
+ @Nullable
+ Expression rewrite(Expression userAggExpression, String materializedViewColumnName);
+
+ /// Extracts trailing LITERAL operands from the user aggregation expression.
+ /// These are configuration parameters (e.g. `log2m` for HLL functions)
+ /// that may have been injected by broker overrides such as
+ /// `handleHLLLog2mOverride` and should be carried over to the rewritten
+ /// expression.
+ ///
+ /// @param userAggExpression the user's aggregation expression
+ /// @return trailing literal operands (after the first operand), or empty list
+ default List<Expression> extractTrailingLiterals(Expression userAggExpression) {
+ Function func = userAggExpression.getFunctionCall();
+ if (func == null || func.getOperands() == null || func.getOperandsSize() <= 1) {
+ return List.of();
+ }
+ List<Expression> trailing = new ArrayList<>();
+ for (int i = 1; i < func.getOperands().size(); i++) {
+ Expression op = func.getOperands().get(i);
+ if (op.getType() == ExpressionType.LITERAL) {
+ trailing.add(op);
+ }
+ }
+ return trailing;
+ }
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/equivalence/AggregationEquivalenceRegistry.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/equivalence/AggregationEquivalenceRegistry.java
new file mode 100644
index 00000000000..7eef8555608
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/equivalence/AggregationEquivalenceRegistry.java
@@ -0,0 +1,97 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.rewrite.equivalence;
+
+import java.util.List;
+import javax.annotation.Nullable;
+
+
+/// Centralized registry of [AggregationEquivalence] rules for
+/// re-aggregating pre-computed MV columns when the MV has finer granularity
+/// than the user query.
+///
+/// All rules are statically initialized at class-load time. No external
+/// registration or lifecycle management is required.
+///
+/// To add support for a new aggregation function equivalence, simply add
+/// the corresponding rule instance to the [#RULES] list.
+///
+/// This class is stateless and thread-safe.
+public final class AggregationEquivalenceRegistry {
+
+ private static final List<AggregationEquivalence> RULES = List.of(
+ /// Distributive: re-aggregate with the same function
+ new PassthroughEquivalence("SUM", "SUM"),
+ new PassthroughEquivalence("MIN", "MIN"),
+ new PassthroughEquivalence("MAX", "MAX"),
+
+ /// Algebraic (simple transformation): COUNT -> SUM
+ new PassthroughEquivalence("COUNT", "SUM"),
+
+ /// Sketch-based: user wants cardinality/result, MV stores raw sketch
+ new SketchMergeEquivalence("DISTINCTCOUNTHLL", "DISTINCTCOUNTRAWHLL", "DISTINCTCOUNTHLL"),
+ new SketchMergeEquivalence("DISTINCTCOUNTHLLPLUS", "DISTINCTCOUNTRAWHLLPLUS", "DISTINCTCOUNTHLLPLUS"),
+ new SketchMergeEquivalence("DISTINCTCOUNTTHETASKETCH", "DISTINCTCOUNTRAWTHETASKETCH",
+ "DISTINCTCOUNTTHETASKETCH")
+ );
+
+ private AggregationEquivalenceRegistry() {
+ }
+
+ /// Finds the first equivalence rule that can handle the given user-function
+ /// and MV-function pair.
+ ///
+ /// @param userFunctionName the aggregation function name from the user query
+ /// (e.g. "SUM", "DISTINCTCOUNTHLL")
+ /// @param materializedViewFunctionName the aggregation function name from the MV
+ /// definition (e.g. "SUM", "DISTINCTCOUNTRAWHLL")
+ /// @return the matching rule, or `null` if no rule supports the
+ /// given function pair (indicating the function cannot be
+ /// re-aggregated)
+ @Nullable
+ public static AggregationEquivalence findRule(String userFunctionName, String materializedViewFunctionName) {
+ for (AggregationEquivalence rule : RULES) {
+ if (rule.matches(userFunctionName, materializedViewFunctionName)) {
+ return rule;
+ }
+ }
+ return null;
+ }
+
+ /// Returns true if any registered equivalence rule can use an MV column produced by
+ /// `materializedViewFunctionName` as input. Used by the analyzer to reject MV definitions
+ /// whose aggregations have no re-aggregation rule, so the operator gets a clear
+ /// create-time error instead of silently observing zero MV rewrites.
+ public static boolean isMaterializedViewFunctionSupported(String materializedViewFunctionName) {
+ for (AggregationEquivalence rule : RULES) {
+ if (rule.supportsMaterializedViewFunction(materializedViewFunctionName)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /// Returns the canonical user-facing aggregation function names that can be served from
+ /// some registered MV function. Used by the analyzer to surface a helpful list when an
+ /// unsupported aggregation is rejected.
+ public static List<String> supportedMaterializedViewFunctions() {
+ return List.of("SUM", "MIN", "MAX", "COUNT",
+ "DISTINCTCOUNTRAWHLL", "DISTINCTCOUNTRAWHLLPLUS", "DISTINCTCOUNTRAWTHETASKETCH");
+ }
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/equivalence/PassthroughEquivalence.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/equivalence/PassthroughEquivalence.java
new file mode 100644
index 00000000000..00caa259320
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/equivalence/PassthroughEquivalence.java
@@ -0,0 +1,94 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.rewrite.equivalence;
+
+import java.util.ArrayList;
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.utils.request.RequestUtils;
+
+
+/// Equivalence for aggregation functions whose MV expression is identical to
+/// the user expression (after stripping aliases), but whose re-aggregation
+/// function may differ from the original.
+///
+/// This covers two categories:
+///
+/// - **Distributive** — the re-aggregation function is the same as the
+/// user function: `SUM(col) -> SUM(materialized_view_sum_col)`,
+/// `MIN(col) -> MIN(materialized_view_min_col)`,
+/// `MAX(col) -> MAX(materialized_view_max_col)`
+/// - **Algebraic (simple transformation)** — the re-aggregation function
+/// differs: `COUNT(col) -> SUM(materialized_view_count_col)`,
+/// `COUNT(*) -> SUM(materialized_view_count_star_col)`
+///
+///
+/// The [#matches] method requires the MV to store the same aggregation
+/// function as the user query (e.g. MV has `SUM(revenue)` for a user
+/// query with `SUM(revenue)`).
+///
+/// Trailing literal parameters injected by broker overrides (e.g. HLL
+/// `log2m`) are preserved in the rewritten expression.
+///
+/// This class is stateless and thread-safe.
+public class PassthroughEquivalence implements AggregationEquivalence {
+
+ private final String _userFunctionName;
+ private final String _reAggFunctionName;
+
+ /// @param userFunctionName the user-side aggregation function name (uppercase)
+ /// @param reAggFunctionName the function to apply during re-aggregation on
+ /// the MV column (uppercase)
+ public PassthroughEquivalence(String userFunctionName, String reAggFunctionName) {
+ _userFunctionName = userFunctionName;
+ _reAggFunctionName = reAggFunctionName;
+ }
+
+ @Override
+ public boolean isSplitSafe() {
+ /// Split-safe only when re-aggregation uses the same function as the user query
+ /// (distributive: SUM->SUM, MIN->MIN, MAX->MAX). COUNT->SUM is NOT split-safe
+ /// because the base side returns COUNT intermediates but the MV side returns
+ /// SUM intermediates, causing a schema mismatch in the broker reducer.
+ return _userFunctionName.equalsIgnoreCase(_reAggFunctionName);
+ }
+
+ @Override
+ public boolean matches(String userFunctionName, String materializedViewFunctionName) {
+ return _userFunctionName.equalsIgnoreCase(userFunctionName)
+ && _userFunctionName.equalsIgnoreCase(materializedViewFunctionName);
+ }
+
+ @Override
+ public boolean supportsMaterializedViewFunction(String materializedViewFunctionName) {
+ return _userFunctionName.equalsIgnoreCase(materializedViewFunctionName);
+ }
+
+ @Nullable
+ @Override
+ public Expression rewrite(Expression userAggExpression, String materializedViewColumnName) {
+ List<Expression> trailingLiterals = extractTrailingLiterals(userAggExpression);
+ List<Expression> operands = new ArrayList<>(1 + trailingLiterals.size());
+ operands.add(RequestUtils.getIdentifierExpression(materializedViewColumnName));
+ operands.addAll(trailingLiterals);
+ return RequestUtils.getFunctionExpression(_reAggFunctionName.toLowerCase(),
+ operands.toArray(new Expression[0]));
+ }
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/equivalence/SketchMergeEquivalence.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/equivalence/SketchMergeEquivalence.java
new file mode 100644
index 00000000000..ed5db03f673
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/equivalence/SketchMergeEquivalence.java
@@ -0,0 +1,94 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.rewrite.equivalence;
+
+import java.util.ArrayList;
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.utils.request.RequestUtils;
+
+
+/// Equivalence for sketch-based aggregation functions where the user query
+/// requests a computed result (e.g. cardinality) but the MV must store the
+/// raw sketch bytes for merging.
+///
+/// The MV stores a "raw" variant of the sketch function (e.g.
+/// `DISTINCTCOUNTRAWHLL(col)`) which serializes the sketch object.
+/// At re-aggregation time, the result function (e.g. `DISTINCTCOUNTHLL`)
+/// is applied on the MV column; it automatically deserializes and merges the
+/// sketch bytes, then returns the computed result.
+///
+/// Supported sketch families:
+///
+/// - `DISTINCTCOUNTHLL` / `DISTINCTCOUNTRAWHLL`
+/// - `DISTINCTCOUNTHLLPLUS` / `DISTINCTCOUNTRAWHLLPLUS`
+/// - `DISTINCTCOUNTTHETASKETCH` / `DISTINCTCOUNTRAWTHETASKETCH`
+///
+///
+/// Trailing literal parameters injected by broker overrides (e.g. HLL
+/// `log2m`) are preserved in the rewritten expression.
+///
+/// This class is stateless and thread-safe.
+public class SketchMergeEquivalence implements AggregationEquivalence {
+
+ private final String _userFunctionName;
+ private final String _materializedViewFunctionName;
+ private final String _reAggFunctionName;
+
+ /// @param userFunctionName the user-side aggregation function name
+ /// (uppercase, e.g. "DISTINCTCOUNTHLL")
+ /// @param materializedViewFunctionName the function the MV must use to store the raw
+ /// sketch (uppercase, e.g. "DISTINCTCOUNTRAWHLL")
+ /// @param reAggFunctionName the function to apply on the MV column during
+ /// re-aggregation (uppercase, e.g. "DISTINCTCOUNTHLL")
+ public SketchMergeEquivalence(String userFunctionName, String materializedViewFunctionName,
+ String reAggFunctionName) {
+ _userFunctionName = userFunctionName;
+ _materializedViewFunctionName = materializedViewFunctionName;
+ _reAggFunctionName = reAggFunctionName;
+ }
+
+ @Override
+ public boolean matches(String userFunctionName, String materializedViewFunctionName) {
+ return _userFunctionName.equalsIgnoreCase(userFunctionName)
+ && _materializedViewFunctionName.equalsIgnoreCase(materializedViewFunctionName);
+ }
+
+ @Override
+ public boolean supportsMaterializedViewFunction(String materializedViewFunctionName) {
+ return _materializedViewFunctionName.equalsIgnoreCase(materializedViewFunctionName);
+ }
+
+ @Override
+ public boolean isSplitSafe() {
+ return true;
+ }
+
+ @Nullable
+ @Override
+ public Expression rewrite(Expression userAggExpression, String materializedViewColumnName) {
+ List<Expression> trailingLiterals = extractTrailingLiterals(userAggExpression);
+ List<Expression> operands = new ArrayList<>(1 + trailingLiterals.size());
+ operands.add(RequestUtils.getIdentifierExpression(materializedViewColumnName));
+ operands.addAll(trailingLiterals);
+ return RequestUtils.getFunctionExpression(_reAggFunctionName.toLowerCase(),
+ operands.toArray(new Expression[0]));
+ }
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/equivalence/package-info.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/equivalence/package-info.java
new file mode 100644
index 00000000000..12dab1a88d9
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/equivalence/package-info.java
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/// **Query rewrite — aggregation equivalence rules.**
+///
+/// An `AggregationEquivalence` rule says "the user's `foo(x)` can be answered from the MV's
+/// `bar(x)`" and, on match, rewrites the user aggregation against the MV column. The
+/// `AggregationSubsumptionStrategy` in
+/// [strategy][org.apache.pinot.materializedview.rewrite.strategy] consults the
+/// `AggregationEquivalenceRegistry` to resolve each user aggregation against an MV-projected
+/// aggregation.
+///
+/// Built-in rules:
+/// - `PassthroughEquivalence` — identity-rewrite for aggregates that are split-safe and
+/// don't need merging (e.g. `MIN`, `MAX`, `SUM`). Maps `f(x) → f(mv_col)` directly.
+/// - `SketchMergeEquivalence` — covers sketch-based aggregations where the MV stores the
+/// raw sketch and the user query merges via the corresponding `*_merge` UDF (e.g.
+/// `DISTINCT_COUNT_HLL → DISTINCT_COUNT_HLL_MERGE`).
+package org.apache.pinot.materializedview.rewrite.equivalence;
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/package-info.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/package-info.java
new file mode 100644
index 00000000000..3e924c6286d
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/package-info.java
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/// **Query rewrite — engine and shared utilities.**
+///
+/// The rewrite engine takes a user `PinotQuery` and decides whether it can be answered by a
+/// materialized view. Top-level pieces in this package:
+///
+/// - `MaterializedViewQueryRewriteEngine` — entry point. Looks up MV candidates by base-
+/// table name (via the metadata cache), applies subsumption strategies in cost order,
+/// and picks the lowest-cost match.
+/// - `MaterializedViewMetadataCache` — mirrors the ZK definition + runtime znodes into an
+/// in-memory cache. Subscribes to child-change and data-change listeners; releases
+/// watcher slots in `close()`.
+/// - `MaterializedViewRewritePlan` / `ExecutionMode` / `MatchType` — value objects
+/// describing the rewrite decision.
+/// - `MaterializedViewQueryShape` — coarse classifier (SCAN / AGGREGATION) used by
+/// strategies to short-circuit shape mismatches.
+/// - `MaterializedViewMatchUtils` — stateless helpers shared across strategies: filter
+/// conjunct flattening, residual extraction, projection-key column resolution.
+///
+/// Sub-packages:
+/// - [rewrite.strategy][org.apache.pinot.materializedview.rewrite.strategy] — concrete
+/// subsumption strategies (exact, scan, aggregation).
+/// - [rewrite.equivalence][org.apache.pinot.materializedview.rewrite.equivalence] —
+/// aggregation equivalence rules (passthrough, sketch-merge).
+package org.apache.pinot.materializedview.rewrite;
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/strategy/AbstractSubsumptionStrategy.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/strategy/AbstractSubsumptionStrategy.java
new file mode 100644
index 00000000000..09ead434aa2
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/strategy/AbstractSubsumptionStrategy.java
@@ -0,0 +1,216 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.rewrite.strategy;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewMatchUtils;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewMetadataCache.MaterializedViewCacheEntry;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewQueryShape;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewRewritePlan;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Base class for materialized view matching strategies that follow the
+/// [subsumption](https://en.wikipedia.org/wiki/Subsumption) model:
+/// a query can be answered by an MV if the MV's definition logically
+/// *subsumes* (covers) the query's requirements.
+///
+/// This class implements the [MaterializedViewMatchStrategy] interface using a
+/// **template method** pattern. The overall matching flow is fixed:
+///
+/// - Shape gate — reject queries whose structural shape is incompatible
+/// - GROUP BY check
+/// - Projection (SELECT) subsumption check
+/// - WHERE filter matching and residual extraction
+/// - Residual filter validation
+/// - ORDER BY compatibility
+/// - HAVING compatibility
+/// - Build the rewritten query and cost
+///
+///
+/// Concrete subclasses customize individual steps by overriding the
+/// `protected abstract` hook methods. This design allows adding new
+/// matching strategies (scan subsumption, aggregation subsumption, etc.)
+/// while reusing the shared matching infrastructure in [MaterializedViewMatchUtils].
+///
+/// Strategies produce plan fragments — [MaterializedViewRewritePlan] instances with
+/// `execMode = null`. The rewrite engine resolves execution mode and split
+/// compatibility after the strategy returns.
+///
+/// Implementations should be stateless and thread-safe.
+///
+/// @see ExactSubsumptionStrategy
+/// @see ScanSubsumptionStrategy
+/// @see AggregationSubsumptionStrategy
+public abstract class AbstractSubsumptionStrategy implements MaterializedViewMatchStrategy {
+ private static final Logger LOGGER = LoggerFactory.getLogger(AbstractSubsumptionStrategy.class);
+
+ /// Template method that drives the subsumption matching pipeline.
+ ///
+ /// This method is `final` to enforce a consistent matching order
+ /// across all strategies. Subclasses customize behavior through the
+ /// `protected abstract` hook methods.
+ @Nullable
+ @Override
+ public final MaterializedViewRewritePlan match(PinotQuery userQuery, MaterializedViewCacheEntry candidateEntry) {
+ PinotQuery viewQuery = candidateEntry.getCompiledQuery();
+ String strategyName = getClass().getSimpleName();
+ String materializedViewName = candidateEntry.getMaterializedViewTableNameWithType();
+
+ if (viewQuery == null) {
+ LOGGER.debug("MV match [{}] strategy={}: compiled query is null", materializedViewName, strategyName);
+ return null;
+ }
+
+ /// Step 1: shape gate
+ if (!acceptsShape(userQuery, viewQuery)) {
+ LOGGER.debug("MV match [{}] strategy={}: rejected at SHAPE_GATE (userShape={}, materializedViewShape={})",
+ materializedViewName, strategyName, MaterializedViewQueryShape.classify(userQuery),
+ MaterializedViewQueryShape.classify(viewQuery));
+ return null;
+ }
+
+ /// Step 2: GROUP BY
+ if (!groupByMatches(userQuery, viewQuery)) {
+ LOGGER.debug("MV match [{}] strategy={}: rejected at GROUP_BY", materializedViewName, strategyName);
+ return null;
+ }
+
+ /// Step 3: projection subsumption — use the projection map cached on the entry.
+ Map<Expression, String> viewProjectionMap = candidateEntry.getViewProjectionMap();
+ if (!projectionSubsumes(userQuery.getSelectList(), viewProjectionMap)) {
+ LOGGER.debug("MV match [{}] strategy={}: rejected at PROJECTION", materializedViewName, strategyName);
+ return null;
+ }
+
+ /// Step 4: WHERE matching + residual extraction (shared logic)
+ Expression userFilter = userQuery.getFilterExpression();
+ Expression materializedViewFilter = viewQuery.getFilterExpression();
+ boolean filtersEqual = Objects.equals(userFilter, materializedViewFilter);
+
+ /// AND is commutative and associative. Two filters with the same conjunct set are equivalent
+ /// even if the underlying List<Expression> order differs (e.g. user wrote "b AND a" but the MV
+ /// was defined with "a AND b"). Promote to filtersEqual so we don't fall into the residual
+ /// extraction branch and misreport "not a superset" when the residual set is in fact empty.
+ /// The MV-side flattening is cached on the entry (computed once per definition update). The
+ /// user-side flatten is computed once per (query, strategy, candidate) triple; for workloads
+ /// with many candidate MVs on the same base table the redundancy could be removed by hoisting
+ /// userFilterConjuncts into the rewrite engine, but the cost is bounded by the number of
+ /// candidates (typically 1–5 per table) and the feature is opt-in via useMaterializedView.
+ if (!filtersEqual && userFilter != null && materializedViewFilter != null) {
+ if (MaterializedViewMatchUtils.flattenAnd(userFilter).equals(candidateEntry.getViewFilterConjuncts())) {
+ filtersEqual = true;
+ }
+ }
+
+ Expression residualFilter = null;
+ if (!filtersEqual) {
+ residualFilter = MaterializedViewMatchUtils.tryExtractResidualFilter(userFilter, materializedViewFilter);
+ if (residualFilter == null && userFilter != null) {
+ LOGGER.debug("MV match [{}] strategy={}: rejected at WHERE_FILTER "
+ + "(user filter is not a superset of MV filter)", materializedViewName, strategyName);
+ return null;
+ }
+ if (userFilter == null) {
+ LOGGER.debug("MV match [{}] strategy={}: rejected at WHERE_FILTER "
+ + "(MV has filter but user query does not)", materializedViewName, strategyName);
+ return null;
+ }
+ }
+
+ /// Step 5: residual validation (subclass-specific)
+ if (!validateResidual(residualFilter, viewQuery, viewProjectionMap)) {
+ LOGGER.debug("MV match [{}] strategy={}: rejected at RESIDUAL_VALIDATION", materializedViewName, strategyName);
+ return null;
+ }
+
+ /// Step 6: ORDER BY
+ if (!orderByCompatible(userQuery, viewQuery, viewProjectionMap)) {
+ LOGGER.debug("MV match [{}] strategy={}: rejected at ORDER_BY", materializedViewName, strategyName);
+ return null;
+ }
+
+ /// Step 7: HAVING
+ if (!havingCompatible(userQuery, viewQuery, viewProjectionMap)) {
+ LOGGER.debug("MV match [{}] strategy={}: rejected at HAVING", materializedViewName, strategyName);
+ return null;
+ }
+
+ /// Step 8: build result
+ return buildResult(userQuery, candidateEntry, residualFilter, viewProjectionMap, filtersEqual);
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Hook methods — to be implemented by concrete strategies
+ /// -----------------------------------------------------------------------
+
+ /// Returns `true` if this strategy is applicable to the given
+ /// combination of user query shape and MV query shape.
+ protected abstract boolean acceptsShape(PinotQuery userQuery, PinotQuery viewQuery);
+
+ /// Returns `true` if the user query's GROUP BY clause is compatible
+ /// with the MV's GROUP BY clause.
+ protected abstract boolean groupByMatches(PinotQuery userQuery, PinotQuery viewQuery);
+
+ /// Returns `true` if the MV's projection (SELECT list) covers all
+ /// expressions required by the user query.
+ ///
+ /// @param userSelectList the user query's SELECT expressions
+ /// @param viewProjectionMap alias-stripped expression → MV column name
+ protected abstract boolean projectionSubsumes(List<Expression> userSelectList,
+ Map<Expression, String> viewProjectionMap);
+
+ /// Returns `true` if the residual filter (extra WHERE predicates beyond
+ /// the MV's definition) is valid for this strategy.
+ ///
+ /// @param residualFilter the residual WHERE filter, or `null` if filters are identical
+ /// @param viewQuery the MV's compiled query
+ /// @param viewProjectionMap alias-stripped expression → MV column name (already computed by caller)
+ protected abstract boolean validateResidual(@Nullable Expression residualFilter, PinotQuery viewQuery,
+ Map<Expression, String> viewProjectionMap);
+
+ /// Returns `true` if the user query's ORDER BY clause can be satisfied
+ /// by the MV table.
+ protected abstract boolean orderByCompatible(PinotQuery userQuery, PinotQuery viewQuery,
+ Map<Expression, String> viewProjectionMap);
+
+ /// Returns `true` if the user query's HAVING clause can be satisfied
+ /// by the MV table.
+ protected abstract boolean havingCompatible(PinotQuery userQuery, PinotQuery viewQuery,
+ Map<Expression, String> viewProjectionMap);
+
+ /// Constructs the [MaterializedViewRewritePlan] fragment containing the rewritten query,
+ /// match type, and cost score. The plan's execution mode is left unset (null)
+ /// — it is resolved by the engine after the strategy returns.
+ ///
+ /// @param userQuery the original user query
+ /// @param candidateEntry the matched MV cache entry
+ /// @param residualFilter residual WHERE filter (null if none)
+ /// @param viewProjectionMap alias-stripped expression → MV column name
+ /// @param filtersEqual true if user and MV filters are identical
+ protected abstract MaterializedViewRewritePlan buildResult(PinotQuery userQuery,
+ MaterializedViewCacheEntry candidateEntry, @Nullable Expression residualFilter,
+ Map<Expression, String> viewProjectionMap, boolean filtersEqual);
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/strategy/AggregationSubsumptionStrategy.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/strategy/AggregationSubsumptionStrategy.java
new file mode 100644
index 00000000000..6a886e8f4f2
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/strategy/AggregationSubsumptionStrategy.java
@@ -0,0 +1,428 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.rewrite.strategy;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.materializedview.rewrite.MatchType;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewMatchUtils;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewMetadataCache.MaterializedViewCacheEntry;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewQueryShape;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewRewritePlan;
+import org.apache.pinot.materializedview.rewrite.equivalence.AggregationEquivalence;
+import org.apache.pinot.materializedview.rewrite.equivalence.AggregationEquivalenceRegistry;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+
+
+/// Subsumption strategy for **aggregation** queries.
+///
+/// The rewritten query always retains GROUP BY and wraps aggregation
+/// columns with re-aggregation functions via
+/// [AggregationEquivalenceRegistry] (e.g. `SUM(col)` on the user
+/// query becomes `SUM(sum_col)` on the MV). When the MV's GROUP BY
+/// granularity matches the user query exactly, each group contains a single
+/// pre-computed row, so the re-aggregation is effectively a no-op — but it
+/// guarantees that the server produces an aggregation intermediate DataTable,
+/// which is critical for split-mode merging where both sides must use the
+/// same schema.
+///
+/// Cost model:
+///
+/// - `6.0` — re-aggregation without residual WHERE
+/// - `7.0` — re-aggregation with residual WHERE
+///
+///
+/// Implementations should be stateless and thread-safe.
+public class AggregationSubsumptionStrategy extends AbstractSubsumptionStrategy {
+
+ private static final double COST_REAGG = 6.0;
+ private static final double COST_REAGG_WITH_RESIDUAL = 7.0;
+
+ @Override
+ protected boolean acceptsShape(PinotQuery userQuery, PinotQuery viewQuery) {
+ return MaterializedViewQueryShape.classify(userQuery) == MaterializedViewQueryShape.AGGREGATION
+ && MaterializedViewQueryShape.classify(viewQuery) == MaterializedViewQueryShape.AGGREGATION;
+ }
+
+ @Override
+ protected boolean groupByMatches(PinotQuery userQuery, PinotQuery viewQuery) {
+ List<Expression> userGroupBy = userQuery.getGroupByList();
+ List<Expression> materializedViewGroupBy = viewQuery.getGroupByList();
+
+ boolean userIsWholeTable = userGroupBy == null || userGroupBy.isEmpty();
+ boolean materializedViewIsWholeTable =
+ materializedViewGroupBy == null || materializedViewGroupBy.isEmpty();
+
+ /// User aggregates over the whole table while the MV pre-aggregates by some grouping —
+ /// legal: re-aggregate the MV's per-group rows into one whole-table result on the MV side.
+ if (userIsWholeTable) {
+ return !materializedViewIsWholeTable;
+ }
+ /// User has a non-empty GROUP BY but the MV is whole-table — the MV cannot supply the
+ /// per-group rows the user wants.
+ if (materializedViewIsWholeTable) {
+ return false;
+ }
+
+ Set<Expression> userSet = new HashSet<>(userGroupBy);
+ Set<Expression> materializedViewSet = new HashSet<>(materializedViewGroupBy);
+ return materializedViewSet.containsAll(userSet);
+ }
+
+ @Override
+ protected boolean projectionSubsumes(List<Expression> userSelectList,
+ Map<Expression, String> viewProjectionMap) {
+ if (userSelectList == null || userSelectList.isEmpty()) {
+ return false;
+ }
+ for (Expression expr : userSelectList) {
+ Expression stripped = MaterializedViewMatchUtils.stripAlias(expr);
+ /// Plain column reference: direct MV projection hit is sufficient.
+ if (stripped.getFunctionCall() == null) {
+ if (!viewProjectionMap.containsKey(stripped)) {
+ return false;
+ }
+ continue;
+ }
+ /// Aggregate function: an exact projection match is NOT enough on its own. We need an
+ /// AggregationEquivalence rule to re-aggregate the pre-computed MV column correctly.
+ /// Without a rule we would fall back to a bare column reference (e.g. AVG(revenue) →
+ /// avg_rev), which produces wrong results for non-distributive functions.
+ if (findEquivalentMaterializedViewEntry(stripped, viewProjectionMap) == null) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ @Nullable
+ private static Object[] findEquivalentMaterializedViewEntry(Expression userExpr,
+ Map<Expression, String> viewProjectionMap) {
+ Function userFunc = userExpr.getFunctionCall();
+ if (userFunc == null) {
+ return null;
+ }
+ String userFuncName = userFunc.getOperator();
+ List<Expression> userOperands = userFunc.getOperands();
+
+ for (Map.Entry<Expression, String> materializedViewEntry : viewProjectionMap.entrySet()) {
+ Expression materializedViewExpr = materializedViewEntry.getKey();
+ Function materializedViewFunc = materializedViewExpr.getFunctionCall();
+ if (materializedViewFunc == null) {
+ continue;
+ }
+ if (!operandsMatch(userOperands, materializedViewFunc.getOperands())) {
+ continue;
+ }
+ AggregationEquivalence rule =
+ AggregationEquivalenceRegistry.findRule(userFuncName, materializedViewFunc.getOperator());
+ if (rule != null) {
+ return new Object[]{materializedViewEntry.getValue(), rule};
+ }
+ }
+ return null;
+ }
+
+ private static boolean operandsMatch(@Nullable List<Expression> a, @Nullable List<Expression> b) {
+ if (a == null && b == null) {
+ return true;
+ }
+ if (a == null || b == null) {
+ return false;
+ }
+ /// MV definitions are authored without broker-injected config literals (e.g. log2m for HLL).
+ /// The broker may append trailing literals to the user query at runtime. We tolerate those
+ /// by comparing only up to the MV's operand count — but only when the extra operands in the
+ /// user query are all literals (so PERCENTILE(col,50) vs PERCENTILE(col,99) still rejects,
+ /// because the MV also carries the literal 99 as a semantically significant operand).
+ int materializedViewSize = b.size();
+ int userSize = a.size();
+ if (userSize < materializedViewSize) {
+ return false;
+ }
+ /// All operands up to materializedViewSize must match exactly.
+ for (int i = 0; i < materializedViewSize; i++) {
+ if (!a.get(i).equals(b.get(i))) {
+ return false;
+ }
+ }
+ /// Extra user operands (beyond the MV operand count) must all be literals —
+ /// otherwise this is a semantically distinct call, not just a config injection.
+ for (int i = materializedViewSize; i < userSize; i++) {
+ if (a.get(i).getType() != ExpressionType.LITERAL) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected boolean validateResidual(@Nullable Expression residualFilter, PinotQuery viewQuery,
+ Map<Expression, String> viewProjectionMap) {
+ if (residualFilter == null) {
+ return true;
+ }
+
+ List<Expression> materializedViewGroupBy = viewQuery.getGroupByList();
+ if (materializedViewGroupBy == null || materializedViewGroupBy.isEmpty()) {
+ return false;
+ }
+
+ Set<String> groupByColumnNames = new HashSet<>(materializedViewGroupBy.size());
+ for (Expression gbExpr : materializedViewGroupBy) {
+ groupByColumnNames.addAll(MaterializedViewMatchUtils.collectReferencedColumns(gbExpr));
+ }
+
+ Set<String> residualColumns = MaterializedViewMatchUtils.collectReferencedColumns(residualFilter);
+ return groupByColumnNames.containsAll(residualColumns);
+ }
+
+ @Override
+ protected boolean orderByCompatible(PinotQuery userQuery, PinotQuery viewQuery,
+ Map<Expression, String> viewProjectionMap) {
+ List<Expression> orderByList = userQuery.getOrderByList();
+ if (orderByList == null || orderByList.isEmpty()) {
+ return true;
+ }
+ for (Expression orderByExpr : orderByList) {
+ Expression inner = CalciteSqlParser.removeOrderByFunctions(orderByExpr);
+ if (!isResolvable(inner, viewProjectionMap)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected boolean havingCompatible(PinotQuery userQuery, PinotQuery viewQuery,
+ Map<Expression, String> viewProjectionMap) {
+ Expression having = userQuery.getHavingExpression();
+ if (having == null) {
+ return true;
+ }
+ return allReferencesResolvableWithEquivalence(having, viewProjectionMap);
+ }
+
+ private boolean allReferencesResolvableWithEquivalence(Expression expr,
+ Map<Expression, String> viewProjectionMap) {
+ /// Resolve via the projection map first: an IDENTIFIER for a grouping column is keyed
+ /// directly in viewProjectionMap (e.g. `carrier` -> `carrier`) and must resolve true here.
+ /// The earlier ordering returned false for any IDENTIFIER before consulting the map, which
+ /// rejected legitimate HAVING / SELECT references like `carrier = 'AA'` even when the MV
+ /// exposed `carrier`.
+ if (isResolvable(expr, viewProjectionMap)) {
+ return true;
+ }
+ if (expr.getType() == ExpressionType.LITERAL) {
+ return true;
+ }
+ if (expr.getType() == ExpressionType.IDENTIFIER) {
+ return false;
+ }
+ if (expr.getFunctionCall() != null && expr.getFunctionCall().getOperands() != null) {
+ for (Expression operand : expr.getFunctionCall().getOperands()) {
+ if (!allReferencesResolvableWithEquivalence(operand, viewProjectionMap)) {
+ return false;
+ }
+ }
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isResolvable(Expression expr, Map<Expression, String> viewProjectionMap) {
+ if (viewProjectionMap.containsKey(expr)) {
+ return true;
+ }
+ return findEquivalentMaterializedViewEntry(expr, viewProjectionMap) != null;
+ }
+
+ @Override
+ protected MaterializedViewRewritePlan buildResult(PinotQuery userQuery, MaterializedViewCacheEntry candidateEntry,
+ @Nullable Expression residualFilter, Map<Expression, String> viewProjectionMap, boolean filtersEqual) {
+ PinotQuery rewritten = userQuery.deepCopy();
+ rewritten.getDataSource().setTableName(candidateEntry.getMaterializedViewTableNameWithType());
+
+ /// Check whether all re-aggregation rules applied to this query are split-safe
+ /// (i.e. produce intermediates compatible with what the original function's reducer expects).
+ /// COUNT->SUM is NOT split-safe: base side produces COUNT intermediates but MV side produces
+ /// SUM intermediates, causing a schema mismatch in the broker reducer during split-mode merge.
+ boolean splitSafe = isSplitSafe(userQuery.getSelectList(), viewProjectionMap);
+
+ rewritten.setSelectList(
+ buildReAggSelectList(userQuery.getSelectList(), viewProjectionMap));
+
+ List<Expression> userGroupBy = userQuery.getGroupByList();
+ if (userGroupBy != null && !userGroupBy.isEmpty()) {
+ List<Expression> remappedGroupBy = new ArrayList<>(userGroupBy.size());
+ for (Expression gbExpr : userGroupBy) {
+ String materializedViewCol = viewProjectionMap.get(gbExpr);
+ remappedGroupBy.add(RequestUtils.getIdentifierExpression(materializedViewCol));
+ }
+ rewritten.setGroupByList(remappedGroupBy);
+ } else {
+ /// Whole-table re-aggregation: user query has no GROUP BY but MV is grouped. Strip the
+ /// GROUP BY from the deepCopy so the re-aggregation sums per-group rows into one whole-
+ /// table result on the MV side. groupByMatches has already validated this is legal.
+ rewritten.setGroupByList(null);
+ }
+
+ Expression originalHaving = userQuery.getHavingExpression();
+ if (originalHaving != null) {
+ rewritten.setHavingExpression(
+ remapExpressionWithEquivalence(originalHaving, viewProjectionMap));
+ } else {
+ rewritten.setHavingExpression(null);
+ }
+
+ Expression remappedResidual = null;
+ if (residualFilter != null) {
+ remappedResidual = MaterializedViewMatchUtils.remapExpression(residualFilter, viewProjectionMap);
+ }
+ rewritten.setFilterExpression(remappedResidual);
+
+ if (userQuery.getOrderByList() != null && !userQuery.getOrderByList().isEmpty()) {
+ List<Expression> remappedOrderBy = new ArrayList<>(userQuery.getOrderByList().size());
+ for (Expression orderByExpr : userQuery.getOrderByList()) {
+ remappedOrderBy.add(remapExpressionWithEquivalence(orderByExpr, viewProjectionMap));
+ }
+ rewritten.setOrderByList(remappedOrderBy);
+ }
+
+ double cost = filtersEqual ? COST_REAGG : COST_REAGG_WITH_RESIDUAL;
+ return new MaterializedViewRewritePlan(candidateEntry.getMaterializedViewTableNameWithType(),
+ MatchType.AGG_REAGG, null, rewritten, cost, splitSafe);
+ }
+
+ /// Returns true if all aggregation expressions in the SELECT list use split-safe
+ /// equivalence rules (i.e. the re-aggregation function is identical to the user
+ /// function, so intermediates from base and MV sides are schema-compatible).
+ private static boolean isSplitSafe(List<Expression> userSelectList,
+ Map<Expression, String> viewProjectionMap) {
+ for (Expression expr : userSelectList) {
+ Expression stripped = MaterializedViewMatchUtils.stripAlias(expr);
+ if (stripped.getFunctionCall() != null) {
+ Object[] match = findEquivalentMaterializedViewEntry(stripped, viewProjectionMap);
+ if (match != null) {
+ AggregationEquivalence rule = (AggregationEquivalence) match[1];
+ if (!rule.isSplitSafe()) {
+ return false;
+ }
+ }
+ }
+ }
+ return true;
+ }
+
+ private List<Expression> buildReAggSelectList(List<Expression> userSelectList,
+ Map<Expression, String> viewProjectionMap) {
+ List<Expression> result = new ArrayList<>(userSelectList.size());
+ for (Expression expr : userSelectList) {
+ Expression stripped = MaterializedViewMatchUtils.stripAlias(expr);
+ String userAlias = MaterializedViewMatchUtils.extractUserAlias(expr);
+ Expression rewritten;
+
+ if (viewProjectionMap.containsKey(stripped)
+ && stripped.getFunctionCall() == null) {
+ rewritten = RequestUtils.getIdentifierExpression(viewProjectionMap.get(stripped));
+ } else {
+ rewritten = rewriteAggregationExpression(stripped, viewProjectionMap);
+ }
+
+ /// Preserve the user's expected result-column name. Bare-identifier dimensions that map 1:1
+ /// to an identically-named MV column need no alias wrap (e.g. city -> city); but when the
+ /// user expression was an aggregation (or a renamed dimension), the rewritten form would
+ /// otherwise surface the MV-side name and silently break clients reading by column name.
+ String resultAlias;
+ if (userAlias != null) {
+ resultAlias = userAlias;
+ } else if (stripped.getType() == ExpressionType.IDENTIFIER
+ && rewritten.getType() == ExpressionType.IDENTIFIER
+ && stripped.getIdentifier().getName().equals(rewritten.getIdentifier().getName())) {
+ resultAlias = null;
+ } else {
+ resultAlias = RequestUtils.prettyPrint(stripped);
+ }
+ if (resultAlias != null) {
+ rewritten = RequestUtils.getFunctionExpression("as", rewritten,
+ RequestUtils.getIdentifierExpression(resultAlias));
+ }
+ result.add(rewritten);
+ }
+ return result;
+ }
+
+ private Expression rewriteAggregationExpression(Expression stripped,
+ Map<Expression, String> viewProjectionMap) {
+ Object[] match = findEquivalentMaterializedViewEntry(stripped, viewProjectionMap);
+ if (match != null) {
+ String materializedViewCol = (String) match[0];
+ AggregationEquivalence rule = (AggregationEquivalence) match[1];
+ return rule.rewrite(stripped, materializedViewCol);
+ }
+
+ throw new IllegalStateException(
+ "Cannot rewrite aggregation expression: " + stripped);
+ }
+
+ private Expression remapExpressionWithEquivalence(Expression expr,
+ Map<Expression, String> viewProjectionMap) {
+ if (viewProjectionMap.containsKey(expr) && expr.getFunctionCall() == null) {
+ return RequestUtils.getIdentifierExpression(viewProjectionMap.get(expr));
+ }
+
+ if (expr.getFunctionCall() != null) {
+ if (viewProjectionMap.containsKey(expr)) {
+ return rewriteAggregationExpression(expr, viewProjectionMap);
+ }
+ Object[] match = findEquivalentMaterializedViewEntry(expr, viewProjectionMap);
+ if (match != null) {
+ return ((AggregationEquivalence) match[1]).rewrite(expr, (String) match[0]);
+ }
+ }
+
+ if (expr.getFunctionCall() != null && expr.getFunctionCall().getOperands() != null) {
+ Function func = expr.getFunctionCall();
+ List<Expression> original = func.getOperands();
+ List<Expression> remapped = new ArrayList<>(original.size());
+ boolean changed = false;
+ for (Expression operand : original) {
+ Expression result = remapExpressionWithEquivalence(operand, viewProjectionMap);
+ remapped.add(result);
+ if (result != operand) {
+ changed = true;
+ }
+ }
+ if (changed) {
+ return RequestUtils.getFunctionExpression(func.getOperator(), remapped);
+ }
+ }
+
+ return expr;
+ }
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/strategy/ExactSubsumptionStrategy.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/strategy/ExactSubsumptionStrategy.java
new file mode 100644
index 00000000000..6264556996d
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/strategy/ExactSubsumptionStrategy.java
@@ -0,0 +1,214 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.rewrite.strategy;
+
+import com.google.common.base.Preconditions;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.materializedview.rewrite.MatchType;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewMatchUtils;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewMetadataCache.MaterializedViewCacheEntry;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewRewritePlan;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+
+
+/// Subsumption strategy that requires an **exact structural match** between
+/// the user query and the MV definition (after stripping aliases and ignoring
+/// column order in the SELECT list).
+///
+/// This is the tightest form of subsumption: the query's SELECT, WHERE,
+/// GROUP BY, ORDER BY, and HAVING must all be semantically identical to the
+/// MV's definition. No residual WHERE filter is allowed — if the WHERE clauses
+/// differ in any way, the match fails.
+///
+/// The rewritten query maps all SELECT expressions to MV table column names
+/// while preserving the user's original aliases.
+///
+/// Cost: `0.0` (perfect match, highest priority).
+///
+/// This strategy accepts any query shape (SCAN or AGGREGATION) because
+/// exact matching is valid regardless of whether aggregation is present.
+public class ExactSubsumptionStrategy extends AbstractSubsumptionStrategy {
+
+ private static final double COST_EXACT = 0.0;
+
+ /// Accepts any query shape — exact matching is universally applicable.
+ @Override
+ protected boolean acceptsShape(PinotQuery userQuery, PinotQuery viewQuery) {
+ return true;
+ }
+
+ /// Requires GROUP BY lists to be identical (same expressions, same order).
+ @Override
+ protected boolean groupByMatches(PinotQuery userQuery, PinotQuery viewQuery) {
+ List<Expression> userList = userQuery.getGroupByList();
+ List<Expression> materializedViewList = viewQuery.getGroupByList();
+ if (userList == null && materializedViewList == null) {
+ return true;
+ }
+ if (userList == null || materializedViewList == null) {
+ return false;
+ }
+ return userList.equals(materializedViewList);
+ }
+
+ /// Requires SELECT lists to contain exactly the same expressions (order-insensitive,
+ /// alias-insensitive). The stripped expression sets must be equal.
+ @Override
+ protected boolean projectionSubsumes(List<Expression> userSelectList,
+ Map<Expression, String> viewProjectionMap) {
+ if (userSelectList == null) {
+ return viewProjectionMap.isEmpty();
+ }
+ if (userSelectList.size() != viewProjectionMap.size()) {
+ return false;
+ }
+ Set<Expression> userStripped = new HashSet<>(userSelectList.size());
+ for (Expression expr : userSelectList) {
+ userStripped.add(MaterializedViewMatchUtils.stripAlias(expr));
+ }
+ return userStripped.equals(viewProjectionMap.keySet());
+ }
+
+ /// Rejects any residual filter — exact match requires WHERE clauses to be
+ /// identical between the user query and the MV definition.
+ @Override
+ protected boolean validateResidual(@Nullable Expression residualFilter, PinotQuery viewQuery,
+ Map<Expression, String> viewProjectionMap) {
+ return residualFilter == null;
+ }
+
+ /// Requires ORDER BY lists to be compatible between user query and MV query.
+ /// If both have ORDER BY, they must be identical. If the user has ORDER BY but the MV
+ /// does not (the common case — MVs rarely define ORDER BY), the check passes as long as
+ /// all referenced columns are present in the MV projection, since the MV table can
+ /// serve any ORDER BY over its columns.
+ @Override
+ protected boolean orderByCompatible(PinotQuery userQuery, PinotQuery viewQuery,
+ Map<Expression, String> viewProjectionMap) {
+ List<Expression> userList = userQuery.getOrderByList();
+ List<Expression> materializedViewList = viewQuery.getOrderByList();
+ if (userList == null || userList.isEmpty()) {
+ return true;
+ }
+ if (materializedViewList != null && !materializedViewList.isEmpty()) {
+ return userList.equals(materializedViewList);
+ }
+ /// MV has no ORDER BY — allow only when each user ORDER BY sort key is rewritable through
+ /// the projection map. An ORDER BY entry comes in as `asc(...)` / `desc(...)`; strip the
+ /// direction wrapper, then accept the sort key if:
+ /// 1. it is present as-is in `viewProjectionMap` (so `remapExpression` will rewrite it
+ /// to the MV column), OR
+ /// 2. it is a simple identifier whose name appears as a key in the projection map
+ /// (covers the aliased-MV case `base_col AS mv_col`).
+ ///
+ /// Critically, a bare `coversReferencedColumns(collectReferencedColumns(orderByExpr), map)`
+ /// is NOT safe: for an aggregate like `ORDER BY count(*)`, `collectReferencedColumns`
+ /// returns the empty set (the `*` identifier is filtered out), `coversReferencedColumns`
+ /// returns true vacuously, and `remapExpression` then leaves `count(*)` unchanged in the
+ /// rewritten query — producing a count over the pre-aggregated MV bucket rows instead of
+ /// the original base-table rows. Restricting the relaxed match to bare-identifier sort
+ /// keys closes that wrong-result hole.
+ for (Expression orderByExpr : userList) {
+ /// Use the canonical wrapper-stripper so `asc` / `desc` / `nullsFirst` / `nullsLast`
+ /// nesting all collapses to the inner sort key. The sibling [AggregationSubsumptionStrategy]
+ /// already uses the same helper; keeping the implementations aligned avoids the silent
+ /// over-rejection that an `asc`-only stripper would produce for `ORDER BY col NULLS LAST`.
+ Expression sortKey = CalciteSqlParser.removeOrderByFunctions(orderByExpr);
+ if (viewProjectionMap.containsKey(sortKey)) {
+ continue;
+ }
+ if (sortKey.getType() == ExpressionType.IDENTIFIER
+ && MaterializedViewMatchUtils.coversReferencedColumns(
+ MaterializedViewMatchUtils.collectReferencedColumns(sortKey), viewProjectionMap)) {
+ continue;
+ }
+ return false;
+ }
+ return true;
+ }
+
+ /// Requires HAVING expressions to be identical between user query and MV query.
+ @Override
+ protected boolean havingCompatible(PinotQuery userQuery, PinotQuery viewQuery,
+ Map<Expression, String> viewProjectionMap) {
+ return Objects.equals(userQuery.getHavingExpression(), viewQuery.getHavingExpression());
+ }
+
+ /// Builds the rewritten query by swapping the table name to the MV table
+ /// and mapping each SELECT expression to its MV column name while preserving
+ /// the user's original alias.
+ ///
+ /// The strategy no longer checks split-mode compatibility here — that
+ /// concern is handled by `MaterializedViewQueryRewriteEngine.resolvePlan`.
+ @Override
+ protected MaterializedViewRewritePlan buildResult(PinotQuery userQuery, MaterializedViewCacheEntry candidateEntry,
+ @Nullable Expression residualFilter, Map<Expression, String> viewProjectionMap, boolean filtersEqual) {
+ PinotQuery rewritten = userQuery.deepCopy();
+ rewritten.getDataSource().setTableName(candidateEntry.getMaterializedViewTableNameWithType());
+ rewritten.setSelectList(MaterializedViewMatchUtils.rewriteSelectList(userQuery.getSelectList(), viewProjectionMap));
+ /// EXACT match implies user filter == MV filter, so the rewritten query carries no WHERE.
+ /// RLS conjuncts injected by the broker before MV rewrite would also be dropped here. Today
+ /// an MV definition cannot encode RLS predicates, so an exact filter equality can NEVER hold
+ /// when RLS was applied — `userFilter` would include the RLS predicate, `viewFilter` would
+ /// not, and `match` would have returned null long before reaching here. Fail-loud guard so
+ /// a future relaxation of filter equivalence (e.g. OR / IN canonicalization in
+ /// MaterializedViewMatchUtils.filtersEqual) cannot silently drop the RLS predicate.
+ if (userQuery.getQueryOptions() != null) {
+ for (String optionKey : userQuery.getQueryOptions().keySet()) {
+ Preconditions.checkState(!optionKey.startsWith(CommonConstants.RLS_FILTERS),
+ "ExactSubsumptionStrategy must not clear the WHERE clause when row-level security "
+ + "filters are present (query option %s set). filtersEqual is structurally false "
+ + "in this case; reaching here indicates a filter-equivalence regression.",
+ optionKey);
+ }
+ }
+ rewritten.setFilterExpression(null);
+ /// Remap GROUP BY and ORDER BY from user column names to MV column names.
+ if (userQuery.getGroupByList() != null) {
+ List<Expression> remappedGroupBy = new ArrayList<>(userQuery.getGroupByList().size());
+ for (Expression expr : userQuery.getGroupByList()) {
+ remappedGroupBy.add(MaterializedViewMatchUtils.remapExpression(expr, viewProjectionMap));
+ }
+ rewritten.setGroupByList(remappedGroupBy);
+ }
+ if (userQuery.getOrderByList() != null) {
+ List<Expression> remappedOrderBy = new ArrayList<>(userQuery.getOrderByList().size());
+ for (Expression expr : userQuery.getOrderByList()) {
+ remappedOrderBy.add(MaterializedViewMatchUtils.remapExpression(expr, viewProjectionMap));
+ }
+ rewritten.setOrderByList(remappedOrderBy);
+ }
+ if (userQuery.getHavingExpression() != null) {
+ rewritten.setHavingExpression(
+ MaterializedViewMatchUtils.remapExpression(userQuery.getHavingExpression(), viewProjectionMap));
+ }
+ return new MaterializedViewRewritePlan(candidateEntry.getMaterializedViewTableNameWithType(),
+ MatchType.EXACT, null, rewritten, COST_EXACT);
+ }
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/strategy/MaterializedViewMatchStrategy.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/strategy/MaterializedViewMatchStrategy.java
new file mode 100644
index 00000000000..834f6e11e6d
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/strategy/MaterializedViewMatchStrategy.java
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.rewrite.strategy;
+
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewMetadataCache.MaterializedViewCacheEntry;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewRewritePlan;
+
+
+/// Strategy interface for matching a user query against a candidate materialized view.
+///
+/// Implementations produce an [MaterializedViewRewritePlan] fragment containing the
+/// rewritten MV query, match type, and cost. Execution mode and base query
+/// template are resolved later by
+/// [MaterializedViewQueryRewriteEngine][org.apache.pinot.materializedview.rewrite.MaterializedViewQueryRewriteEngine].
+///
+/// Implementations should be stateless and thread-safe. New matching algorithms
+/// (partial aggregation, column projection, time-range rollup, etc.) can be added
+/// by implementing this interface and registering with `MaterializedViewQueryRewriteEngine`.
+public interface MaterializedViewMatchStrategy {
+
+ /// Attempts to match the given user query against the candidate MV entry.
+ ///
+ /// @param userQuery the compiled PinotQuery from the user's SQL
+ /// @param candidateEntry the cached MV entry (metadata + pre-compiled definedSql query)
+ /// @return an [MaterializedViewRewritePlan] fragment if the MV can answer the user query,
+ /// `null` otherwise
+ @Nullable
+ MaterializedViewRewritePlan match(PinotQuery userQuery, MaterializedViewCacheEntry candidateEntry);
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/strategy/ScanSubsumptionStrategy.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/strategy/ScanSubsumptionStrategy.java
new file mode 100644
index 00000000000..8c7cabd5241
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/strategy/ScanSubsumptionStrategy.java
@@ -0,0 +1,139 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.rewrite.strategy;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.materializedview.rewrite.MatchType;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewMatchUtils;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewMetadataCache.MaterializedViewCacheEntry;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewQueryShape;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewRewritePlan;
+
+
+/// Subsumption strategy for **non-aggregation (scan)** queries where the
+/// user's SELECT list is a subset of the MV's SELECT list.
+///
+/// Both the user query and the MV must have [MaterializedViewQueryShape#SCAN] shape
+/// (no aggregation functions, no GROUP BY). The MV table is treated as a
+/// pre-filtered / pre-projected physical table whose columns correspond to the
+/// columns listed in the MV's `definedSql`.
+///
+/// Cost model:
+///
+/// - `2.0` — projection-subset match without residual WHERE
+/// - `3.0` — projection-subset match with residual WHERE
+///
+public class ScanSubsumptionStrategy extends AbstractSubsumptionStrategy {
+
+ private static final double COST_SCAN_SUBSUMPTION = 2.0;
+ private static final double COST_SCAN_WITH_RESIDUAL = 3.0;
+
+ @Override
+ protected boolean acceptsShape(PinotQuery userQuery, PinotQuery viewQuery) {
+ return MaterializedViewQueryShape.classify(userQuery) == MaterializedViewQueryShape.SCAN
+ && MaterializedViewQueryShape.classify(viewQuery) == MaterializedViewQueryShape.SCAN;
+ }
+
+ @Override
+ protected boolean groupByMatches(PinotQuery userQuery, PinotQuery viewQuery) {
+ return !userQuery.isSetGroupByList() && !viewQuery.isSetGroupByList();
+ }
+
+ @Override
+ protected boolean projectionSubsumes(List<Expression> userSelectList,
+ Map<Expression, String> viewProjectionMap) {
+ if (userSelectList == null || userSelectList.isEmpty()) {
+ return false;
+ }
+ for (Expression expr : userSelectList) {
+ Expression stripped = MaterializedViewMatchUtils.stripAlias(expr);
+ if (!viewProjectionMap.containsKey(stripped)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected boolean validateResidual(@Nullable Expression residualFilter, PinotQuery viewQuery,
+ Map<Expression, String> viewProjectionMap) {
+ if (residualFilter == null) {
+ return true;
+ }
+ /// Check whether the residual filter's user/base-side columns are all reachable through the
+ /// MV projection map. Comparing against `viewProjectionMap.values()` (MV-side names) would
+ /// over-reject MVs that alias the column — [MaterializedViewMatchUtils#coversReferencedColumns]
+ /// resolves against KEYS so an aliased projection (`base_col AS mv_col`) still matches.
+ Set<String> residualColumns = MaterializedViewMatchUtils.collectReferencedColumns(residualFilter);
+ return MaterializedViewMatchUtils.coversReferencedColumns(residualColumns, viewProjectionMap);
+ }
+
+ @Override
+ protected boolean orderByCompatible(PinotQuery userQuery, PinotQuery viewQuery,
+ Map<Expression, String> viewProjectionMap) {
+ List<Expression> orderByList = userQuery.getOrderByList();
+ if (orderByList == null || orderByList.isEmpty()) {
+ return true;
+ }
+ /// ORDER BY columns are user/base-side names; resolve them against MV projection keys to
+ /// support aliased MV columns — mirrors `validateResidual` above.
+ for (Expression orderByExpr : orderByList) {
+ Set<String> referencedColumns = MaterializedViewMatchUtils.collectReferencedColumns(orderByExpr);
+ if (!MaterializedViewMatchUtils.coversReferencedColumns(referencedColumns, viewProjectionMap)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ @Override
+ protected boolean havingCompatible(PinotQuery userQuery, PinotQuery viewQuery,
+ Map<Expression, String> viewProjectionMap) {
+ return userQuery.getHavingExpression() == null;
+ }
+
+ @Override
+ protected MaterializedViewRewritePlan buildResult(PinotQuery userQuery, MaterializedViewCacheEntry candidateEntry,
+ @Nullable Expression residualFilter, Map<Expression, String> viewProjectionMap, boolean filtersEqual) {
+ PinotQuery rewritten = userQuery.deepCopy();
+ rewritten.getDataSource().setTableName(candidateEntry.getMaterializedViewTableNameWithType());
+ rewritten.setSelectList(MaterializedViewMatchUtils.rewriteSelectList(userQuery.getSelectList(), viewProjectionMap));
+ /// Remap residual filter columns to MV column names in case the MV uses aliased columns.
+ Expression remappedResidual = residualFilter != null
+ ? MaterializedViewMatchUtils.remapExpression(residualFilter, viewProjectionMap) : null;
+ rewritten.setFilterExpression(remappedResidual);
+ if (userQuery.getOrderByList() != null && !userQuery.getOrderByList().isEmpty()) {
+ List<Expression> remappedOrderBy = new ArrayList<>(userQuery.getOrderByList().size());
+ for (Expression orderByExpr : userQuery.getOrderByList()) {
+ remappedOrderBy.add(MaterializedViewMatchUtils.remapExpression(orderByExpr, viewProjectionMap));
+ }
+ rewritten.setOrderByList(remappedOrderBy);
+ }
+
+ double cost = filtersEqual ? COST_SCAN_SUBSUMPTION : COST_SCAN_WITH_RESIDUAL;
+ return new MaterializedViewRewritePlan(candidateEntry.getMaterializedViewTableNameWithType(),
+ MatchType.SCAN_SUBSUME, null, rewritten, cost);
+ }
+}
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/strategy/package-info.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/strategy/package-info.java
new file mode 100644
index 00000000000..a6a96d65f75
--- /dev/null
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/rewrite/strategy/package-info.java
@@ -0,0 +1,37 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/// **Query rewrite — subsumption strategies.**
+///
+/// Each strategy tests whether a user query can be answered by an MV under a specific
+/// match relation. The engine tries them in cost order; the first non-null plan wins:
+///
+/// - `ExactSubsumptionStrategy` — user query is structurally identical to the MV (same
+/// SELECT / WHERE / GROUP BY / HAVING), only the table name differs. Cost 0.0.
+/// - `ScanSubsumptionStrategy` — user query projects a subset of the MV's columns with a
+/// (possibly stricter) residual filter; no aggregation. Cost 2.0 (or 3.0 with residual).
+/// - `AggregationSubsumptionStrategy` — user query's GROUP BY is a subset of the MV's,
+/// aggregates are compatible per
+/// [equivalence][org.apache.pinot.materializedview.rewrite.equivalence] rules, residual
+/// filter is allowed on dimensions that survive the MV's group-by.
+///
+/// All concrete strategies extend `AbstractSubsumptionStrategy` which provides the shared
+/// match skeleton (projection map build, residual extraction, remap-and-rewrite) and
+/// exposes per-step hooks for each subclass to override.
+package org.apache.pinot.materializedview.rewrite.strategy;
diff --git a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/scheduler/MaterializedViewTaskScheduler.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/scheduler/MaterializedViewTaskScheduler.java
index 1f874617ea9..cbb8bc9e4ee 100644
--- a/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/scheduler/MaterializedViewTaskScheduler.java
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/scheduler/MaterializedViewTaskScheduler.java
@@ -571,6 +571,21 @@ public class MaterializedViewTaskScheduler {
return timeColumn;
}
+ /// Resolves the raw format string for the MV table's time column, for persisting in
+ /// [MaterializedViewSplitSpec]. The broker uses this format to convert `watermarkMs` into
+ /// the MV column's native representation before attaching the `materializedViewTime < boundary`
+ /// filter on the MV branch of a split query.
+ private String resolveMaterializedViewTimeFormat(String viewTableWithType, String viewTimeColumn) {
+ Schema viewSchema = _context.getTableSchema(viewTableWithType);
+ Preconditions.checkState(viewSchema != null,
+ "Schema not found for MV table: %s", viewTableWithType);
+ DateTimeFieldSpec fieldSpec = viewSchema.getSpecForTimeColumn(viewTimeColumn);
+ Preconditions.checkState(fieldSpec != null,
+ "No DateTimeFieldSpec found for MV time column '%s' in table: %s",
+ viewTimeColumn, viewTableWithType);
+ return fieldSpec.getFormat();
+ }
+
/// Appends a time-range WHERE clause to the SQL. Window values are raw epoch millis since
/// both base and MV time columns are TIMESTAMP (enforced by [MaterializedViewAnalyzer]).
/// If a WHERE clause already exists, appends with AND; otherwise inserts before GROUP BY /
@@ -922,16 +937,17 @@ public class MaterializedViewTaskScheduler {
? MaterializedViewAnalyzer.extractPartitionExprMaps(definedSQL, viewSchema)
: new HashMap<>();
- // Resolve split spec from both the source and MV tables' time columns. The MV column
- // is enforced TIMESTAMP by MaterializedViewAnalyzer (no format needed there); the base
- // column may use any format, so we persist its `DateTimeFieldSpec.getFormat()` for the
- // broker's base-side filter conversion.
+ /// Resolve split spec from both the source and MV tables' time columns. Both base and MV
+ /// time columns may use any `DateTimeFieldSpec` format; persist each column's `getFormat()`
+ /// so the broker's base-side and MV-side boundary literals are each converted to the right
+ /// native unit at query time.
DateTimeFieldSpec sourceTimeFieldSpec = resolveSourceTimeFieldSpec(sourceTableName);
String sourceTimeColumn = sourceTimeFieldSpec.getName();
String sourceTimeFormat = sourceTimeFieldSpec.getFormat();
String viewTimeColumn = resolveMaterializedViewTimeColumn(viewTableWithType);
- MaterializedViewSplitSpec splitSpec =
- new MaterializedViewSplitSpec(sourceTimeColumn, sourceTimeFormat, viewTimeColumn, bucketMs);
+ String viewTimeFormat = resolveMaterializedViewTimeFormat(viewTableWithType, viewTimeColumn);
+ MaterializedViewSplitSpec splitSpec = new MaterializedViewSplitSpec(sourceTimeColumn,
+ sourceTimeFormat, viewTimeColumn, viewTimeFormat, bucketMs);
long stalenessThresholdMs = parseLong(
taskConfigs.get(MaterializedViewTask.STALENESS_THRESHOLD_MS_KEY),
diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/scheduler/package-info.java
similarity index 50%
copy from pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java
copy to pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/scheduler/package-info.java
index 7d7524b28ab..a2cec0a3e73 100644
--- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/RlsUtils.java
+++ b/pinot-materialized-view/src/main/java/org/apache/pinot/materializedview/scheduler/package-info.java
@@ -16,29 +16,16 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.sql.parsers.rewriter;
-import java.util.Map;
-import java.util.stream.Collectors;
-import org.apache.pinot.spi.utils.CommonConstants;
-
-
-public class RlsUtils {
-
- private RlsUtils() {
- }
-
- public static String buildRlsFilterKey(String tableName) {
- return String.format("%s-%s", CommonConstants.RLS_FILTERS, tableName);
- }
-
- public static Map<String, String> extractRlsFilters(Map<String, String> requestMetadata) {
- return requestMetadata.entrySet().stream()
- .filter(e -> e.getKey().startsWith(CommonConstants.RLS_FILTERS))
- .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
- }
-
- public static String getRlsFilterForTable(Map<String, String> queryOptions, String tableName) {
- return queryOptions.get(buildRlsFilterKey(tableName));
- }
-}
+/// **View generation — task scheduling.**
+///
+/// Controller-side minion task scheduler. Walks the catalog of materialized views, decides
+/// which time windows of the MV need to be (re)materialized (cold-start, watermark advance,
+/// drift recovery), and generates `PinotTaskConfig` records that the minion picks up and
+/// runs through the executor in [org.apache.pinot.materializedview.executor].
+///
+/// The scheduler updates the runtime znode in [org.apache.pinot.materializedview.metadata]
+/// when tasks succeed; the broker rewrite engine in
+/// [org.apache.pinot.materializedview.rewrite] reads that znode to know which time ranges
+/// are queryable from the MV.
+package org.apache.pinot.materializedview.scheduler;
diff --git a/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/analysis/MaterializedViewAnalyzerTest.java b/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/analysis/MaterializedViewAnalyzerTest.java
index f5ab621e90e..d6c3290023b 100644
--- a/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/analysis/MaterializedViewAnalyzerTest.java
+++ b/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/analysis/MaterializedViewAnalyzerTest.java
@@ -638,6 +638,28 @@ public class MaterializedViewAnalyzerTest {
expectError(sql, viewSchema, "does not match any column in the MV table schema");
}
+ @Test
+ public void testRejectsAliasCaseMismatch() {
+ /// Case-sensitive cluster regression: an MV defined with `SUM(x) AS sum_x` against a schema
+ /// that has the column declared as `Sum_X` must be rejected at analyzer time so the broker
+ /// FULL_REWRITE re-canonicalization never has to deal with the mismatch at query time. The
+ /// existing `selectFields.contains` / `schemaColumns.contains` checks are case-sensitive,
+ /// so a case-different SELECT alias fails Check 2 ("SELECT field does not match any column").
+ String sql = "SELECT DaysSinceEpoch, city, sum(amount) AS sum_amount "
+ + "FROM orders GROUP BY DaysSinceEpoch, city";
+ Schema viewSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension("city", FieldSpec.DataType.STRING)
+ .addMetric("Sum_Amount", FieldSpec.DataType.DOUBLE) // case differs from SELECT alias
+ .addDateTime(TIME_COLUMN, FieldSpec.DataType.TIMESTAMP, "1:MILLISECONDS:TIMESTAMP", "1:MILLISECONDS")
+ .build();
+
+ /// The existing `schemaColumns.contains(selectField)` check in Step 3 is case-sensitive
+ /// because both sides are HashSet<String>; the analyzer fails Check 1 ("MV schema column not
+ /// produced by any SELECT expression") before reaching Check 2, but either failure pin the
+ /// case-sensitivity contract.
+ expectError(sql, viewSchema, "is not produced by any SELECT expression");
+ }
+
@Test
public void testAggregateWithoutAlias() {
String sql = "SELECT DaysSinceEpoch, city, count(*) FROM orders GROUP BY DaysSinceEpoch, city";
diff --git a/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/context/MaterializedViewContextTest.java b/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/context/MaterializedViewContextTest.java
new file mode 100644
index 00000000000..d7f5d3d61cd
--- /dev/null
+++ b/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/context/MaterializedViewContextTest.java
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.context;
+
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.materializedview.rewrite.ExecutionMode;
+import org.apache.pinot.materializedview.rewrite.MatchType;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewRewritePlan;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+/// Pins the factory + accessor contracts on `MaterializedViewContext`. The broker keys on
+/// `isFullRewrite()` / `isSplitRewrite()` to decide which post-compile path to take and on
+/// `getMaterializedViewQueriedName()` for the response field, so changes to those contracts
+/// must surface as a test failure here.
+public class MaterializedViewContextTest {
+
+ @Test
+ public void testEmptyContextReportsNoRewrite() {
+ MaterializedViewContext ctx = MaterializedViewContext.empty();
+ Assert.assertFalse(ctx.isFullRewrite());
+ Assert.assertFalse(ctx.isSplitRewrite());
+ Assert.assertNull(ctx.getPlan());
+ Assert.assertNull(ctx.getMaterializedViewQueriedName());
+ Assert.assertNull(ctx.getSplitRewriteContext());
+ }
+
+ @Test
+ public void testFullRewriteContextCarriesPlanAndFullRewriteFlag() {
+ PinotQuery materializedViewQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT ts, SUM(revenue) FROM mv_baseTable_OFFLINE GROUP BY ts");
+ MaterializedViewRewritePlan plan = new MaterializedViewRewritePlan(
+ "mv_baseTable_OFFLINE", MatchType.EXACT, ExecutionMode.FULL_REWRITE, materializedViewQuery, 1.0);
+
+ MaterializedViewContext ctx = MaterializedViewContext.forFullRewrite(plan);
+
+ Assert.assertTrue(ctx.isFullRewrite());
+ Assert.assertFalse(ctx.isSplitRewrite());
+ Assert.assertSame(ctx.getPlan(), plan);
+ Assert.assertEquals(ctx.getMaterializedViewQueriedName(), "mv_baseTable_OFFLINE");
+ Assert.assertNull(ctx.getSplitRewriteContext(),
+ "FULL_REWRITE context must not expose a split-rewrite context");
+ }
+
+ @Test
+ public void testSplitRewriteContextCarriesMvBranchState() {
+ PinotQuery materializedViewQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT ts, SUM(revenue) FROM mv_baseTable_OFFLINE GROUP BY ts");
+ MaterializedViewRewritePlan plan = new MaterializedViewRewritePlan(
+ "mv_baseTable_OFFLINE", MatchType.EXACT, ExecutionMode.SPLIT_REWRITE, materializedViewQuery, 1.0);
+ Schema mvSchema = new Schema.SchemaBuilder()
+ .setSchemaName("mv_baseTable")
+ .addSingleValueDimension("ts", DataType.STRING)
+ .addMetric("revenue", DataType.DOUBLE)
+ .build();
+
+ MaterializedViewContext ctx = MaterializedViewContext.forSplitRewrite(
+ plan, materializedViewQuery, "mv_baseTable_OFFLINE", mvSchema);
+
+ Assert.assertTrue(ctx.isSplitRewrite());
+ Assert.assertFalse(ctx.isFullRewrite());
+ Assert.assertSame(ctx.getPlan(), plan);
+ Assert.assertEquals(ctx.getMaterializedViewQueriedName(), "mv_baseTable_OFFLINE");
+
+ MaterializedViewContext.SplitRewriteContext splitCtx = ctx.getSplitRewriteContext();
+ Assert.assertNotNull(splitCtx, "SPLIT_REWRITE context must expose the MV-branch state");
+ Assert.assertSame(splitCtx.getMaterializedViewServerPinotQuery(), materializedViewQuery);
+ Assert.assertEquals(splitCtx.getMaterializedViewTableNameWithType(), "mv_baseTable_OFFLINE");
+ Assert.assertSame(splitCtx.getMaterializedViewSchema(), mvSchema);
+ }
+}
diff --git a/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/handler/DefaultMaterializedViewHandlerTest.java b/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/handler/DefaultMaterializedViewHandlerTest.java
new file mode 100644
index 00000000000..d28c6dde747
--- /dev/null
+++ b/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/handler/DefaultMaterializedViewHandlerTest.java
@@ -0,0 +1,138 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.handler;
+
+import java.util.Locale;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.core.routing.timeboundary.TimeBoundaryInfo;
+import org.apache.pinot.spi.data.DateTimeFormatSpec;
+import org.apache.pinot.sql.FilterKind;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+/// Unit tests for [DefaultMaterializedViewHandler#attachFilter], the helper that places the
+/// per-branch time-boundary filter on the base (`>= boundary`) and MV (`< boundary`) sides of a
+/// split execution.
+public class DefaultMaterializedViewHandlerTest {
+
+ @Test
+ public void testAttachFilterLessThanOnEmptyFilter() {
+ PinotQuery query = CalciteSqlParser.compileToPinotQuery(
+ "SELECT carrier, SUM(delay) FROM materializedViewTable GROUP BY carrier");
+ Assert.assertNull(query.getFilterExpression(), "precondition: no WHERE clause");
+
+ DefaultMaterializedViewHandler.attachFilter(query, new TimeBoundaryInfo("materializedViewDay", "20000"),
+ FilterKind.LESS_THAN);
+
+ Expression filter = query.getFilterExpression();
+ Assert.assertNotNull(filter, "MV branch must receive the upper-bound filter");
+ Function fn = filter.getFunctionCall();
+ Assert.assertNotNull(fn);
+ Assert.assertEquals(fn.getOperator().toUpperCase(Locale.ROOT), "LESS_THAN",
+ "MV upper-bound filter must use LESS_THAN (exclusive), symmetric to the base branch's >=");
+ Assert.assertEquals(fn.getOperands().get(0).getIdentifier().getName(), "materializedViewDay");
+ }
+
+ @Test
+ public void testAttachFilterLessThanWithExistingFilter() {
+ PinotQuery query = CalciteSqlParser.compileToPinotQuery(
+ "SELECT carrier, SUM(delay) FROM materializedViewTable WHERE carrier = 'AA' GROUP BY carrier");
+ Assert.assertNotNull(query.getFilterExpression(), "precondition: existing WHERE clause");
+
+ DefaultMaterializedViewHandler.attachFilter(query, new TimeBoundaryInfo("materializedViewDay", "20000"),
+ FilterKind.LESS_THAN);
+
+ Expression filter = query.getFilterExpression();
+ Function root = filter.getFunctionCall();
+ Assert.assertNotNull(root);
+ Assert.assertEquals(root.getOperator().toUpperCase(Locale.ROOT), "AND",
+ "Upper-bound filter must be AND-ed with the user's WHERE, not replace it");
+ Assert.assertEquals(root.getOperands().size(), 2);
+
+ /// Second operand is the injected LESS_THAN(materializedViewDay, 20000).
+ Function injected = root.getOperands().get(1).getFunctionCall();
+ Assert.assertNotNull(injected);
+ Assert.assertEquals(injected.getOperator().toUpperCase(Locale.ROOT), "LESS_THAN");
+ Assert.assertEquals(injected.getOperands().get(0).getIdentifier().getName(), "materializedViewDay");
+ }
+
+ @Test
+ public void testAttachFilterGreaterThanOrEqualOnBaseSide() {
+ PinotQuery query = CalciteSqlParser.compileToPinotQuery("SELECT col FROM baseTable");
+ DefaultMaterializedViewHandler.attachFilter(query, new TimeBoundaryInfo("ts", "100"),
+ FilterKind.GREATER_THAN_OR_EQUAL);
+
+ Function fn = query.getFilterExpression().getFunctionCall();
+ Assert.assertEquals(fn.getOperator().toUpperCase(Locale.ROOT), "GREATER_THAN_OR_EQUAL",
+ "Base branch filter must be inclusive >= since the MV uses exclusive <");
+ Assert.assertEquals(fn.getOperands().get(0).getIdentifier().getName(), "ts");
+ }
+
+ /// Pins the contract that `DateTimeFormatSpec.fromMillisToFormat` emits a literal Pinot can
+ /// coerce back to the column's stored type for every time-column format we expect to support on
+ /// the base side of a SPLIT_REWRITE. Each row asserts the round-trip
+ /// (`millis -> formatted literal -> millis`) holds without loss; that round-trip is what makes
+ /// `base.ts >= literal` and `mv.materializedViewTime < literal` partition the timeline exactly.
+ ///
+ /// Regression coverage for the M5 reviewer concern that today only `INT/DAYS:EPOCH` and
+ /// `LONG/MILLISECONDS:EPOCH` are exercised by the integration test.
+ @Test
+ public void testBoundaryLiteralRoundTripAcrossSupportedFormats() {
+ long sampleMillis = 1700000000000L; // 2023-11-14T22:13:20Z — picked to round-trip cleanly in every format.
+ String[] formats = {
+ "1:MILLISECONDS:EPOCH",
+ "1:SECONDS:EPOCH",
+ "1:MINUTES:EPOCH",
+ "1:HOURS:EPOCH",
+ "1:DAYS:EPOCH",
+ "TIMESTAMP",
+ "1:DAYS:SIMPLE_DATE_FORMAT:yyyy-MM-dd"
+ };
+ for (String format : formats) {
+ DateTimeFormatSpec spec = new DateTimeFormatSpec(format);
+ String literal = spec.fromMillisToFormat(sampleMillis);
+ Assert.assertNotNull(literal, "format " + format + " produced null literal");
+ Assert.assertFalse(literal.isEmpty(), "format " + format + " produced empty literal");
+ /// Round-trip: parsing the formatted literal back to millis must produce a value within the
+ /// format's coarsest granularity. For example `1:DAYS:EPOCH` truncates millis to day-granular
+ /// boundaries; for SIMPLE_DATE_FORMAT (yyyy-MM-dd) we also truncate to day granularity.
+ long parsed = spec.fromFormatToMillis(literal);
+ /// Tolerance = format granularity in millis (or 1 day for SIMPLE_DATE_FORMAT yyyy-MM-dd).
+ long toleranceMs;
+ if (format.equals("TIMESTAMP") || format.equals("1:MILLISECONDS:EPOCH")) {
+ toleranceMs = 1L;
+ } else if (format.equals("1:SECONDS:EPOCH")) {
+ toleranceMs = 1_000L;
+ } else if (format.equals("1:MINUTES:EPOCH")) {
+ toleranceMs = 60_000L;
+ } else if (format.equals("1:HOURS:EPOCH")) {
+ toleranceMs = 3_600_000L;
+ } else {
+ toleranceMs = 86_400_000L;
+ }
+ Assert.assertTrue(Math.abs(parsed - sampleMillis) < toleranceMs,
+ "format " + format + " did not round-trip: " + sampleMillis + " -> '" + literal
+ + "' -> " + parsed);
+ }
+ }
+}
diff --git a/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/handler/MaterializedViewHandlerLoadTest.java b/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/handler/MaterializedViewHandlerLoadTest.java
new file mode 100644
index 00000000000..6e529a83a83
--- /dev/null
+++ b/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/handler/MaterializedViewHandlerLoadTest.java
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.handler;
+
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.materializedview.context.MaterializedViewContext;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+/// Tests for [MaterializedViewHandler#loadHandler], the reflective loader that lets operators
+/// configure a custom MV handler class via `pinot.broker.materialized.view.handler.class`.
+public class MaterializedViewHandlerLoadTest {
+
+ /// Public no-arg constructor + observable `init()` call: this is the contract custom handlers
+ /// must satisfy to be loadable by [MaterializedViewHandler#loadHandler].
+ public static class StubMaterializedViewHandler implements MaterializedViewHandler {
+ static volatile PinotConfiguration _lastInitConfig;
+ static volatile Boolean _lastInitSplitSupport;
+
+ public StubMaterializedViewHandler() {
+ }
+
+ @Override
+ public void init(PinotConfiguration configuration, ZkHelixPropertyStore<ZNRecord> propertyStore,
+ boolean supportsSplitRewrite) {
+ _lastInitConfig = configuration;
+ _lastInitSplitSupport = supportsSplitRewrite;
+ }
+
+ @Override
+ public MaterializedViewContext compile(MaterializedViewCompileContext ctx) {
+ return MaterializedViewContext.empty();
+ }
+
+ @Override
+ public BrokerResponseNative executeSplit(MaterializedViewSplitExecutionContext ctx) {
+ throw new UnsupportedOperationException("stub");
+ }
+
+ @Override
+ public void annotateResponse(BrokerResponseNative response, MaterializedViewContext mvContext) {
+ }
+
+ @Override
+ public void invalidateBaseTable(String rawTableName) {
+ }
+
+ @Override
+ public boolean supportsSplitRewrite() {
+ return false;
+ }
+ }
+
+ @Test
+ public void testLoadCustomHandlerByClassName() {
+ StubMaterializedViewHandler._lastInitConfig = null;
+ StubMaterializedViewHandler._lastInitSplitSupport = null;
+
+ Map<String, Object> props = new HashMap<>();
+ props.put("class", StubMaterializedViewHandler.class.getName());
+ props.put("custom.option", "abc");
+ PinotConfiguration cfg = new PinotConfiguration(props);
+
+ MaterializedViewHandler handler = MaterializedViewHandler.loadHandler(cfg, null, true);
+
+ Assert.assertTrue(handler instanceof StubMaterializedViewHandler,
+ "Loader must instantiate the configured class, got: " + handler.getClass().getName());
+ Assert.assertNotNull(StubMaterializedViewHandler._lastInitConfig, "init() must be invoked");
+ Assert.assertEquals(StubMaterializedViewHandler._lastInitConfig.getProperty("custom.option"), "abc",
+ "init() must receive the same config so handler-specific options are reachable");
+ Assert.assertEquals(StubMaterializedViewHandler._lastInitSplitSupport, Boolean.TRUE,
+ "init() must receive the supportsSplitRewrite signal from the broker");
+ }
+
+ @Test(expectedExceptions = RuntimeException.class,
+ expectedExceptionsMessageRegExp = "Failed to load MaterializedViewHandler class:.*")
+ public void testLoadHandlerThrowsForUnknownClass() {
+ Map<String, Object> props = new HashMap<>();
+ props.put("class", "no.such.HandlerClass");
+ MaterializedViewHandler.loadHandler(new PinotConfiguration(props), null, true);
+ }
+}
diff --git a/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/metadata/MaterializedViewMetadataTest.java b/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/metadata/MaterializedViewMetadataTest.java
index 36ae931415c..8568373ddca 100644
--- a/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/metadata/MaterializedViewMetadataTest.java
+++ b/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/metadata/MaterializedViewMetadataTest.java
@@ -41,7 +41,7 @@ public class MaterializedViewMetadataTest {
partitionExprMaps.put("DaysSinceEpoch", "DaysSinceEpoch");
MaterializedViewSplitSpec splitSpec = new MaterializedViewSplitSpec(
- "ts", "1:MILLISECONDS:EPOCH", "DaysSinceEpoch", 86400000L);
+ "ts", "1:MILLISECONDS:EPOCH", "DaysSinceEpoch", "1:DAYS:EPOCH", 86400000L);
MaterializedViewDefinitionMetadata original = new MaterializedViewDefinitionMetadata(
viewTableName,
@@ -63,7 +63,9 @@ public class MaterializedViewMetadataTest {
MaterializedViewSplitSpec restoredSpec = restored.getSplitSpec();
assertNotNull(restoredSpec);
assertEquals(restoredSpec.getSourceTimeColumn(), "ts");
+ assertEquals(restoredSpec.getSourceTimeFormat(), "1:MILLISECONDS:EPOCH");
assertEquals(restoredSpec.getMaterializedViewTimeColumn(), "DaysSinceEpoch");
+ assertEquals(restoredSpec.getMaterializedViewTimeFormat(), "1:DAYS:EPOCH");
assertEquals(restoredSpec.getBucketMs(), 86400000L);
}
diff --git a/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/rewrite/AggregationSubsumptionStrategyTest.java b/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/rewrite/AggregationSubsumptionStrategyTest.java
new file mode 100644
index 00000000000..3b17c594453
--- /dev/null
+++ b/pinot-materialized-view/src/test/java/org/apache/pinot/materializedview/rewrite/AggregationSubsumptionStrategyTest.java
@@ -0,0 +1,897 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.materializedview.rewrite;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.materializedview.metadata.MaterializedViewDefinitionMetadata;
+import org.apache.pinot.materializedview.rewrite.MaterializedViewMetadataCache.MaterializedViewCacheEntry;
+import org.apache.pinot.materializedview.rewrite.strategy.AggregationSubsumptionStrategy;
+import org.apache.pinot.sql.parsers.CalciteSqlParser;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.*;
+
+
+public class AggregationSubsumptionStrategyTest {
+
+ private AggregationSubsumptionStrategy _strategy;
+
+ @BeforeClass
+ public void setUp() {
+ _strategy = new AggregationSubsumptionStrategy();
+ }
+
+ private MaterializedViewCacheEntry createEntry(String viewTableName, String baseTable, String definedSql) {
+ MaterializedViewDefinitionMetadata definition = new MaterializedViewDefinitionMetadata(
+ viewTableName,
+ Collections.singletonList(baseTable),
+ definedSql,
+ new HashMap<>(),
+ null);
+ PinotQuery compiledQuery = CalciteSqlParser.compileToPinotQuery(definedSql);
+ return new MaterializedViewCacheEntry(definition, compiledQuery, 1L, java.util.Map.of());
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Basic aggregation matching
+ /// -----------------------------------------------------------------------
+
+ @Test
+ public void testBasicAggregationMatch() {
+ String definedSql = "SELECT city, SUM(revenue) AS sum_rev FROM orders GROUP BY city";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT city, SUM(revenue) FROM orders GROUP BY city");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNotNull(result);
+ assertEquals(result.getMaterializedViewTableNameWithType(), "mv_orders_OFFLINE");
+ assertEquals(result.getCost(), 6.0);
+ assertEquals(result.getMaterializedViewQuery().getDataSource().getTableName(), "mv_orders_OFFLINE");
+ }
+
+ @Test
+ public void testSelectSubset() {
+ String definedSql =
+ "SELECT city, SUM(revenue) AS sum_rev, COUNT(revenue) AS cnt FROM orders GROUP BY city";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT city, SUM(revenue) FROM orders GROUP BY city");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNotNull(result);
+ assertEquals(result.getCost(), 6.0);
+ assertEquals(result.getMaterializedViewQuery().getSelectList().size(), 2);
+ }
+
+ /// -----------------------------------------------------------------------
+ /// GROUP BY order insensitivity
+ /// -----------------------------------------------------------------------
+
+ @Test
+ public void testGroupByOrderInsensitive() {
+ String definedSql =
+ "SELECT city, state, SUM(revenue) AS sum_rev FROM orders GROUP BY city, state";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT city, state, SUM(revenue) FROM orders GROUP BY state, city");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNotNull(result);
+ assertEquals(result.getCost(), 6.0);
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Alias handling
+ /// -----------------------------------------------------------------------
+
+ @Test
+ public void testAliasInsensitiveMatching() {
+ String definedSql = "SELECT city, SUM(revenue) AS sum_rev FROM orders GROUP BY city";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT city, SUM(revenue) AS total_revenue FROM orders GROUP BY city");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNotNull(result);
+ assertEquals(result.getCost(), 6.0);
+ }
+
+ @Test
+ public void testRewrittenSelectPreservesUserAlias() {
+ String definedSql = "SELECT city, SUM(revenue) AS sum_rev FROM orders GROUP BY city";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT city, SUM(revenue) AS r_sum FROM orders GROUP BY city");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNotNull(result);
+ PinotQuery rewritten = result.getMaterializedViewQuery();
+
+ List<Expression> selectList = rewritten.getSelectList();
+ assertEquals(selectList.size(), 2);
+
+ assertEquals(selectList.get(0).getIdentifier().getName(), "city");
+
+ /// SUM(revenue) AS r_sum → SUM(sum_rev) AS r_sum
+ Function aliasFunc = selectList.get(1).getFunctionCall();
+ assertNotNull(aliasFunc);
+ assertEquals(aliasFunc.getOperator(), "as");
+ Function innerAgg = aliasFunc.getOperands().get(0).getFunctionCall();
+ assertNotNull(innerAgg);
+ assertEquals(innerAgg.getOperator(), "sum");
+ assertEquals(innerAgg.getOperands().get(0).getIdentifier().getName(), "sum_rev");
+ assertEquals(aliasFunc.getOperands().get(1).getIdentifier().getName(), "r_sum");
+ }
+
+ @Test
+ public void testRewrittenSelectNoAlias() {
+ String definedSql = "SELECT city, SUM(revenue) AS sum_rev FROM orders GROUP BY city";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT city, SUM(revenue) FROM orders GROUP BY city");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNotNull(result);
+ List<Expression> selectList = result.getMaterializedViewQuery().getSelectList();
+ assertEquals(selectList.size(), 2);
+
+ assertEquals(selectList.get(0).getIdentifier().getName(), "city");
+ /// SUM(revenue) → SUM(sum_rev) AS sum(revenue) — implicit alias preserves the user's
+ /// expected result column name when there is no explicit AS clause.
+ Function aliasFunc = selectList.get(1).getFunctionCall();
+ assertNotNull(aliasFunc);
+ assertEquals(aliasFunc.getOperator(), "as");
+ Function reAgg = aliasFunc.getOperands().get(0).getFunctionCall();
+ assertNotNull(reAgg);
+ assertEquals(reAgg.getOperator(), "sum");
+ assertEquals(reAgg.getOperands().get(0).getIdentifier().getName(), "sum_rev");
+ assertEquals(aliasFunc.getOperands().get(1).getIdentifier().getName(), "sum(revenue)");
+ }
+
+ /// -----------------------------------------------------------------------
+ /// GROUP BY mismatch
+ /// -----------------------------------------------------------------------
+
+ @Test
+ public void testNoMatchUserHasMoreGroupByColumns() {
+ String definedSql = "SELECT city, SUM(revenue) AS sum_rev FROM orders GROUP BY city";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT city, state, SUM(revenue) FROM orders GROUP BY city, state");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNull(result);
+ }
+
+ @Test
+ public void testFinerMaterializedViewGranularityBasicSum() {
+ String definedSql =
+ "SELECT city, state, SUM(revenue) AS sum_rev FROM orders GROUP BY city, state";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT city, SUM(revenue) FROM orders GROUP BY city");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNotNull(result, "Finer MV granularity should match via re-aggregation");
+ assertEquals(result.getCost(), 6.0);
+
+ PinotQuery rewritten = result.getMaterializedViewQuery();
+ assertNotNull(rewritten.getGroupByList(), "GROUP BY should be retained");
+ assertEquals(rewritten.getGroupByList().size(), 1);
+ assertEquals(rewritten.getGroupByList().get(0).getIdentifier().getName(), "city");
+
+ List<Expression> selectList = rewritten.getSelectList();
+ assertEquals(selectList.size(), 2);
+ assertEquals(selectList.get(0).getIdentifier().getName(), "city");
+ /// SUM(revenue) → SUM(sum_rev) AS sum(revenue) — implicit alias preserves result column name.
+ Function aliasFunc = selectList.get(1).getFunctionCall();
+ assertNotNull(aliasFunc);
+ assertEquals(aliasFunc.getOperator(), "as");
+ Function reAgg = aliasFunc.getOperands().get(0).getFunctionCall();
+ assertNotNull(reAgg);
+ assertEquals(reAgg.getOperator(), "sum");
+ assertEquals(reAgg.getOperands().get(0).getIdentifier().getName(), "sum_rev");
+ assertEquals(aliasFunc.getOperands().get(1).getIdentifier().getName(), "sum(revenue)");
+ }
+
+ @Test
+ public void testWholeTableUserAgainstGroupedMaterializedView() {
+ /// User query has no GROUP BY but contains an aggregation; MV is grouped. Engine should
+ /// re-aggregate the MV's per-group rows into one whole-table result on the MV side. This
+ /// pins the fix for groupByMatches accepting the userIsWholeTable && !mvIsWholeTable case.
+ String definedSql =
+ "SELECT city, SUM(revenue) AS sum_rev FROM orders GROUP BY city";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery("SELECT SUM(revenue) FROM orders");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNotNull(result, "User whole-table over grouped MV should re-aggregate");
+ PinotQuery rewritten = result.getMaterializedViewQuery();
+ assertTrue(rewritten.getGroupByList() == null || rewritten.getGroupByList().isEmpty(),
+ "Rewritten query must not have GROUP BY (whole-table re-aggregation)");
+ List<Expression> selectList = rewritten.getSelectList();
+ assertEquals(selectList.size(), 1);
+ /// SUM(revenue) → SUM(sum_rev) AS sum(revenue) — implicit alias preserves result column name.
+ Function aliasFunc = selectList.get(0).getFunctionCall();
+ assertNotNull(aliasFunc);
+ assertEquals(aliasFunc.getOperator(), "as");
+ Function reAgg = aliasFunc.getOperands().get(0).getFunctionCall();
+ assertNotNull(reAgg);
+ assertEquals(reAgg.getOperator(), "sum");
+ assertEquals(reAgg.getOperands().get(0).getIdentifier().getName(), "sum_rev");
+ assertEquals(aliasFunc.getOperands().get(1).getIdentifier().getName(), "sum(revenue)");
+ }
+
+ @Test
+ public void testNoMatchDifferentGroupByColumns() {
+ String definedSql = "SELECT city, SUM(revenue) AS sum_rev FROM orders GROUP BY city";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT state, SUM(revenue) FROM orders GROUP BY state");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNull(result);
+ }
+
+ /// -----------------------------------------------------------------------
+ /// HAVING remap
+ /// -----------------------------------------------------------------------
+
+ @Test
+ public void testHavingKeptAsHaving() {
+ String definedSql = "SELECT city, SUM(revenue) AS sum_rev FROM orders GROUP BY city";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT city, SUM(revenue) FROM orders GROUP BY city HAVING SUM(revenue) > 1000");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNotNull(result);
+ PinotQuery rewritten = result.getMaterializedViewQuery();
+
+ assertNotNull(rewritten.getGroupByList());
+ assertNotNull(rewritten.getHavingExpression());
+ assertNull(rewritten.getFilterExpression());
+
+ /// HAVING SUM(revenue) > 1000 → HAVING SUM(sum_rev) > 1000
+ assertTrue(rewritten.getHavingExpression().toString().contains("sum_rev"));
+ }
+
+ @Test
+ public void testNoMatchHavingReferencesUnknownAgg() {
+ String definedSql = "SELECT city, SUM(revenue) AS sum_rev FROM orders GROUP BY city";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT city, SUM(revenue) FROM orders GROUP BY city HAVING AVG(revenue) > 100");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNull(result, "HAVING references AVG(revenue) which is not in MV projection");
+ }
+
+ /// -----------------------------------------------------------------------
+ /// ORDER BY remap
+ /// -----------------------------------------------------------------------
+
+ @Test
+ public void testOrderByRemappedToMaterializedViewColumn() {
+ String definedSql = "SELECT city, SUM(revenue) AS sum_rev FROM orders GROUP BY city";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT city, SUM(revenue) FROM orders GROUP BY city ORDER BY SUM(revenue) DESC");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNotNull(result);
+ PinotQuery rewritten = result.getMaterializedViewQuery();
+
+ List<Expression> orderByList = rewritten.getOrderByList();
+ assertNotNull(orderByList);
+ assertEquals(orderByList.size(), 1);
+ assertTrue(orderByList.get(0).toString().contains("sum_rev"));
+ }
+
+ @Test
+ public void testOrderByDimensionColumn() {
+ String definedSql = "SELECT city, SUM(revenue) AS sum_rev FROM orders GROUP BY city";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT city, SUM(revenue) FROM orders GROUP BY city ORDER BY city ASC");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNotNull(result);
+ assertNotNull(result.getMaterializedViewQuery().getOrderByList());
+ }
+
+ @Test
+ public void testNoMatchOrderByUnknownExpression() {
+ String definedSql = "SELECT city, SUM(revenue) AS sum_rev FROM orders GROUP BY city";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT city, SUM(revenue) FROM orders GROUP BY city ORDER BY AVG(revenue)");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNull(result);
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Residual WHERE filter
+ /// -----------------------------------------------------------------------
+
+ @Test
+ public void testResidualWhereOnGroupByColumn() {
+ String definedSql =
+ "SELECT city, SUM(revenue) AS sum_rev FROM orders WHERE region = 'US' GROUP BY city";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT city, SUM(revenue) FROM orders WHERE region = 'US' AND city = 'NYC' GROUP BY city");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNotNull(result);
+ assertEquals(result.getCost(), 7.0);
+ assertNotNull(result.getMaterializedViewQuery().getFilterExpression());
+ }
+
+ @Test
+ public void testNoMatchResidualWhereOnNonGroupByColumn() {
+ String definedSql = "SELECT city, SUM(revenue) AS sum_rev FROM orders GROUP BY city";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT city, SUM(revenue) FROM orders WHERE status = 'active' GROUP BY city");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNull(result, "Residual filter on non-GROUP-BY column 'status' is invalid for aggregation MV");
+ }
+
+ /// -----------------------------------------------------------------------
+ /// Shape rejection
+ /// -----------------------------------------------------------------------
+
+ @Test
+ public void testNoMatchScanUserVsAggMaterializedView() {
+ String definedSql = "SELECT city, SUM(revenue) AS sum_rev FROM orders GROUP BY city";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery("SELECT city FROM orders");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNull(result);
+ }
+
+ @Test
+ public void testNoMatchAggUserVsScanMaterializedView() {
+ String definedSql = "SELECT city, revenue FROM orders";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT city, SUM(revenue) FROM orders GROUP BY city");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
+
+ assertNull(result);
+ }
+
+ /// -----------------------------------------------------------------------
+ /// SELECT expression not in MV
+ /// -----------------------------------------------------------------------
+
+ @Test
+ public void testNoMatchSelectExprNotInMaterializedView() {
+ String definedSql = "SELECT city, SUM(revenue) AS sum_rev FROM orders GROUP BY city";
+ MaterializedViewCacheEntry entry = createEntry("mv_orders_OFFLINE", "orders", definedSql);
+
+ PinotQuery userQuery = CalciteSqlParser.compileToPinotQuery(
+ "SELECT city, AVG(revenue) FROM orders GROUP BY city");
+ MaterializedViewRewritePlan result = _strategy.match(userQuery, entry);
... 2145 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org