You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by ma...@apache.org on 2021/04/14 00:20:24 UTC
[incubator-pinot] branch master updated: Pinot SPI refactor contd:
Added pinot-segment-local module. (#6761)
This is an automated email from the ASF dual-hosted git repository.
mayanks pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new f2e3446 Pinot SPI refactor contd: Added pinot-segment-local module. (#6761)
f2e3446 is described below
commit f2e3446e75f1ec1d553805d03f6504f05b3e3c0e
Author: Mayank Shrivastava <ma...@apache.org>
AuthorDate: Tue Apr 13 17:20:05 2021 -0700
Pinot SPI refactor contd: Added pinot-segment-local module. (#6761)
- Added pinot-segment-local module that contains all the segment related
implementations for the existing local Pinot segment.
- Modules that pinot-segment-local depends on:
- pinot-spi
- pinot-segment-spi
- pinot-common: This is temporary and needed as a lot of common utilities
are within pinot-common.
- External libraries: Datasketches,TDigest, HLL, Lucene, larray.
- For test-scope:
- pinot-avro
- pinot-csv
- pinot-json
- pinot-spi
- pinot-yammer
- Modules dependent on pinot-segment-local:
- pinot-core
- pinot-server, pinot-perf: Only in test-scope
Most moves/refactors done via IDE. Manual changes listed below:
- Fix typo in Logger for RealtimeLuceneIndexReaderRefreshThread.
- Moved builder specific util methods from StarTreeUtils to StarTreeBuilderUtils.
- `shouldRemove()`
- `remove()`
- Made local copy of MAX_DOC_PER_CALL inside of PinotDataBitSetV2, as it is currently
defined in DocIdSetPlanNode, but segment related code depends on it.
- Made local copy of DEFAULT_TDIGEST_COMPRESSION in PercentileTDigestValueAggregator,
originally defined in PercentileTDigestAggregationFunction.
- Made local copy of DEFAULT_MAX_ERROR in PercentileEstValueAggregator, originally defined in
PercentileEstAggregationFunction.
- Split QueryContextRequestUtils into two files, the new one calle RequestContextConverterUtils
is in pinot-common, as it has utility methods related to classes in pinot-common.
- Created CustomSerDeUtils class, that has code from ObjectSerDeUtils related to ValueAggregator.
This is again to avoid pulling in dependencies from pinot-core into pinot-segment-local.
- Removed reference of Operator in javadoc of EarlyTerminationException, as it is
was causing reference dependency to pinot-core.
- Renamed SegmentGenerationWithNullValueTest to SegmentWithNullValueTest as it has test for segment
generation as well as querying, and can only be housed in pinot-core atm.
- The clas RealtimeSegmentStatsHistory is ser/de using Java object ser/de, which makes moving the file
out of its current package backward incompatible. Addressed this issue by using a custom object
input stream that replaces the old package name with new one when deserializing.
Moves done to resolve cross dependency issues:
- Following moved to pinot-segment-local utils package:
- IngestionUtils
- TableConfigUtils
- SchemaUtils
- SegmentTestUtils: This depends on AvroSchemaUtil, added dependency on pinot-avro in test-scope.
- Following moved to pinot-common:
- FilterContext
- ExpressionContext
- FunctionContext
- Predicate
- Following moved to pinot-spi
- ReadMode
- CommonConstants
- ArrayCopyUtils
- BadQueryException moved
- EarlyTerminationException
- Copied necessary data files in resources/data from pinot-core to pinot-segment-local.
---
.../broker/api/resources/PinotClientRequest.java | 2 +-
.../broker/broker/BrokerAdminApiApplication.java | 2 +-
...okerResourceOnlineOfflineStateModelFactory.java | 2 +-
.../broker/broker/helix/HelixBrokerStarter.java | 8 +-
.../broker/pruner/PartitionZKMetadataPruner.java | 2 +-
.../HelixExternalViewBasedQueryQuotaManager.java | 2 +-
.../requesthandler/BaseBrokerRequestHandler.java | 6 +-
.../pinot/broker/routing/RoutingManager.java | 4 +-
.../instanceselector/BaseInstanceSelector.java | 2 +-
.../StrictReplicaGroupInstanceSelector.java | 2 +-
.../routing/segmentpruner/EmptySegmentPruner.java | 2 +-
.../segmentpruner/PartitionSegmentPruner.java | 4 +-
.../routing/segmentpruner/TimeSegmentPruner.java | 4 +-
.../segmentselector/RealtimeSegmentSelector.java | 2 +-
.../routing/timeboundary/TimeBoundaryManager.java | 2 +-
.../broker/broker/HelixBrokerStarterTest.java | 2 +-
...elixExternalViewBasedQueryQuotaManagerTest.java | 2 +-
.../requesthandler/BrokerRequestOptionsTest.java | 2 +-
.../instanceselector/InstanceSelectorTest.java | 8 +-
.../segmentpreselector/SegmentPreSelectorTest.java | 2 +-
.../routing/segmentpruner/SegmentPrunerTest.java | 2 +-
.../segmentselector/SegmentSelectorTest.java | 4 +-
.../assignment/InstanceAssignmentConfigUtils.java | 2 +-
.../pinot/common/metadata/ZKMetadataProvider.java | 3 +-
.../metadata/segment/OfflineSegmentZKMetadata.java | 4 +-
.../segment/RealtimeSegmentZKMetadata.java | 6 +-
.../common/metadata/segment/SegmentZKMetadata.java | 4 +-
.../apache/pinot/common/metrics/BrokerMetrics.java | 4 +-
.../pinot/common/metrics/ControllerMetrics.java | 2 +-
.../apache/pinot/common/metrics/ServerMetrics.java | 4 +-
.../common/request/context/ExpressionContext.java | 118 +
.../common/request/context/FilterContext.java | 116 +
.../common/request/context/FunctionContext.java | 102 +
.../request/context/OrderByExpressionContext.java | 74 +
.../context/RequestContextConvertUtils.java | 302 +
.../request/context/predicate/EqPredicate.java | 72 +
.../request/context/predicate/InPredicate.java | 79 +
.../context/predicate/IsNotNullPredicate.java | 66 +
.../request/context/predicate/IsNullPredicate.java | 66 +
.../context/predicate/JsonMatchPredicate.java | 72 +
.../request/context/predicate/NotEqPredicate.java | 72 +
.../request/context/predicate/NotInPredicate.java | 79 +
.../request/context/predicate/Predicate.java | 47 +
.../request/context/predicate/RangePredicate.java | 141 +
.../context/predicate/RegexpLikePredicate.java | 72 +
.../context/predicate/TextMatchPredicate.java | 72 +
.../org/apache/pinot/common/segment/ReadMode.java | 48 -
.../apache/pinot/common/utils/CommonConstants.java | 509 -
.../common/utils/FileUploadDownloadClient.java | 2 +-
.../org/apache/pinot/common/utils/SchemaUtils.java | 1 +
.../org/apache/pinot/common/utils/URIUtils.java | 1 +
.../pinot/common/utils/config/InstanceUtils.java | 4 +-
.../common/utils/fetcher/BaseSegmentFetcher.java | 2 +-
.../common/utils/fetcher/HttpsSegmentFetcher.java | 5 +-
.../utils/fetcher/SegmentFetcherFactory.java | 3 +-
.../common/utils/grpc/GrpcRequestBuilder.java | 2 +-
.../pinot/common/utils/helix/HelixHelper.java | 2 +-
.../common/utils/helix/LeadControllerUtils.java | 2 +-
.../common/utils/webhdfs/WebHdfsV1Client.java | 2 +-
.../apache/pinot/parsers/utils/ParserUtils.java | 2 +-
.../parsers/pql2/ast/BetweenPredicateAstNode.java | 2 +-
.../pql2/ast/ComparisonPredicateAstNode.java | 2 +-
.../common/metadata/SegmentZKMetadataTest.java | 6 +-
.../pinot/common/tier/TierSegmentSelectorTest.java | 2 +-
.../common/utils/FileUploadDownloadClientTest.java | 1 +
.../common/utils/config/TableConfigSerDeTest.java | 6 +-
.../utils/fetcher/SegmentFetcherFactoryTest.java | 9 +-
.../test/java/org/apache/pinot/util/TestUtils.java | 0
.../apache/pinot/controller/ControllerConf.java | 6 +-
.../apache/pinot/controller/ControllerStarter.java | 10 +-
.../pinot/controller/LeadControllerManager.java | 2 +-
.../api/ControllerAdminApiApplication.java | 2 +-
.../api/resources/ControllerFilePathProvider.java | 4 +-
.../resources/LLCSegmentCompletionHandlers.java | 4 +-
.../api/resources/PinotBrokerRestletResource.java | 2 +-
.../resources/PinotControllerResponseFilter.java | 2 +-
.../resources/PinotIngestionRestletResource.java | 2 +-
.../resources/PinotInstanceRestletResource.java | 2 +-
.../PinotLeadControllerRestletResource.java | 2 +-
.../api/resources/PinotQueryResource.java | 2 +-
.../api/resources/PinotSchemaRestletResource.java | 2 +-
.../PinotSegmentUploadDownloadRestletResource.java | 3 +-
.../api/resources/PinotTableIndexingConfigs.java | 2 +-
.../api/resources/PinotTableMetadataConfigs.java | 2 +-
.../api/resources/PinotTableRestletResource.java | 2 +-
.../api/resources/PinotTableSegmentConfigs.java | 2 +-
.../helix/core/PinotHelixResourceManager.java | 10 +-
.../segment/OfflineDimTableSegmentAssignment.java | 12 +-
.../segment/OfflineSegmentAssignment.java | 2 +-
.../segment/RealtimeSegmentAssignment.java | 21 +-
.../assignment/segment/SegmentAssignmentUtils.java | 2 +-
.../core/minion/MinionInstancesCleanupTask.java | 2 +-
.../core/minion/PinotHelixTaskResourceManager.java | 2 +-
.../realtime/PinotLLCRealtimeSegmentManager.java | 12 +-
.../core/realtime/PinotRealtimeSegmentManager.java | 8 +-
.../core/realtime/SegmentCompletionManager.java | 2 +-
.../segment/CommittingSegmentDescriptor.java | 2 +-
.../helix/core/rebalance/TableRebalancer.java | 2 +-
.../helix/core/retention/RetentionManager.java | 4 +-
.../helix/core/util/HelixSetupUtils.java | 4 +-
.../helix/core/util/ZKMetadataUtils.java | 12 +-
.../realtime/provisioning/MemoryEstimator.java | 20 +-
.../recommender/rules/impl/BloomFilterRule.java | 6 +-
.../NoDictionaryOnHeapDictionaryJointRule.java | 6 +-
.../rules/impl/PinotTablePartitionRule.java | 8 +-
.../utils/QueryInvertedSortedIndexRecommender.java | 10 +-
.../pinot/controller/util/FileIngestionHelper.java | 2 +-
.../controller/util/SegmentIntervalUtils.java | 2 +-
.../pinot/controller/ControllerTestUtils.java | 16 +-
.../api/PinotInstanceRestletResourceTest.java | 45 +-
.../pinot/controller/api/TableViewsTest.java | 2 +-
.../helix/ControllerInstanceToggleTest.java | 2 +-
.../controller/helix/ControllerSentinelTestV2.java | 2 +-
.../helix/ControllerTenantStatelessTest.java | 2 +-
.../pinot/controller/helix/ControllerTest.java | 11 +-
.../helix/PinotControllerModeStatelessTest.java | 4 +-
.../controller/helix/SegmentStatusCheckerTest.java | 4 +-
.../PinotHelixResourceManagerStatelessTest.java | 41 +-
.../helix/core/PinotHelixResourceManagerTest.java | 9 +-
.../instance/InstanceAssignmentTest.java | 2 +-
...fflineNonReplicaGroupSegmentAssignmentTest.java | 5 +-
...NonReplicaGroupTieredSegmentAssignmentTest.java | 2 +-
.../OfflineReplicaGroupSegmentAssignmentTest.java | 4 +-
...altimeNonReplicaGroupSegmentAssignmentTest.java | 2 +-
...NonReplicaGroupTieredSegmentAssignmentTest.java | 2 +-
.../RealtimeReplicaGroupSegmentAssignmentTest.java | 4 +-
.../segment/SegmentAssignmentUtilsTest.java | 2 +-
.../PinotLLCRealtimeSegmentManagerTest.java | 12 +-
.../helix/core/realtime/SegmentCompletionTest.java | 8 +-
.../TableRebalancerClusterStatelessTest.java | 2 +-
.../helix/core/rebalance/TableRebalancerTest.java | 8 +-
.../helix/core/retention/RetentionManagerTest.java | 2 +-
.../core/util/SegmentDeletionManagerTest.java | 16 +-
.../controller/utils/SegmentMetadataMockUtils.java | 7 +-
pinot-core/pom.xml | 11 +
.../apache/pinot/core/common/ObjectSerDeUtils.java | 8 +-
.../org/apache/pinot/core/common/Operator.java | 2 +-
.../core/common/datatable/DataTableUtils.java | 2 +-
.../core/data/aggregator/AvgValueAggregator.java | 84 -
.../core/data/aggregator/CountValueAggregator.java | 72 -
.../DistinctCountBitmapValueAggregator.java | 94 -
.../DistinctCountHLLValueAggregator.java | 105 -
.../core/data/aggregator/MaxValueAggregator.java | 72 -
.../aggregator/MinMaxRangeValueAggregator.java | 86 -
.../core/data/aggregator/MinValueAggregator.java | 72 -
.../aggregator/PercentileEstValueAggregator.java | 95 -
.../PercentileTDigestValueAggregator.java | 95 -
.../aggregator/SumPrecisionValueAggregator.java | 93 -
.../core/data/aggregator/SumValueAggregator.java | 72 -
.../core/data/aggregator/ValueAggregator.java | 79 -
.../data/aggregator/ValueAggregatorFactory.java | 104 -
.../core/data/function/FunctionEvaluator.java | 44 -
.../data/function/FunctionEvaluatorFactory.java | 104 -
.../data/function/GroovyFunctionEvaluator.java | 105 -
.../data/function/InbuiltFunctionEvaluator.java | 172 -
.../data/function/TimeSpecFunctionEvaluator.java | 79 -
.../core/data/manager/BaseTableDataManager.java | 2 +-
.../pinot/core/data/manager/TableDataManager.java | 2 +-
.../manager/config/InstanceDataManagerConfig.java | 57 -
.../manager/config/TableDataManagerConfig.java | 2 +-
.../manager/offline/DimensionTableDataManager.java | 4 +-
.../manager/offline/OfflineTableDataManager.java | 6 +-
.../manager/offline/TableDataManagerProvider.java | 2 +-
.../realtime/HLRealtimeSegmentDataManager.java | 19 +-
.../realtime/LLRealtimeSegmentDataManager.java | 26 +-
.../realtime/PeerSchemeSplitSegmentCommitter.java | 2 +-
.../realtime/RealtimeSegmentDataManager.java | 6 +-
.../manager/realtime/RealtimeTableDataManager.java | 80 +-
.../manager/realtime/SegmentCommitterFactory.java | 2 +-
.../data/partition/ByteArrayPartitionFunction.java | 62 -
.../data/partition/HashCodePartitionFunction.java | 54 -
.../data/partition/ModuloPartitionFunction.java | 73 -
.../data/partition/MurmurPartitionFunction.java | 71 -
.../data/partition/PartitionFunctionFactory.java | 89 -
.../core/data/readers/GenericRowRecordReader.java | 72 -
.../readers/IntermediateSegmentRecordReader.java | 80 -
.../readers/MultiplePinotSegmentRecordReader.java | 280 -
.../data/readers/PinotSegmentColumnReader.java | 112 -
.../data/readers/PinotSegmentRecordReader.java | 171 -
.../core/data/readers/sort/PinotSegmentSorter.java | 121 -
.../core/data/readers/sort/SegmentSorter.java | 35 -
.../recordtransformer/CompositeTransformer.java | 89 -
.../recordtransformer/DataTypeTransformer.java | 200 -
.../recordtransformer/ExpressionTransformer.java | 101 -
.../data/recordtransformer/FilterTransformer.java | 53 -
.../recordtransformer/NullValueTransformer.java | 57 -
.../data/recordtransformer/RecordTransformer.java | 39 -
.../recordtransformer/SanitizationTransformer.java | 77 -
.../apache/pinot/core/data/table/IndexedTable.java | 4 +-
.../apache/pinot/core/data/table/TableResizer.java | 6 +-
.../apache/pinot/core/geospatial/GeometryType.java | 70 -
.../pinot/core/geospatial/GeometryUtils.java | 72 -
.../core/geospatial/serde/GeometrySerializer.java | 526 -
.../function/ConstructFromTextFunction.java | 2 +-
.../function/ConstructFromWKBFunction.java | 2 +-
.../transform/function/ScalarFunctions.java | 4 +-
.../transform/function/StAreaFunction.java | 4 +-
.../transform/function/StAsBinaryFunction.java | 2 +-
.../transform/function/StAsTextFunction.java | 2 +-
.../transform/function/StContainsFunction.java | 4 +-
.../transform/function/StDistanceFunction.java | 4 +-
.../transform/function/StEqualsFunction.java | 2 +-
.../transform/function/StGeogFromTextFunction.java | 2 +-
.../transform/function/StGeogFromWKBFunction.java | 2 +-
.../transform/function/StGeomFromTextFunction.java | 2 +-
.../transform/function/StGeomFromWKBFunction.java | 2 +-
.../transform/function/StGeometryTypeFunction.java | 2 +-
.../transform/function/StPolygonFunction.java | 6 +-
.../indexsegment/immutable/EmptyIndexSegment.java | 121 -
.../immutable/ImmutableSegmentImpl.java | 180 -
.../immutable/ImmutableSegmentLoader.java | 160 -
.../indexsegment/mutable/IntermediateSegment.java | 378 -
.../indexsegment/mutable/MutableSegmentImpl.java | 1179 ---
.../io/compression/ChunkCompressorFactory.java | 74 -
.../core/io/compression/PassThroughCompressor.java | 42 -
.../io/compression/PassThroughDecompressor.java | 39 -
.../core/io/compression/SnappyCompressor.java | 37 -
.../core/io/compression/SnappyDecompressor.java | 36 -
.../core/io/reader/impl/FixedBitIntReader.java | 2591 -----
.../impl/FixedByteSingleValueMultiColReader.java | 223 -
.../readerwriter/PinotDataBufferMemoryManager.java | 46 -
.../RealtimeIndexOffHeapMemoryManager.java | 121 -
...ixedByteSingleValueMultiColumnReaderWriter.java | 194 -
.../core/io/util/FixedBitIntReaderWriter.java | 57 -
.../core/io/util/FixedBitIntReaderWriterV2.java | 149 -
.../core/io/util/FixedByteValueReaderWriter.java | 116 -
.../apache/pinot/core/io/util/PinotDataBitSet.java | 248 -
.../pinot/core/io/util/PinotDataBitSetV2.java | 729 --
.../org/apache/pinot/core/io/util/ValueReader.java | 51 -
.../pinot/core/io/util/VarLengthValueReader.java | 121 -
.../pinot/core/io/util/VarLengthValueWriter.java | 133 -
.../writer/impl/BaseChunkSVForwardIndexWriter.java | 191 -
.../core/io/writer/impl/DirectMemoryManager.java | 60 -
.../writer/impl/FixedBitMVForwardIndexWriter.java | 158 -
.../writer/impl/FixedBitSVForwardIndexWriter.java | 58 -
.../impl/FixedByteChunkSVForwardIndexWriter.java | 113 -
.../impl/FixedByteSingleValueMultiColWriter.java | 115 -
.../core/io/writer/impl/MmapMemoryManager.java | 179 -
.../writer/impl/MutableOffHeapByteArrayStore.java | 267 -
.../impl/VarByteChunkSVForwardIndexWriter.java | 140 -
.../core/metadata/DefaultMetadataExtractor.java | 2 +-
.../pinot/core/minion/RawIndexConverter.java | 24 +-
.../apache/pinot/core/minion/SegmentConverter.java | 4 +-
.../apache/pinot/core/minion/SegmentPurger.java | 6 +-
.../minion/rollup/MergeRollupSegmentConverter.java | 2 +-
.../core/minion/segment/MapperRecordReader.java | 2 +-
.../core/minion/segment/ReducerRecordReader.java | 2 +-
.../apache/pinot/core/operator/BaseOperator.java | 2 +-
.../pinot/core/operator/blocks/TransformBlock.java | 2 +-
.../core/operator/combine/BaseCombineOperator.java | 2 +-
.../operator/combine/GroupByCombineOperator.java | 2 +-
.../combine/GroupByOrderByCombineOperator.java | 2 +-
...xValueBasedSelectionOrderByCombineOperator.java | 6 +-
.../combine/SelectionOrderByCombineOperator.java | 4 +-
.../operator/filter/ExpressionFilterOperator.java | 4 +-
.../core/operator/filter/FilterOperatorUtils.java | 2 +-
.../operator/filter/H3IndexFilterOperator.java | 10 +-
.../filter/RangeIndexBasedFilterOperator.java | 2 +-
.../predicate/EqualsPredicateEvaluatorFactory.java | 4 +-
.../FSTBasedRegexpPredicateEvaluatorFactory.java | 6 +-
.../predicate/InPredicateEvaluatorFactory.java | 4 +-
.../NotEqualsPredicateEvaluatorFactory.java | 4 +-
.../predicate/NotInPredicateEvaluatorFactory.java | 4 +-
.../filter/predicate/PredicateEvaluator.java | 2 +-
.../predicate/PredicateEvaluatorProvider.java | 16 +-
.../predicate/RangePredicateEvaluatorFactory.java | 4 +-
.../RegexpLikePredicateEvaluatorFactory.java | 4 +-
.../operator/query/AggregationGroupByOperator.java | 2 +-
.../query/AggregationGroupByOrderByOperator.java | 2 +-
.../query/DictionaryBasedAggregationOperator.java | 4 +-
.../operator/query/EmptySelectionOperator.java | 2 +-
.../core/operator/query/SelectionOnlyOperator.java | 2 +-
.../operator/query/SelectionOrderByOperator.java | 6 +-
.../operator/streaming/StreamingResponseUtils.java | 2 +-
.../StreamingSelectionOnlyCombineOperator.java | 2 +-
.../streaming/StreamingSelectionOnlyOperator.java | 2 +-
.../core/operator/transform/TransformOperator.java | 2 +-
.../transform/function/BaseTransformFunction.java | 2 +-
.../transform/function/CaseTransformFunction.java | 2 +-
.../transform/function/CastTransformFunction.java | 2 +-
.../function/EqualsTransformFunction.java | 1 -
.../function/GroovyTransformFunction.java | 2 +-
.../function/LogicalOperatorTransformFunction.java | 2 +-
.../function/TransformFunctionFactory.java | 6 +-
.../plan/AggregationGroupByOrderByPlanNode.java | 2 +-
.../core/plan/AggregationGroupByPlanNode.java | 2 +-
.../pinot/core/plan/AggregationPlanNode.java | 2 +-
.../apache/pinot/core/plan/CombinePlanNode.java | 2 +-
.../plan/DictionaryBasedAggregationPlanNode.java | 2 +-
.../org/apache/pinot/core/plan/FilterPlanNode.java | 18 +-
.../plan/MetadataBasedAggregationPlanNode.java | 2 +-
.../apache/pinot/core/plan/SelectionPlanNode.java | 6 +-
.../core/plan/StreamingSelectionPlanNode.java | 2 +-
.../apache/pinot/core/plan/TransformPlanNode.java | 2 +-
.../core/plan/maker/InstancePlanMakerImplV2.java | 4 +-
.../aggregation/function/AggregationFunction.java | 2 +-
.../function/AggregationFunctionFactory.java | 6 +-
.../function/AggregationFunctionUtils.java | 2 +-
.../function/AvgAggregationFunction.java | 4 +-
.../function/AvgMVAggregationFunction.java | 2 +-
.../BaseSingleInputAggregationFunction.java | 2 +-
.../function/CountAggregationFunction.java | 2 +-
.../function/CountMVAggregationFunction.java | 2 +-
.../function/DistinctAggregationFunction.java | 4 +-
.../function/DistinctCountAggregationFunction.java | 2 +-
.../DistinctCountBitmapAggregationFunction.java | 2 +-
.../DistinctCountBitmapMVAggregationFunction.java | 2 +-
.../DistinctCountHLLAggregationFunction.java | 4 +-
.../DistinctCountHLLMVAggregationFunction.java | 2 +-
.../DistinctCountMVAggregationFunction.java | 2 +-
.../DistinctCountRawHLLAggregationFunction.java | 4 +-
.../DistinctCountRawHLLMVAggregationFunction.java | 2 +-
...inctCountRawThetaSketchAggregationFunction.java | 2 +-
...istinctCountThetaSketchAggregationFunction.java | 14 +-
.../function/FastHLLAggregationFunction.java | 2 +-
.../function/IdSetAggregationFunction.java | 2 +-
.../function/MaxAggregationFunction.java | 2 +-
.../function/MaxMVAggregationFunction.java | 2 +-
.../function/MinAggregationFunction.java | 2 +-
.../function/MinMVAggregationFunction.java | 2 +-
.../function/MinMaxRangeAggregationFunction.java | 4 +-
.../function/MinMaxRangeMVAggregationFunction.java | 2 +-
.../function/PercentileAggregationFunction.java | 2 +-
.../function/PercentileEstAggregationFunction.java | 4 +-
.../PercentileEstMVAggregationFunction.java | 4 +-
.../function/PercentileMVAggregationFunction.java | 2 +-
.../PercentileTDigestAggregationFunction.java | 2 +-
.../PercentileTDigestMVAggregationFunction.java | 2 +-
...artitionedDistinctCountAggregationFunction.java | 2 +-
.../function/StUnionAggregationFunction.java | 6 +-
.../function/SumAggregationFunction.java | 2 +-
.../function/SumMVAggregationFunction.java | 2 +-
.../function/SumPrecisionAggregationFunction.java | 2 +-
.../aggregation/function/customobject/AvgPair.java | 94 -
.../function/customobject/MinMaxRangePair.java | 90 -
.../function/customobject/QuantileDigest.java | 1071 --
.../function/customobject/SerializedHLL.java | 42 -
.../groupby/AggregationGroupByTrimmingService.java | 2 +-
.../groupby/DefaultGroupByExecutor.java | 2 +-
.../groupby/DictionaryBasedGroupKeyGenerator.java | 2 +-
.../NoDictionaryMultiColumnGroupKeyGenerator.java | 4 +-
.../NoDictionarySingleColumnGroupKeyGenerator.java | 2 +-
.../query/distinct/DistinctExecutorFactory.java | 4 +-
.../pinot/core/query/distinct/DistinctTable.java | 2 +-
...DictionaryBasedMultiColumnDistinctExecutor.java | 2 +-
...ictionaryBasedSingleColumnDistinctExecutor.java | 2 +-
...ionaryBasedMultiColumnDistinctOnlyExecutor.java | 2 +-
...aryBasedMultiColumnDistinctOrderByExecutor.java | 4 +-
...onaryBasedSingleColumnDistinctOnlyExecutor.java | 2 +-
...ryBasedSingleColumnDistinctOrderByExecutor.java | 4 +-
.../BaseRawBytesSingleColumnDistinctExecutor.java | 2 +-
.../BaseRawDoubleSingleColumnDistinctExecutor.java | 2 +-
.../BaseRawFloatSingleColumnDistinctExecutor.java | 2 +-
.../BaseRawIntSingleColumnDistinctExecutor.java | 2 +-
.../BaseRawLongSingleColumnDistinctExecutor.java | 2 +-
.../BaseRawStringSingleColumnDistinctExecutor.java | 2 +-
.../RawBytesSingleColumnDistinctOnlyExecutor.java | 2 +-
...awBytesSingleColumnDistinctOrderByExecutor.java | 4 +-
.../RawDoubleSingleColumnDistinctOnlyExecutor.java | 2 +-
...wDoubleSingleColumnDistinctOrderByExecutor.java | 4 +-
.../RawFloatSingleColumnDistinctOnlyExecutor.java | 2 +-
...awFloatSingleColumnDistinctOrderByExecutor.java | 4 +-
.../RawIntSingleColumnDistinctOnlyExecutor.java | 2 +-
.../RawIntSingleColumnDistinctOrderByExecutor.java | 4 +-
.../RawLongSingleColumnDistinctOnlyExecutor.java | 2 +-
...RawLongSingleColumnDistinctOrderByExecutor.java | 4 +-
.../raw/RawMultiColumnDistinctExecutor.java | 4 +-
.../RawStringSingleColumnDistinctOnlyExecutor.java | 2 +-
...wStringSingleColumnDistinctOrderByExecutor.java | 4 +-
.../query/exception/BadQueryRequestException.java | 33 -
.../query/exception/EarlyTerminationException.java | 32 -
.../query/executor/ServerQueryExecutorV1Impl.java | 10 +-
.../filter/MergeRangeFilterOptimizer.java | 2 +-
.../query/pruner/ColumnValueSegmentPruner.java | 12 +-
.../query/pruner/SelectionQuerySegmentPruner.java | 6 +-
.../core/query/reduce/BrokerReduceService.java | 4 +-
.../core/query/reduce/GroupByDataTableReducer.java | 4 +-
.../core/query/reduce/HavingFilterHandler.java | 4 +-
.../core/query/reduce/PostAggregationHandler.java | 4 +-
.../core/query/request/ServerQueryRequest.java | 2 +-
.../query/request/context/ExpressionContext.java | 118 -
.../core/query/request/context/FilterContext.java | 116 -
.../query/request/context/FunctionContext.java | 102 -
.../request/context/OrderByExpressionContext.java | 74 -
.../core/query/request/context/QueryContext.java | 4 +
.../request/context/predicate/EqPredicate.java | 72 -
.../request/context/predicate/InPredicate.java | 79 -
.../context/predicate/IsNotNullPredicate.java | 66 -
.../request/context/predicate/IsNullPredicate.java | 66 -
.../context/predicate/JsonMatchPredicate.java | 72 -
.../request/context/predicate/NotEqPredicate.java | 72 -
.../request/context/predicate/NotInPredicate.java | 79 -
.../query/request/context/predicate/Predicate.java | 47 -
.../request/context/predicate/RangePredicate.java | 141 -
.../context/predicate/RegexpLikePredicate.java | 72 -
.../context/predicate/TextMatchPredicate.java | 72 -
.../BrokerRequestToQueryContextConverter.java | 33 +-
.../context/utils/QueryContextConverterUtils.java | 272 -
.../query/selection/SelectionOperatorService.java | 2 +-
.../query/selection/SelectionOperatorUtils.java | 10 +-
.../org/apache/pinot/core/query/utils/Pair.java | 55 -
.../converter/RealtimeSegmentConverter.java | 149 -
.../converter/RealtimeSegmentRecordReader.java | 82 -
.../converter/stats/MutableColumnStatistics.java | 189 -
.../stats/MutableNoDictionaryColStatistics.java | 109 -
.../RealtimeSegmentSegmentCreationDataSource.java | 59 -
.../stats/RealtimeSegmentStatsContainer.java | 62 -
.../core/realtime/impl/RealtimeSegmentConfig.java | 385 -
.../realtime/impl/RealtimeSegmentStatsHistory.java | 393 -
.../impl/ThreadSafeMutableRoaringBitmap.java | 54 -
.../dictionary/BaseOffHeapMutableDictionary.java | 444 -
.../dictionary/BaseOnHeapMutableDictionary.java | 107 -
.../dictionary/BytesOffHeapMutableDictionary.java | 226 -
.../dictionary/BytesOnHeapMutableDictionary.java | 184 -
.../dictionary/DoubleOffHeapMutableDictionary.java | 245 -
.../dictionary/DoubleOnHeapMutableDictionary.java | 202 -
.../dictionary/FloatOffHeapMutableDictionary.java | 245 -
.../dictionary/FloatOnHeapMutableDictionary.java | 202 -
.../dictionary/IntOffHeapMutableDictionary.java | 245 -
.../dictionary/IntOnHeapMutableDictionary.java | 202 -
.../dictionary/LongOffHeapMutableDictionary.java | 246 -
.../dictionary/LongOnHeapMutableDictionary.java | 202 -
.../impl/dictionary/MutableDictionaryFactory.java | 73 -
.../impl/dictionary/OffHeapMutableBytesStore.java | 259 -
.../dictionary/StringOffHeapMutableDictionary.java | 213 -
.../dictionary/StringOnHeapMutableDictionary.java | 170 -
.../forward/FixedByteMVMutableForwardIndex.java | 350 -
.../forward/FixedByteSVMutableForwardIndex.java | 302 -
.../impl/forward/VarByteSVMutableForwardIndex.java | 99 -
.../realtime/impl/geospatial/MutableH3Index.java | 78 -
.../invertedindex/RealtimeInvertedIndexReader.java | 85 -
.../RealtimeLuceneDocIdCollector.java | 64 -
.../RealtimeLuceneIndexReaderRefreshThread.java | 151 -
.../RealtimeLuceneIndexRefreshState.java | 139 -
.../RealtimeLuceneTextIndexReader.java | 160 -
.../core/realtime/impl/json/MutableJsonIndex.java | 279 -
.../impl/nullvalue/MutableNullValueVector.java | 48 -
.../requesthandler/PinotQueryParserFactory.java | 4 +-
.../core/requesthandler/RangeMergeOptimizer.java | 4 +-
...termediateSegmentSegmentCreationDataSource.java | 57 -
.../creator/IntermediateSegmentStatsContainer.java | 54 -
.../RecordReaderSegmentCreationDataSource.java | 100 -
.../pinot/core/segment/creator/TextIndexType.java | 25 -
.../creator/impl/SegmentColumnarIndexCreator.java | 766 --
.../creator/impl/SegmentCreationDriverFactory.java | 35 -
.../creator/impl/SegmentDictionaryCreator.java | 300 -
.../impl/SegmentIndexCreationDriverImpl.java | 434 -
.../core/segment/creator/impl/V1Constants.java | 106 -
.../impl/bloom/OnHeapGuavaBloomFilterCreator.java | 83 -
.../fwd/MultiValueUnsortedForwardIndexCreator.java | 69 -
.../fwd/SingleValueFixedByteRawIndexCreator.java | 118 -
.../fwd/SingleValueSortedForwardIndexCreator.java | 89 -
.../SingleValueUnsortedForwardIndexCreator.java | 67 -
.../fwd/SingleValueVarByteRawIndexCreator.java | 118 -
.../impl/inv/BitmapInvertedIndexWriter.java | 90 -
.../inv/OffHeapBitmapInvertedIndexCreator.java | 250 -
.../impl/inv/OnHeapBitmapInvertedIndexCreator.java | 74 -
.../creator/impl/inv/RangeIndexCreator.java | 574 --
.../impl/inv/geospatial/BaseH3IndexCreator.java | 169 -
.../impl/inv/geospatial/OffHeapH3IndexCreator.java | 219 -
.../impl/inv/geospatial/OnHeapH3IndexCreator.java | 50 -
.../impl/inv/json/BaseJsonIndexCreator.java | 173 -
.../impl/inv/json/OffHeapJsonIndexCreator.java | 283 -
.../impl/inv/json/OnHeapJsonIndexCreator.java | 61 -
.../impl/inv/text/LuceneFSTIndexCreator.java | 102 -
.../impl/nullvalue/NullValueVectorCreator.java | 65 -
.../stats/AbstractColumnStatisticsCollector.java | 163 -
.../stats/BytesColumnPredIndexStatsCollector.java | 113 -
.../stats/DoubleColumnPreIndexStatsCollector.java | 101 -
.../stats/FloatColumnPreIndexStatsCollector.java | 101 -
.../stats/IntColumnPreIndexStatsCollector.java | 101 -
.../stats/LongColumnPreIndexStatsCollector.java | 101 -
.../stats/SegmentPreIndexStatsCollectorImpl.java | 143 -
.../stats/StringColumnPreIndexStatsCollector.java | 120 -
.../creator/impl/text/LuceneTextIndexCreator.java | 161 -
.../index/column/BaseVirtualColumnProvider.java | 47 -
.../DefaultNullValueVirtualColumnProvider.java | 87 -
.../index/column/IntermediateIndexContainer.java | 134 -
.../core/segment/index/column/NumValuesInfo.java | 41 -
.../index/column/PhysicalColumnIndexContainer.java | 327 -
.../converter/SegmentFormatConverterFactory.java | 37 -
.../converter/SegmentV1V2ToV3FormatConverter.java | 288 -
.../segment/index/datasource/BaseDataSource.java | 128 -
.../segment/index/datasource/EmptyDataSource.java | 94 -
.../index/datasource/ImmutableDataSource.java | 119 -
.../index/datasource/MutableDataSource.java | 138 -
.../segment/index/loader/IndexLoadingConfig.java | 445 -
.../core/segment/index/loader/LoaderUtils.java | 186 -
.../segment/index/loader/SegmentPreProcessor.java | 205 -
.../index/loader/V3RemoveIndexException.java | 29 -
.../index/loader/V3UpdateIndexException.java | 29 -
.../loader/bloomfilter/BloomFilterHandler.java | 161 -
.../ColumnMinMaxValueGenerator.java | 155 -
.../ColumnMinMaxValueGeneratorMode.java | 38 -
.../defaultcolumn/BaseDefaultColumnHandler.java | 680 --
.../loader/defaultcolumn/DefaultColumnHandler.java | 31 -
.../defaultcolumn/DefaultColumnHandlerFactory.java | 41 -
.../defaultcolumn/DefaultColumnStatistics.java | 113 -
.../defaultcolumn/V1DefaultColumnHandler.java | 55 -
.../defaultcolumn/V3DefaultColumnHandler.java | 85 -
.../index/loader/invertedindex/H3IndexHandler.java | 161 -
.../loader/invertedindex/InvertedIndexHandler.java | 136 -
.../loader/invertedindex/JsonIndexHandler.java | 158 -
.../invertedindex/LuceneFSTIndexHandler.java | 146 -
.../loader/invertedindex/RangeIndexHandler.java | 215 -
.../loader/invertedindex/TextIndexHandler.java | 180 -
.../segment/index/metadata/ColumnMetadata.java | 593 --
.../index/metadata/SegmentMetadataImpl.java | 575 --
.../index/readers/BaseImmutableDictionary.java | 298 -
.../index/readers/BitmapInvertedIndexReader.java | 94 -
.../segment/index/readers/BytesDictionary.java | 95 -
.../readers/ConstantValueBytesDictionary.java | 108 -
.../readers/ConstantValueDoubleDictionary.java | 96 -
.../readers/ConstantValueFloatDictionary.java | 96 -
.../index/readers/ConstantValueIntDictionary.java | 96 -
.../index/readers/ConstantValueLongDictionary.java | 96 -
.../readers/ConstantValueStringDictionary.java | 102 -
.../segment/index/readers/DocIdDictionary.java | 81 -
.../segment/index/readers/DoubleDictionary.java | 70 -
.../segment/index/readers/FloatDictionary.java | 70 -
.../core/segment/index/readers/IntDictionary.java | 70 -
.../core/segment/index/readers/LongDictionary.java | 70 -
.../index/readers/LuceneFSTIndexReader.java | 82 -
.../index/readers/NullValueVectorReaderImpl.java | 42 -
.../segment/index/readers/OnHeapDictionary.java | 32 -
.../index/readers/OnHeapDoubleDictionary.java | 107 -
.../index/readers/OnHeapFloatDictionary.java | 107 -
.../segment/index/readers/OnHeapIntDictionary.java | 107 -
.../index/readers/OnHeapLongDictionary.java | 107 -
.../index/readers/OnHeapStringDictionary.java | 130 -
.../segment/index/readers/RangeIndexReader.java | 218 -
.../segment/index/readers/StringDictionary.java | 124 -
.../index/readers/ValidDocIndexReaderImpl.java | 37 -
.../readers/bloom/BaseGuavaBloomFilterReader.java | 66 -
.../readers/bloom/BloomFilterReaderFactory.java | 45 -
.../readers/bloom/GuavaBloomFilterReaderUtils.java | 65 -
.../bloom/OffHeapGuavaBloomFilterReader.java | 50 -
.../bloom/OnHeapGuavaBloomFilterReader.java | 54 -
.../constant/ConstantMVForwardIndexReader.java | 55 -
.../constant/ConstantMVInvertedIndexReader.java | 44 -
.../constant/ConstantSortedIndexReader.java | 56 -
.../forward/BaseChunkSVForwardIndexReader.java | 216 -
.../forward/FixedBitMVForwardIndexReader.java | 153 -
.../forward/FixedBitSVForwardIndexReader.java | 70 -
.../forward/FixedBitSVForwardIndexReaderV2.java | 101 -
.../FixedByteChunkSVForwardIndexReader.java | 94 -
.../forward/VarByteChunkSVForwardIndexReader.java | 190 -
.../readers/geospatial/ImmutableH3IndexReader.java | 84 -
.../readers/json/ImmutableJsonIndexReader.java | 253 -
.../readers/sorted/SortedIndexReaderImpl.java | 134 -
.../index/readers/text/LuceneDocIdCollector.java | 76 -
.../index/readers/text/LuceneTextIndexReader.java | 218 -
.../core/segment/memory/BasePinotLBuffer.java | 161 -
.../pinot/core/segment/memory/PinotByteBuffer.java | 336 -
.../pinot/core/segment/memory/PinotDataBuffer.java | 393 -
.../segment/memory/PinotNativeOrderLBuffer.java | 183 -
.../segment/memory/PinotNonNativeOrderLBuffer.java | 183 -
.../filter/FunctionEvaluatorRecordFilter.java | 4 +-
.../processing/framework/SegmentMapper.java | 2 +-
.../framework/SegmentProcessorConfig.java | 4 -
.../framework/SegmentProcessorFramework.java | 2 +-
.../partitioner/TableConfigPartitioner.java | 2 +-
.../partitioner/TransformFunctionPartitioner.java | 4 +-
.../TransformFunctionRecordTransformer.java | 4 +-
.../core/segment/store/ColumnIndexDirectory.java | 114 -
.../pinot/core/segment/store/ColumnIndexType.java | 51 -
.../core/segment/store/FilePerIndexDirectory.java | 174 -
.../pinot/core/segment/store/IndexEntry.java | 44 -
.../apache/pinot/core/segment/store/IndexKey.java | 71 -
.../pinot/core/segment/store/SegmentDirectory.java | 218 -
.../core/segment/store/SegmentDirectoryPaths.java | 127 -
.../segment/store/SegmentLocalFSDirectory.java | 406 -
.../segment/store/SingleFileIndexDirectory.java | 353 -
.../virtualcolumn/DocIdVirtualColumnProvider.java | 84 -
.../virtualcolumn/VirtualColumnContext.java | 44 -
.../virtualcolumn/VirtualColumnIndexContainer.java | 109 -
.../virtualcolumn/VirtualColumnProvider.java | 42 -
.../VirtualColumnProviderFactory.java | 65 -
.../core/segment/virtualcolumn/package-info.java | 22 -
.../pinot/core/startree/OffHeapStarTree.java | 139 -
.../pinot/core/startree/OffHeapStarTreeNode.java | 171 -
.../pinot/core/startree/StarTreeBuilderUtils.java | 219 -
.../apache/pinot/core/startree/StarTreeUtils.java | 67 +-
.../startree/executor/StarTreeGroupByExecutor.java | 2 +-
.../startree/plan/StarTreeTransformPlanNode.java | 2 +-
.../startree/v2/builder/BaseSingleTreeBuilder.java | 541 -
.../startree/v2/builder/MultipleTreesBuilder.java | 175 -
.../v2/builder/OffHeapSingleTreeBuilder.java | 351 -
.../v2/builder/OnHeapSingleTreeBuilder.java | 160 -
.../startree/v2/builder/SingleTreeBuilder.java | 34 -
.../startree/v2/builder/StarTreeIndexCombiner.java | 96 -
.../v2/builder/StarTreeV2BuilderConfig.java | 204 -
.../core/startree/v2/store/StarTreeDataSource.java | 98 -
.../startree/v2/store/StarTreeIndexContainer.java | 72 -
.../startree/v2/store/StarTreeIndexMapUtils.java | 204 -
.../startree/v2/store/StarTreeLoaderUtils.java | 139 -
.../pinot/core/transport/ServerInstance.java | 3 +-
.../upsert/PartitionUpsertMetadataManager.java | 201 -
.../apache/pinot/core/upsert/RecordLocation.java | 55 -
.../core/upsert/TableUpsertMetadataManager.java | 1 +
.../org/apache/pinot/core/util/ArrayCopyUtils.java | 169 -
.../org/apache/pinot/core/util/CleanerUtil.java | 199 -
.../java/org/apache/pinot/core/util/CrcUtils.java | 123 -
.../org/apache/pinot/core/util/FixedIntArray.java | 68 -
.../pinot/core/util/FixedIntArrayOffHeapIdMap.java | 181 -
.../java/org/apache/pinot/core/util/H3Utils.java | 38 -
.../java/org/apache/pinot/core/util/IdMap.java | 74 -
.../org/apache/pinot/core/util/IngestionUtils.java | 366 -
.../apache/pinot/core/util/ListenerConfigUtil.java | 2 +-
.../pinot/core/util/PeerServerSegmentFinder.java | 3 +-
.../org/apache/pinot/core/util/QueryOptions.java | 2 +-
.../org/apache/pinot/core/util/SchemaUtils.java | 162 -
.../apache/pinot/core/util/SegmentPushUtils.java | 347 -
.../apache/pinot/core/util/TableConfigUtils.java | 583 --
.../java/org/apache/pinot/core/util/TlsUtils.java | 2 +-
.../org/apache/pinot/core/util/fst/FSTBuilder.java | 65 -
.../pinot/core/util/fst/PinotBufferIndexInput.java | 89 -
.../apache/pinot/core/util/fst/RegexpMatcher.java | 164 -
.../server/realtime/ControllerLeaderLocator.java | 2 +-
.../ServerSegmentCompletionProtocolHandler.java | 12 +-
.../apache/pinot/core/common/DataFetcherTest.java | 8 +-
.../pinot/core/common/ObjectSerDeUtilsTest.java | 6 +-
.../data/function/ArithmeticFunctionsTest.java | 1 +
.../core/data/function/ArrayFunctionsTest.java | 1 +
.../core/data/function/DateTimeFunctionsTest.java | 1 +
.../data/function/GroovyFunctionEvaluatorTest.java | 1 +
.../function/InbuiltFunctionEvaluatorTest.java | 1 +
.../core/data/function/JsonFunctionsTest.java | 1 +
.../offline/DimensionTableDataManagerTest.java | 14 +-
.../realtime/LLRealtimeSegmentDataManagerTest.java | 59 +-
.../core/data/partition/PartitionFunctionTest.java | 245 -
.../MultiplePinotSegmentRecordReaderTest.java | 167 -
.../data/readers/PinotSegmentRecordReaderTest.java | 145 -
.../pinot/core/data/readers/PinotSegmentUtil.java | 169 -
.../data/readers/RecordReaderSampleDataTest.java | 189 -
.../recordtransformer/DataTypeTransformerTest.java | 202 -
.../ExpressionTransformerTest.java | 261 -
.../ExpressionTransformerTimeTest.java | 177 -
.../recordtransformer/RecordTransformerTest.java | 254 -
.../pinot/core/data/table/TableResizerTest.java | 2 +-
.../core/geospatial/serde/GeometrySerdeTest.java | 3 +-
.../core/geospatial/transform/GeoFunctionTest.java | 14 +-
.../geospatial/transform/StPointFunctionTest.java | 16 +-
.../core/indexsegment/IntermediateSegmentTest.java | 12 +-
.../MutableSegmentImplAggregateMetricsTest.java | 1 +
.../MutableSegmentImplNullValueVectorTest.java | 3 +-
.../mutable/MutableSegmentImplTest.java | 13 +-
.../mutable/MutableSegmentImplTestUtils.java | 9 +-
.../mutable/MutableSegmentImplUpsertTest.java | 5 +-
.../core/io/reader/impl/FixedBitIntReaderTest.java | 83 -
.../pinot/core/io/util/PinotDataBitSetTest.java | 167 -
.../pinot/core/io/util/PinotDataBitSetV2Test.java | 438 -
.../io/util/VarLengthValueReaderWriterTest.java | 117 -
.../impl/MmapMemoryManagerFileCleanupTest.java | 66 -
.../core/io/writer/impl/MmapMemoryManagerTest.java | 185 -
.../impl/MutableOffHeapByteArrayStoreTest.java | 123 -
.../minion/MergeRollupSegmentConverterTest.java | 6 +-
.../pinot/core/minion/SegmentConverterTest.java | 6 +-
.../pinot/core/minion/SegmentPurgerTest.java | 14 +-
.../operator/combine/CombineSlowOperatorsTest.java | 2 +-
.../combine/SelectionCombineOperatorTest.java | 10 +-
.../NoDictionaryEqualsPredicateEvaluatorsTest.java | 6 +-
.../NoDictionaryInPredicateEvaluatorTest.java | 6 +-
.../NoDictionaryRangePredicateEvaluatorTest.java | 4 +-
...ngeOfflineDictionaryPredicateEvaluatorTest.java | 4 +-
.../function/AdditionTransformFunctionTest.java | 12 +-
.../function/ArrayBaseTransformFunctionTest.java | 10 +-
.../function/BaseTransformFunctionTest.java | 8 +-
.../BinaryOperatorTransformFunctionTest.java | 18 +-
.../function/CaseTransformFunctionTest.java | 14 +-
.../function/CastTransformFunctionTest.java | 14 +-
.../DateTimeConversionTransformFunctionTest.java | 10 +-
.../function/DateTruncTransformFunctionTest.java | 19 +-
.../function/DivisionTransformFunctionTest.java | 20 +-
.../function/GroovyTransformFunctionTest.java | 49 +-
.../JsonExtractScalarTransformFunctionTest.java | 66 +-
.../LogicalOperatorTransformFunctionTest.java | 14 +-
.../function/LookupTransformFunctionTest.java | 58 +-
.../function/ModuloTransformFunctionTest.java | 20 +-
.../MultiplicationTransformFunctionTest.java | 12 +-
.../ScalarTransformFunctionWrapperTest.java | 106 +-
.../SingleParamMathTransformFunctionTest.java | 64 +-
.../function/SubtractionTransformFunctionTest.java | 20 +-
.../TimeConversionTransformFunctionTest.java | 10 +-
.../function/ValueInTransformFunctionTest.java | 10 +-
.../pinot/core/plan/CombinePlanNodeTest.java | 2 +-
...adataAndDictionaryAggregationPlanMakerTest.java | 12 +-
.../DefaultAggregationExecutorTest.java | 10 +-
.../function/AggregationFunctionFactoryTest.java | 5 +-
.../DictionaryBasedGroupKeyGeneratorTest.java | 10 +-
.../groupby/NoDictionaryGroupKeyGeneratorTest.java | 10 +-
.../core/query/executor/QueryExecutorTest.java | 8 +-
.../core/query/optimizer/QueryOptimizerTest.java | 2 +-
.../PostAggregationFunctionTest.java | 4 +-
.../query/pruner/ColumnValueSegmentPrunerTest.java | 2 +-
.../request/context/predicate/PredicateTest.java | 14 +-
.../BrokerRequestToQueryContextConverterTest.java | 16 +-
.../resources/BoundedAccountingExecutorTest.java | 11 +-
.../selection/SelectionOperatorServiceTest.java | 9 +-
.../impl/RealtimeSegmentStatsHistoryTest.java | 319 -
.../impl/dictionary/MultiValueDictionaryTest.java | 84 -
.../impl/dictionary/MutableDictionaryTest.java | 366 -
.../dictionary/OffHeapMutableBytesStoreTest.java | 142 -
.../impl/fakestream/FakeStreamConfigUtils.java | 48 +-
.../impl/fakestream/FakeStreamConsumerFactory.java | 2 +-
.../RealtimeInvertedIndexReaderTest.java | 106 -
.../impl/nullvalue/MutableNullValueVectorTest.java | 45 -
.../impl/SegmentColumnarIndexCreatorTest.java | 129 -
.../impl/nullvalue/NullValueVectorCreatorTest.java | 68 -
.../core/segment/index/ColumnMetadataTest.java | 199 -
.../pinot/core/segment/index/H3IndexTest.java | 110 -
.../pinot/core/segment/index/JsonIndexTest.java | 208 -
.../segment/index/SegmentMetadataImplTest.java | 110 -
.../DefaultNullValueVirtualColumnProviderTest.java | 177 -
.../SegmentV1V2ToV3FormatConverterTest.java | 128 -
.../index/creator/BloomFilterCreatorTest.java | 81 -
.../index/creator/LuceneFSTIndexCreatorTest.java | 74 -
.../index/creator/RangeIndexCreatorTest.java | 383 -
.../segment/index/creator/RawIndexCreatorTest.java | 287 -
.../SegmentGenerationWithBytesTypeTest.java | 341 -
.../SegmentGenerationWithFilterRecordsTest.java | 117 -
.../SegmentGenerationWithMultipleRecordsTest.java | 113 -
.../SegmentGenerationWithNoRecordsTest.java | 117 -
.../SegmentGenerationWithNullValueVectorTest.java | 292 -
.../SegmentGenerationWithTimeColumnTest.java | 263 -
.../index/creator/SegmentPartitionTest.java | 231 -
.../core/segment/index/loader/LoaderTest.java | 627 --
.../core/segment/index/loader/LoaderUtilsTest.java | 98 -
.../index/loader/SegmentPreProcessorTest.java | 773 --
.../BaseDefaultColumnHandlerTest.java | 239 -
.../index/readers/ImmutableDictionaryTest.java | 360 -
.../ImmutableDictionaryTypeConversionTest.java | 385 -
.../readers/NullValueVectorReaderImplTest.java | 68 -
.../FixedBitSVForwardIndexReaderV2Test.java | 111 -
.../core/segment/memory/PinotDataBufferTest.java | 641 --
.../processing/framework/SegmentMapperTest.java | 4 +-
.../framework/SegmentProcessingFrameworkTest.java | 8 +-
.../store/ColumnIndexDirectoryTestHelper.java | 124 -
.../segment/store/FilePerIndexDirectoryTest.java | 157 -
.../segment/store/SegmentDirectoryPathsTest.java | 69 -
.../segment/store/SegmentLocalFSDirectoryTest.java | 152 -
.../store/SingleFileIndexDirectoryTest.java | 168 -
.../pinot/core/startree/v2/AvgStarTreeV2Test.java | 6 +-
.../pinot/core/startree/v2/BaseStarTreeV2Test.java | 35 +-
.../core/startree/v2/CountStarTreeV2Test.java | 4 +-
.../v2/DistinctCountBitmapStarTreeV2Test.java | 4 +-
.../v2/DistinctCountHLLStarTreeV2Test.java | 4 +-
.../pinot/core/startree/v2/MaxStarTreeV2Test.java | 4 +-
.../startree/v2/MinMaxRangeStarTreeV2Test.java | 6 +-
.../pinot/core/startree/v2/MinStarTreeV2Test.java | 4 +-
.../startree/v2/PercentileEstStarTreeV2Test.java | 6 +-
.../v2/PercentileTDigestStarTreeV2Test.java | 4 +-
.../v2/PreAggregatedAvgStarTreeV2Test.java | 8 +-
...reAggregatedDistinctCountHLLStarTreeV2Test.java | 6 +-
.../v2/PreAggregatedMinMaxRangeStarTreeV2Test.java | 8 +-
.../PreAggregatedPercentileEstStarTreeV2Test.java | 6 +-
...eAggregatedPercentileTDigestStarTreeV2Test.java | 4 +-
.../startree/v2/SumPrecisionStarTreeV2Test.java | 4 +-
.../pinot/core/startree/v2/SumStarTreeV2Test.java | 4 +-
.../v2/builder/StarTreeV2BuilderConfigTest.java | 99 -
.../upsert/PartitionUpsertMetadataManagerTest.java | 69 +-
.../apache/pinot/core/util/ArrayCopyUtilsTest.java | 125 -
.../org/apache/pinot/core/util/CrcUtilsTest.java | 11 +-
.../apache/pinot/core/util/IngestionUtilsTest.java | 203 -
.../core/util/PeerServerSegmentFinderTest.java | 2 +-
.../apache/pinot/core/util/SchemaUtilsTest.java | 1 +
.../pinot/core/util/SegmentPushUtilsTest.java | 1 +
.../pinot/core/util/TableConfigUtilsTest.java | 9 +-
.../index/forward/FixedBitMVForwardIndexTest.java | 98 -
.../forward/FixedByteChunkSVForwardIndexTest.java | 272 -
.../forward/SortedForwardIndexReaderTest.java | 72 -
.../forward/VarByteChunkSVForwardIndexTest.java | 238 -
.../FixedByteMVMutableForwardIndexTest.java | 267 -
.../FixedByteSVMutableForwardIndexTest.java | 171 -
.../mutable/VarByteSVMutableForwardIndexTest.java | 97 -
.../FixedByteWidthRowColDataFileReaderTest.java | 105 -
...ByteSingleValueMultiColumnReaderWriterTest.java | 137 -
...FixedByteWidthRowColForwardIndexWriterTest.java | 263 -
.../pinot/queries/BaseMultiValueQueriesTest.java | 8 +-
.../org/apache/pinot/queries/BaseQueriesTest.java | 8 +-
.../pinot/queries/BaseSingleValueQueriesTest.java | 8 +-
.../queries/DistinctCountBitmapQueriesTest.java | 14 +-
.../pinot/queries/DistinctCountQueriesTest.java | 14 +-
.../DistinctCountThetaSketchQueriesTest.java | 16 +-
.../apache/pinot/queries/DistinctQueriesTest.java | 137 +-
.../queries/FSTBasedRegexpLikeQueriesTest.java | 14 +-
.../apache/pinot/queries/FastHllQueriesTest.java | 12 +-
.../apache/pinot/queries/H3IndexQueriesTest.java | 16 +-
.../org/apache/pinot/queries/IdSetQueriesTest.java | 14 +-
.../InterSegmentOrderBySingleValueQueriesTest.java | 4 +-
...terSegmentResultTableMultiValueQueriesTest.java | 2 +-
...erSegmentResultTableSingleValueQueriesTest.java | 4 +-
.../pinot/queries/JsonMatchPredicateTest.java | 14 +-
.../queries/PercentileTDigestMVQueriesTest.java | 6 +-
.../queries/PercentileTDigestQueriesTest.java | 14 +-
.../org/apache/pinot/queries/QueriesTestUtils.java | 2 +-
.../RangePredicateWithSortedInvertedIndexTest.java | 16 +-
...SegmentPartitionedDistinctCountQueriesTest.java | 14 +-
.../queries/SegmentWithNullValueVectorTest.java | 293 +
.../pinot/queries/SerializedBytesQueriesTest.java | 20 +-
.../apache/pinot/queries/StUnionQueriesTest.java | 14 +-
.../pinot/queries/SumPrecisionQueriesTest.java | 12 +-
.../pinot/queries/TextSearchQueriesTest.java | 686 +-
.../apache/pinot/queries/TransformQueriesTest.java | 16 +-
.../converter/RealtimeSegmentConverterTest.java | 54 +-
.../v1/creator/BitmapInvertedIndexCreatorTest.java | 171 -
.../v1/creator/BitmapInvertedIndexTest.java | 139 -
.../pinot/segments/v1/creator/BlocksTest.java | 74 -
.../segments/v1/creator/DictionariesTest.java | 468 -
.../v1/creator/FixedIntArrayIdMapTest.java | 111 -
.../pinot/segments/v1/creator/IntArraysTest.java | 128 -
.../v1/creator/OnHeapDictionariesTest.java | 219 -
.../segments/v1/creator/SegmentTestUtils.java | 204 -
.../realtime/ControllerLeaderLocatorTest.java | 4 +-
.../java/org/apache/pinot/util/FSTBuilderTest.java | 107 -
.../util/IntObjectIndexedPriorityQueueTest.java | 4 +-
.../org/apache/pinot/compat/tests/SegmentOp.java | 6 +-
.../tests/AdminConsoleIntegrationTest.java | 2 +-
.../tests/BaseClusterIntegrationTestSet.java | 2 +-
.../tests/ClusterIntegrationTestUtils.java | 8 +-
.../pinot/integration/tests/ClusterTest.java | 9 +-
.../ControllerPeriodicTasksIntegrationTest.java | 11 +-
...vertToRawIndexMinionClusterIntegrationTest.java | 6 +-
...tlyOnceKafkaRealtimeClusterIntegrationTest.java | 2 +-
.../tests/HybridClusterIntegrationTest.java | 6 +-
...ridClusterIntegrationTestCommandLineRunner.java | 11 +-
.../tests/JsonPathClusterIntegrationTest.java | 1 -
.../tests/LLCRealtimeClusterIntegrationTest.java | 12 +-
.../LuceneRealtimeClusterIntegrationTest.java | 1 -
.../tests/MapTypeClusterIntegrationTest.java | 4 +-
.../tests/NullHandlingIntegrationTest.java | 13 +-
.../tests/OfflineClusterIntegrationTest.java | 4 +-
...rDownloadLLCRealtimeClusterIntegrationTest.java | 2 +-
...fflineSegmentsMinionClusterIntegrationTest.java | 2 +-
.../tests/SegmentCompletionIntegrationTest.java | 2 +-
...PartitionLLCRealtimeClusterIntegrationTest.java | 2 +-
.../tests/ServerStarterIntegrationTest.java | 13 +-
.../ControllerLeaderLocatorIntegrationTest.java | 2 +-
.../org/apache/pinot/minion/MinionStarter.java | 4 +-
.../apache/pinot/minion/metrics/MinionMetrics.java | 2 +-
.../minion/taskfactory/TaskFactoryRegistry.java | 1 -
pinot-perf/pom.xml | 6 +
.../apache/pinot/perf/BenchmarkCombineGroupBy.java | 2 +-
.../org/apache/pinot/perf/BenchmarkDictionary.java | 8 +-
.../pinot/perf/BenchmarkDictionaryCreation.java | 2 +-
.../pinot/perf/BenchmarkFixedBitIntReader.java | 8 +-
.../BenchmarkFixedBitSVForwardIndexReader.java | 8 +-
.../perf/BenchmarkFixedIntArrayOffHeapIdMap.java | 10 +-
.../pinot/perf/BenchmarkGeospatialSerde.java | 10 +-
.../perf/BenchmarkOffHeapDictionaryMemory.java | 8 +-
.../perf/BenchmarkOffHeapMutableBytesStore.java | 8 +-
...BenchmarkOffheapBitmapInvertedIndexCreator.java | 18 +-
.../pinot/perf/BenchmarkOfflineIndexReader.java | 32 +-
.../pinot/perf/BenchmarkPinotDataBitSet.java | 16 +-
.../pinot/perf/BenchmarkPinotDataBuffer.java | 2 +-
.../apache/pinot/perf/BenchmarkQueryEngine.java | 4 +-
.../pinot/perf/BenchmarkStringDictionary.java | 8 +-
.../org/apache/pinot/perf/DictionaryDumper.java | 4 +-
.../pinot/perf/ForwardIndexWriterBenchmark.java | 2 +-
.../org/apache/pinot/perf/RawIndexBenchmark.java | 28 +-
.../pinot/perf/StringDictionaryPerfTest.java | 16 +-
.../batch/common/SegmentGenerationTaskRunner.java | 2 +-
.../hadoop/HadoopSegmentMetadataPushJobRunner.java | 2 +-
.../hadoop/HadoopSegmentTarPushJobRunner.java | 2 +-
.../hadoop/HadoopSegmentUriPushJobRunner.java | 3 +-
.../spark/SparkSegmentMetadataPushJobRunner.java | 2 +-
.../batch/spark/SparkSegmentTarPushJobRunner.java | 3 +-
.../batch/spark/SparkSegmentUriPushJobRunner.java | 3 +-
.../standalone/SegmentMetadataPushJobRunner.java | 2 +-
.../batch/standalone/SegmentTarPushJobRunner.java | 3 +-
.../batch/standalone/SegmentUriPushJobRunner.java | 3 +-
.../apache/pinot/hadoop/io/PinotOutputFormat.java | 2 +-
.../apache/pinot/hadoop/io/PinotRecordWriter.java | 2 +-
.../hadoop/job/HadoopSegmentPreprocessingJob.java | 2 +-
.../hadoop/job/mappers/SegmentCreationMapper.java | 12 +-
.../job/partitioners/PartitionFunctionFactory.java | 8 +-
.../pinot/hadoop/io/PinotOutputFormatTest.java | 2 +-
.../spark/jobs/SparkSegmentCreationFunction.java | 10 +-
.../src/test/java/SegmentCreationSparkTest.java | 4 +-
.../RealtimeToOfflineSegmentsTaskGenerator.java | 2 +-
.../SegmentGenerationAndPushTaskExecutor.java | 2 +-
.../merge_rollup/MergeRollupTaskExecutorTest.java | 8 +-
.../minion/tasks/purge/PurgeTaskExecutorTest.java | 8 +-
.../RealtimeToOfflineSegmentsTaskExecutorTest.java | 12 +-
...RealtimeToOfflineSegmentsTaskGeneratorTest.java | 2 +-
.../segmentuploader/SegmentUploaderDefault.java | 2 +-
.../filebased/FileBasedSegmentWriter.java | 6 +-
.../filebased/FileBasedSegmentWriterTest.java | 2 +-
pinot-segment-local/pom.xml | 156 +
.../local/aggregator/AvgValueAggregator.java | 84 +
.../local/aggregator/CountValueAggregator.java | 72 +
.../DistinctCountBitmapValueAggregator.java | 94 +
.../DistinctCountHLLValueAggregator.java | 105 +
.../local/aggregator/MaxValueAggregator.java | 72 +
.../aggregator/MinMaxRangeValueAggregator.java | 86 +
.../local/aggregator/MinValueAggregator.java | 72 +
.../aggregator/PercentileEstValueAggregator.java | 97 +
.../PercentileTDigestValueAggregator.java | 97 +
.../aggregator/SumPrecisionValueAggregator.java | 93 +
.../local/aggregator/SumValueAggregator.java | 72 +
.../segment/local/aggregator/ValueAggregator.java | 79 +
.../local/aggregator/ValueAggregatorFactory.java | 104 +
.../pinot/segment/local/customobject/AvgPair.java | 94 +
.../local/customobject/MinMaxRangePair.java | 90 +
.../segment/local/customobject/QuantileDigest.java | 1071 ++
.../segment/local/customobject/SerializedHLL.java | 42 +
.../segment/local/function/FunctionEvaluator.java | 44 +
.../local/function/FunctionEvaluatorFactory.java | 104 +
.../local/function/GroovyFunctionEvaluator.java | 105 +
.../local/function/InbuiltFunctionEvaluator.java | 172 +
.../local/function/TimeSpecFunctionEvaluator.java | 79 +
.../indexsegment/immutable/EmptyIndexSegment.java | 121 +
.../immutable/ImmutableSegmentImpl.java | 180 +
.../immutable/ImmutableSegmentLoader.java | 160 +
.../indexsegment/mutable/IntermediateSegment.java | 383 +
.../indexsegment/mutable/MutableSegmentImpl.java | 1180 +++
.../io/compression/ChunkCompressorFactory.java | 74 +
.../io/compression/PassThroughCompressor.java | 42 +
.../io/compression/PassThroughDecompressor.java | 39 +
.../local/io/compression/SnappyCompressor.java | 37 +
.../local/io/compression/SnappyDecompressor.java | 36 +
.../local/io/reader/impl/FixedBitIntReader.java | 2591 +++++
.../impl/FixedByteSingleValueMultiColReader.java | 223 +
.../readerwriter/PinotDataBufferMemoryManager.java | 46 +
.../RealtimeIndexOffHeapMemoryManager.java | 121 +
...ixedByteSingleValueMultiColumnReaderWriter.java | 194 +
.../local/io/util/FixedBitIntReaderWriter.java | 57 +
.../local/io/util/FixedBitIntReaderWriterV2.java | 149 +
.../local/io/util/FixedByteValueReaderWriter.java | 116 +
.../segment/local/io/util/PinotDataBitSet.java | 248 +
.../segment/local/io/util/PinotDataBitSetV2.java | 731 ++
.../pinot/segment/local/io/util/ValueReader.java | 51 +
.../local/io/util/VarLengthValueReader.java | 121 +
.../local/io/util/VarLengthValueWriter.java | 133 +
.../writer/impl/BaseChunkSVForwardIndexWriter.java | 191 +
.../local/io/writer/impl/DirectMemoryManager.java | 60 +
.../writer/impl/FixedBitMVForwardIndexWriter.java | 158 +
.../writer/impl/FixedBitSVForwardIndexWriter.java | 58 +
.../impl/FixedByteChunkSVForwardIndexWriter.java | 113 +
.../impl/FixedByteSingleValueMultiColWriter.java | 115 +
.../local/io/writer/impl/MmapMemoryManager.java | 179 +
.../writer/impl/MutableOffHeapByteArrayStore.java | 267 +
.../impl/VarByteChunkSVForwardIndexWriter.java | 140 +
.../partition/ByteArrayPartitionFunction.java | 62 +
.../local/partition/HashCodePartitionFunction.java | 54 +
.../local/partition/ModuloPartitionFunction.java | 73 +
.../local/partition/MurmurPartitionFunction.java | 71 +
.../local/partition/PartitionFunctionFactory.java | 89 +
.../converter/RealtimeSegmentConverter.java | 149 +
.../converter/RealtimeSegmentRecordReader.java | 82 +
.../converter/stats/MutableColumnStatistics.java | 189 +
.../stats/MutableNoDictionaryColStatistics.java | 109 +
.../RealtimeSegmentSegmentCreationDataSource.java | 59 +
.../stats/RealtimeSegmentStatsContainer.java | 62 +
.../local/realtime/impl/RealtimeSegmentConfig.java | 386 +
.../realtime/impl/RealtimeSegmentStatsHistory.java | 432 +
.../impl/ThreadSafeMutableRoaringBitmap.java | 54 +
.../dictionary/BaseOffHeapMutableDictionary.java | 444 +
.../dictionary/BaseOnHeapMutableDictionary.java | 107 +
.../dictionary/BytesOffHeapMutableDictionary.java | 226 +
.../dictionary/BytesOnHeapMutableDictionary.java | 184 +
.../dictionary/DoubleOffHeapMutableDictionary.java | 245 +
.../dictionary/DoubleOnHeapMutableDictionary.java | 202 +
.../dictionary/FloatOffHeapMutableDictionary.java | 245 +
.../dictionary/FloatOnHeapMutableDictionary.java | 202 +
.../dictionary/IntOffHeapMutableDictionary.java | 245 +
.../dictionary/IntOnHeapMutableDictionary.java | 202 +
.../dictionary/LongOffHeapMutableDictionary.java | 246 +
.../dictionary/LongOnHeapMutableDictionary.java | 202 +
.../impl/dictionary/MutableDictionaryFactory.java | 73 +
.../impl/dictionary/OffHeapMutableBytesStore.java | 259 +
.../dictionary/StringOffHeapMutableDictionary.java | 213 +
.../dictionary/StringOnHeapMutableDictionary.java | 170 +
.../forward/FixedByteMVMutableForwardIndex.java | 350 +
.../forward/FixedByteSVMutableForwardIndex.java | 302 +
.../impl/forward/VarByteSVMutableForwardIndex.java | 99 +
.../realtime/impl/geospatial/MutableH3Index.java | 78 +
.../invertedindex/RealtimeInvertedIndexReader.java | 85 +
.../RealtimeLuceneDocIdCollector.java | 64 +
.../RealtimeLuceneIndexReaderRefreshThread.java | 149 +
.../RealtimeLuceneIndexRefreshState.java | 139 +
.../RealtimeLuceneTextIndexReader.java | 161 +
.../local/realtime/impl/json/MutableJsonIndex.java | 281 +
.../impl/nullvalue/MutableNullValueVector.java | 48 +
.../recordtransformer/CompositeTransformer.java | 89 +
.../recordtransformer/DataTypeTransformer.java | 200 +
.../recordtransformer/ExpressionTransformer.java | 101 +
.../local/recordtransformer/FilterTransformer.java | 53 +
.../recordtransformer/NullValueTransformer.java | 57 +
.../local/recordtransformer/RecordTransformer.java | 39 +
.../recordtransformer/SanitizationTransformer.java | 77 +
...termediateSegmentSegmentCreationDataSource.java | 57 +
.../creator/IntermediateSegmentStatsContainer.java | 54 +
.../RecordReaderSegmentCreationDataSource.java | 100 +
.../local/segment/creator/TextIndexType.java | 25 +
.../creator/impl/SegmentColumnarIndexCreator.java | 768 ++
.../creator/impl/SegmentCreationDriverFactory.java | 35 +
.../creator/impl/SegmentDictionaryCreator.java | 300 +
.../impl/SegmentIndexCreationDriverImpl.java | 434 +
.../local/segment/creator/impl/V1Constants.java | 106 +
.../impl/bloom/OnHeapGuavaBloomFilterCreator.java | 83 +
.../fwd/MultiValueUnsortedForwardIndexCreator.java | 69 +
.../fwd/SingleValueFixedByteRawIndexCreator.java | 118 +
.../fwd/SingleValueSortedForwardIndexCreator.java | 89 +
.../SingleValueUnsortedForwardIndexCreator.java | 67 +
.../fwd/SingleValueVarByteRawIndexCreator.java | 118 +
.../impl/inv/BitmapInvertedIndexWriter.java | 90 +
.../inv/OffHeapBitmapInvertedIndexCreator.java | 250 +
.../impl/inv/OnHeapBitmapInvertedIndexCreator.java | 74 +
.../creator/impl/inv/RangeIndexCreator.java | 575 ++
.../impl/inv/geospatial/BaseH3IndexCreator.java | 169 +
.../impl/inv/geospatial/OffHeapH3IndexCreator.java | 219 +
.../impl/inv/geospatial/OnHeapH3IndexCreator.java | 50 +
.../impl/inv/json/BaseJsonIndexCreator.java | 173 +
.../impl/inv/json/OffHeapJsonIndexCreator.java | 283 +
.../impl/inv/json/OnHeapJsonIndexCreator.java | 61 +
.../impl/inv/text/LuceneFSTIndexCreator.java | 101 +
.../impl/nullvalue/NullValueVectorCreator.java | 64 +
.../stats/AbstractColumnStatisticsCollector.java | 163 +
.../stats/BytesColumnPredIndexStatsCollector.java | 113 +
.../stats/DoubleColumnPreIndexStatsCollector.java | 101 +
.../stats/FloatColumnPreIndexStatsCollector.java | 101 +
.../stats/IntColumnPreIndexStatsCollector.java | 101 +
.../stats/LongColumnPreIndexStatsCollector.java | 101 +
.../stats/SegmentPreIndexStatsCollectorImpl.java | 143 +
.../stats/StringColumnPreIndexStatsCollector.java | 120 +
.../creator/impl/text/LuceneTextIndexCreator.java | 163 +
.../index/column/BaseVirtualColumnProvider.java | 47 +
.../DefaultNullValueVirtualColumnProvider.java | 87 +
.../index/column/IntermediateIndexContainer.java | 134 +
.../local/segment/index/column/NumValuesInfo.java | 41 +
.../index/column/PhysicalColumnIndexContainer.java | 328 +
.../converter/SegmentFormatConverterFactory.java | 37 +
.../converter/SegmentV1V2ToV3FormatConverter.java | 288 +
.../segment/index/datasource/BaseDataSource.java | 128 +
.../segment/index/datasource/EmptyDataSource.java | 94 +
.../index/datasource/ImmutableDataSource.java | 119 +
.../index/datasource/MutableDataSource.java | 138 +
.../segment/index/loader/IndexLoadingConfig.java | 445 +
.../local/segment/index/loader/LoaderUtils.java | 186 +
.../segment/index/loader/SegmentPreProcessor.java | 204 +
.../index/loader/V3RemoveIndexException.java | 29 +
.../index/loader/V3UpdateIndexException.java | 29 +
.../loader/bloomfilter/BloomFilterHandler.java | 161 +
.../ColumnMinMaxValueGenerator.java | 156 +
.../ColumnMinMaxValueGeneratorMode.java | 38 +
.../defaultcolumn/BaseDefaultColumnHandler.java | 685 ++
.../loader/defaultcolumn/DefaultColumnHandler.java | 31 +
.../defaultcolumn/DefaultColumnHandlerFactory.java | 41 +
.../defaultcolumn/DefaultColumnStatistics.java | 113 +
.../defaultcolumn/V1DefaultColumnHandler.java | 55 +
.../defaultcolumn/V3DefaultColumnHandler.java | 85 +
.../index/loader/invertedindex/H3IndexHandler.java | 161 +
.../loader/invertedindex/InvertedIndexHandler.java | 136 +
.../loader/invertedindex/JsonIndexHandler.java | 158 +
.../invertedindex/LuceneFSTIndexHandler.java | 147 +
.../loader/invertedindex/RangeIndexHandler.java | 215 +
.../loader/invertedindex/TextIndexHandler.java | 182 +
.../segment/index/metadata/ColumnMetadata.java | 620 ++
.../index/metadata/SegmentMetadataImpl.java | 579 ++
.../index/readers/BaseImmutableDictionary.java | 298 +
.../index/readers/BitmapInvertedIndexReader.java | 95 +
.../segment/index/readers/BytesDictionary.java | 95 +
.../readers/ConstantValueBytesDictionary.java | 108 +
.../readers/ConstantValueDoubleDictionary.java | 96 +
.../readers/ConstantValueFloatDictionary.java | 96 +
.../index/readers/ConstantValueIntDictionary.java | 96 +
.../index/readers/ConstantValueLongDictionary.java | 96 +
.../readers/ConstantValueStringDictionary.java | 102 +
.../segment/index/readers/DocIdDictionary.java | 81 +
.../segment/index/readers/DoubleDictionary.java | 70 +
.../segment/index/readers/FloatDictionary.java | 70 +
.../local/segment/index/readers/IntDictionary.java | 70 +
.../segment/index/readers/LongDictionary.java | 70 +
.../index/readers/LuceneFSTIndexReader.java | 82 +
.../index/readers/NullValueVectorReaderImpl.java | 42 +
.../segment/index/readers/OnHeapDictionary.java | 32 +
.../index/readers/OnHeapDoubleDictionary.java | 108 +
.../index/readers/OnHeapFloatDictionary.java | 108 +
.../segment/index/readers/OnHeapIntDictionary.java | 108 +
.../index/readers/OnHeapLongDictionary.java | 108 +
.../index/readers/OnHeapStringDictionary.java | 131 +
.../segment/index/readers/RangeIndexReader.java | 220 +
.../segment/index/readers/StringDictionary.java | 124 +
.../index/readers/ValidDocIndexReaderImpl.java | 37 +
.../readers/bloom/BaseGuavaBloomFilterReader.java | 66 +
.../readers/bloom/BloomFilterReaderFactory.java | 45 +
.../readers/bloom/GuavaBloomFilterReaderUtils.java | 65 +
.../bloom/OffHeapGuavaBloomFilterReader.java | 50 +
.../bloom/OnHeapGuavaBloomFilterReader.java | 54 +
.../constant/ConstantMVForwardIndexReader.java | 55 +
.../constant/ConstantMVInvertedIndexReader.java | 44 +
.../constant/ConstantSortedIndexReader.java | 56 +
.../forward/BaseChunkSVForwardIndexReader.java | 216 +
.../forward/FixedBitMVForwardIndexReader.java | 153 +
.../forward/FixedBitSVForwardIndexReader.java | 70 +
.../forward/FixedBitSVForwardIndexReaderV2.java | 101 +
.../FixedByteChunkSVForwardIndexReader.java | 94 +
.../forward/VarByteChunkSVForwardIndexReader.java | 190 +
.../readers/geospatial/ImmutableH3IndexReader.java | 84 +
.../readers/json/ImmutableJsonIndexReader.java | 255 +
.../readers/sorted/SortedIndexReaderImpl.java | 134 +
.../index/readers/text/LuceneDocIdCollector.java | 76 +
.../index/readers/text/LuceneTextIndexReader.java | 220 +
.../local/segment/memory/BasePinotLBuffer.java | 161 +
.../local/segment/memory/PinotByteBuffer.java | 336 +
.../local/segment/memory/PinotDataBuffer.java | 393 +
.../segment/memory/PinotNativeOrderLBuffer.java | 183 +
.../segment/memory/PinotNonNativeOrderLBuffer.java | 183 +
.../segment/readers/GenericRowRecordReader.java | 72 +
.../readers/IntermediateSegmentRecordReader.java | 80 +
.../readers/MultiplePinotSegmentRecordReader.java | 280 +
.../segment/readers/PinotSegmentColumnReader.java | 112 +
.../segment/readers/PinotSegmentRecordReader.java | 171 +
.../segment/readers/sort/PinotSegmentSorter.java | 121 +
.../local/segment/readers/sort/SegmentSorter.java | 35 +
.../local/segment/store/ColumnIndexDirectory.java | 114 +
.../local/segment/store/ColumnIndexType.java | 51 +
.../local/segment/store/FilePerIndexDirectory.java | 174 +
.../segment/local/segment/store/IndexEntry.java | 44 +
.../segment/local/segment/store/IndexKey.java | 71 +
.../local/segment/store/SegmentDirectory.java | 218 +
.../local/segment/store/SegmentDirectoryPaths.java | 127 +
.../segment/store/SegmentLocalFSDirectory.java | 411 +
.../segment/store/SingleFileIndexDirectory.java | 353 +
.../virtualcolumn/DocIdVirtualColumnProvider.java | 84 +
.../virtualcolumn/VirtualColumnContext.java | 44 +
.../virtualcolumn/VirtualColumnIndexContainer.java | 109 +
.../virtualcolumn/VirtualColumnProvider.java | 42 +
.../VirtualColumnProviderFactory.java | 65 +
.../local/segment/virtualcolumn/package-info.java | 22 +
.../segment/local/startree/OffHeapStarTree.java | 139 +
.../local/startree/OffHeapStarTreeNode.java | 171 +
.../local/startree/StarTreeBuilderUtils.java | 279 +
.../startree/v2/builder/BaseSingleTreeBuilder.java | 541 +
.../startree/v2/builder/MultipleTreesBuilder.java | 175 +
.../v2/builder/OffHeapSingleTreeBuilder.java | 350 +
.../v2/builder/OnHeapSingleTreeBuilder.java | 159 +
.../startree/v2/builder/SingleTreeBuilder.java | 34 +
.../startree/v2/builder/StarTreeIndexCombiner.java | 95 +
.../v2/builder/StarTreeV2BuilderConfig.java | 204 +
.../startree/v2/store/StarTreeDataSource.java | 98 +
.../startree/v2/store/StarTreeIndexContainer.java | 72 +
.../startree/v2/store/StarTreeIndexMapUtils.java | 201 +
.../startree/v2/store/StarTreeLoaderUtils.java | 139 +
.../upsert/PartitionUpsertMetadataManager.java | 205 +
.../pinot/segment/local/upsert/RecordLocation.java | 55 +
.../pinot/segment/local/utils/CleanerUtil.java | 199 +
.../apache/pinot/segment/local/utils/CrcUtils.java | 123 +
.../segment/local/utils/CustomSerDeUtils.java | 274 +
.../local/utils/FixedIntArrayOffHeapIdMap.java | 182 +
.../segment/local/utils/GeometrySerializer.java | 526 +
.../pinot/segment/local/utils/GeometryType.java | 70 +
.../pinot/segment/local/utils/GeometryUtils.java | 71 +
.../apache/pinot/segment/local/utils/H3Utils.java | 38 +
.../apache/pinot/segment/local/utils/IdMap.java | 74 +
.../pinot/segment/local/utils/IngestionUtils.java | 366 +
.../pinot/segment/local/utils/SchemaUtils.java | 162 +
.../segment/local/utils/SegmentPushUtils.java | 347 +
.../segment/local/utils/TableConfigUtils.java | 583 ++
.../pinot/segment/local/utils/fst/FSTBuilder.java | 64 +
.../local/utils/fst/PinotBufferIndexInput.java | 89 +
.../segment/local/utils/fst/RegexpMatcher.java | 163 +
.../io/reader/impl/FixedBitIntReaderTest.java | 83 +
.../segment/local/io/util/PinotDataBitSetTest.java | 167 +
.../local/io/util/PinotDataBitSetV2Test.java | 443 +
.../io/util/VarLengthValueReaderWriterTest.java | 117 +
.../impl/MmapMemoryManagerFileCleanupTest.java | 66 +
.../io/writer/impl/MmapMemoryManagerTest.java | 185 +
.../impl/MutableOffHeapByteArrayStoreTest.java | 123 +
.../local/partition/PartitionFunctionTest.java | 245 +
.../impl/RealtimeSegmentStatsHistoryTest.java | 319 +
.../impl/dictionary/MultiValueDictionaryTest.java | 84 +
.../impl/dictionary/MutableDictionaryTest.java | 366 +
.../dictionary/OffHeapMutableBytesStoreTest.java | 142 +
.../RealtimeInvertedIndexReaderTest.java | 106 +
.../impl/nullvalue/MutableNullValueVectorTest.java | 45 +
.../recordtransformer/DataTypeTransformerTest.java | 202 +
.../ExpressionTransformerTest.java | 260 +
.../ExpressionTransformerTimeTest.java | 175 +
.../recordtransformer/RecordTransformerTest.java | 254 +
.../creator/BitmapInvertedIndexCreatorTest.java | 171 +
.../segment/creator/BitmapInvertedIndexTest.java | 139 +
.../segment/local/segment/creator/BlocksTest.java | 74 +
.../local/segment/creator/DictionariesTest.java | 468 +
.../segment/creator/FixedIntArrayIdMapTest.java | 111 +
.../local/segment/creator/IntArraysTest.java | 128 +
.../segment/creator/OnHeapDictionariesTest.java | 219 +
.../local/segment/creator/SegmentTestUtils.java | 204 +
.../impl/SegmentColumnarIndexCreatorTest.java | 126 +
.../impl/nullvalue/NullValueVectorCreatorTest.java | 67 +
.../local/segment/index/ColumnMetadataTest.java | 199 +
.../segment/local/segment/index/H3IndexTest.java | 110 +
.../segment/local/segment/index/JsonIndexTest.java | 207 +
.../segment/index/SegmentMetadataImplTest.java | 109 +
.../DefaultNullValueVirtualColumnProviderTest.java | 177 +
.../SegmentV1V2ToV3FormatConverterTest.java | 128 +
.../index/creator/BloomFilterCreatorTest.java | 81 +
.../index/creator/LuceneFSTIndexCreatorTest.java | 73 +
.../index/creator/RangeIndexCreatorTest.java | 382 +
.../segment/index/creator/RawIndexCreatorTest.java | 288 +
.../SegmentGenerationWithBytesTypeTest.java | 341 +
.../SegmentGenerationWithFilterRecordsTest.java | 117 +
.../SegmentGenerationWithMultipleRecordsTest.java | 113 +
.../SegmentGenerationWithNoRecordsTest.java | 117 +
.../SegmentGenerationWithTimeColumnTest.java | 263 +
.../index/creator/SegmentPartitionTest.java | 227 +
.../index/forward/FixedBitMVForwardIndexTest.java | 98 +
.../forward/FixedByteChunkSVForwardIndexTest.java | 280 +
.../forward/SortedForwardIndexReaderTest.java | 72 +
.../forward/VarByteChunkSVForwardIndexTest.java | 238 +
.../FixedByteMVMutableForwardIndexTest.java | 267 +
.../FixedByteSVMutableForwardIndexTest.java | 171 +
.../mutable/VarByteSVMutableForwardIndexTest.java | 97 +
.../local/segment/index/loader/LoaderTest.java | 626 ++
.../segment/index/loader/LoaderUtilsTest.java | 98 +
.../index/loader/SegmentPreProcessorTest.java | 773 ++
.../BaseDefaultColumnHandlerTest.java | 238 +
.../FixedByteWidthRowColDataFileReaderTest.java | 105 +
.../index/readers/ImmutableDictionaryTest.java | 361 +
.../ImmutableDictionaryTypeConversionTest.java | 385 +
.../readers/NullValueVectorReaderImplTest.java | 68 +
.../FixedBitSVForwardIndexReaderV2Test.java | 112 +
...ByteSingleValueMultiColumnReaderWriterTest.java | 137 +
...FixedByteWidthRowColForwardIndexWriterTest.java | 263 +
.../local/segment/memory/PinotDataBufferTest.java | 641 ++
.../MultiplePinotSegmentRecordReaderTest.java | 162 +
.../readers/PinotSegmentRecordReaderTest.java | 141 +
.../local/segment/readers/PinotSegmentUtil.java | 169 +
.../readers/RecordReaderSampleDataTest.java | 189 +
.../store/ColumnIndexDirectoryTestHelper.java | 122 +
.../segment/store/FilePerIndexDirectoryTest.java | 157 +
.../segment/store/SegmentDirectoryPathsTest.java | 69 +
.../segment/store/SegmentLocalFSDirectoryTest.java | 152 +
.../store/SingleFileIndexDirectoryTest.java | 166 +
.../v2/builder/StarTreeV2BuilderConfigTest.java | 99 +
.../pinot/segment/local/util/FSTBuilderTest.java | 104 +
.../segment/local/util/IngestionUtilsTest.java | 204 +
pinot-segment-local/src/test/resources/data/README | 22 +
.../src/test/resources/data/fixedByteCompressed.v2 | Bin 0 -> 8098 bytes
.../src/test/resources/data/fixedByteRaw.v2 | Bin 0 -> 16036 bytes
.../src/test/resources/data/fixedByteSVRDoubles.v1 | Bin 0 -> 44397 bytes
.../src/test/resources/data/log4j2.xml | 35 +
.../src/test/resources/data/newColumnsSchema1.json | 37 +
.../src/test/resources/data/newColumnsSchema2.json | 38 +
.../src/test/resources/data/newColumnsSchema3.json | 48 +
.../src/test/resources/data/newColumnsSchema4.json | 84 +
.../resources/data/newColumnsWithTextSchema.json | 76 +
.../src/test/resources/data/paddingNull.tar.gz | Bin 0 -> 1170 bytes
.../src/test/resources/data/paddingOld.tar.gz | Bin 0 -> 1157 bytes
.../src/test/resources/data/paddingPercent.tar.gz | Bin 0 -> 1174 bytes
.../src/test/resources/data/rawhllresults.txt | 8 +
.../data/realtime-segment-stats-history-v1.ser | Bin
.../src/test/resources/data/testDataMVSchema.json | 68 +
.../src/test/resources/data/test_data-mv.avro | Bin 0 -> 7700192 bytes
.../src/test/resources/data/test_data-sv.avro | Bin 0 -> 3397502 bytes
.../src/test/resources/data/test_data-sv_hll.avro | Bin 0 -> 9751470 bytes
.../src/test/resources/data/test_empty_data.json | 0
.../data/test_null_value_vector_data.json | 12 +
.../data/test_null_value_vector_pinot_schema.json | 26 +
.../src/test/resources/data/test_sample_data.avro | Bin 0 -> 917973 bytes
.../src/test/resources/data/test_sample_data.csv | 10002 +++++++++++++++++++
.../src/test/resources/data/test_sample_data.json | 10001 ++++++++++++++++++
.../data/test_sample_data_multi_value.avro | Bin 0 -> 12222227 bytes
.../test/resources/data/test_sample_pv_data.avro | Bin 0 -> 15706752 bytes
.../test/resources/data/test_upsert_schema.json | 21 +
.../src/test/resources/data/varByteStrings.v1 | Bin 0 -> 22447 bytes
.../resources/data/varByteStringsCompressed.v2 | Bin 0 -> 17355 bytes
.../src/test/resources/data/varByteStringsRaw.v2 | Bin 0 -> 286902 bytes
pinot-server/pom.xml | 7 +
.../server/api/resources/MmapDebugResource.java | 2 +-
.../api/resources/SegmentMetadataFetcher.java | 4 +-
.../pinot/server/api/resources/TablesResource.java | 29 +-
.../org/apache/pinot/server/conf/ServerConf.java | 10 +-
.../pinot/server/starter/ServerInstance.java | 4 +-
.../starter/ServerQueriesDisabledTracker.java | 2 +-
.../server/starter/helix/AdminApiApplication.java | 2 +-
.../helix/DefaultHelixStarterServerConfig.java | 2 +-
.../starter/helix/HelixInstanceDataManager.java | 12 +-
.../helix/HelixInstanceDataManagerConfig.java | 10 +-
.../server/starter/helix/HelixServerStarter.java | 16 +-
.../starter/helix/SegmentFetcherAndLoader.java | 16 +-
.../apache/pinot/server/api/AccessControlTest.java | 2 +-
.../apache/pinot/server/api/BaseResourceTest.java | 45 +-
.../pinot/server/api/TablesResourceTest.java | 6 +-
.../config/instance/InstanceDataManagerConfig.java | 57 +
.../spi/exception/BadQueryRequestException.java | 33 +
.../spi/exception/EarlyTerminationException.java | 30 +
.../org/apache/pinot/spi/utils/ArrayCopyUtils.java | 168 +
.../apache/pinot/spi/utils/CommonConstants.java | 509 +
.../org/apache/pinot/spi/utils/FixedIntArray.java | 68 +
.../main/java/org/apache/pinot/spi/utils/Pair.java | 55 +
.../java/org/apache/pinot/spi/utils/ReadMode.java | 45 +
.../apache/pinot/spi/utils/ArrayCopyUtilsTest.java | 125 +
.../org/apache/pinot/tools/SegmentDumpTool.java | 6 +-
.../apache/pinot/tools/StarTreeIndexViewer.java | 4 +-
.../org/apache/pinot/tools/UpdateSegmentState.java | 2 +-
.../tools/admin/command/AddSchemaCommand.java | 9 +-
.../pinot/tools/admin/command/AddTableCommand.java | 2 +-
.../tools/admin/command/AddTenantCommand.java | 4 +-
.../tools/admin/command/BootstrapTableCommand.java | 6 +-
.../tools/admin/command/ChangeTableState.java | 5 +-
.../tools/admin/command/CreateSegmentCommand.java | 8 +-
.../admin/command/OperateClusterConfigCommand.java | 2 +-
.../tools/admin/command/PostQueryCommand.java | 4 +-
.../tools/admin/command/QuickstartRunner.java | 2 +-
.../admin/command/ShowClusterInfoCommand.java | 2 +-
.../tools/admin/command/StartBrokerCommand.java | 3 +-
.../tools/admin/command/StartMinionCommand.java | 4 +-
.../tools/admin/command/StartServerCommand.java | 3 +-
.../admin/command/StartServiceManagerCommand.java | 7 +-
.../tools/admin/command/UploadSegmentCommand.java | 2 +-
.../anonymizer/ArrayBasedGlobalDictionaries.java | 2 +-
.../pinot/tools/anonymizer/GlobalDictionaries.java | 2 +-
.../anonymizer/MapBasedGlobalDictionaries.java | 2 +-
.../anonymizer/PinotDataAndQueryAnonymizer.java | 16 +-
.../pinot/tools/backfill/BackfillSegmentUtils.java | 2 +-
.../tools/config/validator/SchemaValidator.java | 2 +-
.../pinot/tools/perf/PerfBenchmarkDriver.java | 11 +-
.../pinot/tools/perf/PerfBenchmarkDriverConf.java | 5 +-
.../pinot/tools/perf/PerfBenchmarkRunner.java | 2 +-
.../query/comparison/QueryComparisonConfig.java | 2 +-
.../query/comparison/SegmentInfoProvider.java | 4 +-
.../apache/pinot/tools/scan/query/Aggregation.java | 4 +-
.../apache/pinot/tools/scan/query/AvgFunction.java | 2 +-
.../pinot/tools/scan/query/CountFunction.java | 2 +-
.../tools/scan/query/DistinctCountFunction.java | 2 +-
.../apache/pinot/tools/scan/query/MaxFunction.java | 2 +-
.../apache/pinot/tools/scan/query/MinFunction.java | 2 +-
.../tools/scan/query/MinMaxRangeFunction.java | 2 +-
.../apache/pinot/tools/scan/query/Projection.java | 6 +-
.../pinot/tools/scan/query/QueryResponse.java | 2 +-
.../tools/scan/query/RangePredicateFilter.java | 2 +-
.../apache/pinot/tools/scan/query/ResultTable.java | 2 +-
.../tools/scan/query/SegmentQueryProcessor.java | 12 +-
.../apache/pinot/tools/scan/query/Selection.java | 4 +-
.../apache/pinot/tools/scan/query/SumFunction.java | 2 +-
.../converter/DictionaryToRawIndexConverter.java | 15 +-
.../converter/PinotSegmentToAvroConverter.java | 2 +-
.../converter/PinotSegmentToCsvConverter.java | 2 +-
.../converter/PinotSegmentToJsonConverter.java | 2 +-
.../segment/converter/SegmentMergeCommand.java | 2 +-
.../PinotServiceManagerAdminApiApplication.java | 2 +-
.../PinotServiceManagerInstanceResource.java | 23 +-
.../apache/pinot/tools/utils/PinotConfigUtils.java | 3 +-
.../converter/PinotSegmentConverterTest.java | 6 +-
pom.xml | 6 +
1347 files changed, 86028 insertions(+), 64906 deletions(-)
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/api/resources/PinotClientRequest.java b/pinot-broker/src/main/java/org/apache/pinot/broker/api/resources/PinotClientRequest.java
index 216a7c4..d85e907 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/api/resources/PinotClientRequest.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/api/resources/PinotClientRequest.java
@@ -45,7 +45,7 @@ import org.apache.pinot.broker.requesthandler.BrokerRequestHandler;
import org.apache.pinot.common.metrics.BrokerMeter;
import org.apache.pinot.common.metrics.BrokerMetrics;
import org.apache.pinot.common.response.BrokerResponse;
-import org.apache.pinot.common.utils.CommonConstants.Broker.Request;
+import org.apache.pinot.spi.utils.CommonConstants.Broker.Request;
import org.apache.pinot.spi.utils.JsonUtils;
import org.glassfish.jersey.server.ManagedAsync;
import org.slf4j.Logger;
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/broker/BrokerAdminApiApplication.java b/pinot-broker/src/main/java/org/apache/pinot/broker/broker/BrokerAdminApiApplication.java
index 8a2fe80..c87e619 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/broker/BrokerAdminApiApplication.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/broker/BrokerAdminApiApplication.java
@@ -26,9 +26,9 @@ import java.util.List;
import org.apache.pinot.broker.requesthandler.BrokerRequestHandler;
import org.apache.pinot.broker.routing.RoutingManager;
import org.apache.pinot.common.metrics.BrokerMetrics;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.core.transport.ListenerConfig;
import org.apache.pinot.core.util.ListenerConfigUtil;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.glassfish.grizzly.http.server.CLStaticHttpHandler;
import org.glassfish.grizzly.http.server.HttpHandler;
import org.glassfish.grizzly.http.server.HttpServer;
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BrokerResourceOnlineOfflineStateModelFactory.java b/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BrokerResourceOnlineOfflineStateModelFactory.java
index 9ea3573..b6639e2 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BrokerResourceOnlineOfflineStateModelFactory.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BrokerResourceOnlineOfflineStateModelFactory.java
@@ -34,7 +34,7 @@ import org.apache.pinot.spi.config.table.TableConfig;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.BROKER_RESOURCE_INSTANCE;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.BROKER_RESOURCE_INSTANCE;
/**
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/HelixBrokerStarter.java b/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/HelixBrokerStarter.java
index 8cd41c3..2b58791 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/HelixBrokerStarter.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/HelixBrokerStarter.java
@@ -51,11 +51,7 @@ import org.apache.pinot.common.function.FunctionRegistry;
import org.apache.pinot.common.metadata.ZKMetadataProvider;
import org.apache.pinot.common.metrics.BrokerMeter;
import org.apache.pinot.common.metrics.BrokerMetrics;
-import org.apache.pinot.spi.metrics.PinotMetricsRegistry;
import org.apache.pinot.common.metrics.PinotMetricUtils;
-import org.apache.pinot.common.utils.CommonConstants;
-import org.apache.pinot.common.utils.CommonConstants.Broker;
-import org.apache.pinot.common.utils.CommonConstants.Helix;
import org.apache.pinot.common.utils.NetUtil;
import org.apache.pinot.common.utils.ServiceStatus;
import org.apache.pinot.common.utils.config.TagNameUtils;
@@ -65,8 +61,12 @@ import org.apache.pinot.core.transport.TlsConfig;
import org.apache.pinot.core.util.ListenerConfigUtil;
import org.apache.pinot.core.util.TlsUtils;
import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.metrics.PinotMetricsRegistry;
import org.apache.pinot.spi.services.ServiceRole;
import org.apache.pinot.spi.services.ServiceStartable;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants.Broker;
+import org.apache.pinot.spi.utils.CommonConstants.Helix;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/pruner/PartitionZKMetadataPruner.java b/pinot-broker/src/main/java/org/apache/pinot/broker/pruner/PartitionZKMetadataPruner.java
index 34fcb3f..3761c22 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/pruner/PartitionZKMetadataPruner.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/pruner/PartitionZKMetadataPruner.java
@@ -26,7 +26,7 @@ import org.apache.pinot.common.metadata.segment.SegmentPartitionMetadata;
import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
import org.apache.pinot.common.request.FilterOperator;
import org.apache.pinot.common.utils.request.FilterQueryTree;
-import org.apache.pinot.core.data.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.local.partition.PartitionFunctionFactory;
import org.apache.pinot.segment.spi.partition.PartitionFunction;
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java b/pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java
index a33815d..21b08cc 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java
@@ -36,11 +36,11 @@ import org.apache.pinot.broker.broker.helix.ClusterChangeHandler;
import org.apache.pinot.common.metadata.ZKMetadataProvider;
import org.apache.pinot.common.metrics.BrokerGauge;
import org.apache.pinot.common.metrics.BrokerMetrics;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.helix.HelixHelper;
import org.apache.pinot.spi.config.table.QuotaConfig;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.apache.zookeeper.data.Stat;
import org.slf4j.Logger;
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
index 7b83884..d2b47ee 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
@@ -72,9 +72,6 @@ import org.apache.pinot.common.request.transform.TransformExpressionTree;
import org.apache.pinot.common.response.BrokerResponse;
import org.apache.pinot.common.response.broker.BrokerResponseNative;
import org.apache.pinot.common.response.broker.ResultTable;
-import org.apache.pinot.common.utils.CommonConstants;
-import org.apache.pinot.common.utils.CommonConstants.Broker;
-import org.apache.pinot.common.utils.CommonConstants.Query.Range;
import org.apache.pinot.common.utils.DataSchema;
import org.apache.pinot.common.utils.helix.TableCache;
import org.apache.pinot.common.utils.request.RequestUtils;
@@ -92,6 +89,9 @@ import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.spi.env.PinotConfiguration;
import org.apache.pinot.spi.utils.BytesUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants.Broker;
+import org.apache.pinot.spi.utils.CommonConstants.Query.Range;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.apache.pinot.sql.parsers.CalciteSqlParser;
import org.slf4j.Logger;
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/RoutingManager.java b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/RoutingManager.java
index 423fbc0..d097dc8 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/RoutingManager.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/RoutingManager.java
@@ -53,12 +53,12 @@ import org.apache.pinot.common.metadata.ZKMetadataProvider;
import org.apache.pinot.common.metrics.BrokerMeter;
import org.apache.pinot.common.metrics.BrokerMetrics;
import org.apache.pinot.common.request.BrokerRequest;
-import org.apache.pinot.common.utils.CommonConstants;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.apache.pinot.common.utils.HashUtil;
import org.apache.pinot.core.transport.ServerInstance;
import org.apache.pinot.spi.config.table.QueryConfig;
import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.apache.zookeeper.data.Stat;
import org.slf4j.Logger;
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
index 0410180..e927f52 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
@@ -32,8 +32,8 @@ import org.apache.helix.model.IdealState;
import org.apache.pinot.common.metrics.BrokerMeter;
import org.apache.pinot.common.metrics.BrokerMetrics;
import org.apache.pinot.common.request.BrokerRequest;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.apache.pinot.common.utils.HashUtil;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/StrictReplicaGroupInstanceSelector.java b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/StrictReplicaGroupInstanceSelector.java
index a368c79..730335f 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/StrictReplicaGroupInstanceSelector.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/StrictReplicaGroupInstanceSelector.java
@@ -27,8 +27,8 @@ import java.util.TreeSet;
import org.apache.helix.model.ExternalView;
import org.apache.helix.model.IdealState;
import org.apache.pinot.common.metrics.BrokerMetrics;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.apache.pinot.common.utils.HashUtil;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
/**
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/EmptySegmentPruner.java b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/EmptySegmentPruner.java
index 8e45331..a43bb4b 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/EmptySegmentPruner.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/EmptySegmentPruner.java
@@ -32,8 +32,8 @@ import org.apache.helix.model.IdealState;
import org.apache.helix.store.zk.ZkHelixPropertyStore;
import org.apache.pinot.common.metadata.ZKMetadataProvider;
import org.apache.pinot.common.request.BrokerRequest;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/PartitionSegmentPruner.java b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/PartitionSegmentPruner.java
index e7a1078..41c9028 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/PartitionSegmentPruner.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/PartitionSegmentPruner.java
@@ -34,11 +34,11 @@ import org.apache.pinot.common.metadata.ZKMetadataProvider;
import org.apache.pinot.common.metadata.segment.ColumnPartitionMetadata;
import org.apache.pinot.common.metadata.segment.SegmentPartitionMetadata;
import org.apache.pinot.common.request.BrokerRequest;
-import org.apache.pinot.common.utils.CommonConstants.Segment;
import org.apache.pinot.common.utils.request.FilterQueryTree;
import org.apache.pinot.common.utils.request.RequestUtils;
-import org.apache.pinot.core.data.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.local.partition.PartitionFunctionFactory;
import org.apache.pinot.segment.spi.partition.PartitionFunction;
+import org.apache.pinot.spi.utils.CommonConstants.Segment;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/TimeSegmentPruner.java b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/TimeSegmentPruner.java
index c994483..77abd71 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/TimeSegmentPruner.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/TimeSegmentPruner.java
@@ -38,14 +38,14 @@ import org.apache.pinot.broker.routing.segmentpruner.interval.Interval;
import org.apache.pinot.broker.routing.segmentpruner.interval.IntervalTree;
import org.apache.pinot.common.metadata.ZKMetadataProvider;
import org.apache.pinot.common.request.BrokerRequest;
-import org.apache.pinot.common.utils.CommonConstants;
-import org.apache.pinot.common.utils.CommonConstants.Query.Range;
import org.apache.pinot.common.utils.request.FilterQueryTree;
import org.apache.pinot.common.utils.request.RequestUtils;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.data.DateTimeFieldSpec;
import org.apache.pinot.spi.data.DateTimeFormatSpec;
import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants.Query.Range;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentselector/RealtimeSegmentSelector.java b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentselector/RealtimeSegmentSelector.java
index 2d778c6..185abfa 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentselector/RealtimeSegmentSelector.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentselector/RealtimeSegmentSelector.java
@@ -30,10 +30,10 @@ import java.util.concurrent.atomic.AtomicLong;
import org.apache.helix.model.ExternalView;
import org.apache.helix.model.IdealState;
import org.apache.pinot.common.request.BrokerRequest;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.apache.pinot.common.utils.HLCSegmentName;
import org.apache.pinot.common.utils.LLCSegmentName;
import org.apache.pinot.common.utils.SegmentName;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
/**
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/timeboundary/TimeBoundaryManager.java b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/timeboundary/TimeBoundaryManager.java
index ddd5dbb..c31e311 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/timeboundary/TimeBoundaryManager.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/timeboundary/TimeBoundaryManager.java
@@ -32,12 +32,12 @@ import org.apache.helix.model.ExternalView;
import org.apache.helix.model.IdealState;
import org.apache.helix.store.zk.ZkHelixPropertyStore;
import org.apache.pinot.common.metadata.ZKMetadataProvider;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.data.DateTimeFieldSpec;
import org.apache.pinot.spi.data.DateTimeFormatSpec;
import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.IngestionConfigUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-broker/src/test/java/org/apache/pinot/broker/broker/HelixBrokerStarterTest.java b/pinot-broker/src/test/java/org/apache/pinot/broker/broker/HelixBrokerStarterTest.java
index 99052ff..0d97e20 100644
--- a/pinot-broker/src/test/java/org/apache/pinot/broker/broker/HelixBrokerStarterTest.java
+++ b/pinot-broker/src/test/java/org/apache/pinot/broker/broker/HelixBrokerStarterTest.java
@@ -29,7 +29,6 @@ import org.apache.pinot.broker.routing.RoutingTable;
import org.apache.pinot.broker.routing.timeboundary.TimeBoundaryInfo;
import org.apache.pinot.common.metadata.segment.OfflineSegmentZKMetadata;
import org.apache.pinot.common.request.BrokerRequest;
-import org.apache.pinot.common.utils.CommonConstants.Helix;
import org.apache.pinot.common.utils.ZkStarter;
import org.apache.pinot.common.utils.config.TagNameUtils;
import org.apache.pinot.controller.helix.ControllerTest;
@@ -44,6 +43,7 @@ import org.apache.pinot.spi.data.DateTimeGranularitySpec;
import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.CommonConstants.Helix;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.apache.pinot.util.TestUtils;
diff --git a/pinot-broker/src/test/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManagerTest.java b/pinot-broker/src/test/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManagerTest.java
index e61c487..a302eed 100644
--- a/pinot-broker/src/test/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManagerTest.java
+++ b/pinot-broker/src/test/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManagerTest.java
@@ -37,13 +37,13 @@ import org.apache.helix.model.HelixConfigScope;
import org.apache.helix.store.zk.ZkHelixPropertyStore;
import org.apache.pinot.common.metadata.ZKMetadataProvider;
import org.apache.pinot.common.metrics.BrokerMetrics;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.StringUtil;
import org.apache.pinot.common.utils.ZkStarter;
import org.apache.pinot.common.utils.config.TableConfigUtils;
import org.apache.pinot.spi.config.table.QuotaConfig;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.mockito.Mockito;
diff --git a/pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/BrokerRequestOptionsTest.java b/pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/BrokerRequestOptionsTest.java
index c9f7dc5..5792d68 100644
--- a/pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/BrokerRequestOptionsTest.java
+++ b/pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/BrokerRequestOptionsTest.java
@@ -22,8 +22,8 @@ import com.fasterxml.jackson.databind.node.ObjectNode;
import java.util.HashMap;
import java.util.Map;
import org.apache.pinot.common.request.BrokerRequest;
-import org.apache.pinot.common.utils.CommonConstants.Broker.Request;
import org.apache.pinot.pql.parsers.Pql2Compiler;
+import org.apache.pinot.spi.utils.CommonConstants.Broker.Request;
import org.apache.pinot.spi.utils.JsonUtils;
import org.testng.Assert;
import org.testng.annotations.Test;
diff --git a/pinot-broker/src/test/java/org/apache/pinot/broker/routing/instanceselector/InstanceSelectorTest.java b/pinot-broker/src/test/java/org/apache/pinot/broker/routing/instanceselector/InstanceSelectorTest.java
index 79fd518..1ef82a7 100644
--- a/pinot-broker/src/test/java/org/apache/pinot/broker/routing/instanceselector/InstanceSelectorTest.java
+++ b/pinot-broker/src/test/java/org/apache/pinot/broker/routing/instanceselector/InstanceSelectorTest.java
@@ -35,10 +35,10 @@ import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.testng.annotations.Test;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel.CONSUMING;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel.ERROR;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel.OFFLINE;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel.ONLINE;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel.CONSUMING;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel.ERROR;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel.OFFLINE;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel.ONLINE;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
import static org.testng.Assert.assertEquals;
diff --git a/pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentpreselector/SegmentPreSelectorTest.java b/pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentpreselector/SegmentPreSelectorTest.java
index d19def7..545845c 100644
--- a/pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentpreselector/SegmentPreSelectorTest.java
+++ b/pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentpreselector/SegmentPreSelectorTest.java
@@ -35,7 +35,7 @@ import org.apache.pinot.common.lineage.SegmentLineageUtils;
import org.testng.Assert;
import org.testng.annotations.Test;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel.ONLINE;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel.ONLINE;
public class SegmentPreSelectorTest {
diff --git a/pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentpruner/SegmentPrunerTest.java b/pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentpruner/SegmentPrunerTest.java
index a639709..e850b76 100644
--- a/pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentpruner/SegmentPrunerTest.java
+++ b/pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentpruner/SegmentPrunerTest.java
@@ -39,7 +39,6 @@ import org.apache.pinot.common.metadata.segment.OfflineSegmentZKMetadata;
import org.apache.pinot.common.metadata.segment.RealtimeSegmentZKMetadata;
import org.apache.pinot.common.metadata.segment.SegmentPartitionMetadata;
import org.apache.pinot.common.request.BrokerRequest;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.ZkStarter;
import org.apache.pinot.pql.parsers.Pql2Compiler;
import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
@@ -52,6 +51,7 @@ import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.data.DateTimeFormatSpec;
import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
import org.mockito.Mockito;
diff --git a/pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentselector/SegmentSelectorTest.java b/pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentselector/SegmentSelectorTest.java
index 051f8c6..9542b90 100644
--- a/pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentselector/SegmentSelectorTest.java
+++ b/pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentselector/SegmentSelectorTest.java
@@ -33,8 +33,8 @@ import org.testng.annotations.Test;
import static org.apache.pinot.broker.routing.segmentselector.RealtimeSegmentSelector.FORCE_HLC;
import static org.apache.pinot.broker.routing.segmentselector.RealtimeSegmentSelector.ROUTING_OPTIONS_KEY;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel.CONSUMING;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel.ONLINE;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel.CONSUMING;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel.ONLINE;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
import static org.testng.Assert.assertEqualsNoOrder;
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/assignment/InstanceAssignmentConfigUtils.java b/pinot-common/src/main/java/org/apache/pinot/common/assignment/InstanceAssignmentConfigUtils.java
index 710f805..8a8b08b 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/assignment/InstanceAssignmentConfigUtils.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/assignment/InstanceAssignmentConfigUtils.java
@@ -20,7 +20,6 @@ package org.apache.pinot.common.assignment;
import com.google.common.base.Preconditions;
import java.util.Map;
-import org.apache.pinot.common.utils.CommonConstants.Segment.AssignmentStrategy;
import org.apache.pinot.common.utils.config.TagNameUtils;
import org.apache.pinot.spi.config.table.ReplicaGroupStrategyConfig;
import org.apache.pinot.spi.config.table.SegmentsValidationAndRetentionConfig;
@@ -30,6 +29,7 @@ import org.apache.pinot.spi.config.table.assignment.InstanceAssignmentConfig;
import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
import org.apache.pinot.spi.config.table.assignment.InstanceReplicaGroupPartitionConfig;
import org.apache.pinot.spi.config.table.assignment.InstanceTagPoolConfig;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.AssignmentStrategy;
public class InstanceAssignmentConfigUtils {
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java b/pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java
index 7cce1eb..df4e58b 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java
@@ -31,16 +31,15 @@ import org.apache.pinot.common.metadata.instance.InstanceZKMetadata;
import org.apache.pinot.common.metadata.segment.LLCRealtimeSegmentZKMetadata;
import org.apache.pinot.common.metadata.segment.OfflineSegmentZKMetadata;
import org.apache.pinot.common.metadata.segment.RealtimeSegmentZKMetadata;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.SchemaUtils;
import org.apache.pinot.common.utils.SegmentName;
import org.apache.pinot.common.utils.StringUtil;
import org.apache.pinot.common.utils.config.TableConfigUtils;
-import org.apache.pinot.spi.config.BaseJsonConfig;
import org.apache.pinot.spi.config.ConfigUtils;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.apache.zookeeper.data.Stat;
import org.slf4j.Logger;
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/metadata/segment/OfflineSegmentZKMetadata.java b/pinot-common/src/main/java/org/apache/pinot/common/metadata/segment/OfflineSegmentZKMetadata.java
index 58f5b4a..c4da443 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/metadata/segment/OfflineSegmentZKMetadata.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/metadata/segment/OfflineSegmentZKMetadata.java
@@ -20,8 +20,8 @@ package org.apache.pinot.common.metadata.segment;
import java.util.Map;
import org.apache.helix.ZNRecord;
-import org.apache.pinot.common.utils.CommonConstants;
-import org.apache.pinot.common.utils.CommonConstants.Segment.SegmentType;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.SegmentType;
import static org.apache.pinot.spi.utils.EqualityUtils.hashCodeOf;
import static org.apache.pinot.spi.utils.EqualityUtils.isEqual;
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/metadata/segment/RealtimeSegmentZKMetadata.java b/pinot-common/src/main/java/org/apache/pinot/common/metadata/segment/RealtimeSegmentZKMetadata.java
index d88be18..100879e 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/metadata/segment/RealtimeSegmentZKMetadata.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/metadata/segment/RealtimeSegmentZKMetadata.java
@@ -20,9 +20,9 @@ package org.apache.pinot.common.metadata.segment;
import java.util.Map;
import org.apache.helix.ZNRecord;
-import org.apache.pinot.common.utils.CommonConstants;
-import org.apache.pinot.common.utils.CommonConstants.Segment.Realtime.Status;
-import org.apache.pinot.common.utils.CommonConstants.Segment.SegmentType;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.Realtime.Status;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.SegmentType;
import static org.apache.pinot.spi.utils.EqualityUtils.hashCodeOf;
import static org.apache.pinot.spi.utils.EqualityUtils.isEqual;
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/metadata/segment/SegmentZKMetadata.java b/pinot-common/src/main/java/org/apache/pinot/common/metadata/segment/SegmentZKMetadata.java
index 8fa7a9a..42e9c01 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/metadata/segment/SegmentZKMetadata.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/metadata/segment/SegmentZKMetadata.java
@@ -26,8 +26,8 @@ import java.util.Objects;
import java.util.concurrent.TimeUnit;
import org.apache.helix.ZNRecord;
import org.apache.pinot.common.metadata.ZKMetadata;
-import org.apache.pinot.common.utils.CommonConstants.Segment;
-import org.apache.pinot.common.utils.CommonConstants.Segment.SegmentType;
+import org.apache.pinot.spi.utils.CommonConstants.Segment;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.SegmentType;
import org.apache.pinot.spi.utils.JsonUtils;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.joda.time.Duration;
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMetrics.java b/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMetrics.java
index f79924a..850d7f3 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMetrics.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMetrics.java
@@ -22,8 +22,8 @@ import java.util.Collection;
import java.util.Collections;
import org.apache.pinot.spi.metrics.PinotMetricsRegistry;
-import static org.apache.pinot.common.utils.CommonConstants.Broker.DEFAULT_ENABLE_TABLE_LEVEL_METRICS;
-import static org.apache.pinot.common.utils.CommonConstants.Broker.DEFAULT_METRICS_NAME_PREFIX;
+import static org.apache.pinot.spi.utils.CommonConstants.Broker.DEFAULT_ENABLE_TABLE_LEVEL_METRICS;
+import static org.apache.pinot.spi.utils.CommonConstants.Broker.DEFAULT_METRICS_NAME_PREFIX;
/**
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerMetrics.java b/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerMetrics.java
index 8a30c19..6735de2 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerMetrics.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerMetrics.java
@@ -20,7 +20,7 @@ package org.apache.pinot.common.metrics;
import org.apache.pinot.spi.metrics.PinotMetricsRegistry;
-import static org.apache.pinot.common.utils.CommonConstants.Controller.DEFAULT_METRICS_PREFIX;
+import static org.apache.pinot.spi.utils.CommonConstants.Controller.DEFAULT_METRICS_PREFIX;
/**
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerMetrics.java b/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerMetrics.java
index 77a1a99..2784c4f 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerMetrics.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerMetrics.java
@@ -22,8 +22,8 @@ import java.util.Collection;
import java.util.Collections;
import org.apache.pinot.spi.metrics.PinotMetricsRegistry;
-import static org.apache.pinot.common.utils.CommonConstants.Server.DEFAULT_ENABLE_TABLE_LEVEL_METRICS;
-import static org.apache.pinot.common.utils.CommonConstants.Server.DEFAULT_METRICS_PREFIX;
+import static org.apache.pinot.spi.utils.CommonConstants.Server.DEFAULT_ENABLE_TABLE_LEVEL_METRICS;
+import static org.apache.pinot.spi.utils.CommonConstants.Server.DEFAULT_METRICS_PREFIX;
/**
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/request/context/ExpressionContext.java b/pinot-common/src/main/java/org/apache/pinot/common/request/context/ExpressionContext.java
new file mode 100644
index 0000000..35f87ab
--- /dev/null
+++ b/pinot-common/src/main/java/org/apache/pinot/common/request/context/ExpressionContext.java
@@ -0,0 +1,118 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.request.context;
+
+import java.util.Objects;
+import java.util.Set;
+
+
+/**
+ * The {@code ExpressionContext} class represents an expression in the query.
+ * <p>The expression can be a LITERAL (e.g. 1, "abc"), an IDENTIFIER (e.g. memberId, timestamp), or a FUNCTION (e.g.
+ * SUM(price), ADD(foo, bar)).
+ * <p>Currently the query engine processes all literals as strings, so we store literals in string format (1 is stored
+ * as "1").
+ */
+public class ExpressionContext {
+ public enum Type {
+ LITERAL, IDENTIFIER, FUNCTION
+ }
+
+ private final Type _type;
+ private final String _value;
+ private final FunctionContext _function;
+
+ public static ExpressionContext forLiteral(String literal) {
+ return new ExpressionContext(Type.LITERAL, literal, null);
+ }
+
+ public static ExpressionContext forIdentifier(String identifier) {
+ return new ExpressionContext(Type.IDENTIFIER, identifier, null);
+ }
+
+ public static ExpressionContext forFunction(FunctionContext function) {
+ return new ExpressionContext(Type.FUNCTION, null, function);
+ }
+
+ private ExpressionContext(Type type, String value, FunctionContext function) {
+ _type = type;
+ _value = value;
+ _function = function;
+ }
+
+ public Type getType() {
+ return _type;
+ }
+
+ public String getLiteral() {
+ return _value;
+ }
+
+ public String getIdentifier() {
+ return _value;
+ }
+
+ public FunctionContext getFunction() {
+ return _function;
+ }
+
+ /**
+ * Adds the columns (IDENTIFIER expressions) in the expression to the given set.
+ */
+ public void getColumns(Set<String> columns) {
+ if (_type == Type.IDENTIFIER) {
+ if (!_value.equals("*")) {
+ columns.add(_value);
+ }
+ } else if (_type == Type.FUNCTION) {
+ _function.getColumns(columns);
+ }
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof ExpressionContext)) {
+ return false;
+ }
+ ExpressionContext that = (ExpressionContext) o;
+ return _type == that._type && Objects.equals(_value, that._value) && Objects.equals(_function, that._function);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(_type, _value, _function);
+ }
+
+ @Override
+ public String toString() {
+ switch (_type) {
+ case LITERAL:
+ return '\'' + _value + '\'';
+ case IDENTIFIER:
+ return _value;
+ case FUNCTION:
+ return _function.toString();
+ default:
+ throw new IllegalStateException();
+ }
+ }
+}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/request/context/FilterContext.java b/pinot-common/src/main/java/org/apache/pinot/common/request/context/FilterContext.java
new file mode 100644
index 0000000..90e12d2
--- /dev/null
+++ b/pinot-common/src/main/java/org/apache/pinot/common/request/context/FilterContext.java
@@ -0,0 +1,116 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.request.context;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import org.apache.pinot.common.request.context.predicate.Predicate;
+
+
+/**
+ * The {@code FilterContext} class encapsulates the information of a filter in the query. Both WHERE clause and HAVING
+ * clause are modeled as a filter.
+ */
+public class FilterContext {
+ public enum Type {
+ AND, OR, PREDICATE
+ }
+
+ private final Type _type;
+
+ // For AND and OR
+ private final List<FilterContext> _children;
+
+ // For Predicate
+ private final Predicate _predicate;
+
+ public FilterContext(Type type, List<FilterContext> children, Predicate predicate) {
+ _type = type;
+ _children = children;
+ _predicate = predicate;
+ }
+
+ public Type getType() {
+ return _type;
+ }
+
+ public List<FilterContext> getChildren() {
+ return _children;
+ }
+
+ public Predicate getPredicate() {
+ return _predicate;
+ }
+
+ /**
+ * Adds the columns (IDENTIFIER expressions) in the filter to the given set.
+ */
+ public void getColumns(Set<String> columns) {
+ if (_children != null) {
+ for (FilterContext child : _children) {
+ child.getColumns(columns);
+ }
+ } else {
+ _predicate.getLhs().getColumns(columns);
+ }
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof FilterContext)) {
+ return false;
+ }
+ FilterContext that = (FilterContext) o;
+ return _type == that._type && Objects.equals(_children, that._children) && Objects
+ .equals(_predicate, that._predicate);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(_type, _children, _predicate);
+ }
+
+ @Override
+ public String toString() {
+ switch (_type) {
+ case AND:
+ StringBuilder stringBuilder = new StringBuilder().append('(').append(_children.get(0).toString());
+ int numChildren = _children.size();
+ for (int i = 1; i < numChildren; i++) {
+ stringBuilder.append(" AND ").append(_children.get(i).toString());
+ }
+ return stringBuilder.append(')').toString();
+ case OR:
+ stringBuilder = new StringBuilder().append('(').append(_children.get(0).toString());
+ numChildren = _children.size();
+ for (int i = 1; i < numChildren; i++) {
+ stringBuilder.append(" OR ").append(_children.get(i).toString());
+ }
+ return stringBuilder.append(')').toString();
+ case PREDICATE:
+ return _predicate.toString();
+ default:
+ throw new IllegalStateException();
+ }
+ }
+}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/request/context/FunctionContext.java b/pinot-common/src/main/java/org/apache/pinot/common/request/context/FunctionContext.java
new file mode 100644
index 0000000..066c315
--- /dev/null
+++ b/pinot-common/src/main/java/org/apache/pinot/common/request/context/FunctionContext.java
@@ -0,0 +1,102 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.request.context;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+
+
+/**
+ * The {@code FunctionContext} class represents the function in the expression.
+ * <p>Pinot currently supports 2 types of functions: Aggregation (e.g. SUM, MAX) and Transform (e.g. ADD, SUB).
+ */
+public class FunctionContext {
+ public enum Type {
+ AGGREGATION, TRANSFORM
+ }
+
+ private final Type _type;
+ private String _functionName;
+ private final List<ExpressionContext> _arguments;
+
+ public FunctionContext(Type type, String functionName, List<ExpressionContext> arguments) {
+ _type = type;
+ // NOTE: Standardize the function name to lower case
+ _functionName = functionName.toLowerCase();
+ _arguments = arguments;
+ }
+
+ public Type getType() {
+ return _type;
+ }
+
+ public void setFunctionName(String functionName) {
+ // NOTE: Standardize the function name to lower case
+ _functionName = functionName.toLowerCase();
+ }
+
+ public String getFunctionName() {
+ return _functionName;
+ }
+
+ public List<ExpressionContext> getArguments() {
+ return _arguments;
+ }
+
+ /**
+ * Adds the columns (IDENTIFIER expressions) in the function to the given set.
+ */
+ public void getColumns(Set<String> columns) {
+ for (ExpressionContext argument : _arguments) {
+ argument.getColumns(columns);
+ }
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof FunctionContext)) {
+ return false;
+ }
+ FunctionContext that = (FunctionContext) o;
+ return _type == that._type && Objects.equals(_functionName, that._functionName) && Objects
+ .equals(_arguments, that._arguments);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(_type, _functionName, _arguments);
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder stringBuilder = new StringBuilder(_functionName).append('(');
+ int numArguments = _arguments.size();
+ if (numArguments > 0) {
+ stringBuilder.append(_arguments.get(0).toString());
+ for (int i = 1; i < numArguments; i++) {
+ stringBuilder.append(',').append(_arguments.get(i).toString());
+ }
+ }
+ return stringBuilder.append(')').toString();
+ }
+}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/request/context/OrderByExpressionContext.java b/pinot-common/src/main/java/org/apache/pinot/common/request/context/OrderByExpressionContext.java
new file mode 100644
index 0000000..f47ed35
--- /dev/null
+++ b/pinot-common/src/main/java/org/apache/pinot/common/request/context/OrderByExpressionContext.java
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.request.context;
+
+import java.util.Objects;
+import java.util.Set;
+
+
+/**
+ * The {@code OrderByExpressionContext} class represents an expression in the ORDER-BY clause. It encapsulates an
+ * expression and the expected ordering of the expression.
+ */
+public class OrderByExpressionContext {
+ private final ExpressionContext _expression;
+ private final boolean _isAsc;
+
+ public OrderByExpressionContext(ExpressionContext expression, boolean isAsc) {
+ _expression = expression;
+ _isAsc = isAsc;
+ }
+
+ public ExpressionContext getExpression() {
+ return _expression;
+ }
+
+ public boolean isAsc() {
+ return _isAsc;
+ }
+
+ /**
+ * Adds the columns (IDENTIFIER expressions) in the order-by expression to the given set.
+ */
+ public void getColumns(Set<String> columns) {
+ _expression.getColumns(columns);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof OrderByExpressionContext)) {
+ return false;
+ }
+ OrderByExpressionContext that = (OrderByExpressionContext) o;
+ return _isAsc == that._isAsc && Objects.equals(_expression, that._expression);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(_expression, _isAsc);
+ }
+
+ @Override
+ public String toString() {
+ return _expression.toString() + (_isAsc ? " ASC" : " DESC");
+ }
+}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/request/context/RequestContextConvertUtils.java b/pinot-common/src/main/java/org/apache/pinot/common/request/context/RequestContextConvertUtils.java
new file mode 100644
index 0000000..8e475358
--- /dev/null
+++ b/pinot-common/src/main/java/org/apache/pinot/common/request/context/RequestContextConvertUtils.java
@@ -0,0 +1,302 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.request.context;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import org.apache.pinot.common.function.FunctionDefinitionRegistry;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.ExpressionType;
+import org.apache.pinot.common.request.FilterOperator;
+import org.apache.pinot.common.request.Function;
+import org.apache.pinot.common.request.context.predicate.EqPredicate;
+import org.apache.pinot.common.request.context.predicate.InPredicate;
+import org.apache.pinot.common.request.context.predicate.IsNotNullPredicate;
+import org.apache.pinot.common.request.context.predicate.IsNullPredicate;
+import org.apache.pinot.common.request.context.predicate.JsonMatchPredicate;
+import org.apache.pinot.common.request.context.predicate.NotEqPredicate;
+import org.apache.pinot.common.request.context.predicate.NotInPredicate;
+import org.apache.pinot.common.request.context.predicate.RangePredicate;
+import org.apache.pinot.common.request.context.predicate.RegexpLikePredicate;
+import org.apache.pinot.common.request.context.predicate.TextMatchPredicate;
+import org.apache.pinot.common.utils.request.FilterQueryTree;
+import org.apache.pinot.pql.parsers.Pql2Compiler;
+import org.apache.pinot.pql.parsers.pql2.ast.AstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.pql.parsers.pql2.ast.FunctionCallAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.IdentifierAstNode;
+import org.apache.pinot.pql.parsers.pql2.ast.LiteralAstNode;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
+
+
+public class RequestContextConvertUtils {
+
+ private RequestContextConvertUtils() {
+
+ }
+
+ private static final Pql2Compiler PQL_COMPILER = new Pql2Compiler();
+ private static final CalciteSqlCompiler SQL_COMPILER = new CalciteSqlCompiler();
+
+ /**
+ * Converts the given Thrift {@link Expression} into an {@link ExpressionContext}.
+ */
+ public static ExpressionContext getExpression(Expression thriftExpression) {
+ switch (thriftExpression.getType()) {
+ case LITERAL:
+ return ExpressionContext.forLiteral(thriftExpression.getLiteral().getFieldValue().toString());
+ case IDENTIFIER:
+ return ExpressionContext.forIdentifier(thriftExpression.getIdentifier().getName());
+ case FUNCTION:
+ return ExpressionContext.forFunction(getFunction(thriftExpression.getFunctionCall()));
+ default:
+ throw new IllegalStateException();
+ }
+ }
+
+ /**
+ * Converts the given string representation of the expression into an {@link ExpressionContext}.
+ */
+ public static ExpressionContext getExpression(String stringExpression) {
+ if (stringExpression.equals("*")) {
+ // For 'SELECT *' and 'SELECT COUNT(*)'
+ return ExpressionContext.forIdentifier("*");
+ } else {
+ return getExpression(PQL_COMPILER.parseToAstNode(stringExpression));
+ }
+ }
+
+ /**
+ * Converts the given {@link AstNode} into an {@link ExpressionContext}.
+ */
+ public static ExpressionContext getExpression(AstNode astNode) {
+ if (astNode instanceof IdentifierAstNode) {
+ return ExpressionContext.forIdentifier(((IdentifierAstNode) astNode).getName());
+ }
+ if (astNode instanceof FunctionCallAstNode) {
+ return ExpressionContext.forFunction(getFunction((FunctionCallAstNode) astNode));
+ }
+ if (astNode instanceof LiteralAstNode) {
+ return ExpressionContext.forLiteral(((LiteralAstNode) astNode).getValueAsString());
+ }
+ throw new IllegalStateException();
+ }
+
+ /**
+ * Converts the given Thrift {@link Function} into a {@link FunctionContext}.
+ */
+ public static FunctionContext getFunction(Function thriftFunction) {
+ String functionName = thriftFunction.getOperator();
+ if (functionName.equalsIgnoreCase(AggregationFunctionType.COUNT.getName())) {
+ // NOTE: COUNT always take one single argument "*"
+ return new FunctionContext(FunctionContext.Type.AGGREGATION, AggregationFunctionType.COUNT.getName(),
+ Collections.singletonList(ExpressionContext.forIdentifier("*")));
+ }
+ FunctionContext.Type functionType =
+ FunctionDefinitionRegistry.isAggFunc(functionName) ? FunctionContext.Type.AGGREGATION
+ : FunctionContext.Type.TRANSFORM;
+ List<Expression> operands = thriftFunction.getOperands();
+ if (operands != null) {
+ List<ExpressionContext> arguments = new ArrayList<>(operands.size());
+ for (Expression operand : operands) {
+ arguments.add(getExpression(operand));
+ }
+ return new FunctionContext(functionType, functionName, arguments);
+ } else {
+ return new FunctionContext(functionType, functionName, Collections.emptyList());
+ }
+ }
+
+ /**
+ * Converts the given {@link FunctionCallAstNode} into a {@link FunctionContext}.
+ */
+ public static FunctionContext getFunction(FunctionCallAstNode astNode) {
+ String functionName = astNode.getName();
+ if (functionName.equalsIgnoreCase(AggregationFunctionType.COUNT.getName())) {
+ // NOTE: COUNT always take one single argument "*"
+ return new FunctionContext(FunctionContext.Type.AGGREGATION, AggregationFunctionType.COUNT.getName(),
+ Collections.singletonList(ExpressionContext.forIdentifier("*")));
+ }
+ FunctionContext.Type functionType =
+ FunctionDefinitionRegistry.isAggFunc(functionName) ? FunctionContext.Type.AGGREGATION
+ : FunctionContext.Type.TRANSFORM;
+ List<? extends AstNode> children = astNode.getChildren();
+ if (children != null) {
+ List<ExpressionContext> arguments = new ArrayList<>(children.size());
+ for (AstNode child : children) {
+ arguments.add(getExpression(child));
+ }
+ return new FunctionContext(functionType, functionName, arguments);
+ } else {
+ return new FunctionContext(functionType, functionName, Collections.emptyList());
+ }
+ }
+
+ /**
+ * Converts the given Thrift {@link Expression} into a {@link FilterContext}.
+ * <p>NOTE: Currently the query engine only accepts string literals as the right-hand side of the predicate, so we
+ * always convert the right-hand side expressions into strings.
+ */
+ public static FilterContext getFilter(Expression thriftExpression) {
+ Function thriftFunction = thriftExpression.getFunctionCall();
+ FilterKind filterKind = FilterKind.valueOf(thriftFunction.getOperator().toUpperCase());
+ List<Expression> operands = thriftFunction.getOperands();
+ int numOperands = operands.size();
+ switch (filterKind) {
+ case AND:
+ List<FilterContext> children = new ArrayList<>(numOperands);
+ for (Expression operand : operands) {
+ children.add(getFilter(operand));
+ }
+ return new FilterContext(FilterContext.Type.AND, children, null);
+ case OR:
+ children = new ArrayList<>(numOperands);
+ for (Expression operand : operands) {
+ children.add(getFilter(operand));
+ }
+ return new FilterContext(FilterContext.Type.OR, children, null);
+ case EQUALS:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new EqPredicate(getExpression(operands.get(0)), getStringValue(operands.get(1))));
+ case NOT_EQUALS:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new NotEqPredicate(getExpression(operands.get(0)), getStringValue(operands.get(1))));
+ case IN:
+ List<String> values = new ArrayList<>(numOperands - 1);
+ for (int i = 1; i < numOperands; i++) {
+ values.add(getStringValue(operands.get(i)));
+ }
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new InPredicate(getExpression(operands.get(0)), values));
+ case NOT_IN:
+ values = new ArrayList<>(numOperands - 1);
+ for (int i = 1; i < numOperands; i++) {
+ values.add(getStringValue(operands.get(i)));
+ }
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new NotInPredicate(getExpression(operands.get(0)), values));
+ case GREATER_THAN:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new RangePredicate(getExpression(operands.get(0)), false, getStringValue(operands.get(1)), false,
+ RangePredicate.UNBOUNDED));
+ case GREATER_THAN_OR_EQUAL:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new RangePredicate(getExpression(operands.get(0)), true, getStringValue(operands.get(1)), false,
+ RangePredicate.UNBOUNDED));
+ case LESS_THAN:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new RangePredicate(getExpression(operands.get(0)), false, RangePredicate.UNBOUNDED, false,
+ getStringValue(operands.get(1))));
+ case LESS_THAN_OR_EQUAL:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new RangePredicate(getExpression(operands.get(0)), false, RangePredicate.UNBOUNDED, true,
+ getStringValue(operands.get(1))));
+ case BETWEEN:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new RangePredicate(getExpression(operands.get(0)), true, getStringValue(operands.get(1)), true,
+ getStringValue(operands.get(2))));
+ case RANGE:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new RangePredicate(getExpression(operands.get(0)), getStringValue(operands.get(1))));
+ case REGEXP_LIKE:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new RegexpLikePredicate(getExpression(operands.get(0)), getStringValue(operands.get(1))));
+ case TEXT_MATCH:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new TextMatchPredicate(getExpression(operands.get(0)), getStringValue(operands.get(1))));
+ case JSON_MATCH:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new JsonMatchPredicate(getExpression(operands.get(0)), getStringValue(operands.get(1))));
+ case IS_NULL:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new IsNullPredicate(getExpression(operands.get(0))));
+ case IS_NOT_NULL:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new IsNotNullPredicate(getExpression(operands.get(0))));
+ default:
+ throw new IllegalStateException();
+ }
+ }
+
+ private static String getStringValue(Expression thriftExpression) {
+ if (thriftExpression.getType() != ExpressionType.LITERAL) {
+ throw new BadQueryRequestException(
+ "Pinot does not support column or function on the right-hand side of the predicate");
+ }
+ return thriftExpression.getLiteral().getFieldValue().toString();
+ }
+
+ /**
+ * Converts the given {@link FilterQueryTree} into a {@link FilterContext}.
+ */
+ public static FilterContext getFilter(FilterQueryTree node) {
+ FilterOperator filterOperator = node.getOperator();
+ switch (filterOperator) {
+ case AND:
+ List<FilterQueryTree> childNodes = node.getChildren();
+ List<FilterContext> children = new ArrayList<>(childNodes.size());
+ for (FilterQueryTree childNode : childNodes) {
+ children.add(getFilter(childNode));
+ }
+ return new FilterContext(FilterContext.Type.AND, children, null);
+ case OR:
+ childNodes = node.getChildren();
+ children = new ArrayList<>(childNodes.size());
+ for (FilterQueryTree childNode : childNodes) {
+ children.add(getFilter(childNode));
+ }
+ return new FilterContext(FilterContext.Type.OR, children, null);
+ case EQUALITY:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new EqPredicate(getExpression(node.getColumn()), node.getValue().get(0)));
+ case NOT:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new NotEqPredicate(getExpression(node.getColumn()), node.getValue().get(0)));
+ case IN:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new InPredicate(getExpression(node.getColumn()), node.getValue()));
+ case NOT_IN:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new NotInPredicate(getExpression(node.getColumn()), node.getValue()));
+ case RANGE:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new RangePredicate(getExpression(node.getColumn()), node.getValue().get(0)));
+ case REGEXP_LIKE:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new RegexpLikePredicate(getExpression(node.getColumn()), node.getValue().get(0)));
+ case TEXT_MATCH:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new TextMatchPredicate(getExpression(node.getColumn()), node.getValue().get(0)));
+ case JSON_MATCH:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new JsonMatchPredicate(getExpression(node.getColumn()), node.getValue().get(0)));
+ case IS_NULL:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new IsNullPredicate(getExpression(node.getColumn())));
+ case IS_NOT_NULL:
+ return new FilterContext(FilterContext.Type.PREDICATE, null,
+ new IsNotNullPredicate(getExpression(node.getColumn())));
+ default:
+ throw new IllegalStateException();
+ }
+ }
+}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/EqPredicate.java b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/EqPredicate.java
new file mode 100644
index 0000000..dab9ee0
--- /dev/null
+++ b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/EqPredicate.java
@@ -0,0 +1,72 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.request.context.predicate;
+
+import java.util.Objects;
+import org.apache.pinot.common.request.context.ExpressionContext;
+
+
+/**
+ * Predicate for EQ.
+ */
+public class EqPredicate implements Predicate {
+ private final ExpressionContext _lhs;
+ private final String _value;
+
+ public EqPredicate(ExpressionContext lhs, String value) {
+ _lhs = lhs;
+ _value = value;
+ }
+
+ @Override
+ public Type getType() {
+ return Type.EQ;
+ }
+
+ @Override
+ public ExpressionContext getLhs() {
+ return _lhs;
+ }
+
+ public String getValue() {
+ return _value;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof EqPredicate)) {
+ return false;
+ }
+ EqPredicate that = (EqPredicate) o;
+ return Objects.equals(_lhs, that._lhs) && Objects.equals(_value, that._value);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(_lhs, _value);
+ }
+
+ @Override
+ public String toString() {
+ return _lhs + " = '" + _value + '\'';
+ }
+}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/InPredicate.java b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/InPredicate.java
new file mode 100644
index 0000000..ce9c75b
--- /dev/null
+++ b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/InPredicate.java
@@ -0,0 +1,79 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.request.context.predicate;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.pinot.common.request.context.ExpressionContext;
+
+
+/**
+ * Predicate for IN.
+ */
+public class InPredicate implements Predicate {
+ private final ExpressionContext _lhs;
+ private final List<String> _values;
+
+ public InPredicate(ExpressionContext lhs, List<String> values) {
+ _lhs = lhs;
+ _values = values;
+ }
+
+ @Override
+ public Type getType() {
+ return Type.IN;
+ }
+
+ @Override
+ public ExpressionContext getLhs() {
+ return _lhs;
+ }
+
+ public List<String> getValues() {
+ return _values;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof InPredicate)) {
+ return false;
+ }
+ InPredicate that = (InPredicate) o;
+ return Objects.equals(_lhs, that._lhs) && Objects.equals(_values, that._values);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(_lhs, _values);
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder stringBuilder =
+ new StringBuilder(_lhs.toString()).append(" IN ('").append(_values.get(0)).append('\'');
+ int numValues = _values.size();
+ for (int i = 1; i < numValues; i++) {
+ stringBuilder.append(",'").append(_values.get(i)).append('\'');
+ }
+ return stringBuilder.append(')').toString();
+ }
+}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/IsNotNullPredicate.java b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/IsNotNullPredicate.java
new file mode 100644
index 0000000..17567ad
--- /dev/null
+++ b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/IsNotNullPredicate.java
@@ -0,0 +1,66 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.request.context.predicate;
+
+import java.util.Objects;
+import org.apache.pinot.common.request.context.ExpressionContext;
+
+
+/**
+ * Predicate for IS_NOT_NULL.
+ */
+public class IsNotNullPredicate implements Predicate {
+ private final ExpressionContext _lhs;
+
+ public IsNotNullPredicate(ExpressionContext lhs) {
+ _lhs = lhs;
+ }
+
+ @Override
+ public Type getType() {
+ return Type.IS_NOT_NULL;
+ }
+
+ @Override
+ public ExpressionContext getLhs() {
+ return _lhs;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof IsNotNullPredicate)) {
+ return false;
+ }
+ IsNotNullPredicate that = (IsNotNullPredicate) o;
+ return Objects.equals(_lhs, that._lhs);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(_lhs);
+ }
+
+ @Override
+ public String toString() {
+ return _lhs + " IS NOT NULL";
+ }
+}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/IsNullPredicate.java b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/IsNullPredicate.java
new file mode 100644
index 0000000..cd8e8ab
--- /dev/null
+++ b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/IsNullPredicate.java
@@ -0,0 +1,66 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.request.context.predicate;
+
+import java.util.Objects;
+import org.apache.pinot.common.request.context.ExpressionContext;
+
+
+/**
+ * Predicate for IS_NULL.
+ */
+public class IsNullPredicate implements Predicate {
+ private final ExpressionContext _lhs;
+
+ public IsNullPredicate(ExpressionContext lhs) {
+ _lhs = lhs;
+ }
+
+ @Override
+ public Type getType() {
+ return Type.IS_NULL;
+ }
+
+ @Override
+ public ExpressionContext getLhs() {
+ return _lhs;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof IsNullPredicate)) {
+ return false;
+ }
+ IsNullPredicate that = (IsNullPredicate) o;
+ return Objects.equals(_lhs, that._lhs);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(_lhs);
+ }
+
+ @Override
+ public String toString() {
+ return _lhs + " IS NULL";
+ }
+}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/JsonMatchPredicate.java b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/JsonMatchPredicate.java
new file mode 100644
index 0000000..a08a7dc
--- /dev/null
+++ b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/JsonMatchPredicate.java
@@ -0,0 +1,72 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.request.context.predicate;
+
+import java.util.Objects;
+import org.apache.pinot.common.request.context.ExpressionContext;
+
+
+/**
+ * Predicate for JSON_MATCH.
+ */
+public class JsonMatchPredicate implements Predicate {
+ private final ExpressionContext _lhs;
+ private final String _value;
+
+ public JsonMatchPredicate(ExpressionContext lhs, String value) {
+ _lhs = lhs;
+ _value = value;
+ }
+
+ @Override
+ public Type getType() {
+ return Type.JSON_MATCH;
+ }
+
+ @Override
+ public ExpressionContext getLhs() {
+ return _lhs;
+ }
+
+ public String getValue() {
+ return _value;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof JsonMatchPredicate)) {
+ return false;
+ }
+ JsonMatchPredicate that = (JsonMatchPredicate) o;
+ return Objects.equals(_lhs, that._lhs) && Objects.equals(_value, that._value);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(_lhs, _value);
+ }
+
+ @Override
+ public String toString() {
+ return "json_match(" + _lhs + ",'" + _value + "')";
+ }
+}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/NotEqPredicate.java b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/NotEqPredicate.java
new file mode 100644
index 0000000..b289666
--- /dev/null
+++ b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/NotEqPredicate.java
@@ -0,0 +1,72 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.request.context.predicate;
+
+import java.util.Objects;
+import org.apache.pinot.common.request.context.ExpressionContext;
+
+
+/**
+ * Predicate for NOT_EQ.
+ */
+public class NotEqPredicate implements Predicate {
+ private final ExpressionContext _lhs;
+ private final String _value;
+
+ public NotEqPredicate(ExpressionContext lhs, String value) {
+ _lhs = lhs;
+ _value = value;
+ }
+
+ @Override
+ public Type getType() {
+ return Type.NOT_EQ;
+ }
+
+ @Override
+ public ExpressionContext getLhs() {
+ return _lhs;
+ }
+
+ public String getValue() {
+ return _value;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof NotEqPredicate)) {
+ return false;
+ }
+ NotEqPredicate that = (NotEqPredicate) o;
+ return Objects.equals(_lhs, that._lhs) && Objects.equals(_value, that._value);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(_lhs, _value);
+ }
+
+ @Override
+ public String toString() {
+ return _lhs + " != '" + _value + '\'';
+ }
+}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/NotInPredicate.java b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/NotInPredicate.java
new file mode 100644
index 0000000..58a2b51
--- /dev/null
+++ b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/NotInPredicate.java
@@ -0,0 +1,79 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.request.context.predicate;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.pinot.common.request.context.ExpressionContext;
+
+
+/**
+ * Predicate for NOT_IN.
+ */
+public class NotInPredicate implements Predicate {
+ private final ExpressionContext _lhs;
+ private final List<String> _values;
+
+ public NotInPredicate(ExpressionContext lhs, List<String> values) {
+ _lhs = lhs;
+ _values = values;
+ }
+
+ @Override
+ public Type getType() {
+ return Type.NOT_IN;
+ }
+
+ @Override
+ public ExpressionContext getLhs() {
+ return _lhs;
+ }
+
+ public List<String> getValues() {
+ return _values;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof NotInPredicate)) {
+ return false;
+ }
+ NotInPredicate that = (NotInPredicate) o;
+ return Objects.equals(_lhs, that._lhs) && Objects.equals(_values, that._values);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(_lhs, _values);
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder stringBuilder =
+ new StringBuilder(_lhs.toString()).append(" NOT IN ('").append(_values.get(0)).append('\'');
+ int numValues = _values.size();
+ for (int i = 1; i < numValues; i++) {
+ stringBuilder.append(",'").append(_values.get(i)).append('\'');
+ }
+ return stringBuilder.append(')').toString();
+ }
+}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/Predicate.java b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/Predicate.java
new file mode 100644
index 0000000..0444975
--- /dev/null
+++ b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/Predicate.java
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.request.context.predicate;
+
+import org.apache.pinot.common.request.context.ExpressionContext;
+
+
+/**
+ * The {@code Predicate} class represents the predicate in the filter.
+ * <p>Currently the query engine only accepts string literals as the right-hand side of the predicate, so we store the
+ * right-hand side of the predicate as string or list of strings.
+ */
+public interface Predicate {
+ enum Type {
+ EQ, NOT_EQ, IN, NOT_IN, RANGE, REGEXP_LIKE, TEXT_MATCH, JSON_MATCH, IS_NULL, IS_NOT_NULL;
+
+ public boolean isExclusive() {
+ return this == NOT_EQ || this == NOT_IN || this == IS_NOT_NULL;
+ }
+ }
+
+ /**
+ * Returns the type of the predicate.
+ */
+ Type getType();
+
+ /**
+ * Returns the left-hand side expression of the predicate.
+ */
+ ExpressionContext getLhs();
+}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/RangePredicate.java b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/RangePredicate.java
new file mode 100644
index 0000000..0f22efd
--- /dev/null
+++ b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/RangePredicate.java
@@ -0,0 +1,141 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.request.context.predicate;
+
+import java.util.Objects;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.spi.utils.CommonConstants.Query.Range;
+
+
+/**
+ * Predicate for RANGE.
+ * <p>Pinot uses RANGE to represent '>', '>=', '<', '<=', BETWEEN so that intersection of multiple ranges can be merged.
+ */
+public class RangePredicate implements Predicate {
+ public static final char DELIMITER = Range.DELIMITER;
+ public static final char LOWER_EXCLUSIVE = Range.LOWER_EXCLUSIVE;
+ public static final char LOWER_INCLUSIVE = Range.LOWER_INCLUSIVE;
+ public static final char UPPER_EXCLUSIVE = Range.UPPER_EXCLUSIVE;
+ public static final char UPPER_INCLUSIVE = Range.UPPER_INCLUSIVE;
+ public static final String UNBOUNDED = Range.UNBOUNDED;
+
+ // For backward-compatibility
+ private static final String LEGACY_DELIMITER = "\t\t";
+
+ private final ExpressionContext _lhs;
+ private final boolean _lowerInclusive;
+ private final String _lowerBound;
+ private final boolean _upperInclusive;
+ private final String _upperBound;
+
+ /**
+ * The range is formatted as 5 parts:
+ * <ul>
+ * <li>Lower inclusive '[' or exclusive '('</li>
+ * <li>Lower bound ('*' for unbounded)</li>
+ * <li>Delimiter ('\0')</li>
+ * <li>Upper bound ('*' for unbounded)</li>
+ * <li>Upper inclusive ']' or exclusive ')'</li>
+ * </ul>
+ */
+ public RangePredicate(ExpressionContext lhs, String range) {
+ _lhs = lhs;
+ String[] split = StringUtils.split(range, DELIMITER);
+ if (split.length != 2) {
+ split = StringUtils.split(range, LEGACY_DELIMITER);
+ }
+ String lower = split[0];
+ String upper = split[1];
+ _lowerInclusive = lower.charAt(0) == LOWER_INCLUSIVE;
+ _lowerBound = lower.substring(1);
+ int upperLength = upper.length();
+ _upperInclusive = upper.charAt(upperLength - 1) == UPPER_INCLUSIVE;
+ _upperBound = upper.substring(0, upperLength - 1);
+ }
+
+ public RangePredicate(ExpressionContext lhs, boolean lowerInclusive, String lowerBound, boolean upperInclusive,
+ String upperBound) {
+ _lhs = lhs;
+ _lowerInclusive = lowerInclusive;
+ _lowerBound = lowerBound;
+ _upperInclusive = upperInclusive;
+ _upperBound = upperBound;
+ }
+
+ @Override
+ public Type getType() {
+ return Type.RANGE;
+ }
+
+ @Override
+ public ExpressionContext getLhs() {
+ return _lhs;
+ }
+
+ public boolean isLowerInclusive() {
+ return _lowerInclusive;
+ }
+
+ public String getLowerBound() {
+ return _lowerBound;
+ }
+
+ public boolean isUpperInclusive() {
+ return _upperInclusive;
+ }
+
+ public String getUpperBound() {
+ return _upperBound;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof RangePredicate)) {
+ return false;
+ }
+ RangePredicate that = (RangePredicate) o;
+ return _lowerInclusive == that._lowerInclusive && _upperInclusive == that._upperInclusive && Objects
+ .equals(_lhs, that._lhs) && Objects.equals(_lowerBound, that._lowerBound) && Objects
+ .equals(_upperBound, that._upperBound);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(_lhs, _lowerInclusive, _lowerBound, _upperInclusive, _upperBound);
+ }
+
+ @Override
+ public String toString() {
+ if (_lowerBound.equals(UNBOUNDED)) {
+ return _lhs + (_upperInclusive ? " <= '" : " < '") + _upperBound + '\'';
+ }
+ if (_upperBound.equals(UNBOUNDED)) {
+ return _lhs + (_lowerInclusive ? " >= '" : " > '") + _lowerBound + '\'';
+ }
+ if (_lowerInclusive && _upperInclusive) {
+ return _lhs + " BETWEEN '" + _lowerBound + "' AND '" + _upperBound + '\'';
+ }
+ return "(" + _lhs + (_lowerInclusive ? " >= '" : " > '") + _lowerBound + "' AND " + _lhs + (_upperInclusive
+ ? " <= '" : " < '") + _upperBound + "')";
+ }
+}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/RegexpLikePredicate.java b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/RegexpLikePredicate.java
new file mode 100644
index 0000000..c6039fb
--- /dev/null
+++ b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/RegexpLikePredicate.java
@@ -0,0 +1,72 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.request.context.predicate;
+
+import java.util.Objects;
+import org.apache.pinot.common.request.context.ExpressionContext;
+
+
+/**
+ * Predicate for REGEXP_LIKE.
+ */
+public class RegexpLikePredicate implements Predicate {
+ private final ExpressionContext _lhs;
+ private final String _value;
+
+ public RegexpLikePredicate(ExpressionContext lhs, String value) {
+ _lhs = lhs;
+ _value = value;
+ }
+
+ @Override
+ public Type getType() {
+ return Type.REGEXP_LIKE;
+ }
+
+ @Override
+ public ExpressionContext getLhs() {
+ return _lhs;
+ }
+
+ public String getValue() {
+ return _value;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof RegexpLikePredicate)) {
+ return false;
+ }
+ RegexpLikePredicate that = (RegexpLikePredicate) o;
+ return Objects.equals(_lhs, that._lhs) && Objects.equals(_value, that._value);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(_lhs, _value);
+ }
+
+ @Override
+ public String toString() {
+ return "regexp_like(" + _lhs + ",'" + _value + "')";
+ }
+}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/TextMatchPredicate.java b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/TextMatchPredicate.java
new file mode 100644
index 0000000..27a93b5
--- /dev/null
+++ b/pinot-common/src/main/java/org/apache/pinot/common/request/context/predicate/TextMatchPredicate.java
@@ -0,0 +1,72 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.request.context.predicate;
+
+import java.util.Objects;
+import org.apache.pinot.common.request.context.ExpressionContext;
+
+
+/**
+ * Predicate for TEXT_MATCH.
+ */
+public class TextMatchPredicate implements Predicate {
+ private final ExpressionContext _lhs;
+ private final String _value;
+
+ public TextMatchPredicate(ExpressionContext lhs, String value) {
+ _lhs = lhs;
+ _value = value;
+ }
+
+ @Override
+ public Type getType() {
+ return Type.TEXT_MATCH;
+ }
+
+ @Override
+ public ExpressionContext getLhs() {
+ return _lhs;
+ }
+
+ public String getValue() {
+ return _value;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof TextMatchPredicate)) {
+ return false;
+ }
+ TextMatchPredicate that = (TextMatchPredicate) o;
+ return Objects.equals(_lhs, that._lhs) && Objects.equals(_value, that._value);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(_lhs, _value);
+ }
+
+ @Override
+ public String toString() {
+ return "text_match(" + _lhs + ",'" + _value + "')";
+ }
+}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/segment/ReadMode.java b/pinot-common/src/main/java/org/apache/pinot/common/segment/ReadMode.java
deleted file mode 100644
index e71904f..0000000
--- a/pinot-common/src/main/java/org/apache/pinot/common/segment/ReadMode.java
+++ /dev/null
@@ -1,48 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.common.segment;
-
-import org.apache.pinot.common.utils.CommonConstants;
-
-
-/**
- * Enum class for segment read mode:
- * <ul>
- * <li> heap: Segments are loaded on direct-memory. Note, 'heap' here is a legacy misnomer, and it does not
- * imply JVM heap. This mode should only be used when we want faster performance than memory-mapped files,
- * and are also sure that we will never run into OOM. </li>
- * <li> mmap: Segments are loaded on memory-mapped file. This is the default mode. </li>
- * </ul>
- */
-public enum ReadMode {
- heap, mmap;
-
- public static final ReadMode DEFAULT_MODE = ReadMode.valueOf(CommonConstants.Server.DEFAULT_READ_MODE);
-
- public static ReadMode getEnum(String strVal) {
- if (strVal.equalsIgnoreCase("heap")) {
- return heap;
- }
- if (strVal.equalsIgnoreCase("mmap") || strVal.equalsIgnoreCase("memorymapped") || strVal
- .equalsIgnoreCase("memorymap")) {
- return mmap;
- }
- throw new IllegalArgumentException("Unknown String Value: " + strVal);
- }
-}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/utils/CommonConstants.java b/pinot-common/src/main/java/org/apache/pinot/common/utils/CommonConstants.java
deleted file mode 100644
index 8da4c2d..0000000
--- a/pinot-common/src/main/java/org/apache/pinot/common/utils/CommonConstants.java
+++ /dev/null
@@ -1,509 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.common.utils;
-
-import java.io.File;
-
-
-public class CommonConstants {
-
- public static final String PREFIX_OF_SSL_SUBSET = "ssl";
- public static final String HTTP_PROTOCOL = "http";
- public static final String HTTPS_PROTOCOL = "https";
-
- public static final String KEY_OF_AUTH_TOKEN = "auth.token";
-
- public static class Table {
- public static final String PUSH_FREQUENCY_HOURLY = "hourly";
- public static final String PUSH_FREQUENCY_DAILY = "daily";
- public static final String PUSH_FREQUENCY_WEEKLY = "weekly";
- public static final String PUSH_FREQUENCY_MONTHLY = "monthly";
- }
-
- public static class Helix {
- public static final String IS_SHUTDOWN_IN_PROGRESS = "shutdownInProgress";
- public static final String QUERIES_DISABLED = "queriesDisabled";
- public static final String QUERY_RATE_LIMIT_DISABLED = "queryRateLimitDisabled";
-
- public static final String INSTANCE_CONNECTED_METRIC_NAME = "helix.connected";
-
- public static final String PREFIX_OF_CONTROLLER_INSTANCE = "Controller_";
- public static final String PREFIX_OF_BROKER_INSTANCE = "Broker_";
- public static final String PREFIX_OF_SERVER_INSTANCE = "Server_";
- public static final String PREFIX_OF_MINION_INSTANCE = "Minion_";
-
- public static final String BROKER_RESOURCE_INSTANCE = "brokerResource";
- public static final String LEAD_CONTROLLER_RESOURCE_NAME = "leadControllerResource";
-
- public static final String LEAD_CONTROLLER_RESOURCE_ENABLED_KEY = "RESOURCE_ENABLED";
-
- public static final String ENABLE_CASE_INSENSITIVE_KEY = "enable.case.insensitive";
- public static final String DEPRECATED_ENABLE_CASE_INSENSITIVE_KEY = "enable.case.insensitive.pql";
-
- public static final String DEFAULT_HYPERLOGLOG_LOG2M_KEY = "default.hyperloglog.log2m";
- public static final int DEFAULT_HYPERLOGLOG_LOG2M = 8;
-
- // Whether to rewrite DistinctCount to DistinctCountBitmap
- public static final String ENABLE_DISTINCT_COUNT_BITMAP_OVERRIDE_KEY = "enable.distinct.count.bitmap.override";
-
- // More information on why these numbers are set can be found in the following doc:
- // https://cwiki.apache.org/confluence/display/PINOT/Controller+Separation+between+Helix+and+Pinot
- public static final int NUMBER_OF_PARTITIONS_IN_LEAD_CONTROLLER_RESOURCE = 24;
- public static final int LEAD_CONTROLLER_RESOURCE_REPLICA_COUNT = 1;
- public static final int MIN_ACTIVE_REPLICAS = 0;
- public static final int REBALANCE_DELAY_MS = 300_000; // 5 minutes.
-
- // Instance tags
- public static final String CONTROLLER_INSTANCE = "controller";
- public static final String UNTAGGED_BROKER_INSTANCE = "broker_untagged";
- public static final String UNTAGGED_SERVER_INSTANCE = "server_untagged";
- public static final String UNTAGGED_MINION_INSTANCE = "minion_untagged";
-
- public static class StateModel {
- public static class SegmentStateModel {
- public static final String ONLINE = "ONLINE";
- public static final String OFFLINE = "OFFLINE";
- public static final String ERROR = "ERROR";
- public static final String CONSUMING = "CONSUMING";
- }
-
- public static class BrokerResourceStateModel {
- public static final String ONLINE = "ONLINE";
- public static final String OFFLINE = "OFFLINE";
- public static final String ERROR = "ERROR";
- }
- }
-
- public static class ZkClient {
- public static final long DEFAULT_CONNECT_TIMEOUT_SEC = 60L;
- // Retry interval and count for ZK operations where we would rather fail than get an empty (wrong) result back
- public static final int RETRY_INTERVAL_MS = 50;
- public static final int RETRY_COUNT = 2;
- }
-
- public static class DataSource {
- public enum SegmentAssignmentStrategyType {
- RandomAssignmentStrategy,
- BalanceNumSegmentAssignmentStrategy,
- BucketizedSegmentAssignmentStrategy,
- ReplicaGroupSegmentAssignmentStrategy
- }
- }
-
- public static class Instance {
- public static final String INSTANCE_ID_KEY = "instanceId";
- public static final String DATA_DIR_KEY = "dataDir";
- public static final String ADMIN_PORT_KEY = "adminPort";
- public static final String ADMIN_HTTPS_PORT_KEY = "adminHttpsPort";
- public static final String GRPC_PORT_KEY = "grpcPort";
- public static final String NETTYTLS_PORT_KEY = "nettyTlsPort";
- }
-
- public static final String SET_INSTANCE_ID_TO_HOSTNAME_KEY = "pinot.set.instance.id.to.hostname";
-
- public static final String KEY_OF_SERVER_NETTY_PORT = "pinot.server.netty.port";
- public static final int DEFAULT_SERVER_NETTY_PORT = 8098;
- public static final String KEY_OF_SERVER_NETTYTLS_PORT = Server.SERVER_NETTYTLS_PREFIX + ".port";
- public static final int DEFAULT_SERVER_NETTYTLS_PORT = 8091;
- public static final String KEY_OF_BROKER_QUERY_PORT = "pinot.broker.client.queryPort";
- public static final int DEFAULT_BROKER_QUERY_PORT = 8099;
- public static final String KEY_OF_SERVER_NETTY_HOST = "pinot.server.netty.host";
- public static final String KEY_OF_MINION_HOST = "pinot.minion.host";
- public static final String KEY_OF_MINION_PORT = "pinot.minion.port";
-
- // NOTE: Helix will disconnect the manager and disable the instance if it detects flapping (too frequent disconnect
- // from ZooKeeper). Setting flapping time window to a small value can avoid this from happening. Helix ignores the
- // non-positive value, so set the default value as 1.
- public static final String CONFIG_OF_CONTROLLER_FLAPPING_TIME_WINDOW_MS = "pinot.controller.flapping.timeWindowMs";
- public static final String CONFIG_OF_BROKER_FLAPPING_TIME_WINDOW_MS = "pinot.broker.flapping.timeWindowMs";
- public static final String CONFIG_OF_SERVER_FLAPPING_TIME_WINDOW_MS = "pinot.server.flapping.timeWindowMs";
- public static final String CONFIG_OF_MINION_FLAPPING_TIME_WINDOW_MS = "pinot.minion.flapping.timeWindowMs";
- public static final String CONFIG_OF_HELIX_INSTANCE_MAX_STATE_TRANSITIONS =
- "pinot.helix.instance.state.maxStateTransitions";
- public static final String DEFAULT_HELIX_INSTANCE_MAX_STATE_TRANSITIONS = "100000";
- public static final String DEFAULT_FLAPPING_TIME_WINDOW_MS = "1";
-
- public static final String PINOT_SERVICE_ROLE = "pinot.service.role";
- }
-
- public static class Broker {
- public static final String ROUTING_TABLE_CONFIG_PREFIX = "pinot.broker.routing.table";
- public static final String ACCESS_CONTROL_CONFIG_PREFIX = "pinot.broker.access.control";
- public static final String METRICS_CONFIG_PREFIX = "pinot.broker.metrics";
- public static final String CONFIG_OF_METRICS_NAME_PREFIX = "pinot.broker.metrics.prefix";
- public static final String DEFAULT_METRICS_NAME_PREFIX = "pinot.broker.";
-
- public static final String CONFIG_OF_DELAY_SHUTDOWN_TIME_MS = "pinot.broker.delayShutdownTimeMs";
- public static final long DEFAULT_DELAY_SHUTDOWN_TIME_MS = 10_000L;
- public static final String CONFIG_OF_ENABLE_TABLE_LEVEL_METRICS = "pinot.broker.enableTableLevelMetrics";
- public static final boolean DEFAULT_ENABLE_TABLE_LEVEL_METRICS = true;
- public static final String CONFIG_OF_ALLOWED_TABLES_FOR_EMITTING_METRICS =
- "pinot.broker.allowedTablesForEmittingMetrics";
-
- public static final String CONFIG_OF_BROKER_QUERY_RESPONSE_LIMIT = "pinot.broker.query.response.limit";
- public static final int DEFAULT_BROKER_QUERY_RESPONSE_LIMIT = Integer.MAX_VALUE;
- public static final String CONFIG_OF_BROKER_QUERY_LOG_LENGTH = "pinot.broker.query.log.length";
- public static final int DEFAULT_BROKER_QUERY_LOG_LENGTH = Integer.MAX_VALUE;
- public static final String CONFIG_OF_BROKER_QUERY_LOG_MAX_RATE_PER_SECOND =
- "pinot.broker.query.log.maxRatePerSecond";
- public static final double DEFAULT_BROKER_QUERY_LOG_MAX_RATE_PER_SECOND = 10_000d;
- public static final String CONFIG_OF_BROKER_TIMEOUT_MS = "pinot.broker.timeoutMs";
- public static final long DEFAULT_BROKER_TIMEOUT_MS = 10_000L;
- public static final String CONFIG_OF_BROKER_ID = "pinot.broker.id";
- // Configuration to consider the broker ServiceStatus as being STARTED if the percent of resources (tables) that
- // are ONLINE for this this broker has crossed the threshold percentage of the total number of tables
- // that it is expected to serve.
- public static final String CONFIG_OF_BROKER_MIN_RESOURCE_PERCENT_FOR_START =
- "pinot.broker.startup.minResourcePercent";
- public static final double DEFAULT_BROKER_MIN_RESOURCE_PERCENT_FOR_START = 100.0;
- public static final String CONFIG_OF_ENABLE_QUERY_LIMIT_OVERRIDE = "pinot.broker.enable.query.limit.override";
-
- // Config for number of threads to use for Broker reduce-phase.
- public static final String CONFIG_OF_MAX_REDUCE_THREADS_PER_QUERY = "pinot.broker.max.reduce.threads.per.query";
- public static final int DEFAULT_MAX_REDUCE_THREADS_PER_QUERY =
- Math.max(1, Math.min(10, Runtime.getRuntime().availableProcessors() / 2)); // Same logic as CombineOperatorUtils
-
- // used for SQL GROUP BY during broker reduce
- public static final String CONFIG_OF_BROKER_GROUPBY_TRIM_THRESHOLD = "pinot.broker.groupby.trim.threshold";
- public static final int DEFAULT_BROKER_GROUPBY_TRIM_THRESHOLD = 1_000_000;
-
- public static final String BROKER_TLS_PREFIX = "pinot.broker.tls";
- public static final String BROKER_NETTYTLS_ENABLED = "pinot.broker.nettytls.enabled";
-
- public static class Request {
- public static final String PQL = "pql";
- public static final String SQL = "sql";
- public static final String TRACE = "trace";
- public static final String DEBUG_OPTIONS = "debugOptions";
- public static final String QUERY_OPTIONS = "queryOptions";
-
- public static class QueryOptionKey {
- public static final String TIMEOUT_MS = "timeoutMs";
- public static final String PRESERVE_TYPE = "preserveType";
- public static final String RESPONSE_FORMAT = "responseFormat";
- public static final String GROUP_BY_MODE = "groupByMode";
- public static final String SKIP_UPSERT = "skipUpsert";
- }
- }
- }
-
- public static class Server {
- public static final String CONFIG_OF_INSTANCE_ID = "pinot.server.instance.id";
- public static final String CONFIG_OF_INSTANCE_DATA_DIR = "pinot.server.instance.dataDir";
- public static final String CONFIG_OF_CONSUMER_DIR = "pinot.server.instance.consumerDir";
- public static final String CONFIG_OF_INSTANCE_SEGMENT_TAR_DIR = "pinot.server.instance.segmentTarDir";
- public static final String CONFIG_OF_INSTANCE_READ_MODE = "pinot.server.instance.readMode";
- public static final String CONFIG_OF_INSTANCE_RELOAD_CONSUMING_SEGMENT =
- "pinot.server.instance.reload.consumingSegment";
- public static final String CONFIG_OF_INSTANCE_DATA_MANAGER_CLASS = "pinot.server.instance.data.manager.class";
- public static final String CONFIG_OF_QUERY_EXECUTOR_PRUNER_CLASS = "pinot.server.query.executor.pruner.class";
- public static final String CONFIG_OF_QUERY_EXECUTOR_TIMEOUT = "pinot.server.query.executor.timeout";
- public static final String CONFIG_OF_QUERY_EXECUTOR_CLASS = "pinot.server.query.executor.class";
- public static final String CONFIG_OF_REQUEST_HANDLER_FACTORY_CLASS = "pinot.server.requestHandlerFactory.class";
- public static final String CONFIG_OF_NETTY_SERVER_ENABLED = "pinot.server.netty.enabled";
- public static final boolean DEFAULT_NETTY_SERVER_ENABLED = true;
- public static final String CONFIG_OF_NETTY_PORT = "pinot.server.netty.port";
- public static final String CONFIG_OF_ENABLE_GRPC_SERVER = "pinot.server.grpc.enable";
- public static final boolean DEFAULT_ENABLE_GRPC_SERVER = false;
- public static final String CONFIG_OF_GRPC_PORT = "pinot.server.grpc.port";
- public static final int DEFAULT_GRPC_PORT = 8090;
- public static final String CONFIG_OF_NETTYTLS_SERVER_ENABLED = "pinot.server.nettytls.enabled";
- public static final boolean DEFAULT_NETTYTLS_SERVER_ENABLED = false;
- public static final String CONFIG_OF_ADMIN_API_PORT = "pinot.server.adminapi.port";
- public static final int DEFAULT_ADMIN_API_PORT = 8097;
-
- public static final String CONFIG_OF_SEGMENT_FORMAT_VERSION = "pinot.server.instance.segment.format.version";
- public static final String CONFIG_OF_ENABLE_SPLIT_COMMIT = "pinot.server.instance.enable.split.commit";
- public static final String CONFIG_OF_ENABLE_COMMIT_END_WITH_METADATA =
- "pinot.server.instance.enable.commitend.metadata";
- public static final String CONFIG_OF_REALTIME_OFFHEAP_ALLOCATION = "pinot.server.instance.realtime.alloc.offheap";
- public static final String CONFIG_OF_REALTIME_OFFHEAP_DIRECT_ALLOCATION =
- "pinot.server.instance.realtime.alloc.offheap.direct";
- public static final String PREFIX_OF_CONFIG_OF_PINOT_FS_FACTORY = "pinot.server.storage.factory";
- public static final String PREFIX_OF_CONFIG_OF_PINOT_CRYPTER = "pinot.server.crypter";
-
- /**
- * Service token for accessing protected controller APIs.
- * E.g. null (auth disabled), "Basic abcdef..." (basic auth), "Bearer 123def..." (oauth2)
- */
- public static final String CONFIG_OF_AUTH_TOKEN = KEY_OF_AUTH_TOKEN;
-
- // Configuration to consider the server ServiceStatus as being STARTED if the percent of resources (tables) that
- // are ONLINE for this this server has crossed the threshold percentage of the total number of tables
- // that it is expected to serve.
- public static final String CONFIG_OF_SERVER_MIN_RESOURCE_PERCENT_FOR_START =
- "pinot.server.startup.minResourcePercent";
- public static final double DEFAULT_SERVER_MIN_RESOURCE_PERCENT_FOR_START = 100.0;
- public static final String CONFIG_OF_STARTUP_REALTIME_CONSUMPTION_CATCHUP_WAIT_MS =
- "pinot.server.starter.realtimeConsumptionCatchupWaitMs";
- public static final int DEFAULT_STARTUP_REALTIME_CONSUMPTION_CATCHUP_WAIT_MS = 0;
-
- public static final String DEFAULT_READ_MODE = "mmap";
- // Whether to reload consuming segment on scheme update. Will change default behavior to true when this feature is stabilized
- public static final boolean DEFAULT_RELOAD_CONSUMING_SEGMENT = false;
- public static final String DEFAULT_INSTANCE_BASE_DIR =
- System.getProperty("java.io.tmpdir") + File.separator + "PinotServer";
- public static final String DEFAULT_INSTANCE_DATA_DIR = DEFAULT_INSTANCE_BASE_DIR + File.separator + "index";
- public static final String DEFAULT_INSTANCE_SEGMENT_TAR_DIR =
- DEFAULT_INSTANCE_BASE_DIR + File.separator + "segmentTar";
- public static final String DEFAULT_DATA_MANAGER_CLASS =
- "org.apache.pinot.server.starter.helix.HelixInstanceDataManager";
- public static final String DEFAULT_QUERY_EXECUTOR_CLASS =
- "org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl";
- public static final long DEFAULT_QUERY_EXECUTOR_TIMEOUT_MS = 15_000L;
- public static final String DEFAULT_REQUEST_HANDLER_FACTORY_CLASS =
- "org.apache.pinot.server.request.SimpleRequestHandlerFactory";
- public static final String PREFIX_OF_CONFIG_OF_SEGMENT_FETCHER_FACTORY = "pinot.server.segment.fetcher";
-
- // Configs for server starter startup/shutdown checks
- // Startup: timeout for the startup checks
- public static final String CONFIG_OF_STARTUP_TIMEOUT_MS = "pinot.server.startup.timeoutMs";
- public static final long DEFAULT_STARTUP_TIMEOUT_MS = 600_000L;
- // Startup: enable service status check before claiming server up
- public static final String CONFIG_OF_STARTUP_ENABLE_SERVICE_STATUS_CHECK =
- "pinot.server.startup.enableServiceStatusCheck";
- public static final boolean DEFAULT_STARTUP_ENABLE_SERVICE_STATUS_CHECK = true;
- public static final String CONFIG_OF_STARTUP_SERVICE_STATUS_CHECK_INTERVAL_MS =
- "pinot.server.startup.serviceStatusCheckIntervalMs";
- public static final long DEFAULT_STARTUP_SERVICE_STATUS_CHECK_INTERVAL_MS = 10_000L;
- // Shutdown: timeout for the shutdown checks
- public static final String CONFIG_OF_SHUTDOWN_TIMEOUT_MS = "pinot.server.shutdown.timeoutMs";
- public static final long DEFAULT_SHUTDOWN_TIMEOUT_MS = 600_000L;
- // Shutdown: enable query check before shutting down the server
- // Will drain queries (no incoming queries and all existing queries finished)
- public static final String CONFIG_OF_SHUTDOWN_ENABLE_QUERY_CHECK = "pinot.server.shutdown.enableQueryCheck";
- public static final boolean DEFAULT_SHUTDOWN_ENABLE_QUERY_CHECK = true;
- // Shutdown: threshold to mark that there is no incoming queries, use max query time as the default threshold
- public static final String CONFIG_OF_SHUTDOWN_NO_QUERY_THRESHOLD_MS = "pinot.server.shutdown.noQueryThresholdMs";
- // Shutdown: enable resource check before shutting down the server
- // Will wait until all the resources in the external view are neither ONLINE nor CONSUMING
- // No need to enable this check if startup service status check is enabled
- public static final String CONFIG_OF_SHUTDOWN_ENABLE_RESOURCE_CHECK = "pinot.server.shutdown.enableResourceCheck";
- public static final boolean DEFAULT_SHUTDOWN_ENABLE_RESOURCE_CHECK = false;
- public static final String CONFIG_OF_SHUTDOWN_RESOURCE_CHECK_INTERVAL_MS =
- "pinot.server.shutdown.resourceCheckIntervalMs";
- public static final long DEFAULT_SHUTDOWN_RESOURCE_CHECK_INTERVAL_MS = 10_000L;
-
- public static final String DEFAULT_COLUMN_MIN_MAX_VALUE_GENERATOR_MODE = "ALL";
-
- public static final String PINOT_SERVER_METRICS_PREFIX = "pinot.server.metrics.prefix";
-
- public static final String SERVER_TLS_PREFIX = "pinot.server.tls";
- public static final String SERVER_NETTYTLS_PREFIX = "pinot.server.nettytls";
-
- // The complete config key is pinot.server.instance.segment.store.uri
- public static final String CONFIG_OF_SEGMENT_STORE_URI = "segment.store.uri";
-
- public static class SegmentCompletionProtocol {
- public static final String PREFIX_OF_CONFIG_OF_SEGMENT_UPLOADER = "pinot.server.segment.uploader";
-
- public static final String CONFIG_OF_CONTROLLER_HTTPS_ENABLED = "enabled";
- public static final String CONFIG_OF_CONTROLLER_HTTPS_PORT = "controller.port";
- public static final String CONFIG_OF_SEGMENT_UPLOAD_REQUEST_TIMEOUT_MS = "upload.request.timeout.ms";
-
- /**
- * Service token for accessing protected controller APIs.
- * E.g. null (auth disabled), "Basic abcdef..." (basic auth), "Bearer 123def..." (oauth2)
- */
- public static final String CONFIG_OF_SEGMENT_UPLOADER_AUTH_TOKEN = KEY_OF_AUTH_TOKEN;
-
- public static final int DEFAULT_SEGMENT_UPLOAD_REQUEST_TIMEOUT_MS = 300_000;
- public static final int DEFAULT_OTHER_REQUESTS_TIMEOUT = 10_000;
- }
-
- public static final String DEFAULT_METRICS_PREFIX = "pinot.server.";
- public static final String CONFIG_OF_ENABLE_TABLE_LEVEL_METRICS = "pinot.server.enableTableLevelMetrics";
- public static final boolean DEFAULT_ENABLE_TABLE_LEVEL_METRICS = true;
- public static final String CONFIG_OF_ALLOWED_TABLES_FOR_EMITTING_METRICS =
- "pinot.server.allowedTablesForEmittingMetrics";
- public static final String ACCESS_CONTROL_FACTORY_CLASS = "pinot.server.admin.access.control.factory.class";
- public static final String DEFAULT_ACCESS_CONTROL_FACTORY_CLASS =
- "org.apache.pinot.server.api.access.AllowAllAccessFactory";
-
- public static final String CONFIG_OF_ENABLE_THREAD_CPU_TIME_MEASUREMENT =
- "pinot.server.instance.enableThreadCpuTimeMeasurement";
- public static final boolean DEFAULT_ENABLE_THREAD_CPU_TIME_MEASUREMENT = false;
-
- public static final String CONFIG_OF_CURRENT_DATA_TABLE_VERSION = "pinot.server.instance.currentDataTableVersion";
- public static final int DEFAULT_CURRENT_DATA_TABLE_VERSION = 3;
- }
-
- public static class Controller {
- public static final String PREFIX_OF_CONFIG_OF_SEGMENT_FETCHER_FACTORY = "pinot.controller.segment.fetcher";
- public static final String PREFIX_OF_CONFIG_OF_PINOT_FS_FACTORY = "pinot.controller.storage.factory";
- public static final String HOST_HTTP_HEADER = "Pinot-Controller-Host";
- public static final String VERSION_HTTP_HEADER = "Pinot-Controller-Version";
- public static final String SEGMENT_NAME_HTTP_HEADER = "Pinot-Segment-Name";
- public static final String TABLE_NAME_HTTP_HEADER = "Pinot-Table-Name";
- public static final String INGESTION_DESCRIPTOR = "Pinot-Ingestion-Descriptor";
- public static final String PREFIX_OF_CONFIG_OF_PINOT_CRYPTER = "pinot.controller.crypter";
-
- public static final String CONFIG_OF_CONTROLLER_METRICS_PREFIX = "controller.metrics.prefix";
- // FYI this is incorrect as it generate metrics named without a dot after pinot.controller part,
- // but we keep this default for backward compatibility in case someone relies on this format
- // see Server or Broker class for correct prefix format you should use
- public static final String DEFAULT_METRICS_PREFIX = "pinot.controller.";
- }
-
- public static class Minion {
- public static final String CONFIG_OF_METRICS_PREFIX = "pinot.minion.";
- public static final String METADATA_EVENT_OBSERVER_PREFIX = "metadata.event.notifier";
-
- // Config keys
- public static final String CONFIG_OF_METRICS_PREFIX_KEY = "metricsPrefix";
- public static final String METRICS_REGISTRY_REGISTRATION_LISTENERS_KEY = "metricsRegistryRegistrationListeners";
-
- // Default settings
- public static final int DEFAULT_HELIX_PORT = 9514;
- public static final String DEFAULT_INSTANCE_BASE_DIR =
- System.getProperty("java.io.tmpdir") + File.separator + "PinotMinion";
- public static final String DEFAULT_INSTANCE_DATA_DIR = DEFAULT_INSTANCE_BASE_DIR + File.separator + "data";
- public static final String PREFIX_OF_CONFIG_OF_PINOT_FS_FACTORY = "storage.factory";
- public static final String PREFIX_OF_CONFIG_OF_SEGMENT_FETCHER_FACTORY = "segment.fetcher";
- public static final String PREFIX_OF_CONFIG_OF_SEGMENT_UPLOADER = "segment.uploader";
- public static final String PREFIX_OF_CONFIG_OF_PINOT_CRYPTER = "crypter";
-
- /**
- * Service token for accessing protected controller APIs.
- * E.g. null (auth disabled), "Basic abcdef..." (basic auth), "Bearer 123def..." (oauth2)
- */
- public static final String CONFIG_OF_TASK_AUTH_TOKEN = "task.auth.token";
- }
-
- public static class Segment {
- public static class Realtime {
- public enum Status {
- // Means the segment is in CONSUMING state.
- IN_PROGRESS,
- // Means the segment is in ONLINE state (segment completed consuming and has been saved in segment store).
- DONE,
- // Means the segment is uploaded to a Pinot controller by an external party.
- UPLOADED
- }
-
- /**
- * During realtime segment completion, the value of this enum decides how non-winner servers should replace the completed segment.
- */
- public enum CompletionMode {
- // default behavior - if the in memory segment in the non-winner server is equivalent to the committed segment, then build and replace, else download
- DEFAULT,
- // non-winner servers always download the segment, never build it
- DOWNLOAD
- }
-
- public static final String STATUS = "segment.realtime.status";
- }
-
- public static class Offline {
- public static final String DOWNLOAD_URL = "segment.offline.download.url";
- public static final String PUSH_TIME = "segment.offline.push.time";
- public static final String REFRESH_TIME = "segment.offline.refresh.time";
- }
-
- public static final String SEGMENT_NAME = "segment.name";
- public static final String SEGMENT_TYPE = "segment.type";
- public static final String START_TIME = "segment.start.time";
- public static final String END_TIME = "segment.end.time";
- public static final String TIME_UNIT = "segment.time.unit";
- public static final String INDEX_VERSION = "segment.index.version";
- public static final String TOTAL_DOCS = "segment.total.docs";
- public static final String CRC = "segment.crc";
- public static final String CREATION_TIME = "segment.creation.time";
- public static final String FLUSH_THRESHOLD_SIZE = "segment.flush.threshold.size";
- public static final String FLUSH_THRESHOLD_TIME = "segment.flush.threshold.time";
- public static final String PARTITION_METADATA = "segment.partition.metadata";
- /**
- * This field is used for parallel push protection to lock the segment globally.
- * We put the segment upload start timestamp so that if the previous push failed without unlock the segment, the
- * next upload won't be blocked forever.
- */
- public static final String SEGMENT_UPLOAD_START_TIME = "segment.upload.start.time";
-
- public static final String CRYPTER_NAME = "segment.crypter";
- public static final String CUSTOM_MAP = "custom.map";
-
- @Deprecated
- public static final String TABLE_NAME = "segment.table.name";
-
- public static final String SEGMENT_BACKUP_DIR_SUFFIX = ".segment.bak";
- public static final String SEGMENT_TEMP_DIR_SUFFIX = ".segment.tmp";
-
- public static final String LOCAL_SEGMENT_SCHEME = "file";
- public static final String PEER_SEGMENT_DOWNLOAD_SCHEME = "peer://";
- public static final String METADATA_URI_FOR_PEER_DOWNLOAD = "";
-
- public enum SegmentType {
- OFFLINE, REALTIME
- }
-
- public static class AssignmentStrategy {
- public static String BALANCE_NUM_SEGMENT_ASSIGNMENT_STRATEGY = "BalanceNumSegmentAssignmentStrategy";
- public static String REPLICA_GROUP_SEGMENT_ASSIGNMENT_STRATEGY = "ReplicaGroupSegmentAssignmentStrategy";
- }
-
- public static class BuiltInVirtualColumn {
- public static final String DOCID = "$docId";
- public static final String HOSTNAME = "$hostName";
- public static final String SEGMENTNAME = "$segmentName";
- }
- }
-
- public static class Query {
- public static class Request {
- public static class MetadataKeys {
- public static final String REQUEST_ID = "requestId";
- public static final String BROKER_ID = "brokerId";
- public static final String ENABLE_TRACE = "enableTrace";
- public static final String ENABLE_STREAMING = "enableStreaming";
- public static final String PAYLOAD_TYPE = "payloadType";
- }
-
- public static class PayloadType {
- public static final String SQL = "sql";
- public static final String BROKER_REQUEST = "brokerRequest";
- }
- }
-
- public static class Response {
- public static class MetadataKeys {
- public static final String RESPONSE_TYPE = "responseType";
- }
-
- public static class ResponseType {
- // For streaming response, multiple (could be 0 if no data should be returned, or query encounters exception)
- // data responses will be returned, followed by one single metadata response
- public static final String DATA = "data";
- public static final String METADATA = "metadata";
- // For non-streaming response
- public static final String NON_STREAMING = "nonStreaming";
- }
- }
-
- public static class Range {
- public static final char DELIMITER = '\0';
- public static final char LOWER_EXCLUSIVE = '(';
- public static final char LOWER_INCLUSIVE = '[';
- public static final char UPPER_EXCLUSIVE = ')';
- public static final char UPPER_INCLUSIVE = ']';
- public static final String UNBOUNDED = "*";
- public static final String LOWER_UNBOUNDED = LOWER_EXCLUSIVE + UNBOUNDED + DELIMITER;
- public static final String UPPER_UNBOUNDED = DELIMITER + UNBOUNDED + UPPER_EXCLUSIVE;
- }
- }
-}
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/utils/FileUploadDownloadClient.java b/pinot-common/src/main/java/org/apache/pinot/common/utils/FileUploadDownloadClient.java
index 5da6a96..ba32144 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/utils/FileUploadDownloadClient.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/utils/FileUploadDownloadClient.java
@@ -28,7 +28,6 @@ import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;
import java.net.URISyntaxException;
-import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
@@ -64,6 +63,7 @@ import org.apache.http.message.BasicNameValuePair;
import org.apache.http.util.EntityUtils;
import org.apache.pinot.common.exception.HttpErrorStatusException;
import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.JsonUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/utils/SchemaUtils.java b/pinot-common/src/main/java/org/apache/pinot/common/utils/SchemaUtils.java
index 45ac7b3..c7f3c99 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/utils/SchemaUtils.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/utils/SchemaUtils.java
@@ -36,6 +36,7 @@ import org.apache.helix.ZNRecord;
import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.spi.data.TimeGranularitySpec;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/utils/URIUtils.java b/pinot-common/src/main/java/org/apache/pinot/common/utils/URIUtils.java
index d39c3b6..b2617a3 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/utils/URIUtils.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/utils/URIUtils.java
@@ -26,6 +26,7 @@ import java.net.URLEncoder;
import java.util.Map;
import java.util.StringJoiner;
import org.apache.http.client.utils.URIBuilder;
+import org.apache.pinot.spi.utils.CommonConstants;
public class URIUtils {
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/utils/config/InstanceUtils.java b/pinot-common/src/main/java/org/apache/pinot/common/utils/config/InstanceUtils.java
index 6ef3bbd..b460089 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/utils/config/InstanceUtils.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/utils/config/InstanceUtils.java
@@ -22,9 +22,9 @@ import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import org.apache.helix.model.InstanceConfig;
-import org.apache.pinot.common.utils.CommonConstants;
-import org.apache.pinot.common.utils.CommonConstants.Helix;
import org.apache.pinot.spi.config.instance.Instance;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants.Helix;
public class InstanceUtils {
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/BaseSegmentFetcher.java b/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/BaseSegmentFetcher.java
index 0a80d94..bc1a975 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/BaseSegmentFetcher.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/BaseSegmentFetcher.java
@@ -22,8 +22,8 @@ import java.io.File;
import java.net.URI;
import java.util.List;
import java.util.Random;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.retry.RetryPolicies;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/HttpsSegmentFetcher.java b/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/HttpsSegmentFetcher.java
index 7f44d13..8fec2e6 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/HttpsSegmentFetcher.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/HttpsSegmentFetcher.java
@@ -18,15 +18,12 @@
*/
package org.apache.pinot.common.utils.fetcher;
-import java.util.Iterator;
import java.util.Set;
-
import javax.net.ssl.SSLContext;
-
import org.apache.pinot.common.utils.ClientSSLContextGenerator;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.FileUploadDownloadClient;
import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.CommonConstants;
/*
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/SegmentFetcherFactory.java b/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/SegmentFetcherFactory.java
index 064256c..e6a51ed 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/SegmentFetcherFactory.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/SegmentFetcherFactory.java
@@ -25,11 +25,10 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.commons.lang3.StringUtils;
-
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.spi.crypt.PinotCrypter;
import org.apache.pinot.spi.crypt.PinotCrypterFactory;
import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/utils/grpc/GrpcRequestBuilder.java b/pinot-common/src/main/java/org/apache/pinot/common/utils/grpc/GrpcRequestBuilder.java
index 0d6daeb..51280be 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/utils/grpc/GrpcRequestBuilder.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/utils/grpc/GrpcRequestBuilder.java
@@ -26,7 +26,7 @@ import java.util.Map;
import org.apache.commons.collections.CollectionUtils;
import org.apache.pinot.common.proto.Server;
import org.apache.pinot.common.request.BrokerRequest;
-import org.apache.pinot.common.utils.CommonConstants.Query.Request;
+import org.apache.pinot.spi.utils.CommonConstants.Query.Request;
import org.apache.thrift.TException;
import org.apache.thrift.TSerializer;
import org.apache.thrift.protocol.TCompactProtocol;
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/utils/helix/HelixHelper.java b/pinot-common/src/main/java/org/apache/pinot/common/utils/helix/HelixHelper.java
index 95a2925..f5bdb62 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/utils/helix/HelixHelper.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/utils/helix/HelixHelper.java
@@ -42,8 +42,8 @@ import org.apache.helix.model.HelixConfigScope.ConfigScopeProperty;
import org.apache.helix.model.IdealState;
import org.apache.helix.model.InstanceConfig;
import org.apache.helix.model.builder.HelixConfigScopeBuilder;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.config.TagNameUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.retry.RetryPolicies;
import org.apache.pinot.spi.utils.retry.RetryPolicy;
import org.slf4j.Logger;
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/utils/helix/LeadControllerUtils.java b/pinot-common/src/main/java/org/apache/pinot/common/utils/helix/LeadControllerUtils.java
index e07f384..7d2f8ef 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/utils/helix/LeadControllerUtils.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/utils/helix/LeadControllerUtils.java
@@ -24,9 +24,9 @@ import org.apache.helix.HelixManager;
import org.apache.helix.PropertyKey;
import org.apache.helix.model.LiveInstance;
import org.apache.helix.model.ResourceConfig;
-import org.apache.pinot.common.utils.CommonConstants.Helix;
import org.apache.pinot.common.utils.HashUtil;
import org.apache.pinot.common.utils.StringUtil;
+import org.apache.pinot.spi.utils.CommonConstants.Helix;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/utils/webhdfs/WebHdfsV1Client.java b/pinot-common/src/main/java/org/apache/pinot/common/utils/webhdfs/WebHdfsV1Client.java
index 7dc1a22..2deaf20 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/utils/webhdfs/WebHdfsV1Client.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/utils/webhdfs/WebHdfsV1Client.java
@@ -25,7 +25,7 @@ import org.apache.commons.httpclient.HttpMethod;
import org.apache.commons.httpclient.methods.FileRequestEntity;
import org.apache.commons.httpclient.methods.PutMethod;
import org.apache.commons.httpclient.methods.RequestEntity;
-import org.apache.pinot.common.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-common/src/main/java/org/apache/pinot/parsers/utils/ParserUtils.java b/pinot-common/src/main/java/org/apache/pinot/parsers/utils/ParserUtils.java
index 6a67991..c1d0781 100644
--- a/pinot-common/src/main/java/org/apache/pinot/parsers/utils/ParserUtils.java
+++ b/pinot-common/src/main/java/org/apache/pinot/parsers/utils/ParserUtils.java
@@ -27,8 +27,8 @@ import org.apache.pinot.common.request.Expression;
import org.apache.pinot.common.request.FilterOperator;
import org.apache.pinot.common.request.Function;
import org.apache.pinot.common.request.Literal;
-import org.apache.pinot.common.utils.CommonConstants.Query.Range;
import org.apache.pinot.pql.parsers.pql2.ast.FilterKind;
+import org.apache.pinot.spi.utils.CommonConstants.Query.Range;
/**
diff --git a/pinot-common/src/main/java/org/apache/pinot/pql/parsers/pql2/ast/BetweenPredicateAstNode.java b/pinot-common/src/main/java/org/apache/pinot/pql/parsers/pql2/ast/BetweenPredicateAstNode.java
index 60720af..0e793a8 100644
--- a/pinot-common/src/main/java/org/apache/pinot/pql/parsers/pql2/ast/BetweenPredicateAstNode.java
+++ b/pinot-common/src/main/java/org/apache/pinot/pql/parsers/pql2/ast/BetweenPredicateAstNode.java
@@ -23,10 +23,10 @@ import org.apache.pinot.common.request.Expression;
import org.apache.pinot.common.request.FilterOperator;
import org.apache.pinot.common.request.Function;
import org.apache.pinot.common.request.transform.TransformExpressionTree;
-import org.apache.pinot.common.utils.CommonConstants.Query.Range;
import org.apache.pinot.common.utils.request.FilterQueryTree;
import org.apache.pinot.common.utils.request.RequestUtils;
import org.apache.pinot.pql.parsers.Pql2CompilationException;
+import org.apache.pinot.spi.utils.CommonConstants.Query.Range;
/**
diff --git a/pinot-common/src/main/java/org/apache/pinot/pql/parsers/pql2/ast/ComparisonPredicateAstNode.java b/pinot-common/src/main/java/org/apache/pinot/pql/parsers/pql2/ast/ComparisonPredicateAstNode.java
index 450839a..a6bce42 100644
--- a/pinot-common/src/main/java/org/apache/pinot/pql/parsers/pql2/ast/ComparisonPredicateAstNode.java
+++ b/pinot-common/src/main/java/org/apache/pinot/pql/parsers/pql2/ast/ComparisonPredicateAstNode.java
@@ -22,10 +22,10 @@ import java.util.Collections;
import org.apache.pinot.common.request.Expression;
import org.apache.pinot.common.request.FilterOperator;
import org.apache.pinot.common.request.transform.TransformExpressionTree;
-import org.apache.pinot.common.utils.CommonConstants.Query.Range;
import org.apache.pinot.common.utils.request.FilterQueryTree;
import org.apache.pinot.common.utils.request.RequestUtils;
import org.apache.pinot.pql.parsers.Pql2CompilationException;
+import org.apache.pinot.spi.utils.CommonConstants.Query.Range;
/**
diff --git a/pinot-common/src/test/java/org/apache/pinot/common/metadata/SegmentZKMetadataTest.java b/pinot-common/src/test/java/org/apache/pinot/common/metadata/SegmentZKMetadataTest.java
index 8bf7b45..918fd1d 100644
--- a/pinot-common/src/test/java/org/apache/pinot/common/metadata/SegmentZKMetadataTest.java
+++ b/pinot-common/src/test/java/org/apache/pinot/common/metadata/SegmentZKMetadataTest.java
@@ -31,9 +31,9 @@ import org.apache.pinot.common.metadata.segment.OfflineSegmentZKMetadata;
import org.apache.pinot.common.metadata.segment.RealtimeSegmentZKMetadata;
import org.apache.pinot.common.metadata.segment.SegmentPartitionMetadata;
import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
-import org.apache.pinot.common.utils.CommonConstants;
-import org.apache.pinot.common.utils.CommonConstants.Segment.Realtime.Status;
-import org.apache.pinot.common.utils.CommonConstants.Segment.SegmentType;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.Realtime.Status;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.SegmentType;
import org.testng.Assert;
import org.testng.annotations.Test;
diff --git a/pinot-common/src/test/java/org/apache/pinot/common/tier/TierSegmentSelectorTest.java b/pinot-common/src/test/java/org/apache/pinot/common/tier/TierSegmentSelectorTest.java
index 36210e0..a9caa8b 100644
--- a/pinot-common/src/test/java/org/apache/pinot/common/tier/TierSegmentSelectorTest.java
+++ b/pinot-common/src/test/java/org/apache/pinot/common/tier/TierSegmentSelectorTest.java
@@ -25,7 +25,7 @@ import org.apache.helix.store.zk.ZkHelixPropertyStore;
import org.apache.pinot.common.metadata.ZKMetadataProvider;
import org.apache.pinot.common.metadata.segment.LLCRealtimeSegmentZKMetadata;
import org.apache.pinot.common.metadata.segment.OfflineSegmentZKMetadata;
-import org.apache.pinot.common.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.testng.Assert;
import org.testng.annotations.Test;
diff --git a/pinot-common/src/test/java/org/apache/pinot/common/utils/FileUploadDownloadClientTest.java b/pinot-common/src/test/java/org/apache/pinot/common/utils/FileUploadDownloadClientTest.java
index b8a70d8..ac896fd 100644
--- a/pinot-common/src/test/java/org/apache/pinot/common/utils/FileUploadDownloadClientTest.java
+++ b/pinot-common/src/test/java/org/apache/pinot/common/utils/FileUploadDownloadClientTest.java
@@ -36,6 +36,7 @@ import org.apache.http.HttpStatus;
import org.apache.http.NameValuePair;
import org.apache.http.message.BasicHeader;
import org.apache.pinot.common.utils.FileUploadDownloadClient.FileUploadType;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.JsonUtils;
import org.testng.Assert;
import org.testng.annotations.AfterClass;
diff --git a/pinot-common/src/test/java/org/apache/pinot/common/utils/config/TableConfigSerDeTest.java b/pinot-common/src/test/java/org/apache/pinot/common/utils/config/TableConfigSerDeTest.java
index d38435f..435c182 100644
--- a/pinot-common/src/test/java/org/apache/pinot/common/utils/config/TableConfigSerDeTest.java
+++ b/pinot-common/src/test/java/org/apache/pinot/common/utils/config/TableConfigSerDeTest.java
@@ -28,11 +28,8 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.pinot.common.tier.TierFactory;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.spi.config.table.CompletionConfig;
import org.apache.pinot.spi.config.table.FieldConfig;
-import org.apache.pinot.spi.config.table.ingestion.BatchIngestionConfig;
-import org.apache.pinot.spi.config.table.ingestion.IngestionConfig;
import org.apache.pinot.spi.config.table.QueryConfig;
import org.apache.pinot.spi.config.table.QuotaConfig;
import org.apache.pinot.spi.config.table.ReplicaGroupStrategyConfig;
@@ -50,9 +47,12 @@ import org.apache.pinot.spi.config.table.assignment.InstanceConstraintConfig;
import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
import org.apache.pinot.spi.config.table.assignment.InstanceReplicaGroupPartitionConfig;
import org.apache.pinot.spi.config.table.assignment.InstanceTagPoolConfig;
+import org.apache.pinot.spi.config.table.ingestion.BatchIngestionConfig;
import org.apache.pinot.spi.config.table.ingestion.FilterConfig;
+import org.apache.pinot.spi.config.table.ingestion.IngestionConfig;
import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
import org.apache.pinot.spi.config.table.ingestion.TransformConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.JsonUtils;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.testng.annotations.Test;
diff --git a/pinot-common/src/test/java/org/apache/pinot/common/utils/fetcher/SegmentFetcherFactoryTest.java b/pinot-common/src/test/java/org/apache/pinot/common/utils/fetcher/SegmentFetcherFactoryTest.java
index 1472bf5..54679a3 100644
--- a/pinot-common/src/test/java/org/apache/pinot/common/utils/fetcher/SegmentFetcherFactoryTest.java
+++ b/pinot-common/src/test/java/org/apache/pinot/common/utils/fetcher/SegmentFetcherFactoryTest.java
@@ -18,22 +18,21 @@
*/
package org.apache.pinot.common.utils.fetcher;
-import static org.apache.pinot.common.utils.CommonConstants.HTTPS_PROTOCOL;
-import static org.apache.pinot.common.utils.CommonConstants.HTTP_PROTOCOL;
-import static org.testng.Assert.assertEquals;
-
import java.io.File;
import java.net.URI;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-
import org.apache.pinot.spi.crypt.PinotCrypter;
import org.apache.pinot.spi.crypt.PinotCrypterFactory;
import org.apache.pinot.spi.env.PinotConfiguration;
import org.testng.annotations.Test;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTPS_PROTOCOL;
+import static org.apache.pinot.spi.utils.CommonConstants.HTTP_PROTOCOL;
+import static org.testng.Assert.assertEquals;
+
public class SegmentFetcherFactoryTest {
private static final String FILE_PROTOCOL = "file";
diff --git a/pinot-core/src/test/java/org/apache/pinot/util/TestUtils.java b/pinot-common/src/test/java/org/apache/pinot/util/TestUtils.java
similarity index 100%
rename from pinot-core/src/test/java/org/apache/pinot/util/TestUtils.java
rename to pinot-common/src/test/java/org/apache/pinot/util/TestUtils.java
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java b/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
index 739dd28..ce91c01 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
@@ -28,13 +28,13 @@ import java.util.Random;
import org.apache.commons.configuration.Configuration;
import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
import org.apache.pinot.common.protocols.SegmentCompletionProtocol;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.spi.env.PinotConfiguration;
import org.apache.pinot.spi.filesystem.LocalPinotFS;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.TimeUtils;
-import static org.apache.pinot.common.utils.CommonConstants.Controller.CONFIG_OF_CONTROLLER_METRICS_PREFIX;
-import static org.apache.pinot.common.utils.CommonConstants.Controller.DEFAULT_METRICS_PREFIX;
+import static org.apache.pinot.spi.utils.CommonConstants.Controller.CONFIG_OF_CONTROLLER_METRICS_PREFIX;
+import static org.apache.pinot.spi.utils.CommonConstants.Controller.DEFAULT_METRICS_PREFIX;
public class ControllerConf extends PinotConfiguration {
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerStarter.java b/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerStarter.java
index f05a863..49fc615 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerStarter.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerStarter.java
@@ -51,10 +51,8 @@ import org.apache.pinot.common.Utils;
import org.apache.pinot.common.function.FunctionRegistry;
import org.apache.pinot.common.metrics.ControllerMeter;
import org.apache.pinot.common.metrics.ControllerMetrics;
-import org.apache.pinot.spi.metrics.PinotMetricsRegistry;
import org.apache.pinot.common.metrics.PinotMetricUtils;
import org.apache.pinot.common.metrics.ValidationMetrics;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.NetUtil;
import org.apache.pinot.common.utils.ServiceStatus;
import org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory;
@@ -62,12 +60,11 @@ import org.apache.pinot.common.utils.helix.LeadControllerUtils;
import org.apache.pinot.controller.api.ControllerAdminApiApplication;
import org.apache.pinot.controller.api.access.AccessControlFactory;
import org.apache.pinot.controller.api.events.MetadataEventNotifierFactory;
-import org.apache.pinot.controller.helix.core.minion.MinionInstancesCleanupTask;
-import org.apache.pinot.core.transport.ListenerConfig;
import org.apache.pinot.controller.api.resources.ControllerFilePathProvider;
import org.apache.pinot.controller.api.resources.InvalidControllerConfigException;
import org.apache.pinot.controller.helix.SegmentStatusChecker;
import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
+import org.apache.pinot.controller.helix.core.minion.MinionInstancesCleanupTask;
import org.apache.pinot.controller.helix.core.minion.PinotHelixTaskResourceManager;
import org.apache.pinot.controller.helix.core.minion.PinotTaskManager;
import org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager;
@@ -78,19 +75,22 @@ import org.apache.pinot.controller.helix.core.retention.RetentionManager;
import org.apache.pinot.controller.helix.core.statemodel.LeadControllerResourceMasterSlaveStateModelFactory;
import org.apache.pinot.controller.helix.core.util.HelixSetupUtils;
import org.apache.pinot.controller.helix.starter.HelixConfig;
-import org.apache.pinot.core.util.ListenerConfigUtil;
import org.apache.pinot.controller.validation.BrokerResourceValidationManager;
import org.apache.pinot.controller.validation.OfflineSegmentIntervalChecker;
import org.apache.pinot.controller.validation.RealtimeSegmentValidationManager;
import org.apache.pinot.core.periodictask.PeriodicTask;
import org.apache.pinot.core.periodictask.PeriodicTaskScheduler;
+import org.apache.pinot.core.transport.ListenerConfig;
import org.apache.pinot.core.transport.TlsConfig;
+import org.apache.pinot.core.util.ListenerConfigUtil;
import org.apache.pinot.core.util.TlsUtils;
import org.apache.pinot.spi.crypt.PinotCrypterFactory;
import org.apache.pinot.spi.env.PinotConfiguration;
import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.metrics.PinotMetricsRegistry;
import org.apache.pinot.spi.services.ServiceRole;
import org.apache.pinot.spi.services.ServiceStartable;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.glassfish.hk2.utilities.binding.AbstractBinder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/LeadControllerManager.java b/pinot-controller/src/main/java/org/apache/pinot/controller/LeadControllerManager.java
index c103f82..f7e3643 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/LeadControllerManager.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/LeadControllerManager.java
@@ -25,8 +25,8 @@ import org.apache.helix.HelixManager;
import org.apache.pinot.common.metrics.ControllerGauge;
import org.apache.pinot.common.metrics.ControllerMeter;
import org.apache.pinot.common.metrics.ControllerMetrics;
-import org.apache.pinot.common.utils.CommonConstants.Helix;
import org.apache.pinot.common.utils.helix.LeadControllerUtils;
+import org.apache.pinot.spi.utils.CommonConstants.Helix;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/ControllerAdminApiApplication.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/ControllerAdminApiApplication.java
index 0fea2bd..6c3389b 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/ControllerAdminApiApplication.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/ControllerAdminApiApplication.java
@@ -26,10 +26,10 @@ import java.util.List;
import javax.ws.rs.container.ContainerRequestContext;
import javax.ws.rs.container.ContainerResponseContext;
import javax.ws.rs.container.ContainerResponseFilter;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.controller.api.access.AuthenticationFilter;
import org.apache.pinot.core.transport.ListenerConfig;
import org.apache.pinot.core.util.ListenerConfigUtil;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.glassfish.grizzly.http.server.CLStaticHttpHandler;
import org.glassfish.grizzly.http.server.HttpServer;
import org.glassfish.hk2.utilities.binding.AbstractBinder;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ControllerFilePathProvider.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ControllerFilePathProvider.java
index 4df9b30..c75b98f 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ControllerFilePathProvider.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ControllerFilePathProvider.java
@@ -23,11 +23,11 @@ import java.io.File;
import java.io.IOException;
import java.net.URI;
import org.apache.commons.io.FileUtils;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.URIUtils;
import org.apache.pinot.controller.ControllerConf;
-import org.apache.pinot.spi.filesystem.PinotFSFactory;
import org.apache.pinot.spi.filesystem.PinotFS;
+import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/LLCSegmentCompletionHandlers.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/LLCSegmentCompletionHandlers.java
index 2f79aa9..3b6c3d3 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/LLCSegmentCompletionHandlers.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/LLCSegmentCompletionHandlers.java
@@ -47,8 +47,8 @@ import org.apache.pinot.controller.api.access.Authenticate;
import org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager;
import org.apache.pinot.controller.helix.core.realtime.segment.CommittingSegmentDescriptor;
import org.apache.pinot.controller.util.SegmentCompletionUtils;
-import org.apache.pinot.core.segment.creator.impl.V1Constants;
-import org.apache.pinot.core.segment.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.local.segment.creator.impl.V1Constants;
+import org.apache.pinot.segment.local.segment.index.metadata.SegmentMetadataImpl;
import org.apache.pinot.spi.filesystem.PinotFS;
import org.apache.pinot.spi.filesystem.PinotFSFactory;
import org.glassfish.jersey.media.multipart.FormDataBodyPart;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotBrokerRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotBrokerRestletResource.java
index 600a41f..052e394 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotBrokerRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotBrokerRestletResource.java
@@ -42,10 +42,10 @@ import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import org.apache.helix.model.InstanceConfig;
import org.apache.pinot.common.exception.TableNotFoundException;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.controller.api.access.AccessType;
import org.apache.pinot.controller.api.access.Authenticate;
import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotControllerResponseFilter.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotControllerResponseFilter.java
index 2d01c85..f723dd9 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotControllerResponseFilter.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotControllerResponseFilter.java
@@ -25,8 +25,8 @@ import javax.ws.rs.container.ContainerResponseContext;
import javax.ws.rs.container.ContainerResponseFilter;
import javax.ws.rs.ext.Provider;
import org.apache.pinot.common.Utils;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.NetUtil;
+import org.apache.pinot.spi.utils.CommonConstants;
// A class to add the controller host and version in the response headers for all APIs.
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotIngestionRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotIngestionRestletResource.java
index 0695db5..00f1e68 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotIngestionRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotIngestionRestletResource.java
@@ -37,7 +37,6 @@ import javax.ws.rs.container.AsyncResponse;
import javax.ws.rs.container.Suspended;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.controller.ControllerConf;
import org.apache.pinot.controller.api.access.AccessType;
import org.apache.pinot.controller.api.access.Authenticate;
@@ -48,6 +47,7 @@ import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.spi.ingestion.batch.BatchConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.JsonUtils;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.glassfish.jersey.media.multipart.FormDataMultiPart;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotInstanceRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotInstanceRestletResource.java
index 8769d7d..00711e6 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotInstanceRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotInstanceRestletResource.java
@@ -39,13 +39,13 @@ import javax.ws.rs.QueryParam;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import org.apache.helix.model.InstanceConfig;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.config.InstanceUtils;
import org.apache.pinot.controller.api.access.AccessType;
import org.apache.pinot.controller.api.access.Authenticate;
import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
import org.apache.pinot.controller.helix.core.PinotResourceManagerResponse;
import org.apache.pinot.spi.config.instance.Instance;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.JsonUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotLeadControllerRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotLeadControllerRestletResource.java
index 5fb9b15..247c6b1 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotLeadControllerRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotLeadControllerRestletResource.java
@@ -40,9 +40,9 @@ import javax.ws.rs.core.Response;
import org.apache.helix.HelixManager;
import org.apache.helix.model.ExternalView;
import org.apache.helix.model.MasterSlaveSMD;
-import org.apache.pinot.common.utils.CommonConstants.Helix;
import org.apache.pinot.common.utils.helix.LeadControllerUtils;
import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
+import org.apache.pinot.spi.utils.CommonConstants.Helix;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java
index 41a2f02..a37fc81 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java
@@ -46,12 +46,12 @@ import org.apache.commons.lang3.tuple.Pair;
import org.apache.helix.model.InstanceConfig;
import org.apache.pinot.common.Utils;
import org.apache.pinot.common.exception.QueryException;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.controller.ControllerConf;
import org.apache.pinot.controller.api.access.AccessControl;
import org.apache.pinot.controller.api.access.AccessControlFactory;
import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
import org.apache.pinot.pql.parsers.Pql2Compiler;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.JsonUtils;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.apache.pinot.sql.parsers.CalciteSqlCompiler;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSchemaRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSchemaRestletResource.java
index ee7a760..9e829f6 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSchemaRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSchemaRestletResource.java
@@ -55,7 +55,7 @@ import org.apache.pinot.controller.api.access.Authenticate;
import org.apache.pinot.controller.api.events.MetadataEventNotifierFactory;
import org.apache.pinot.controller.api.events.SchemaEventType;
import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
-import org.apache.pinot.core.util.SchemaUtils;
+import org.apache.pinot.segment.local.utils.SchemaUtils;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.spi.utils.JsonUtils;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
index f4e6807..1c260ef 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
@@ -32,7 +32,6 @@ import java.net.InetAddress;
import java.net.URI;
import java.nio.file.Files;
import java.util.List;
-import java.util.Locale;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.Executor;
@@ -63,7 +62,6 @@ import org.apache.commons.lang3.tuple.Pair;
import org.apache.pinot.common.metrics.ControllerMeter;
import org.apache.pinot.common.metrics.ControllerMetrics;
import org.apache.pinot.common.restlet.resources.StartReplaceSegmentsRequest;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.FileUploadDownloadClient;
import org.apache.pinot.common.utils.StringUtil;
import org.apache.pinot.common.utils.URIUtils;
@@ -85,6 +83,7 @@ import org.apache.pinot.spi.crypt.PinotCrypter;
import org.apache.pinot.spi.crypt.PinotCrypterFactory;
import org.apache.pinot.spi.filesystem.PinotFS;
import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.JsonUtils;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.glassfish.grizzly.http.server.Request;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableIndexingConfigs.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableIndexingConfigs.java
index 3162e63..f5ad68a 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableIndexingConfigs.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableIndexingConfigs.java
@@ -33,7 +33,7 @@ import javax.ws.rs.core.Response;
import org.apache.pinot.controller.api.access.AccessType;
import org.apache.pinot.controller.api.access.Authenticate;
import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
-import org.apache.pinot.core.util.TableConfigUtils;
+import org.apache.pinot.segment.local.utils.TableConfigUtils;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.spi.utils.JsonUtils;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableMetadataConfigs.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableMetadataConfigs.java
index 66cc10d..639d5dd 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableMetadataConfigs.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableMetadataConfigs.java
@@ -32,7 +32,7 @@ import javax.ws.rs.core.Response;
import org.apache.pinot.controller.api.access.AccessType;
import org.apache.pinot.controller.api.access.Authenticate;
import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
-import org.apache.pinot.core.util.TableConfigUtils;
+import org.apache.pinot.segment.local.utils.TableConfigUtils;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.spi.utils.JsonUtils;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableRestletResource.java
index 8559542..d4d06f7 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableRestletResource.java
@@ -67,7 +67,7 @@ import org.apache.pinot.controller.helix.core.rebalance.RebalanceResult;
import org.apache.pinot.controller.recommender.RecommenderDriver;
import org.apache.pinot.core.common.MinionConstants;
import org.apache.pinot.core.util.ReplicationUtils;
-import org.apache.pinot.core.util.TableConfigUtils;
+import org.apache.pinot.segment.local.utils.TableConfigUtils;
import org.apache.pinot.spi.config.table.QuotaConfig;
import org.apache.pinot.spi.config.table.SegmentsValidationAndRetentionConfig;
import org.apache.pinot.spi.config.table.TableConfig;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableSegmentConfigs.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableSegmentConfigs.java
index 2a9c630..6000f66 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableSegmentConfigs.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableSegmentConfigs.java
@@ -34,7 +34,7 @@ import org.apache.pinot.common.metrics.ControllerMetrics;
import org.apache.pinot.controller.api.access.AccessType;
import org.apache.pinot.controller.api.access.Authenticate;
import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
-import org.apache.pinot.core.util.TableConfigUtils;
+import org.apache.pinot.segment.local.utils.TableConfigUtils;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.spi.utils.JsonUtils;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
index 446c541..42c83b0 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
@@ -89,11 +89,6 @@ import org.apache.pinot.common.metadata.instance.InstanceZKMetadata;
import org.apache.pinot.common.metadata.segment.LLCRealtimeSegmentZKMetadata;
import org.apache.pinot.common.metadata.segment.OfflineSegmentZKMetadata;
import org.apache.pinot.common.metadata.segment.RealtimeSegmentZKMetadata;
-import org.apache.pinot.common.utils.CommonConstants;
-import org.apache.pinot.common.utils.CommonConstants.Helix;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.BrokerResourceStateModel;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
-import org.apache.pinot.common.utils.CommonConstants.Server;
import org.apache.pinot.common.utils.HashUtil;
import org.apache.pinot.common.utils.SchemaUtils;
import org.apache.pinot.common.utils.config.InstanceUtils;
@@ -129,6 +124,11 @@ import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
import org.apache.pinot.spi.config.tenant.Tenant;
import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants.Helix;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.BrokerResourceStateModel;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.apache.pinot.spi.utils.CommonConstants.Server;
import org.apache.pinot.spi.utils.IngestionConfigUtils;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.apache.pinot.spi.utils.retry.RetryPolicies;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineDimTableSegmentAssignment.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineDimTableSegmentAssignment.java
index 3ed2184..5b4f88f 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineDimTableSegmentAssignment.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineDimTableSegmentAssignment.java
@@ -20,23 +20,21 @@ package org.apache.pinot.controller.helix.core.assignment.segment;
import com.google.common.base.Preconditions;
import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
import java.util.Set;
+import java.util.TreeMap;
+import javax.annotation.Nullable;
import org.apache.commons.configuration.Configuration;
import org.apache.helix.HelixManager;
import org.apache.pinot.common.assignment.InstancePartitions;
import org.apache.pinot.common.tier.Tier;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.config.TagNameUtils;
import org.apache.pinot.common.utils.helix.HelixHelper;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TenantConfig;
import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
-
-import javax.annotation.Nullable;
-
-import java.util.List;
-import java.util.Map;
-import java.util.TreeMap;
+import org.apache.pinot.spi.utils.CommonConstants;
/**
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineSegmentAssignment.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineSegmentAssignment.java
index 3955bb5..fbc20a2 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineSegmentAssignment.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineSegmentAssignment.java
@@ -36,11 +36,11 @@ import org.apache.pinot.common.metadata.ZKMetadataProvider;
import org.apache.pinot.common.metadata.segment.ColumnPartitionMetadata;
import org.apache.pinot.common.metadata.segment.OfflineSegmentZKMetadata;
import org.apache.pinot.common.tier.Tier;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.apache.pinot.controller.helix.core.rebalance.RebalanceConfigConstants;
import org.apache.pinot.spi.config.table.ReplicaGroupStrategyConfig;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeSegmentAssignment.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeSegmentAssignment.java
index 99fb465..0e1974a 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeSegmentAssignment.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeSegmentAssignment.java
@@ -32,12 +32,12 @@ import org.apache.commons.configuration.Configuration;
import org.apache.helix.HelixManager;
import org.apache.pinot.common.assignment.InstancePartitions;
import org.apache.pinot.common.tier.Tier;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
-import org.apache.pinot.controller.helix.core.rebalance.RebalanceConfigConstants;
import org.apache.pinot.common.utils.SegmentUtils;
+import org.apache.pinot.controller.helix.core.rebalance.RebalanceConfigConstants;
import org.apache.pinot.spi.config.table.ReplicaGroupStrategyConfig;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -94,7 +94,7 @@ public class RealtimeSegmentAssignment implements SegmentAssignment {
_realtimeTableName = tableConfig.getTableName();
_replication = tableConfig.getValidationConfig().getReplicasPerPartitionNumber();
ReplicaGroupStrategyConfig replicaGroupStrategyConfig =
- tableConfig.getValidationConfig().getReplicaGroupStrategyConfig();
+ tableConfig.getValidationConfig().getReplicaGroupStrategyConfig();
_partitionColumn = replicaGroupStrategyConfig != null ? replicaGroupStrategyConfig.getPartitionColumn() : null;
LOGGER.info("Initialized RealtimeSegmentAssignment with replication: {}, partitionColumn: {} for table: {}",
@@ -180,9 +180,8 @@ public class RealtimeSegmentAssignment implements SegmentAssignment {
@Override
public Map<String, Map<String, String>> rebalanceTable(Map<String, Map<String, String>> currentAssignment,
- Map<InstancePartitionsType, InstancePartitions> instancePartitionsMap,
- @Nullable List<Tier> sortedTiers, @Nullable Map<String, InstancePartitions> tierInstancePartitionsMap,
- Configuration config) {
+ Map<InstancePartitionsType, InstancePartitions> instancePartitionsMap, @Nullable List<Tier> sortedTiers,
+ @Nullable Map<String, InstancePartitions> tierInstancePartitionsMap, Configuration config) {
InstancePartitions completedInstancePartitions = instancePartitionsMap.get(InstancePartitionsType.COMPLETED);
InstancePartitions consumingInstancePartitions = instancePartitionsMap.get(InstancePartitionsType.CONSUMING);
@@ -304,8 +303,7 @@ public class RealtimeSegmentAssignment implements SegmentAssignment {
* Rebalances segments in the current assignment using the instancePartitions and returns new assignment
*/
private Map<String, Map<String, String>> reassignSegments(String instancePartitionType,
- Map<String, Map<String, String>> currentAssignment, InstancePartitions instancePartitions,
- boolean bootstrap) {
+ Map<String, Map<String, String>> currentAssignment, InstancePartitions instancePartitions, boolean bootstrap) {
Map<String, Map<String, String>> newAssignment;
if (bootstrap) {
LOGGER.info("Bootstrapping segment assignment for {} segments of table: {}", instancePartitionType,
@@ -322,8 +320,8 @@ public class RealtimeSegmentAssignment implements SegmentAssignment {
if (instancePartitions.getNumReplicaGroups() == 1) {
// Non-replica-group based assignment
- List<String> instances = SegmentAssignmentUtils
- .getInstancesForNonReplicaGroupBasedAssignment(instancePartitions, _replication);
+ List<String> instances =
+ SegmentAssignmentUtils.getInstancesForNonReplicaGroupBasedAssignment(instancePartitions, _replication);
newAssignment = SegmentAssignmentUtils
.rebalanceTableWithHelixAutoRebalanceStrategy(currentAssignment, instances, _replication);
} else {
@@ -345,8 +343,7 @@ public class RealtimeSegmentAssignment implements SegmentAssignment {
}
newAssignment = SegmentAssignmentUtils
- .rebalanceReplicaGroupBasedTable(currentAssignment, instancePartitions,
- partitionGroupIdToSegmentsMap);
+ .rebalanceReplicaGroupBasedTable(currentAssignment, instancePartitions, partitionGroupIdToSegmentsMap);
}
}
return newAssignment;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentUtils.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentUtils.java
index b0568e7..f50ba0f 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentUtils.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentUtils.java
@@ -31,7 +31,7 @@ import java.util.TreeMap;
import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
import org.apache.pinot.common.assignment.InstancePartitions;
import org.apache.pinot.common.tier.Tier;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.apache.pinot.spi.utils.Pairs;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/MinionInstancesCleanupTask.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/MinionInstancesCleanupTask.java
index 0d0feed..87fd80b 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/MinionInstancesCleanupTask.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/MinionInstancesCleanupTask.java
@@ -22,11 +22,11 @@ import java.util.ArrayList;
import java.util.List;
import org.apache.pinot.common.metrics.ControllerGauge;
import org.apache.pinot.common.metrics.ControllerMetrics;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.controller.ControllerConf;
import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
import org.apache.pinot.controller.helix.core.PinotResourceManagerResponse;
import org.apache.pinot.core.periodictask.BasePeriodicTask;
+import org.apache.pinot.spi.utils.CommonConstants;
/**
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotHelixTaskResourceManager.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotHelixTaskResourceManager.java
index 096176b..ee08384 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotHelixTaskResourceManager.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotHelixTaskResourceManager.java
@@ -34,8 +34,8 @@ import org.apache.helix.task.TaskConfig;
import org.apache.helix.task.TaskDriver;
import org.apache.helix.task.TaskState;
import org.apache.helix.task.WorkflowConfig;
-import org.apache.pinot.common.utils.CommonConstants.Helix;
import org.apache.pinot.core.minion.PinotTaskConfig;
+import org.apache.pinot.spi.utils.CommonConstants.Helix;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
index b3f6e1f..2e1dcc3 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
@@ -49,9 +49,6 @@ import org.apache.pinot.common.metadata.segment.SegmentPartitionMetadata;
import org.apache.pinot.common.metrics.ControllerMeter;
import org.apache.pinot.common.metrics.ControllerMetrics;
import org.apache.pinot.common.protocols.SegmentCompletionProtocol;
-import org.apache.pinot.common.utils.CommonConstants;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
-import org.apache.pinot.common.utils.CommonConstants.Segment.Realtime.Status;
import org.apache.pinot.common.utils.LLCSegmentName;
import org.apache.pinot.common.utils.SegmentName;
import org.apache.pinot.common.utils.URIUtils;
@@ -67,8 +64,8 @@ import org.apache.pinot.controller.helix.core.realtime.segment.CommittingSegment
import org.apache.pinot.controller.helix.core.realtime.segment.FlushThresholdUpdateManager;
import org.apache.pinot.controller.helix.core.realtime.segment.FlushThresholdUpdater;
import org.apache.pinot.controller.util.SegmentCompletionUtils;
-import org.apache.pinot.core.segment.index.metadata.ColumnMetadata;
-import org.apache.pinot.core.segment.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.local.segment.index.metadata.ColumnMetadata;
+import org.apache.pinot.segment.local.segment.index.metadata.SegmentMetadataImpl;
import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
import org.apache.pinot.spi.config.table.TableConfig;
@@ -76,14 +73,17 @@ import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
import org.apache.pinot.spi.filesystem.PinotFS;
import org.apache.pinot.spi.filesystem.PinotFSFactory;
import org.apache.pinot.spi.stream.OffsetCriteria;
-import org.apache.pinot.spi.stream.PartitionGroupMetadata;
import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.PartitionGroupMetadata;
import org.apache.pinot.spi.stream.PartitionLevelStreamConfig;
import org.apache.pinot.spi.stream.StreamConfig;
import org.apache.pinot.spi.stream.StreamConfigProperties;
import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.Realtime.Status;
import org.apache.pinot.spi.utils.IngestionConfigUtils;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.apache.pinot.spi.utils.retry.RetryPolicies;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotRealtimeSegmentManager.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotRealtimeSegmentManager.java
index b5e9ef7..76537a0 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotRealtimeSegmentManager.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotRealtimeSegmentManager.java
@@ -42,9 +42,6 @@ import org.apache.pinot.common.metadata.instance.InstanceZKMetadata;
import org.apache.pinot.common.metadata.segment.RealtimeSegmentZKMetadata;
import org.apache.pinot.common.metrics.ControllerMeter;
import org.apache.pinot.common.metrics.ControllerMetrics;
-import org.apache.pinot.common.utils.CommonConstants;
-import org.apache.pinot.common.utils.CommonConstants.Segment.Realtime.Status;
-import org.apache.pinot.common.utils.CommonConstants.Segment.SegmentType;
import org.apache.pinot.common.utils.HLCSegmentName;
import org.apache.pinot.common.utils.SegmentName;
import org.apache.pinot.common.utils.config.TableConfigUtils;
@@ -52,11 +49,14 @@ import org.apache.pinot.common.utils.helix.HelixHelper;
import org.apache.pinot.controller.LeadControllerManager;
import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
import org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder;
-import org.apache.pinot.core.query.utils.Pair;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.Realtime.Status;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.SegmentType;
import org.apache.pinot.spi.utils.IngestionConfigUtils;
+import org.apache.pinot.spi.utils.Pair;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.apache.pinot.spi.utils.retry.RetryPolicies;
import org.apache.zookeeper.data.Stat;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java
index 143d35c..1338f1a 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java
@@ -32,7 +32,6 @@ import org.apache.pinot.common.metadata.segment.LLCRealtimeSegmentZKMetadata;
import org.apache.pinot.common.metrics.ControllerMeter;
import org.apache.pinot.common.metrics.ControllerMetrics;
import org.apache.pinot.common.protocols.SegmentCompletionProtocol;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.LLCSegmentName;
import org.apache.pinot.common.utils.URIUtils;
import org.apache.pinot.controller.LeadControllerManager;
@@ -42,6 +41,7 @@ import org.apache.pinot.spi.stream.PartitionLevelStreamConfig;
import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.IngestionConfigUtils;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.slf4j.Logger;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/segment/CommittingSegmentDescriptor.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/segment/CommittingSegmentDescriptor.java
index cf6b916..36989be 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/segment/CommittingSegmentDescriptor.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/segment/CommittingSegmentDescriptor.java
@@ -19,7 +19,7 @@
package org.apache.pinot.controller.helix.core.realtime.segment;
import org.apache.pinot.common.protocols.SegmentCompletionProtocol;
-import org.apache.pinot.core.segment.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.local.segment.index.metadata.SegmentMetadataImpl;
/**
* Class to hold properties of the committing segment
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java
index bc1fe0e..3ff3b61 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java
@@ -42,7 +42,6 @@ import org.apache.pinot.common.assignment.InstancePartitionsUtils;
import org.apache.pinot.common.tier.PinotServerTierStorage;
import org.apache.pinot.common.tier.Tier;
import org.apache.pinot.common.tier.TierFactory;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.apache.pinot.common.utils.config.TierConfigUtils;
import org.apache.pinot.controller.helix.core.assignment.instance.InstanceAssignmentDriver;
import org.apache.pinot.controller.helix.core.assignment.segment.SegmentAssignment;
@@ -53,6 +52,7 @@ import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.apache.pinot.spi.utils.IngestionConfigUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java
index a7a5f5a..bfe82f8 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java
@@ -33,8 +33,6 @@ import org.apache.pinot.common.lineage.SegmentLineageAccessHelper;
import org.apache.pinot.common.metadata.segment.OfflineSegmentZKMetadata;
import org.apache.pinot.common.metadata.segment.RealtimeSegmentZKMetadata;
import org.apache.pinot.common.metrics.ControllerMetrics;
-import org.apache.pinot.common.utils.CommonConstants;
-import org.apache.pinot.common.utils.CommonConstants.Segment.Realtime.Status;
import org.apache.pinot.common.utils.SegmentName;
import org.apache.pinot.controller.ControllerConf;
import org.apache.pinot.controller.LeadControllerManager;
@@ -45,6 +43,8 @@ import org.apache.pinot.controller.helix.core.retention.strategy.TimeRetentionSt
import org.apache.pinot.spi.config.table.SegmentsValidationAndRetentionConfig;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.Realtime.Status;
import org.apache.pinot.spi.utils.IngestionConfigUtils;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.apache.pinot.spi.utils.retry.RetryPolicies;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/util/HelixSetupUtils.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/util/HelixSetupUtils.java
index baa094d..41da3c4 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/util/HelixSetupUtils.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/util/HelixSetupUtils.java
@@ -43,14 +43,14 @@ import org.apache.helix.model.StateModelDefinition;
import org.apache.helix.model.builder.CustomModeISBuilder;
import org.apache.helix.model.builder.FullAutoModeISBuilder;
import org.apache.helix.model.builder.HelixConfigScopeBuilder;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.helix.LeadControllerUtils;
import org.apache.pinot.controller.helix.core.PinotHelixBrokerResourceOnlineOfflineStateModelGenerator;
import org.apache.pinot.controller.helix.core.PinotHelixSegmentOnlineOfflineStateModelGenerator;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.*;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.*;
public class HelixSetupUtils {
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/util/ZKMetadataUtils.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/util/ZKMetadataUtils.java
index cbee3ff..324ea00 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/util/ZKMetadataUtils.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/util/ZKMetadataUtils.java
@@ -20,13 +20,15 @@ package org.apache.pinot.controller.helix.core.util;
import java.util.HashMap;
import java.util.Map;
-
-import org.apache.pinot.common.metadata.segment.*;
-import org.apache.pinot.common.utils.CommonConstants.Segment.SegmentType;
-import org.apache.pinot.core.segment.index.metadata.ColumnMetadata;
-import org.apache.pinot.core.segment.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.common.metadata.segment.ColumnPartitionMetadata;
+import org.apache.pinot.common.metadata.segment.SegmentPartitionMetadata;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadataCustomMapModifier;
+import org.apache.pinot.segment.local.segment.index.metadata.ColumnMetadata;
+import org.apache.pinot.segment.local.segment.index.metadata.SegmentMetadataImpl;
import org.apache.pinot.segment.spi.SegmentMetadata;
import org.apache.pinot.segment.spi.partition.PartitionFunction;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.SegmentType;
public class ZKMetadataUtils {
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/realtime/provisioning/MemoryEstimator.java b/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/realtime/provisioning/MemoryEstimator.java
index 2a9811a..f0bc20c 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/realtime/provisioning/MemoryEstimator.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/realtime/provisioning/MemoryEstimator.java
@@ -35,7 +35,6 @@ import java.util.stream.Stream;
import org.apache.commons.collections.CollectionUtils;
import org.apache.commons.io.FileUtils;
import org.apache.pinot.common.metadata.segment.RealtimeSegmentZKMetadata;
-import org.apache.pinot.common.segment.ReadMode;
import org.apache.pinot.controller.recommender.data.generator.DataGenerator;
import org.apache.pinot.controller.recommender.data.generator.DataGeneratorSpec;
import org.apache.pinot.controller.recommender.io.metadata.DateTimeFieldSpecMetadata;
@@ -43,15 +42,15 @@ import org.apache.pinot.controller.recommender.io.metadata.FieldMetadata;
import org.apache.pinot.controller.recommender.io.metadata.SchemaWithMetaData;
import org.apache.pinot.controller.recommender.io.metadata.TimeFieldSpecMetadata;
import org.apache.pinot.controller.recommender.io.metadata.TimeGranularitySpecMetadata;
-import org.apache.pinot.core.data.readers.PinotSegmentRecordReader;
-import org.apache.pinot.core.indexsegment.immutable.ImmutableSegmentLoader;
-import org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl;
-import org.apache.pinot.core.io.readerwriter.RealtimeIndexOffHeapMemoryManager;
-import org.apache.pinot.core.io.writer.impl.DirectMemoryManager;
-import org.apache.pinot.core.realtime.impl.RealtimeSegmentConfig;
-import org.apache.pinot.core.realtime.impl.RealtimeSegmentStatsHistory;
-import org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl;
-import org.apache.pinot.core.segment.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.readerwriter.RealtimeIndexOffHeapMemoryManager;
+import org.apache.pinot.segment.local.io.writer.impl.DirectMemoryManager;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.local.segment.readers.PinotSegmentRecordReader;
import org.apache.pinot.segment.spi.ImmutableSegment;
import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
import org.apache.pinot.segment.spi.creator.SegmentIndexCreationDriver;
@@ -62,6 +61,7 @@ import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.spi.data.readers.FileFormat;
import org.apache.pinot.spi.data.readers.GenericRow;
import org.apache.pinot.spi.utils.DataSizeUtils;
+import org.apache.pinot.spi.utils.ReadMode;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/BloomFilterRule.java b/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/BloomFilterRule.java
index ad165cc..f1a8d7e 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/BloomFilterRule.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/BloomFilterRule.java
@@ -19,15 +19,15 @@
package org.apache.pinot.controller.recommender.rules.impl;
import com.google.common.util.concurrent.AtomicDouble;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.common.request.context.FilterContext;
+import org.apache.pinot.common.request.context.predicate.Predicate;
import org.apache.pinot.controller.recommender.io.ConfigManager;
import org.apache.pinot.controller.recommender.io.InputManager;
import org.apache.pinot.controller.recommender.rules.AbstractRule;
import org.apache.pinot.controller.recommender.rules.io.params.BloomFilterRuleParams;
import org.apache.pinot.controller.recommender.rules.utils.FixedLenBitset;
-import org.apache.pinot.core.query.request.context.ExpressionContext;
-import org.apache.pinot.core.query.request.context.FilterContext;
import org.apache.pinot.core.query.request.context.QueryContext;
-import org.apache.pinot.core.query.request.context.predicate.Predicate;
import org.apache.pinot.core.requesthandler.BrokerRequestOptimizer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/NoDictionaryOnHeapDictionaryJointRule.java b/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/NoDictionaryOnHeapDictionaryJointRule.java
index eb3e345..e57377d 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/NoDictionaryOnHeapDictionaryJointRule.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/NoDictionaryOnHeapDictionaryJointRule.java
@@ -23,16 +23,16 @@ import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.common.request.context.FilterContext;
+import org.apache.pinot.common.request.context.predicate.Predicate;
import org.apache.pinot.controller.recommender.exceptions.InvalidInputException;
import org.apache.pinot.controller.recommender.io.ConfigManager;
import org.apache.pinot.controller.recommender.io.InputManager;
import org.apache.pinot.controller.recommender.rules.AbstractRule;
import org.apache.pinot.controller.recommender.rules.io.params.NoDictionaryOnHeapDictionaryJointRuleParams;
import org.apache.pinot.controller.recommender.rules.utils.FixedLenBitset;
-import org.apache.pinot.core.query.request.context.ExpressionContext;
-import org.apache.pinot.core.query.request.context.FilterContext;
import org.apache.pinot.core.query.request.context.QueryContext;
-import org.apache.pinot.core.query.request.context.predicate.Predicate;
import org.apache.pinot.core.requesthandler.BrokerRequestOptimizer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/PinotTablePartitionRule.java b/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/PinotTablePartitionRule.java
index 0e7cf57..5900c4d 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/PinotTablePartitionRule.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/PinotTablePartitionRule.java
@@ -23,17 +23,17 @@ import java.util.Comparator;
import java.util.List;
import java.util.Optional;
import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.common.request.context.FilterContext;
+import org.apache.pinot.common.request.context.predicate.InPredicate;
+import org.apache.pinot.common.request.context.predicate.Predicate;
import org.apache.pinot.controller.recommender.exceptions.InvalidInputException;
import org.apache.pinot.controller.recommender.io.ConfigManager;
import org.apache.pinot.controller.recommender.io.InputManager;
import org.apache.pinot.controller.recommender.rules.AbstractRule;
import org.apache.pinot.controller.recommender.rules.io.params.PartitionRuleParams;
import org.apache.pinot.controller.recommender.rules.utils.FixedLenBitset;
-import org.apache.pinot.core.query.request.context.ExpressionContext;
-import org.apache.pinot.core.query.request.context.FilterContext;
import org.apache.pinot.core.query.request.context.QueryContext;
-import org.apache.pinot.core.query.request.context.predicate.InPredicate;
-import org.apache.pinot.core.query.request.context.predicate.Predicate;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/utils/QueryInvertedSortedIndexRecommender.java b/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/utils/QueryInvertedSortedIndexRecommender.java
index fe43267..8fa8c4c 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/utils/QueryInvertedSortedIndexRecommender.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/utils/QueryInvertedSortedIndexRecommender.java
@@ -28,18 +28,18 @@ import java.util.Map;
import java.util.Optional;
import java.util.stream.Collectors;
import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.common.request.context.FilterContext;
+import org.apache.pinot.common.request.context.predicate.InPredicate;
+import org.apache.pinot.common.request.context.predicate.NotInPredicate;
+import org.apache.pinot.common.request.context.predicate.Predicate;
import org.apache.pinot.controller.recommender.io.InputManager;
import org.apache.pinot.controller.recommender.rules.io.configs.IndexConfig;
import org.apache.pinot.controller.recommender.rules.io.params.InvertedSortedIndexJointRuleParams;
import org.apache.pinot.controller.recommender.rules.io.params.RecommenderConstants;
import org.apache.pinot.core.operator.docidsets.AndDocIdSet;
import org.apache.pinot.core.operator.filter.FilterOperatorUtils;
-import org.apache.pinot.core.query.request.context.ExpressionContext;
-import org.apache.pinot.core.query.request.context.FilterContext;
import org.apache.pinot.core.query.request.context.QueryContext;
-import org.apache.pinot.core.query.request.context.predicate.InPredicate;
-import org.apache.pinot.core.query.request.context.predicate.NotInPredicate;
-import org.apache.pinot.core.query.request.context.predicate.Predicate;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/util/FileIngestionHelper.java b/pinot-controller/src/main/java/org/apache/pinot/controller/util/FileIngestionHelper.java
index 73615f4..c02fa5b 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/util/FileIngestionHelper.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/util/FileIngestionHelper.java
@@ -33,7 +33,7 @@ import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.StringUtils;
import org.apache.pinot.common.utils.TarGzCompressionUtils;
import org.apache.pinot.controller.api.resources.SuccessResponse;
-import org.apache.pinot.core.util.IngestionUtils;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
import org.apache.pinot.spi.auth.AuthContext;
import org.apache.pinot.spi.config.table.TableConfig;
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/util/SegmentIntervalUtils.java b/pinot-controller/src/main/java/org/apache/pinot/controller/util/SegmentIntervalUtils.java
index cdd3291..428bdb1 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/util/SegmentIntervalUtils.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/util/SegmentIntervalUtils.java
@@ -19,8 +19,8 @@
package org.apache.pinot.controller.util;
import org.apache.commons.lang3.StringUtils;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.spi.config.table.SegmentsValidationAndRetentionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.joda.time.Duration;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/ControllerTestUtils.java b/pinot-controller/src/test/java/org/apache/pinot/controller/ControllerTestUtils.java
index 1c4a67c..5a01175 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/ControllerTestUtils.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/ControllerTestUtils.java
@@ -61,14 +61,12 @@ import org.apache.helix.participant.statemachine.StateModelFactory;
import org.apache.helix.participant.statemachine.StateModelInfo;
import org.apache.helix.participant.statemachine.Transition;
import org.apache.helix.store.zk.ZkHelixPropertyStore;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.ZkStarter;
import org.apache.pinot.common.utils.config.TagNameUtils;
import org.apache.pinot.controller.api.AccessControlTest;
import org.apache.pinot.controller.helix.ControllerRequestURLBuilder;
import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
import org.apache.pinot.spi.config.table.TableConfig;
-import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.config.tenant.Tenant;
import org.apache.pinot.spi.config.tenant.TenantRole;
import org.apache.pinot.spi.data.DateTimeFieldSpec;
@@ -76,15 +74,19 @@ import org.apache.pinot.spi.data.DimensionFieldSpec;
import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.MetricFieldSpec;
import org.apache.pinot.spi.data.Schema;
-import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testng.Assert;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.Instance.*;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.*;
-import static org.apache.pinot.common.utils.CommonConstants.Server.*;
-import static org.testng.Assert.*;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.Instance.ADMIN_PORT_KEY;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.LEAD_CONTROLLER_RESOURCE_ENABLED_KEY;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.LEAD_CONTROLLER_RESOURCE_NAME;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.UNTAGGED_BROKER_INSTANCE;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.UNTAGGED_SERVER_INSTANCE;
+import static org.apache.pinot.spi.utils.CommonConstants.Server.DEFAULT_ADMIN_API_PORT;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotNull;
/**
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotInstanceRestletResourceTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotInstanceRestletResourceTest.java
index 94c7f49..156b388 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotInstanceRestletResourceTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotInstanceRestletResourceTest.java
@@ -27,17 +27,17 @@ import java.util.Collections;
import java.util.Map;
import java.util.TreeMap;
import javax.annotation.Nullable;
-import org.apache.pinot.common.utils.CommonConstants.Helix;
import org.apache.pinot.controller.ControllerTestUtils;
import org.apache.pinot.spi.config.instance.Instance;
import org.apache.pinot.spi.config.instance.InstanceType;
+import org.apache.pinot.spi.utils.CommonConstants.Helix;
import org.apache.pinot.spi.utils.JsonUtils;
import org.apache.pinot.util.TestUtils;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
-import static org.testng.Assert.*;
+import static org.testng.Assert.fail;
/**
@@ -48,12 +48,14 @@ public class PinotInstanceRestletResourceTest {
private static final long GET_CALL_TIMEOUT_MS = 10000;
@BeforeClass
- public void setUp() throws Exception {
+ public void setUp()
+ throws Exception {
ControllerTestUtils.setupClusterAndValidate();
}
@Test
- public void testInstanceListingAndCreation() throws Exception {
+ public void testInstanceListingAndCreation()
+ throws Exception {
// Check that there is only one CONTROLLER instance in the cluster
String listInstancesUrl = ControllerTestUtils.getControllerRequestURLBuilder().forInstanceList();
@@ -85,7 +87,7 @@ public class PinotInstanceRestletResourceTest {
Instance brokerInstance = new Instance("1.2.3.4", 1234, InstanceType.BROKER, null, null, 0, 0, false);
ControllerTestUtils.sendPostRequest(createInstanceUrl, brokerInstance.toJsonString());
- Instance serverInstance = new Instance("1.2.3.4", 2345, InstanceType.SERVER, null, null, 8090, 8091,false);
+ Instance serverInstance = new Instance("1.2.3.4", 2345, InstanceType.SERVER, null, null, 8090, 8091, false);
ControllerTestUtils.sendPostRequest(createInstanceUrl, serverInstance.toJsonString());
// Check that we have added two more instances
@@ -142,30 +144,34 @@ public class PinotInstanceRestletResourceTest {
String newServerTag = "new-server-tag";
Instance newServerInstance =
- new Instance("1.2.3.4", 2345, InstanceType.SERVER, Collections.singletonList(newServerTag), null, 28090, 28091, true);
+ new Instance("1.2.3.4", 2345, InstanceType.SERVER, Collections.singletonList(newServerTag), null, 28090, 28091,
+ true);
String serverInstanceId = "Server_1.2.3.4_2345";
String serverInstanceUrl = ControllerTestUtils.getControllerRequestURLBuilder().forInstance(serverInstanceId);
ControllerTestUtils.sendPutRequest(serverInstanceUrl, newServerInstance.toJsonString());
checkInstanceInfo(brokerInstanceId, "Broker_1.2.3.4", 1234, new String[]{newBrokerTag}, null, null, false);
- checkInstanceInfo(serverInstanceId, "Server_1.2.3.4", 2345, new String[]{newServerTag}, null, null, 28090, 28091, true);
+ checkInstanceInfo(serverInstanceId, "Server_1.2.3.4", 2345, new String[]{newServerTag}, null, null, 28090, 28091,
+ true);
// Test Instance updateTags API
- String brokerInstanceUpdateTagsUrl = ControllerTestUtils.getControllerRequestURLBuilder().forInstanceUpdateTags(brokerInstanceId,
- Lists.newArrayList("tag_BROKER", "newTag_BROKER"));
+ String brokerInstanceUpdateTagsUrl = ControllerTestUtils.getControllerRequestURLBuilder()
+ .forInstanceUpdateTags(brokerInstanceId, Lists.newArrayList("tag_BROKER", "newTag_BROKER"));
ControllerTestUtils.sendPutRequest(brokerInstanceUpdateTagsUrl);
- String serverInstanceUpdateTagsUrl = ControllerTestUtils.getControllerRequestURLBuilder().forInstanceUpdateTags(serverInstanceId,
- Lists.newArrayList("tag_REALTIME", "newTag_OFFLINE", "newTag_REALTIME"));
+ String serverInstanceUpdateTagsUrl = ControllerTestUtils.getControllerRequestURLBuilder()
+ .forInstanceUpdateTags(serverInstanceId,
+ Lists.newArrayList("tag_REALTIME", "newTag_OFFLINE", "newTag_REALTIME"));
ControllerTestUtils.sendPutRequest(serverInstanceUpdateTagsUrl);
- checkInstanceInfo(brokerInstanceId, "Broker_1.2.3.4", 1234, new String[]{"tag_BROKER", "newTag_BROKER"}, null,
- null, false);
+ checkInstanceInfo(brokerInstanceId, "Broker_1.2.3.4", 1234, new String[]{"tag_BROKER", "newTag_BROKER"}, null, null,
+ false);
checkInstanceInfo(serverInstanceId, "Server_1.2.3.4", 2345,
new String[]{"tag_REALTIME", "newTag_OFFLINE", "newTag_REALTIME"}, null, null, 28090, 28091, true);
}
private void checkInstanceInfo(String instanceName, String hostName, int port, String[] tags, String[] pools,
int[] poolValues, boolean queriesDisabled) {
- checkInstanceInfo(instanceName, hostName, port, tags, pools, poolValues, Instance.NOT_SET_GRPC_PORT_VALUE, Instance.NOT_SET_ADMIN_PORT_VALUE, queriesDisabled);
+ checkInstanceInfo(instanceName, hostName, port, tags, pools, poolValues, Instance.NOT_SET_GRPC_PORT_VALUE,
+ Instance.NOT_SET_ADMIN_PORT_VALUE, queriesDisabled);
}
private void checkInstanceInfo(String instanceName, String hostName, int port, String[] tags, String[] pools,
@@ -175,16 +181,19 @@ public class PinotInstanceRestletResourceTest {
@Override
public Boolean apply(@Nullable Void aVoid) {
try {
- String getResponse = ControllerTestUtils.sendGetRequest(ControllerTestUtils.getControllerRequestURLBuilder().forInstance(instanceName));
+ String getResponse = ControllerTestUtils
+ .sendGetRequest(ControllerTestUtils.getControllerRequestURLBuilder().forInstance(instanceName));
JsonNode instance = JsonUtils.stringToJsonNode(getResponse);
boolean result =
(instance.get("instanceName") != null) && (instance.get("instanceName").asText().equals(instanceName))
&& (instance.get("hostName") != null) && (instance.get("hostName").asText().equals(hostName)) && (
instance.get("port") != null) && (instance.get("port").asText().equals(String.valueOf(port)))
&& (instance.get("enabled").asBoolean()) && (instance.get("tags") != null) && (
- instance.get("tags").size() == tags.length) && (instance.get("grpcPort").asText().equals(String.valueOf(grpcPort))
- && (instance.get("queriesDisabled") == null && !queriesDisabled ||
- instance.get("queriesDisabled") != null && instance.get("queriesDisabled").asBoolean() == queriesDisabled));
+ instance.get("tags").size() == tags.length) && (
+ instance.get("grpcPort").asText().equals(String.valueOf(grpcPort)) && (
+ instance.get("queriesDisabled") == null && !queriesDisabled
+ || instance.get("queriesDisabled") != null
+ && instance.get("queriesDisabled").asBoolean() == queriesDisabled));
for (int i = 0; i < tags.length; i++) {
result = result && instance.get("tags").get(i).asText().equals(tags[i]);
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/api/TableViewsTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/api/TableViewsTest.java
index a6bcac1..3c46bda 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/api/TableViewsTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/api/TableViewsTest.java
@@ -22,7 +22,6 @@ import java.net.HttpURLConnection;
import java.net.URL;
import java.util.Map;
import org.apache.helix.InstanceType;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.controller.ControllerTestUtils;
import org.apache.pinot.controller.api.resources.TableViews;
import org.apache.pinot.controller.utils.SegmentMetadataMockUtils;
@@ -30,6 +29,7 @@ import org.apache.pinot.core.realtime.impl.fakestream.FakeStreamConfigUtils;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.JsonUtils;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerInstanceToggleTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerInstanceToggleTest.java
index a33d2d5..14523a3 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerInstanceToggleTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerInstanceToggleTest.java
@@ -21,13 +21,13 @@ package org.apache.pinot.controller.helix;
import java.io.IOException;
import java.util.Set;
import org.apache.helix.model.ExternalView;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.config.TagNameUtils;
import org.apache.pinot.common.utils.helix.HelixHelper;
import org.apache.pinot.controller.ControllerTestUtils;
import org.apache.pinot.controller.utils.SegmentMetadataMockUtils;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.apache.pinot.util.TestUtils;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerSentinelTestV2.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerSentinelTestV2.java
index 5d7aebe..929d651 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerSentinelTestV2.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerSentinelTestV2.java
@@ -19,12 +19,12 @@
package org.apache.pinot.controller.helix;
import java.io.IOException;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.config.TagNameUtils;
import org.apache.pinot.controller.ControllerTestUtils;
import org.apache.pinot.controller.utils.SegmentMetadataMockUtils;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.testng.Assert;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerTenantStatelessTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerTenantStatelessTest.java
index 80c6041..d80e7d3 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerTenantStatelessTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerTenantStatelessTest.java
@@ -21,8 +21,8 @@ package org.apache.pinot.controller.helix;
import com.fasterxml.jackson.databind.JsonNode;
import java.io.IOException;
import org.apache.pinot.common.metadata.ZKMetadataProvider;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.config.TagNameUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.JsonUtils;
import org.testng.Assert;
import org.testng.annotations.AfterClass;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerTest.java
index 478b784..cec4544 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerTest.java
@@ -60,7 +60,6 @@ import org.apache.helix.participant.statemachine.StateModelFactory;
import org.apache.helix.participant.statemachine.StateModelInfo;
import org.apache.helix.participant.statemachine.Transition;
import org.apache.helix.store.zk.ZkHelixPropertyStore;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.ZkStarter;
import org.apache.pinot.common.utils.config.TagNameUtils;
import org.apache.pinot.controller.ControllerConf;
@@ -75,15 +74,17 @@ import org.apache.pinot.spi.data.DimensionFieldSpec;
import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.MetricFieldSpec;
import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testng.Assert;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.Instance.*;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.*;
-import static org.apache.pinot.common.utils.CommonConstants.Server.*;
-import static org.testng.Assert.*;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.Instance.ADMIN_PORT_KEY;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.*;
+import static org.apache.pinot.spi.utils.CommonConstants.Server.DEFAULT_ADMIN_API_PORT;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotNull;
public abstract class ControllerTest {
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/PinotControllerModeStatelessTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/PinotControllerModeStatelessTest.java
index 595c657..e1252c4 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/PinotControllerModeStatelessTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/PinotControllerModeStatelessTest.java
@@ -28,14 +28,14 @@ import org.apache.helix.model.ConstraintItem;
import org.apache.helix.model.ExternalView;
import org.apache.helix.model.IdealState;
import org.apache.helix.model.Message;
-import org.apache.pinot.common.utils.CommonConstants;
-import org.apache.pinot.common.utils.CommonConstants.Helix;
import org.apache.pinot.common.utils.ZkStarter;
import org.apache.pinot.common.utils.helix.LeadControllerUtils;
import org.apache.pinot.controller.ControllerConf;
import org.apache.pinot.controller.ControllerStarter;
import org.apache.pinot.controller.LeadControllerManager;
import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants.Helix;
import org.apache.pinot.util.TestUtils;
import org.testng.Assert;
import org.testng.annotations.AfterClass;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/SegmentStatusCheckerTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/SegmentStatusCheckerTest.java
index 9f88082..f9c516e 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/SegmentStatusCheckerTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/SegmentStatusCheckerTest.java
@@ -30,13 +30,13 @@ import org.apache.helix.store.zk.ZkHelixPropertyStore;
import org.apache.pinot.common.metadata.segment.OfflineSegmentZKMetadata;
import org.apache.pinot.common.metrics.ControllerGauge;
import org.apache.pinot.common.metrics.ControllerMetrics;
-import org.apache.pinot.spi.metrics.PinotMetricsRegistry;
import org.apache.pinot.common.metrics.PinotMetricUtils;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.LLCSegmentName;
import org.apache.pinot.controller.ControllerConf;
import org.apache.pinot.controller.LeadControllerManager;
import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
+import org.apache.pinot.spi.metrics.PinotMetricsRegistry;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.testng.Assert;
import org.testng.annotations.Test;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManagerStatelessTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManagerStatelessTest.java
index 4c1569e..71fbedd 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManagerStatelessTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManagerStatelessTest.java
@@ -18,63 +18,24 @@
*/
package org.apache.pinot.controller.helix.core;
-import com.google.common.collect.BiMap;
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.LinkedHashMap;
-import java.util.List;
import java.util.Map;
-import java.util.Random;
-import java.util.Set;
-import org.I0Itec.zkclient.ZkClient;
-import org.apache.helix.HelixDataAccessor;
-import org.apache.helix.PropertyKey;
-import org.apache.helix.PropertyPathBuilder;
-import org.apache.helix.ZNRecord;
-import org.apache.helix.controller.rebalancer.strategy.CrushEdRebalanceStrategy;
-import org.apache.helix.controller.stages.ClusterDataCache;
-import org.apache.helix.manager.zk.ZNRecordSerializer;
-import org.apache.helix.model.ClusterConfig;
-import org.apache.helix.model.ExternalView;
-import org.apache.helix.model.IdealState;
-import org.apache.helix.model.InstanceConfig;
-import org.apache.helix.model.MasterSlaveSMD;
-import org.apache.pinot.common.exception.InvalidConfigException;
-import org.apache.pinot.common.lineage.LineageEntryState;
-import org.apache.pinot.common.lineage.SegmentLineage;
-import org.apache.pinot.common.lineage.SegmentLineageAccessHelper;
-import org.apache.pinot.common.metadata.ZKMetadataProvider;
-import org.apache.pinot.common.metadata.segment.OfflineSegmentZKMetadata;
-import org.apache.pinot.common.metadata.segment.RealtimeSegmentZKMetadata;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.config.TagNameUtils;
-import org.apache.pinot.common.utils.helix.LeadControllerUtils;
import org.apache.pinot.controller.ControllerConf;
import org.apache.pinot.controller.helix.ControllerTest;
-import org.apache.pinot.controller.utils.SegmentMetadataMockUtils;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.config.table.TagOverrideConfig;
import org.apache.pinot.spi.config.table.TenantConfig;
-import org.apache.pinot.spi.config.instance.Instance;
-import org.apache.pinot.spi.config.instance.InstanceType;
import org.apache.pinot.spi.config.tenant.Tenant;
import org.apache.pinot.spi.config.tenant.TenantRole;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
-import org.apache.pinot.util.TestUtils;
import org.testng.Assert;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.LEAD_CONTROLLER_RESOURCE_NAME;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.LEAD_CONTROLLER_RESOURCE_REPLICA_COUNT;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.NUMBER_OF_PARTITIONS_IN_LEAD_CONTROLLER_RESOURCE;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.UNTAGGED_SERVER_INSTANCE;
import static org.apache.pinot.controller.helix.core.PinotHelixResourceManager.InvalidTableConfigException;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManagerTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManagerTest.java
index bd1ebda..ff36034 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManagerTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManagerTest.java
@@ -49,7 +49,6 @@ import org.apache.pinot.common.lineage.SegmentLineageAccessHelper;
import org.apache.pinot.common.metadata.ZKMetadataProvider;
import org.apache.pinot.common.metadata.segment.OfflineSegmentZKMetadata;
import org.apache.pinot.common.metadata.segment.RealtimeSegmentZKMetadata;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.config.TagNameUtils;
import org.apache.pinot.common.utils.helix.LeadControllerUtils;
import org.apache.pinot.controller.ControllerTestUtils;
@@ -60,6 +59,7 @@ import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.config.tenant.Tenant;
import org.apache.pinot.spi.config.tenant.TenantRole;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.apache.pinot.util.TestUtils;
@@ -68,8 +68,11 @@ import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.*;
-import static org.apache.pinot.common.utils.CommonConstants.Server.*;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.LEAD_CONTROLLER_RESOURCE_NAME;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.LEAD_CONTROLLER_RESOURCE_REPLICA_COUNT;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.NUMBER_OF_PARTITIONS_IN_LEAD_CONTROLLER_RESOURCE;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.UNTAGGED_SERVER_INSTANCE;
+import static org.apache.pinot.spi.utils.CommonConstants.Server.DEFAULT_ADMIN_API_PORT;
public class PinotHelixResourceManagerTest {
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentTest.java
index c21b2a3..7925008 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentTest.java
@@ -25,7 +25,6 @@ import java.util.List;
import org.apache.helix.model.InstanceConfig;
import org.apache.pinot.common.assignment.InstanceAssignmentConfigUtils;
import org.apache.pinot.common.assignment.InstancePartitions;
-import org.apache.pinot.common.utils.CommonConstants.Segment.AssignmentStrategy;
import org.apache.pinot.common.utils.config.InstanceUtils;
import org.apache.pinot.common.utils.config.TagNameUtils;
import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
@@ -37,6 +36,7 @@ import org.apache.pinot.spi.config.table.assignment.InstanceAssignmentConfig;
import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
import org.apache.pinot.spi.config.table.assignment.InstanceReplicaGroupPartitionConfig;
import org.apache.pinot.spi.config.table.assignment.InstanceTagPoolConfig;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.AssignmentStrategy;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.testng.annotations.Test;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineNonReplicaGroupSegmentAssignmentTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineNonReplicaGroupSegmentAssignmentTest.java
index ee7c4de..fc1b90a 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineNonReplicaGroupSegmentAssignmentTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineNonReplicaGroupSegmentAssignmentTest.java
@@ -27,16 +27,17 @@ import java.util.TreeMap;
import org.apache.commons.configuration.BaseConfiguration;
import org.apache.commons.configuration.Configuration;
import org.apache.pinot.common.assignment.InstancePartitions;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.apache.pinot.controller.helix.core.rebalance.RebalanceConfigConstants;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
-import static org.testng.Assert.*;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertTrue;
public class OfflineNonReplicaGroupSegmentAssignmentTest {
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineNonReplicaGroupTieredSegmentAssignmentTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineNonReplicaGroupTieredSegmentAssignmentTest.java
index 538d32c..9654d6b 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineNonReplicaGroupTieredSegmentAssignmentTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineNonReplicaGroupTieredSegmentAssignmentTest.java
@@ -33,12 +33,12 @@ import org.apache.pinot.common.tier.PinotServerTierStorage;
import org.apache.pinot.common.tier.Tier;
import org.apache.pinot.common.tier.TierFactory;
import org.apache.pinot.common.tier.TierSegmentSelector;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.apache.pinot.controller.helix.core.rebalance.RebalanceConfigConstants;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.config.table.TierConfig;
import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.testng.Assert;
import org.testng.annotations.BeforeClass;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineReplicaGroupSegmentAssignmentTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineReplicaGroupSegmentAssignmentTest.java
index f6728c4..fa62039 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineReplicaGroupSegmentAssignmentTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineReplicaGroupSegmentAssignmentTest.java
@@ -34,13 +34,13 @@ import org.apache.pinot.common.metadata.ZKMetadataProvider;
import org.apache.pinot.common.metadata.segment.ColumnPartitionMetadata;
import org.apache.pinot.common.metadata.segment.OfflineSegmentZKMetadata;
import org.apache.pinot.common.metadata.segment.SegmentPartitionMetadata;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
-import org.apache.pinot.common.utils.CommonConstants.Segment.AssignmentStrategy;
import org.apache.pinot.controller.helix.core.rebalance.RebalanceConfigConstants;
import org.apache.pinot.spi.config.table.ReplicaGroupStrategyConfig;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.AssignmentStrategy;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.testng.annotations.BeforeClass;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeNonReplicaGroupSegmentAssignmentTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeNonReplicaGroupSegmentAssignmentTest.java
index 3b76ba3..303abfe 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeNonReplicaGroupSegmentAssignmentTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeNonReplicaGroupSegmentAssignmentTest.java
@@ -24,12 +24,12 @@ import java.util.Map;
import java.util.TreeMap;
import org.apache.commons.configuration.BaseConfiguration;
import org.apache.pinot.common.assignment.InstancePartitions;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.apache.pinot.common.utils.LLCSegmentName;
import org.apache.pinot.controller.helix.core.rebalance.RebalanceConfigConstants;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeNonReplicaGroupTieredSegmentAssignmentTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeNonReplicaGroupTieredSegmentAssignmentTest.java
index 47d6b57..547301a 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeNonReplicaGroupTieredSegmentAssignmentTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeNonReplicaGroupTieredSegmentAssignmentTest.java
@@ -31,13 +31,13 @@ import org.apache.pinot.common.tier.PinotServerTierStorage;
import org.apache.pinot.common.tier.Tier;
import org.apache.pinot.common.tier.TierFactory;
import org.apache.pinot.common.tier.TierSegmentSelector;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.apache.pinot.common.utils.LLCSegmentName;
import org.apache.pinot.controller.helix.core.rebalance.RebalanceConfigConstants;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.config.table.TierConfig;
import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeReplicaGroupSegmentAssignmentTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeReplicaGroupSegmentAssignmentTest.java
index d78f928..d75c766 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeReplicaGroupSegmentAssignmentTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeReplicaGroupSegmentAssignmentTest.java
@@ -25,13 +25,13 @@ import java.util.Map;
import java.util.TreeMap;
import org.apache.commons.configuration.BaseConfiguration;
import org.apache.pinot.common.assignment.InstancePartitions;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
-import org.apache.pinot.common.utils.CommonConstants.Segment.AssignmentStrategy;
import org.apache.pinot.common.utils.LLCSegmentName;
import org.apache.pinot.controller.helix.core.rebalance.RebalanceConfigConstants;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.AssignmentStrategy;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentUtilsTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentUtilsTest.java
index 6e867c6..92fba17 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentUtilsTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentUtilsTest.java
@@ -25,7 +25,7 @@ import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import org.apache.pinot.common.assignment.InstancePartitions;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
import org.testng.annotations.Test;
import static org.testng.Assert.assertEquals;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManagerTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManagerTest.java
index 5a63b4d..c3d58da 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManagerTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManagerTest.java
@@ -39,10 +39,6 @@ import org.apache.helix.model.IdealState;
import org.apache.pinot.common.assignment.InstancePartitions;
import org.apache.pinot.common.metadata.segment.LLCRealtimeSegmentZKMetadata;
import org.apache.pinot.common.metrics.ControllerMetrics;
-import org.apache.pinot.common.utils.CommonConstants;
-import org.apache.pinot.common.utils.CommonConstants.Helix;
-import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
-import org.apache.pinot.common.utils.CommonConstants.Segment.Realtime.Status;
import org.apache.pinot.common.utils.LLCSegmentName;
import org.apache.pinot.common.utils.URIUtils;
import org.apache.pinot.controller.ControllerConf;
@@ -51,7 +47,7 @@ import org.apache.pinot.controller.helix.core.assignment.segment.SegmentAssignme
import org.apache.pinot.controller.helix.core.realtime.segment.CommittingSegmentDescriptor;
import org.apache.pinot.controller.util.SegmentCompletionUtils;
import org.apache.pinot.core.realtime.impl.fakestream.FakeStreamConfigUtils;
-import org.apache.pinot.core.segment.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.local.segment.index.metadata.SegmentMetadataImpl;
import org.apache.pinot.segment.spi.creator.SegmentVersion;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
@@ -59,10 +55,14 @@ import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
import org.apache.pinot.spi.env.PinotConfiguration;
import org.apache.pinot.spi.filesystem.PinotFSFactory;
import org.apache.pinot.spi.stream.LongMsgOffset;
-import org.apache.pinot.spi.stream.PartitionGroupMetadata;
import org.apache.pinot.spi.stream.PartitionGroupConsumptionStatus;
+import org.apache.pinot.spi.stream.PartitionGroupMetadata;
import org.apache.pinot.spi.stream.PartitionLevelStreamConfig;
import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants.Helix;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.Realtime.Status;
import org.apache.pinot.spi.utils.IngestionConfigUtils;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionTest.java
index f0a141d..adc86c2 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionTest.java
@@ -26,7 +26,6 @@ import org.apache.pinot.common.metadata.segment.LLCRealtimeSegmentZKMetadata;
import org.apache.pinot.common.metrics.ControllerMetrics;
import org.apache.pinot.common.metrics.PinotMetricUtils;
import org.apache.pinot.common.protocols.SegmentCompletionProtocol;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.LLCSegmentName;
import org.apache.pinot.controller.ControllerConf;
import org.apache.pinot.controller.LeadControllerManager;
@@ -36,13 +35,16 @@ import org.apache.pinot.spi.stream.LongMsgOffset;
import org.apache.pinot.spi.stream.LongMsgOffsetFactory;
import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.zookeeper.data.Stat;
import org.testng.Assert;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
-import static org.apache.pinot.common.protocols.SegmentCompletionProtocol.*;
-import static org.mockito.Mockito.*;
+import static org.apache.pinot.common.protocols.SegmentCompletionProtocol.ControllerResponseStatus;
+import static org.apache.pinot.common.protocols.SegmentCompletionProtocol.Request;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
public class SegmentCompletionTest {
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancerClusterStatelessTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancerClusterStatelessTest.java
index 2776fac..196633a 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancerClusterStatelessTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancerClusterStatelessTest.java
@@ -47,7 +47,7 @@ import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel.ONLINE;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel.ONLINE;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertNull;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancerTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancerTest.java
index 681ed2c..58bd878 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancerTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancerTest.java
@@ -26,10 +26,10 @@ import java.util.TreeSet;
import org.apache.pinot.controller.helix.core.assignment.segment.SegmentAssignmentUtils;
import org.testng.annotations.Test;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel.CONSUMING;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel.ERROR;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel.OFFLINE;
-import static org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.SegmentStateModel.ONLINE;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel.CONSUMING;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel.ERROR;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel.OFFLINE;
+import static org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel.ONLINE;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertTrue;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/retention/RetentionManagerTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/retention/RetentionManagerTest.java
index af90e10..cc7ecaf 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/retention/RetentionManagerTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/retention/RetentionManagerTest.java
@@ -29,7 +29,6 @@ import org.apache.pinot.common.metadata.segment.OfflineSegmentZKMetadata;
import org.apache.pinot.common.metadata.segment.RealtimeSegmentZKMetadata;
import org.apache.pinot.common.metrics.ControllerMetrics;
import org.apache.pinot.common.metrics.PinotMetricUtils;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.LLCSegmentName;
import org.apache.pinot.controller.ControllerConf;
import org.apache.pinot.controller.LeadControllerManager;
@@ -41,6 +40,7 @@ import org.apache.pinot.segment.spi.SegmentMetadata;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.stream.LongMsgOffset;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
import org.joda.time.Duration;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/util/SegmentDeletionManagerTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/util/SegmentDeletionManagerTest.java
index e85a9b7..3a6f533 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/util/SegmentDeletionManagerTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/util/SegmentDeletionManagerTest.java
@@ -18,12 +18,7 @@
*/
package org.apache.pinot.controller.helix.core.util;
-import static org.mockito.ArgumentMatchers.anyInt;
-import static org.mockito.ArgumentMatchers.anyList;
-import static org.mockito.ArgumentMatchers.anyString;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.when;
-
+import com.google.common.io.Files;
import java.io.File;
import java.util.ArrayList;
import java.util.Collection;
@@ -32,24 +27,27 @@ import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
-
import org.apache.helix.HelixAdmin;
import org.apache.helix.ZNRecord;
import org.apache.helix.model.ExternalView;
import org.apache.helix.model.IdealState;
import org.apache.helix.store.zk.ZkHelixPropertyStore;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.controller.helix.core.SegmentDeletionManager;
import org.apache.pinot.spi.env.PinotConfiguration;
import org.apache.pinot.spi.filesystem.LocalPinotFS;
import org.apache.pinot.spi.filesystem.PinotFSFactory;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.joda.time.DateTime;
import org.mockito.invocation.InvocationOnMock;
import org.mockito.stubbing.Answer;
import org.testng.Assert;
import org.testng.annotations.Test;
-import com.google.common.io.Files;
+import static org.mockito.ArgumentMatchers.anyInt;
+import static org.mockito.ArgumentMatchers.anyList;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
public class SegmentDeletionManagerTest {
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/utils/SegmentMetadataMockUtils.java b/pinot-controller/src/test/java/org/apache/pinot/controller/utils/SegmentMetadataMockUtils.java
index d694af6..95e400a 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/utils/SegmentMetadataMockUtils.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/utils/SegmentMetadataMockUtils.java
@@ -23,11 +23,10 @@ import java.util.HashMap;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.TimeUnit;
-
import org.apache.pinot.common.metadata.segment.RealtimeSegmentZKMetadata;
-import org.apache.pinot.core.data.partition.MurmurPartitionFunction;
-import org.apache.pinot.core.segment.index.metadata.ColumnMetadata;
-import org.apache.pinot.core.segment.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.local.partition.MurmurPartitionFunction;
+import org.apache.pinot.segment.local.segment.index.metadata.ColumnMetadata;
+import org.apache.pinot.segment.local.segment.index.metadata.SegmentMetadataImpl;
import org.apache.pinot.segment.spi.SegmentMetadata;
import org.joda.time.Interval;
import org.mockito.Mockito;
diff --git a/pinot-core/pom.xml b/pinot-core/pom.xml
index b2beea2..1caa2d1 100644
--- a/pinot-core/pom.xml
+++ b/pinot-core/pom.xml
@@ -75,6 +75,10 @@
</dependency>
<dependency>
<groupId>org.apache.pinot</groupId>
+ <artifactId>pinot-segment-local</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.pinot</groupId>
<artifactId>pinot-common</artifactId>
</dependency>
<dependency>
@@ -183,6 +187,13 @@
<scope>test</scope>
</dependency>
<dependency>
+ <groupId>org.apache.pinot</groupId>
+ <artifactId>pinot-segment-local</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<scope>test</scope>
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/common/ObjectSerDeUtils.java b/pinot-core/src/main/java/org/apache/pinot/core/common/ObjectSerDeUtils.java
index 5d1fc54..7417daa 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/common/ObjectSerDeUtils.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/common/ObjectSerDeUtils.java
@@ -51,13 +51,13 @@ import java.util.Set;
import org.apache.datasketches.memory.Memory;
import org.apache.datasketches.theta.Sketch;
import org.apache.pinot.common.utils.StringUtil;
-import org.apache.pinot.core.geospatial.serde.GeometrySerializer;
-import org.apache.pinot.core.query.aggregation.function.customobject.AvgPair;
-import org.apache.pinot.core.query.aggregation.function.customobject.MinMaxRangePair;
-import org.apache.pinot.core.query.aggregation.function.customobject.QuantileDigest;
import org.apache.pinot.core.query.distinct.DistinctTable;
import org.apache.pinot.core.query.utils.idset.IdSet;
import org.apache.pinot.core.query.utils.idset.IdSets;
+import org.apache.pinot.segment.local.customobject.AvgPair;
+import org.apache.pinot.segment.local.customobject.MinMaxRangePair;
+import org.apache.pinot.segment.local.customobject.QuantileDigest;
+import org.apache.pinot.segment.local.utils.GeometrySerializer;
import org.apache.pinot.spi.utils.BigDecimalUtils;
import org.apache.pinot.spi.utils.ByteArray;
import org.apache.pinot.spi.utils.StringUtils;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/common/Operator.java b/pinot-core/src/main/java/org/apache/pinot/core/common/Operator.java
index 77e8965..e5c36de 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/common/Operator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/common/Operator.java
@@ -19,7 +19,7 @@
package org.apache.pinot.core.common;
import org.apache.pinot.core.operator.ExecutionStatistics;
-import org.apache.pinot.core.query.exception.EarlyTerminationException;
+import org.apache.pinot.spi.exception.EarlyTerminationException;
public interface Operator<T extends Block> {
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/common/datatable/DataTableUtils.java b/pinot-core/src/main/java/org/apache/pinot/core/common/datatable/DataTableUtils.java
index f59dc0c..f6cf43e 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/common/datatable/DataTableUtils.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/common/datatable/DataTableUtils.java
@@ -26,6 +26,7 @@ import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import org.apache.commons.lang3.StringUtils;
+import org.apache.pinot.common.request.context.ExpressionContext;
import org.apache.pinot.common.utils.DataSchema;
import org.apache.pinot.common.utils.DataSchema.ColumnDataType;
import org.apache.pinot.common.utils.DataTable;
@@ -33,7 +34,6 @@ import org.apache.pinot.common.utils.StringUtil;
import org.apache.pinot.core.query.aggregation.function.AggregationFunction;
import org.apache.pinot.core.query.aggregation.function.DistinctAggregationFunction;
import org.apache.pinot.core.query.distinct.DistinctTable;
-import org.apache.pinot.core.query.request.context.ExpressionContext;
import org.apache.pinot.core.query.request.context.QueryContext;
import org.apache.pinot.core.query.request.context.utils.QueryContextUtils;
import org.apache.pinot.core.util.QueryOptions;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/AvgValueAggregator.java b/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/AvgValueAggregator.java
deleted file mode 100644
index b977dc2..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/AvgValueAggregator.java
+++ /dev/null
@@ -1,84 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.aggregator;
-
-import org.apache.pinot.core.common.ObjectSerDeUtils;
-import org.apache.pinot.core.query.aggregation.function.customobject.AvgPair;
-import org.apache.pinot.segment.spi.AggregationFunctionType;
-import org.apache.pinot.spi.data.FieldSpec.DataType;
-
-
-public class AvgValueAggregator implements ValueAggregator<Object, AvgPair> {
- public static final DataType AGGREGATED_VALUE_TYPE = DataType.BYTES;
-
- @Override
- public AggregationFunctionType getAggregationType() {
- return AggregationFunctionType.AVG;
- }
-
- @Override
- public DataType getAggregatedValueType() {
- return AGGREGATED_VALUE_TYPE;
- }
-
- @Override
- public AvgPair getInitialAggregatedValue(Object rawValue) {
- if (rawValue instanceof byte[]) {
- return deserializeAggregatedValue((byte[]) rawValue);
- } else {
- return new AvgPair(((Number) rawValue).doubleValue(), 1L);
- }
- }
-
- @Override
- public AvgPair applyRawValue(AvgPair value, Object rawValue) {
- if (rawValue instanceof byte[]) {
- value.apply(deserializeAggregatedValue((byte[]) rawValue));
- } else {
- value.apply(((Number) rawValue).doubleValue(), 1L);
- }
- return value;
- }
-
- @Override
- public AvgPair applyAggregatedValue(AvgPair value, AvgPair aggregatedValue) {
- value.apply(aggregatedValue);
- return value;
- }
-
- @Override
- public AvgPair cloneAggregatedValue(AvgPair value) {
- return new AvgPair(value.getSum(), value.getCount());
- }
-
- @Override
- public int getMaxAggregatedValueByteSize() {
- return Double.BYTES + Long.BYTES;
- }
-
- @Override
- public byte[] serializeAggregatedValue(AvgPair value) {
- return ObjectSerDeUtils.AVG_PAIR_SER_DE.serialize(value);
- }
-
- @Override
- public AvgPair deserializeAggregatedValue(byte[] bytes) {
- return ObjectSerDeUtils.AVG_PAIR_SER_DE.deserialize(bytes);
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/CountValueAggregator.java b/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/CountValueAggregator.java
deleted file mode 100644
index eec69f3..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/CountValueAggregator.java
+++ /dev/null
@@ -1,72 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.aggregator;
-
-import org.apache.pinot.segment.spi.AggregationFunctionType;
-import org.apache.pinot.spi.data.FieldSpec.DataType;
-
-
-public class CountValueAggregator implements ValueAggregator<Void, Long> {
- public static final DataType AGGREGATED_VALUE_TYPE = DataType.LONG;
-
- @Override
- public AggregationFunctionType getAggregationType() {
- return AggregationFunctionType.COUNT;
- }
-
- @Override
- public DataType getAggregatedValueType() {
- return AGGREGATED_VALUE_TYPE;
- }
-
- @Override
- public Long getInitialAggregatedValue(Void rawValue) {
- return 1L;
- }
-
- @Override
- public Long applyRawValue(Long value, Void rawValue) {
- return value + 1;
- }
-
- @Override
- public Long applyAggregatedValue(Long value, Long aggregatedValue) {
- return value + aggregatedValue;
- }
-
- @Override
- public Long cloneAggregatedValue(Long value) {
- return value;
- }
-
- @Override
- public int getMaxAggregatedValueByteSize() {
- return Long.BYTES;
- }
-
- @Override
- public byte[] serializeAggregatedValue(Long value) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public Long deserializeAggregatedValue(byte[] bytes) {
- throw new UnsupportedOperationException();
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/DistinctCountBitmapValueAggregator.java b/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/DistinctCountBitmapValueAggregator.java
deleted file mode 100644
index 40f2f4f..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/DistinctCountBitmapValueAggregator.java
+++ /dev/null
@@ -1,94 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.aggregator;
-
-import org.apache.pinot.core.common.ObjectSerDeUtils;
-import org.apache.pinot.segment.spi.AggregationFunctionType;
-import org.apache.pinot.spi.data.FieldSpec.DataType;
-import org.roaringbitmap.RoaringBitmap;
-
-
-public class DistinctCountBitmapValueAggregator implements ValueAggregator<Object, RoaringBitmap> {
- public static final DataType AGGREGATED_VALUE_TYPE = DataType.BYTES;
-
- private int _maxByteSize;
-
- @Override
- public AggregationFunctionType getAggregationType() {
- return AggregationFunctionType.DISTINCTCOUNTBITMAP;
- }
-
- @Override
- public DataType getAggregatedValueType() {
- return AGGREGATED_VALUE_TYPE;
- }
-
- @Override
- public RoaringBitmap getInitialAggregatedValue(Object rawValue) {
- RoaringBitmap initialValue;
- if (rawValue instanceof byte[]) {
- byte[] bytes = (byte[]) rawValue;
- initialValue = deserializeAggregatedValue(bytes);
- _maxByteSize = Math.max(_maxByteSize, bytes.length);
- } else {
- initialValue = new RoaringBitmap();
- initialValue.add(rawValue.hashCode());
- _maxByteSize = Math.max(_maxByteSize, initialValue.serializedSizeInBytes());
- }
- return initialValue;
- }
-
- @Override
- public RoaringBitmap applyRawValue(RoaringBitmap value, Object rawValue) {
- if (rawValue instanceof byte[]) {
- value.or(deserializeAggregatedValue((byte[]) rawValue));
- } else {
- value.add(rawValue.hashCode());
- }
- _maxByteSize = Math.max(_maxByteSize, value.serializedSizeInBytes());
- return value;
- }
-
- @Override
- public RoaringBitmap applyAggregatedValue(RoaringBitmap value, RoaringBitmap aggregatedValue) {
- value.or(aggregatedValue);
- _maxByteSize = Math.max(_maxByteSize, value.serializedSizeInBytes());
- return value;
- }
-
- @Override
- public RoaringBitmap cloneAggregatedValue(RoaringBitmap value) {
- return value.clone();
- }
-
- @Override
- public int getMaxAggregatedValueByteSize() {
- return _maxByteSize;
- }
-
- @Override
- public byte[] serializeAggregatedValue(RoaringBitmap value) {
- return ObjectSerDeUtils.ROARING_BITMAP_SER_DE.serialize(value);
- }
-
- @Override
- public RoaringBitmap deserializeAggregatedValue(byte[] bytes) {
- return ObjectSerDeUtils.ROARING_BITMAP_SER_DE.deserialize(bytes);
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/DistinctCountHLLValueAggregator.java b/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/DistinctCountHLLValueAggregator.java
deleted file mode 100644
index 979eca6..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/DistinctCountHLLValueAggregator.java
+++ /dev/null
@@ -1,105 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.aggregator;
-
-import com.clearspring.analytics.stream.cardinality.CardinalityMergeException;
-import com.clearspring.analytics.stream.cardinality.HyperLogLog;
-import org.apache.pinot.common.utils.CommonConstants;
-import org.apache.pinot.core.common.ObjectSerDeUtils;
-import org.apache.pinot.segment.spi.AggregationFunctionType;
-import org.apache.pinot.spi.data.FieldSpec.DataType;
-
-
-public class DistinctCountHLLValueAggregator implements ValueAggregator<Object, HyperLogLog> {
- public static final DataType AGGREGATED_VALUE_TYPE = DataType.BYTES;
- private static final int DEFAULT_LOG2M_BYTE_SIZE = 180;
-
- // Byte size won't change once we get the initial aggregated value
- private int _maxByteSize;
-
- @Override
- public AggregationFunctionType getAggregationType() {
- return AggregationFunctionType.DISTINCTCOUNTHLL;
- }
-
- @Override
- public DataType getAggregatedValueType() {
- return AGGREGATED_VALUE_TYPE;
- }
-
- @Override
- public HyperLogLog getInitialAggregatedValue(Object rawValue) {
- HyperLogLog initialValue;
- if (rawValue instanceof byte[]) {
- byte[] bytes = (byte[]) rawValue;
- initialValue = deserializeAggregatedValue(bytes);
- _maxByteSize = Math.max(_maxByteSize, bytes.length);
- } else {
- // TODO: Handle configurable log2m for StarTreeBuilder
- initialValue = new HyperLogLog(CommonConstants.Helix.DEFAULT_HYPERLOGLOG_LOG2M);
- initialValue.offer(rawValue);
- _maxByteSize = Math.max(_maxByteSize, DEFAULT_LOG2M_BYTE_SIZE);
- }
- return initialValue;
- }
-
- @Override
- public HyperLogLog applyRawValue(HyperLogLog value, Object rawValue) {
- if (rawValue instanceof byte[]) {
- try {
- value.addAll(deserializeAggregatedValue((byte[]) rawValue));
- } catch (CardinalityMergeException e) {
- throw new RuntimeException(e);
- }
- } else {
- value.offer(rawValue);
- }
- return value;
- }
-
- @Override
- public HyperLogLog applyAggregatedValue(HyperLogLog value, HyperLogLog aggregatedValue) {
- try {
- value.addAll(aggregatedValue);
- return value;
- } catch (CardinalityMergeException e) {
- throw new RuntimeException(e);
- }
- }
-
- @Override
- public HyperLogLog cloneAggregatedValue(HyperLogLog value) {
- return deserializeAggregatedValue(serializeAggregatedValue(value));
- }
-
- @Override
- public int getMaxAggregatedValueByteSize() {
- return _maxByteSize;
- }
-
- @Override
- public byte[] serializeAggregatedValue(HyperLogLog value) {
- return ObjectSerDeUtils.HYPER_LOG_LOG_SER_DE.serialize(value);
- }
-
- @Override
- public HyperLogLog deserializeAggregatedValue(byte[] bytes) {
- return ObjectSerDeUtils.HYPER_LOG_LOG_SER_DE.deserialize(bytes);
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/MaxValueAggregator.java b/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/MaxValueAggregator.java
deleted file mode 100644
index 5bbcb82..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/MaxValueAggregator.java
+++ /dev/null
@@ -1,72 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.aggregator;
-
-import org.apache.pinot.segment.spi.AggregationFunctionType;
-import org.apache.pinot.spi.data.FieldSpec.DataType;
-
-
-public class MaxValueAggregator implements ValueAggregator<Number, Double> {
- public static final DataType AGGREGATED_VALUE_TYPE = DataType.DOUBLE;
-
- @Override
- public AggregationFunctionType getAggregationType() {
- return AggregationFunctionType.MAX;
- }
-
- @Override
- public DataType getAggregatedValueType() {
- return AGGREGATED_VALUE_TYPE;
- }
-
- @Override
- public Double getInitialAggregatedValue(Number rawValue) {
- return rawValue.doubleValue();
- }
-
- @Override
- public Double applyRawValue(Double value, Number rawValue) {
- return Math.max(value, rawValue.doubleValue());
- }
-
- @Override
- public Double applyAggregatedValue(Double value, Double aggregatedValue) {
- return Math.max(value, aggregatedValue);
- }
-
- @Override
- public Double cloneAggregatedValue(Double value) {
- return value;
- }
-
- @Override
- public int getMaxAggregatedValueByteSize() {
- return Double.BYTES;
- }
-
- @Override
- public byte[] serializeAggregatedValue(Double value) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public Double deserializeAggregatedValue(byte[] bytes) {
- throw new UnsupportedOperationException();
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/MinMaxRangeValueAggregator.java b/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/MinMaxRangeValueAggregator.java
deleted file mode 100644
index 5420106..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/MinMaxRangeValueAggregator.java
+++ /dev/null
@@ -1,86 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.aggregator;
-
-import org.apache.pinot.core.common.ObjectSerDeUtils;
-import org.apache.pinot.core.query.aggregation.function.customobject.MinMaxRangePair;
-import org.apache.pinot.segment.spi.AggregationFunctionType;
-import org.apache.pinot.spi.data.FieldSpec.DataType;
-
-
-public class MinMaxRangeValueAggregator implements ValueAggregator<Object, MinMaxRangePair> {
- public static final DataType AGGREGATED_VALUE_TYPE = DataType.BYTES;
-
- @Override
- public AggregationFunctionType getAggregationType() {
- return AggregationFunctionType.MINMAXRANGE;
- }
-
- @Override
- public DataType getAggregatedValueType() {
- return AGGREGATED_VALUE_TYPE;
- }
-
- @Override
- public MinMaxRangePair getInitialAggregatedValue(Object rawValue) {
- if (rawValue instanceof byte[]) {
- return deserializeAggregatedValue((byte[]) rawValue);
- } else {
- double doubleValue = ((Number) rawValue).doubleValue();
- return new MinMaxRangePair(doubleValue, doubleValue);
- }
- }
-
- @Override
- public MinMaxRangePair applyRawValue(MinMaxRangePair value, Object rawValue) {
- if (rawValue instanceof byte[]) {
- value.apply(deserializeAggregatedValue((byte[]) rawValue));
- } else {
- double doubleValue = ((Number) rawValue).doubleValue();
- value.apply(doubleValue, doubleValue);
- }
- return value;
- }
-
- @Override
- public MinMaxRangePair applyAggregatedValue(MinMaxRangePair value, MinMaxRangePair aggregatedValue) {
- value.apply(aggregatedValue);
- return value;
- }
-
- @Override
- public MinMaxRangePair cloneAggregatedValue(MinMaxRangePair value) {
- return new MinMaxRangePair(value.getMin(), value.getMax());
- }
-
- @Override
- public int getMaxAggregatedValueByteSize() {
- return Double.BYTES + Double.BYTES;
- }
-
- @Override
- public byte[] serializeAggregatedValue(MinMaxRangePair value) {
- return ObjectSerDeUtils.MIN_MAX_RANGE_PAIR_SER_DE.serialize(value);
- }
-
- @Override
- public MinMaxRangePair deserializeAggregatedValue(byte[] bytes) {
- return ObjectSerDeUtils.MIN_MAX_RANGE_PAIR_SER_DE.deserialize(bytes);
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/MinValueAggregator.java b/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/MinValueAggregator.java
deleted file mode 100644
index 3dad924..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/MinValueAggregator.java
+++ /dev/null
@@ -1,72 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.aggregator;
-
-import org.apache.pinot.segment.spi.AggregationFunctionType;
-import org.apache.pinot.spi.data.FieldSpec.DataType;
-
-
-public class MinValueAggregator implements ValueAggregator<Number, Double> {
- public static final DataType AGGREGATED_VALUE_TYPE = DataType.DOUBLE;
-
- @Override
- public AggregationFunctionType getAggregationType() {
- return AggregationFunctionType.MIN;
- }
-
- @Override
- public DataType getAggregatedValueType() {
- return AGGREGATED_VALUE_TYPE;
- }
-
- @Override
- public Double getInitialAggregatedValue(Number rawValue) {
- return rawValue.doubleValue();
- }
-
- @Override
- public Double applyRawValue(Double value, Number rawValue) {
- return Math.min(value, rawValue.doubleValue());
- }
-
- @Override
- public Double applyAggregatedValue(Double value, Double aggregatedValue) {
- return Math.min(value, aggregatedValue);
- }
-
- @Override
- public Double cloneAggregatedValue(Double value) {
- return value;
- }
-
- @Override
- public int getMaxAggregatedValueByteSize() {
- return Double.BYTES;
- }
-
- @Override
- public byte[] serializeAggregatedValue(Double value) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public Double deserializeAggregatedValue(byte[] bytes) {
- throw new UnsupportedOperationException();
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/PercentileEstValueAggregator.java b/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/PercentileEstValueAggregator.java
deleted file mode 100644
index 1d389d9..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/PercentileEstValueAggregator.java
+++ /dev/null
@@ -1,95 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.aggregator;
-
-import org.apache.pinot.core.common.ObjectSerDeUtils;
-import org.apache.pinot.core.query.aggregation.function.PercentileEstAggregationFunction;
-import org.apache.pinot.core.query.aggregation.function.customobject.QuantileDigest;
-import org.apache.pinot.segment.spi.AggregationFunctionType;
-import org.apache.pinot.spi.data.FieldSpec.DataType;
-
-
-public class PercentileEstValueAggregator implements ValueAggregator<Object, QuantileDigest> {
- public static final DataType AGGREGATED_VALUE_TYPE = DataType.BYTES;
-
- private int _maxByteSize;
-
- @Override
- public AggregationFunctionType getAggregationType() {
- return AggregationFunctionType.PERCENTILEEST;
- }
-
- @Override
- public DataType getAggregatedValueType() {
- return AGGREGATED_VALUE_TYPE;
- }
-
- @Override
- public QuantileDigest getInitialAggregatedValue(Object rawValue) {
- QuantileDigest initialValue;
- if (rawValue instanceof byte[]) {
- byte[] bytes = (byte[]) rawValue;
- initialValue = deserializeAggregatedValue(bytes);
- _maxByteSize = Math.max(_maxByteSize, bytes.length);
- } else {
- initialValue = new QuantileDigest(PercentileEstAggregationFunction.DEFAULT_MAX_ERROR);
- initialValue.add(((Number) rawValue).longValue());
- _maxByteSize = Math.max(_maxByteSize, initialValue.getByteSize());
- }
- return initialValue;
- }
-
- @Override
- public QuantileDigest applyRawValue(QuantileDigest value, Object rawValue) {
- if (rawValue instanceof byte[]) {
- value.merge(deserializeAggregatedValue((byte[]) rawValue));
- } else {
- value.add(((Number) rawValue).longValue());
- }
- _maxByteSize = Math.max(_maxByteSize, value.getByteSize());
- return value;
- }
-
- @Override
- public QuantileDigest applyAggregatedValue(QuantileDigest value, QuantileDigest aggregatedValue) {
- value.merge(aggregatedValue);
- _maxByteSize = Math.max(_maxByteSize, value.getByteSize());
- return value;
- }
-
- @Override
- public QuantileDigest cloneAggregatedValue(QuantileDigest value) {
- return deserializeAggregatedValue(serializeAggregatedValue(value));
- }
-
- @Override
- public int getMaxAggregatedValueByteSize() {
- return _maxByteSize;
- }
-
- @Override
- public byte[] serializeAggregatedValue(QuantileDigest value) {
- return ObjectSerDeUtils.QUANTILE_DIGEST_SER_DE.serialize(value);
- }
-
- @Override
- public QuantileDigest deserializeAggregatedValue(byte[] bytes) {
- return ObjectSerDeUtils.QUANTILE_DIGEST_SER_DE.deserialize(bytes);
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/PercentileTDigestValueAggregator.java b/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/PercentileTDigestValueAggregator.java
deleted file mode 100644
index a917ea8..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/PercentileTDigestValueAggregator.java
+++ /dev/null
@@ -1,95 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.aggregator;
-
-import com.tdunning.math.stats.TDigest;
-import org.apache.pinot.core.common.ObjectSerDeUtils;
-import org.apache.pinot.core.query.aggregation.function.PercentileTDigestAggregationFunction;
-import org.apache.pinot.segment.spi.AggregationFunctionType;
-import org.apache.pinot.spi.data.FieldSpec.DataType;
-
-
-public class PercentileTDigestValueAggregator implements ValueAggregator<Object, TDigest> {
- public static final DataType AGGREGATED_VALUE_TYPE = DataType.BYTES;
-
- private int _maxByteSize;
-
- @Override
- public AggregationFunctionType getAggregationType() {
- return AggregationFunctionType.PERCENTILETDIGEST;
- }
-
- @Override
- public DataType getAggregatedValueType() {
- return AGGREGATED_VALUE_TYPE;
- }
-
- @Override
- public TDigest getInitialAggregatedValue(Object rawValue) {
- TDigest initialValue;
- if (rawValue instanceof byte[]) {
- byte[] bytes = (byte[]) rawValue;
- initialValue = deserializeAggregatedValue(bytes);
- _maxByteSize = Math.max(_maxByteSize, bytes.length);
- } else {
- initialValue = TDigest.createMergingDigest(PercentileTDigestAggregationFunction.DEFAULT_TDIGEST_COMPRESSION);
- initialValue.add(((Number) rawValue).doubleValue());
- _maxByteSize = Math.max(_maxByteSize, initialValue.byteSize());
- }
- return initialValue;
- }
-
- @Override
- public TDigest applyRawValue(TDigest value, Object rawValue) {
- if (rawValue instanceof byte[]) {
- value.add(deserializeAggregatedValue((byte[]) rawValue));
- } else {
- value.add(((Number) rawValue).doubleValue());
- }
- _maxByteSize = Math.max(_maxByteSize, value.byteSize());
- return value;
- }
-
- @Override
- public TDigest applyAggregatedValue(TDigest value, TDigest aggregatedValue) {
- value.add(aggregatedValue);
- _maxByteSize = Math.max(_maxByteSize, value.byteSize());
- return value;
- }
-
- @Override
- public TDigest cloneAggregatedValue(TDigest value) {
- return deserializeAggregatedValue(serializeAggregatedValue(value));
- }
-
- @Override
- public int getMaxAggregatedValueByteSize() {
- return _maxByteSize;
- }
-
- @Override
- public byte[] serializeAggregatedValue(TDigest value) {
- return ObjectSerDeUtils.TDIGEST_SER_DE.serialize(value);
- }
-
- @Override
- public TDigest deserializeAggregatedValue(byte[] bytes) {
- return ObjectSerDeUtils.TDIGEST_SER_DE.deserialize(bytes);
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/SumPrecisionValueAggregator.java b/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/SumPrecisionValueAggregator.java
deleted file mode 100644
index 3d4e230..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/SumPrecisionValueAggregator.java
+++ /dev/null
@@ -1,93 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.aggregator;
-
-import java.math.BigDecimal;
-import org.apache.pinot.segment.spi.AggregationFunctionType;
-import org.apache.pinot.spi.data.FieldSpec.DataType;
-import org.apache.pinot.spi.utils.BigDecimalUtils;
-
-
-public class SumPrecisionValueAggregator implements ValueAggregator<Object, BigDecimal> {
- public static final DataType AGGREGATED_VALUE_TYPE = DataType.BYTES;
-
- private int _maxByteSize;
-
- @Override
- public AggregationFunctionType getAggregationType() {
- return AggregationFunctionType.SUMPRECISION;
- }
-
- @Override
- public DataType getAggregatedValueType() {
- return AGGREGATED_VALUE_TYPE;
- }
-
- @Override
- public BigDecimal getInitialAggregatedValue(Object rawValue) {
- BigDecimal initialValue = toBigDecimal(rawValue);
- _maxByteSize = Math.max(_maxByteSize, BigDecimalUtils.byteSize(initialValue));
- return initialValue;
- }
-
- @Override
- public BigDecimal applyRawValue(BigDecimal value, Object rawValue) {
- value = value.add(toBigDecimal(rawValue));
- _maxByteSize = Math.max(_maxByteSize, BigDecimalUtils.byteSize(value));
- return value;
- }
-
- private static BigDecimal toBigDecimal(Object rawValue) {
- if (rawValue instanceof byte[]) {
- return BigDecimalUtils.deserialize((byte[]) rawValue);
- }
- if (rawValue instanceof Integer || rawValue instanceof Long) {
- return BigDecimal.valueOf(((Number) rawValue).longValue());
- }
- return new BigDecimal(rawValue.toString());
- }
-
- @Override
- public BigDecimal applyAggregatedValue(BigDecimal value, BigDecimal aggregatedValue) {
- value = value.add(aggregatedValue);
- _maxByteSize = Math.max(_maxByteSize, BigDecimalUtils.byteSize(value));
- return value;
- }
-
- @Override
- public BigDecimal cloneAggregatedValue(BigDecimal value) {
- // NOTE: No need to clone because BigDecimal is immutable
- return value;
- }
-
- @Override
- public int getMaxAggregatedValueByteSize() {
- return _maxByteSize;
- }
-
- @Override
- public byte[] serializeAggregatedValue(BigDecimal value) {
- return BigDecimalUtils.serialize(value);
- }
-
- @Override
- public BigDecimal deserializeAggregatedValue(byte[] bytes) {
- return BigDecimalUtils.deserialize(bytes);
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/SumValueAggregator.java b/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/SumValueAggregator.java
deleted file mode 100644
index 881fe2a..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/SumValueAggregator.java
+++ /dev/null
@@ -1,72 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.aggregator;
-
-import org.apache.pinot.segment.spi.AggregationFunctionType;
-import org.apache.pinot.spi.data.FieldSpec.DataType;
-
-
-public class SumValueAggregator implements ValueAggregator<Number, Double> {
- public static final DataType AGGREGATED_VALUE_TYPE = DataType.DOUBLE;
-
- @Override
- public AggregationFunctionType getAggregationType() {
- return AggregationFunctionType.SUM;
- }
-
- @Override
- public DataType getAggregatedValueType() {
- return AGGREGATED_VALUE_TYPE;
- }
-
- @Override
- public Double getInitialAggregatedValue(Number rawValue) {
- return rawValue.doubleValue();
- }
-
- @Override
- public Double applyRawValue(Double value, Number rawValue) {
- return value + rawValue.doubleValue();
- }
-
- @Override
- public Double applyAggregatedValue(Double value, Double aggregatedValue) {
- return value + aggregatedValue;
- }
-
- @Override
- public Double cloneAggregatedValue(Double value) {
- return value;
- }
-
- @Override
- public int getMaxAggregatedValueByteSize() {
- return Double.BYTES;
- }
-
- @Override
- public byte[] serializeAggregatedValue(Double value) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public Double deserializeAggregatedValue(byte[] bytes) {
- throw new UnsupportedOperationException();
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/ValueAggregator.java b/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/ValueAggregator.java
deleted file mode 100644
index e950449..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/ValueAggregator.java
+++ /dev/null
@@ -1,79 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.aggregator;
-
-import org.apache.pinot.segment.spi.AggregationFunctionType;
-import org.apache.pinot.spi.data.FieldSpec.DataType;
-
-
-/**
- * A value aggregator that pre-aggregates on the input values for a specific type of aggregation.
- *
- * @param <R> Type of the raw value (non-aggregated)
- * @param <A> Type of the aggregated value
- */
-public interface ValueAggregator<R, A> {
-
- /**
- * Returns the type of the aggregation.
- */
- AggregationFunctionType getAggregationType();
-
- /**
- * Returns the data type of the aggregated value.
- */
- DataType getAggregatedValueType();
-
- /**
- * Returns the initial aggregated value.
- */
- A getInitialAggregatedValue(R rawValue);
-
- /**
- * Applies a raw value to the current aggregated value.
- * <p>NOTE: if value is mutable, will directly modify the value.
- */
- A applyRawValue(A value, R rawValue);
-
- /**
- * Applies an aggregated value to the current aggregated value.
- * <p>NOTE: if value is mutable, will directly modify the value.
- */
- A applyAggregatedValue(A value, A aggregatedValue);
-
- /**
- * Clones an aggregated value.
- */
- A cloneAggregatedValue(A value);
-
- /**
- * Returns the maximum size in bytes of the aggregated values seen so far.
- */
- int getMaxAggregatedValueByteSize();
-
- /**
- * Serializes an aggregated value into a byte array.
- */
- byte[] serializeAggregatedValue(A value);
-
- /**
- * De-serializes an aggregated value from a byte array.
- */
- A deserializeAggregatedValue(byte[] bytes);
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/ValueAggregatorFactory.java b/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/ValueAggregatorFactory.java
deleted file mode 100644
index 809dc81..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/aggregator/ValueAggregatorFactory.java
+++ /dev/null
@@ -1,104 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.aggregator;
-
-import org.apache.pinot.segment.spi.AggregationFunctionType;
-import org.apache.pinot.spi.data.FieldSpec.DataType;
-
-
-/**
- * The {@code ValueAggregatorFactory} class is the factory for all value aggregators.
- */
-@SuppressWarnings("rawtypes")
-public class ValueAggregatorFactory {
- private ValueAggregatorFactory() {
- }
-
- /**
- * Returns a new instance of value aggregator for the given aggregation type.
- *
- * @param aggregationType Aggregation type
- * @return Value aggregator
- */
- public static ValueAggregator getValueAggregator(AggregationFunctionType aggregationType) {
- switch (aggregationType) {
- case COUNT:
- return new CountValueAggregator();
- case MIN:
- return new MinValueAggregator();
- case MAX:
- return new MaxValueAggregator();
- case SUM:
- return new SumValueAggregator();
- case SUMPRECISION:
- return new SumPrecisionValueAggregator();
- case AVG:
- return new AvgValueAggregator();
- case MINMAXRANGE:
- return new MinMaxRangeValueAggregator();
- case DISTINCTCOUNTBITMAP:
- return new DistinctCountBitmapValueAggregator();
- case DISTINCTCOUNTHLL:
- case DISTINCTCOUNTRAWHLL:
- return new DistinctCountHLLValueAggregator();
- case PERCENTILEEST:
- return new PercentileEstValueAggregator();
- case PERCENTILETDIGEST:
- return new PercentileTDigestValueAggregator();
- default:
- throw new IllegalStateException("Unsupported aggregation type: " + aggregationType);
- }
- }
-
- /**
- * Returns the data type of the aggregated value for the given aggregation type.
- *
- * @param aggregationType Aggregation type
- * @return Data type of the aggregated value
- */
- public static DataType getAggregatedValueType(AggregationFunctionType aggregationType) {
- switch (aggregationType) {
- case COUNT:
- return CountValueAggregator.AGGREGATED_VALUE_TYPE;
- case MIN:
- return MinValueAggregator.AGGREGATED_VALUE_TYPE;
- case MAX:
- return MaxValueAggregator.AGGREGATED_VALUE_TYPE;
- case SUM:
- return SumValueAggregator.AGGREGATED_VALUE_TYPE;
- case SUMPRECISION:
- return SumPrecisionValueAggregator.AGGREGATED_VALUE_TYPE;
- case AVG:
- return AvgValueAggregator.AGGREGATED_VALUE_TYPE;
- case MINMAXRANGE:
- return MinMaxRangeValueAggregator.AGGREGATED_VALUE_TYPE;
- case DISTINCTCOUNTBITMAP:
- return DistinctCountBitmapValueAggregator.AGGREGATED_VALUE_TYPE;
- case DISTINCTCOUNTHLL:
- case DISTINCTCOUNTRAWHLL:
- return DistinctCountHLLValueAggregator.AGGREGATED_VALUE_TYPE;
- case PERCENTILEEST:
- return PercentileEstValueAggregator.AGGREGATED_VALUE_TYPE;
- case PERCENTILETDIGEST:
- return PercentileTDigestValueAggregator.AGGREGATED_VALUE_TYPE;
- default:
- throw new IllegalStateException("Unsupported aggregation type: " + aggregationType);
- }
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/function/FunctionEvaluator.java b/pinot-core/src/main/java/org/apache/pinot/core/data/function/FunctionEvaluator.java
deleted file mode 100644
index f6b586f..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/function/FunctionEvaluator.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.function;
-
-import java.util.List;
-import org.apache.pinot.spi.data.readers.GenericRow;
-
-
-/**
- * Interface for evaluators of transform function expressions of schema field specs
- */
-public interface FunctionEvaluator {
-
- /**
- * Get the arguments of the function
- */
- List<String> getArguments();
-
- /**
- * Evaluate the function on the generic row and return the result
- */
- Object evaluate(GenericRow genericRow);
-
- /**
- * Evaluates the function on the given values (same order as the arguments) and returns the result.
- */
- Object evaluate(Object[] values);
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/function/FunctionEvaluatorFactory.java b/pinot-core/src/main/java/org/apache/pinot/core/data/function/FunctionEvaluatorFactory.java
deleted file mode 100644
index 6b3db98..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/function/FunctionEvaluatorFactory.java
+++ /dev/null
@@ -1,104 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.function;
-
-import javax.annotation.Nullable;
-import org.apache.pinot.core.util.SchemaUtils;
-import org.apache.pinot.spi.data.FieldSpec;
-import org.apache.pinot.spi.data.TimeFieldSpec;
-import org.apache.pinot.spi.data.TimeGranularitySpec;
-
-
-/**
- * Factory class to create an {@link FunctionEvaluator} for the field spec based on the {@link FieldSpec#getTransformFunction()}
- */
-public class FunctionEvaluatorFactory {
- /**
- * Creates the {@link FunctionEvaluator} for the given field spec
- *
- * 1. If transform expression is defined, use it to create the appropriate {@link FunctionEvaluator}
- * 2. For TIME column, if conversion is needed, {@link TimeSpecFunctionEvaluator} for backward compatible handling of time spec. This is needed until we migrate to {@link org.apache.pinot.spi.data.DateTimeFieldSpec}
- * 3. For columns ending with __KEYS or __VALUES (used for interpreting Map column in Avro), create default groovy functions for handing the Map
- * 4. Return null, if none of the above
- */
- @Nullable
- public static FunctionEvaluator getExpressionEvaluator(FieldSpec fieldSpec) {
- FunctionEvaluator functionEvaluator = null;
-
- String columnName = fieldSpec.getName();
- // TODO: once we have published a release w/ IngestionConfig#TransformConfigs, stop reading transform function from schema in next release
- String transformExpression = fieldSpec.getTransformFunction();
- if (transformExpression != null && !transformExpression.isEmpty()) {
-
- // if transform function expression present, use it to generate function evaluator
- try {
- functionEvaluator = getExpressionEvaluator(transformExpression);
- } catch (Exception e) {
- throw new IllegalStateException(
- "Caught exception while constructing expression evaluator for transform expression:" + transformExpression
- + ", of column:" + columnName);
- }
- } else if (fieldSpec.getFieldType().equals(FieldSpec.FieldType.TIME)) {
-
- // Time conversions should be done using DateTimeFieldSpec and transformFunctions
- // But we need below lines for converting TimeFieldSpec's incoming to outgoing
- TimeFieldSpec timeFieldSpec = (TimeFieldSpec) fieldSpec;
- TimeGranularitySpec incomingGranularitySpec = timeFieldSpec.getIncomingGranularitySpec();
- TimeGranularitySpec outgoingGranularitySpec = timeFieldSpec.getOutgoingGranularitySpec();
- if (!incomingGranularitySpec.equals(outgoingGranularitySpec)) {
- if (!incomingGranularitySpec.getName().equals(outgoingGranularitySpec.getName())) {
- functionEvaluator = new TimeSpecFunctionEvaluator(incomingGranularitySpec, outgoingGranularitySpec);
- } else {
- throw new IllegalStateException(
- "Invalid timeSpec - Incoming and outgoing field specs are different, but name " + incomingGranularitySpec
- .getName() + " is same");
- }
- }
- } else if (columnName.endsWith(SchemaUtils.MAP_KEY_COLUMN_SUFFIX)) {
-
- // for backward compatible handling of Map type (currently only in Avro)
- String sourceMapName = columnName.substring(0, columnName.length() - SchemaUtils.MAP_KEY_COLUMN_SUFFIX.length());
- String defaultMapKeysTransformExpression = getDefaultMapKeysTransformExpression(sourceMapName);
- functionEvaluator = getExpressionEvaluator(defaultMapKeysTransformExpression);
- } else if (columnName.endsWith(SchemaUtils.MAP_VALUE_COLUMN_SUFFIX)) {
- // for backward compatible handling of Map type in avro (currently only in Avro)
- String sourceMapName =
- columnName.substring(0, columnName.length() - SchemaUtils.MAP_VALUE_COLUMN_SUFFIX.length());
- String defaultMapValuesTransformExpression = getDefaultMapValuesTransformExpression(sourceMapName);
- functionEvaluator = getExpressionEvaluator(defaultMapValuesTransformExpression);
- }
- return functionEvaluator;
- }
-
- public static FunctionEvaluator getExpressionEvaluator(String transformExpression) {
- if (transformExpression.startsWith(GroovyFunctionEvaluator.getGroovyExpressionPrefix())) {
- return new GroovyFunctionEvaluator(transformExpression);
- } else {
- return new InbuiltFunctionEvaluator(transformExpression);
- }
- }
-
- private static String getDefaultMapKeysTransformExpression(String mapColumnName) {
- return String.format("Groovy({%s.sort()*.key}, %s)", mapColumnName, mapColumnName);
- }
-
- private static String getDefaultMapValuesTransformExpression(String mapColumnName) {
- return String.format("Groovy({%s.sort()*.value}, %s)", mapColumnName, mapColumnName);
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/function/GroovyFunctionEvaluator.java b/pinot-core/src/main/java/org/apache/pinot/core/data/function/GroovyFunctionEvaluator.java
deleted file mode 100644
index 30a597c..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/function/GroovyFunctionEvaluator.java
+++ /dev/null
@@ -1,105 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.function;
-
-import com.google.common.base.Preconditions;
-import com.google.common.base.Splitter;
-import groovy.lang.Binding;
-import groovy.lang.GroovyShell;
-import groovy.lang.Script;
-import java.util.Collections;
-import java.util.List;
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-import org.apache.pinot.spi.data.readers.GenericRow;
-
-
-/**
- * An {@link FunctionEvaluator} for evaluating transform function expressions of a Schema field spec written in Groovy.
- * GroovyShell is used to execute expressions.
- *
- * The transform expression must follow the convention Groovy({expression}, arguments1, argument2...)
- * For example:
- * "dimensionFieldSpecs": [
- * {
- * "name": "fullName",
- * "dataType": "STRING",
- * "transformFunction": "Groovy({firstName+' '+lastName}, firstName, lastName)"
- * }
- * ]
- */
-public class GroovyFunctionEvaluator implements FunctionEvaluator {
-
- private static final String GROOVY_EXPRESSION_PREFIX = "Groovy";
- private static final String GROOVY_FUNCTION_REGEX = "Groovy\\(\\{(?<script>.+)}(,(?<arguments>.+))?\\)";
- private static final Pattern GROOVY_FUNCTION_PATTERN =
- Pattern.compile(GROOVY_FUNCTION_REGEX, Pattern.CASE_INSENSITIVE);
- private static final String ARGUMENTS_GROUP_NAME = "arguments";
- private static final String SCRIPT_GROUP_NAME = "script";
- private static final String ARGUMENTS_SEPARATOR = ",";
-
- private final List<String> _arguments;
- private final int _numArguments;
- private final Binding _binding;
- private final Script _script;
-
- public GroovyFunctionEvaluator(String closure) {
- Matcher matcher = GROOVY_FUNCTION_PATTERN.matcher(closure);
- Preconditions.checkState(matcher.matches(), "Invalid transform expression: %s", closure);
- String arguments = matcher.group(ARGUMENTS_GROUP_NAME);
- if (arguments != null) {
- _arguments = Splitter.on(ARGUMENTS_SEPARATOR).trimResults().splitToList(arguments);
- } else {
- _arguments = Collections.emptyList();
- }
- _numArguments = _arguments.size();
- _binding = new Binding();
- _script = new GroovyShell(_binding).parse(matcher.group(SCRIPT_GROUP_NAME));
- }
-
- public static String getGroovyExpressionPrefix() {
- return GROOVY_EXPRESSION_PREFIX;
- }
-
- @Override
- public List<String> getArguments() {
- return _arguments;
- }
-
- @Override
- public Object evaluate(GenericRow genericRow) {
- for (String argument : _arguments) {
- Object value = genericRow.getValue(argument);
- if (value == null) {
- // FIXME: if any param is null a) exit OR b) assume function handles it ?
- return null;
- }
- _binding.setVariable(argument, value);
- }
- return _script.run();
- }
-
- @Override
- public Object evaluate(Object[] values) {
- for (int i = 0; i < _numArguments; i++) {
- _binding.setVariable(_arguments.get(i), values[i]);
- }
- return _script.run();
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/function/InbuiltFunctionEvaluator.java b/pinot-core/src/main/java/org/apache/pinot/core/data/function/InbuiltFunctionEvaluator.java
deleted file mode 100644
index ab6775d..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/function/InbuiltFunctionEvaluator.java
+++ /dev/null
@@ -1,172 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.function;
-
-import com.google.common.base.Preconditions;
-import java.util.ArrayList;
-import java.util.List;
-import org.apache.pinot.common.function.FunctionInfo;
-import org.apache.pinot.common.function.FunctionInvoker;
-import org.apache.pinot.common.function.FunctionRegistry;
-import org.apache.pinot.core.query.request.context.ExpressionContext;
-import org.apache.pinot.core.query.request.context.FunctionContext;
-import org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils;
-import org.apache.pinot.spi.data.readers.GenericRow;
-
-
-/**
- * Evaluates an expression.
- * <p>This is optimized for evaluating an expression multiple times with different inputs.
- * <p>Overall idea: parse the expression into an ExecutableNode, where an ExecutableNode can be:
- * <ul>
- * <li>FunctionNode - executes a function</li>
- * <li>ColumnNode - fetches the value of the column from the input GenericRow</li>
- * <li>ConstantNode - returns the literal value</li>
- * </ul>
- */
-public class InbuiltFunctionEvaluator implements FunctionEvaluator {
- // Root of the execution tree
- private final ExecutableNode _rootNode;
- private final List<String> _arguments;
-
- public InbuiltFunctionEvaluator(String functionExpression) {
- _arguments = new ArrayList<>();
- ExpressionContext expression = QueryContextConverterUtils.getExpression(functionExpression);
- _rootNode = planExecution(expression);
- }
-
- private ExecutableNode planExecution(ExpressionContext expression) {
- switch (expression.getType()) {
- case LITERAL:
- return new ConstantExecutionNode(expression.getLiteral());
- case IDENTIFIER:
- String columnName = expression.getIdentifier();
- ColumnExecutionNode columnExecutionNode = new ColumnExecutionNode(columnName, _arguments.size());
- _arguments.add(columnName);
- return columnExecutionNode;
- case FUNCTION:
- FunctionContext function = expression.getFunction();
- List<ExpressionContext> arguments = function.getArguments();
- int numArguments = arguments.size();
- ExecutableNode[] childNodes = new ExecutableNode[numArguments];
- for (int i = 0; i < numArguments; i++) {
- childNodes[i] = planExecution(arguments.get(i));
- }
- String functionName = function.getFunctionName();
- FunctionInfo functionInfo = FunctionRegistry.getFunctionInfo(functionName, numArguments);
- Preconditions.checkState(functionInfo != null, "Unsupported function: %s with %s parameters", functionName,
- numArguments);
- return new FunctionExecutionNode(functionInfo, childNodes);
- default:
- throw new IllegalStateException();
- }
- }
-
- @Override
- public List<String> getArguments() {
- return _arguments;
- }
-
- @Override
- public Object evaluate(GenericRow row) {
- return _rootNode.execute(row);
- }
-
- @Override
- public Object evaluate(Object[] values) {
- return _rootNode.execute(values);
- }
-
- private interface ExecutableNode {
-
- Object execute(GenericRow row);
-
- Object execute(Object[] values);
- }
-
- private static class FunctionExecutionNode implements ExecutableNode {
- final FunctionInvoker _functionInvoker;
- final ExecutableNode[] _argumentNodes;
- final Object[] _arguments;
-
- FunctionExecutionNode(FunctionInfo functionInfo, ExecutableNode[] argumentNodes) {
- _functionInvoker = new FunctionInvoker(functionInfo);
- _argumentNodes = argumentNodes;
- _arguments = new Object[_argumentNodes.length];
- }
-
- @Override
- public Object execute(GenericRow row) {
- int numArguments = _argumentNodes.length;
- for (int i = 0; i < numArguments; i++) {
- _arguments[i] = _argumentNodes[i].execute(row);
- }
- _functionInvoker.convertTypes(_arguments);
- return _functionInvoker.invoke(_arguments);
- }
-
- @Override
- public Object execute(Object[] values) {
- int numArguments = _argumentNodes.length;
- for (int i = 0; i < numArguments; i++) {
- _arguments[i] = _argumentNodes[i].execute(values);
- }
- _functionInvoker.convertTypes(_arguments);
- return _functionInvoker.invoke(_arguments);
- }
- }
-
- private static class ConstantExecutionNode implements ExecutableNode {
- final String _value;
-
- ConstantExecutionNode(String value) {
- _value = value;
- }
-
- @Override
- public String execute(GenericRow row) {
- return _value;
- }
-
- @Override
- public Object execute(Object[] values) {
- return _value;
- }
- }
-
- private static class ColumnExecutionNode implements ExecutableNode {
- final String _column;
- final int _id;
-
- ColumnExecutionNode(String column, int id) {
- _column = column;
- _id = id;
- }
-
- @Override
- public Object execute(GenericRow row) {
- return row.getValue(_column);
- }
-
- @Override
- public Object execute(Object[] values) {
- return values[_id];
- }
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/function/TimeSpecFunctionEvaluator.java b/pinot-core/src/main/java/org/apache/pinot/core/data/function/TimeSpecFunctionEvaluator.java
deleted file mode 100644
index bc79b97..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/function/TimeSpecFunctionEvaluator.java
+++ /dev/null
@@ -1,79 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.function;
-
-import com.google.common.base.Preconditions;
-import java.util.Collections;
-import java.util.List;
-import org.apache.pinot.spi.data.TimeFieldSpec;
-import org.apache.pinot.spi.data.TimeGranularitySpec;
-import org.apache.pinot.spi.data.readers.GenericRow;
-import org.apache.pinot.spi.utils.TimeConverter;
-import org.apache.pinot.spi.utils.TimeUtils;
-
-
-/**
- * An implementation of {@link FunctionEvaluator} for converting the time value based on the {@link TimeFieldSpec}.
- */
-public class TimeSpecFunctionEvaluator implements FunctionEvaluator {
- private final String _incomingTimeColumn;
- private final String _outgoingTimeColumn;
- private final TimeConverter _incomingTimeConverter;
- private final TimeConverter _outgoingTimeConverter;
- private boolean _isValidated = false;
-
- public TimeSpecFunctionEvaluator(TimeGranularitySpec incomingGranularitySpec,
- TimeGranularitySpec outgoingGranularitySpec) {
- Preconditions.checkState(!incomingGranularitySpec.equals(outgoingGranularitySpec));
- _incomingTimeColumn = incomingGranularitySpec.getName();
- _outgoingTimeColumn = outgoingGranularitySpec.getName();
- Preconditions.checkState(!_incomingTimeColumn.equals(_outgoingTimeColumn));
- _incomingTimeConverter = new TimeConverter(incomingGranularitySpec);
- _outgoingTimeConverter = new TimeConverter(outgoingGranularitySpec);
- }
-
- @Override
- public List<String> getArguments() {
- return Collections.singletonList(_incomingTimeColumn);
- }
-
- /**
- * Performs time transformation
- */
- @Override
- public Object evaluate(GenericRow genericRow) {
- return evaluate(genericRow.getValue(_incomingTimeColumn));
- }
-
- @Override
- public Object evaluate(Object[] values) {
- return evaluate(values[0]);
- }
-
- private Object evaluate(Object incomingTimeValue) {
- if (!_isValidated) {
- if (_incomingTimeColumn == null || !TimeUtils
- .timeValueInValidRange(_incomingTimeConverter.toMillisSinceEpoch(incomingTimeValue))) {
- throw new IllegalStateException("No valid time value found in incoming time column: " + _incomingTimeColumn);
- }
- _isValidated = true;
- }
- return _outgoingTimeConverter.fromMillisSinceEpoch(_incomingTimeConverter.toMillisSinceEpoch(incomingTimeValue));
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
index 52c037b..7214418 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
@@ -33,7 +33,7 @@ import org.apache.pinot.common.metrics.ServerMeter;
import org.apache.pinot.common.metrics.ServerMetrics;
import org.apache.pinot.core.data.manager.config.TableDataManagerConfig;
import org.apache.pinot.core.data.manager.offline.ImmutableSegmentDataManager;
-import org.apache.pinot.core.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
import org.apache.pinot.segment.spi.ImmutableSegment;
import org.apache.pinot.spi.config.table.TableConfig;
import org.slf4j.Logger;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/TableDataManager.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/TableDataManager.java
index 42e8477..9357ba6 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/TableDataManager.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/TableDataManager.java
@@ -27,7 +27,7 @@ import org.apache.helix.ZNRecord;
import org.apache.helix.store.zk.ZkHelixPropertyStore;
import org.apache.pinot.common.metrics.ServerMetrics;
import org.apache.pinot.core.data.manager.config.TableDataManagerConfig;
-import org.apache.pinot.core.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
import org.apache.pinot.segment.spi.ImmutableSegment;
import org.apache.pinot.spi.config.table.TableConfig;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/config/InstanceDataManagerConfig.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/config/InstanceDataManagerConfig.java
deleted file mode 100644
index c03fb20..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/config/InstanceDataManagerConfig.java
+++ /dev/null
@@ -1,57 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.manager.config;
-
-import org.apache.pinot.common.segment.ReadMode;
-import org.apache.pinot.spi.env.PinotConfiguration;
-
-
-public interface InstanceDataManagerConfig {
- PinotConfiguration getConfig();
-
- String getInstanceId();
-
- String getInstanceDataDir();
-
- String getConsumerDir();
-
- String getInstanceSegmentTarDir();
-
- String getInstanceBootstrapSegmentDir();
-
- String getSegmentStoreUri();
-
- ReadMode getReadMode();
-
- String getSegmentFormatVersion();
-
- String getAvgMultiValueCount();
-
- boolean isEnableSplitCommit();
-
- boolean isEnableSplitCommitEndWithMetadata();
-
- boolean isRealtimeOffHeapAllocation();
-
- boolean isDirectRealtimeOffHeapAllocation();
-
- int getMaxParallelSegmentBuilds();
-
- String getAuthToken();
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/config/TableDataManagerConfig.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/config/TableDataManagerConfig.java
index 577d795..3f89263 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/config/TableDataManagerConfig.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/config/TableDataManagerConfig.java
@@ -22,7 +22,7 @@ import com.google.common.base.Preconditions;
import javax.annotation.Nonnull;
import org.apache.commons.configuration.Configuration;
import org.apache.commons.configuration.PropertiesConfiguration;
-import org.apache.pinot.common.utils.CommonConstants;
+import org.apache.pinot.spi.config.instance.InstanceDataManagerConfig;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.utils.builder.TableNameBuilder;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/DimensionTableDataManager.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/DimensionTableDataManager.java
index 65d7990..c3c0d7b 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/DimensionTableDataManager.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/DimensionTableDataManager.java
@@ -31,8 +31,8 @@ import javax.annotation.concurrent.GuardedBy;
import javax.annotation.concurrent.ThreadSafe;
import org.apache.pinot.common.metadata.ZKMetadataProvider;
import org.apache.pinot.core.data.manager.SegmentDataManager;
-import org.apache.pinot.core.data.readers.PinotSegmentRecordReader;
-import org.apache.pinot.core.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.readers.PinotSegmentRecordReader;
import org.apache.pinot.segment.spi.IndexSegment;
import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.Schema;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/OfflineTableDataManager.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/OfflineTableDataManager.java
index c2c9351..e3cf839 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/OfflineTableDataManager.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/OfflineTableDataManager.java
@@ -20,11 +20,11 @@ package org.apache.pinot.core.data.manager.offline;
import java.io.File;
import javax.annotation.concurrent.ThreadSafe;
-import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.common.metadata.ZKMetadataProvider;
import org.apache.pinot.core.data.manager.BaseTableDataManager;
-import org.apache.pinot.core.indexsegment.immutable.ImmutableSegmentLoader;
-import org.apache.pinot.core.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.spi.data.Schema;
/**
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/TableDataManagerProvider.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/TableDataManagerProvider.java
index 010d872..80f2917 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/TableDataManagerProvider.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/TableDataManagerProvider.java
@@ -25,9 +25,9 @@ import org.apache.helix.ZNRecord;
import org.apache.helix.store.zk.ZkHelixPropertyStore;
import org.apache.pinot.common.metrics.ServerMetrics;
import org.apache.pinot.core.data.manager.TableDataManager;
-import org.apache.pinot.core.data.manager.config.InstanceDataManagerConfig;
import org.apache.pinot.core.data.manager.config.TableDataManagerConfig;
import org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager;
+import org.apache.pinot.spi.config.instance.InstanceDataManagerConfig;
import org.apache.pinot.spi.config.table.TableType;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/HLRealtimeSegmentDataManager.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/HLRealtimeSegmentDataManager.java
index 9e09a91..bb52290 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/HLRealtimeSegmentDataManager.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/HLRealtimeSegmentDataManager.java
@@ -35,15 +35,13 @@ import org.apache.pinot.common.metadata.segment.RealtimeSegmentZKMetadata;
import org.apache.pinot.common.metrics.ServerGauge;
import org.apache.pinot.common.metrics.ServerMeter;
import org.apache.pinot.common.metrics.ServerMetrics;
-import org.apache.pinot.common.utils.CommonConstants.Segment.Realtime.Status;
-import org.apache.pinot.common.utils.CommonConstants.Segment.SegmentType;
-import org.apache.pinot.core.data.recordtransformer.CompositeTransformer;
-import org.apache.pinot.core.data.recordtransformer.RecordTransformer;
-import org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl;
-import org.apache.pinot.core.realtime.converter.RealtimeSegmentConverter;
-import org.apache.pinot.core.realtime.impl.RealtimeSegmentConfig;
-import org.apache.pinot.core.segment.index.loader.IndexLoadingConfig;
-import org.apache.pinot.core.util.IngestionUtils;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.recordtransformer.CompositeTransformer;
+import org.apache.pinot.segment.local.recordtransformer.RecordTransformer;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
import org.apache.pinot.segment.spi.MutableSegment;
import org.apache.pinot.segment.spi.creator.SegmentVersion;
import org.apache.pinot.spi.config.table.IndexingConfig;
@@ -57,6 +55,8 @@ import org.apache.pinot.spi.stream.StreamConfig;
import org.apache.pinot.spi.stream.StreamConsumerFactory;
import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
import org.apache.pinot.spi.stream.StreamLevelConsumer;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.Realtime.Status;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.SegmentType;
import org.apache.pinot.spi.utils.IngestionConfigUtils;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
@@ -286,6 +286,7 @@ public class HLRealtimeSegmentDataManager extends RealtimeSegmentDataManager {
File tempSegmentFolder = new File(_resourceTmpDir, "tmp-" + System.currentTimeMillis());
// lets convert the segment now
+
RealtimeSegmentConverter converter =
new RealtimeSegmentConverter(_realtimeSegment, tempSegmentFolder.getAbsolutePath(), schema,
_tableNameWithType, tableConfig, realtimeSegmentZKMetadata.getSegmentName(), _sortedColumn,
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
index e7808de..b182f45 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
@@ -45,21 +45,20 @@ import org.apache.pinot.common.metrics.ServerGauge;
import org.apache.pinot.common.metrics.ServerMeter;
import org.apache.pinot.common.metrics.ServerMetrics;
import org.apache.pinot.common.protocols.SegmentCompletionProtocol;
-import org.apache.pinot.common.utils.CommonConstants.Segment.Realtime.CompletionMode;
import org.apache.pinot.common.utils.LLCSegmentName;
import org.apache.pinot.common.utils.TarGzCompressionUtils;
-import org.apache.pinot.core.data.partition.PartitionFunctionFactory;
-import org.apache.pinot.core.data.recordtransformer.CompositeTransformer;
-import org.apache.pinot.core.data.recordtransformer.RecordTransformer;
-import org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl;
-import org.apache.pinot.core.io.readerwriter.PinotDataBufferMemoryManager;
-import org.apache.pinot.core.realtime.converter.RealtimeSegmentConverter;
-import org.apache.pinot.core.realtime.impl.RealtimeSegmentConfig;
-import org.apache.pinot.core.segment.creator.impl.V1Constants;
-import org.apache.pinot.core.segment.index.loader.IndexLoadingConfig;
-import org.apache.pinot.core.segment.store.SegmentDirectoryPaths;
-import org.apache.pinot.core.upsert.PartitionUpsertMetadataManager;
-import org.apache.pinot.core.util.IngestionUtils;
+import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl;
+import org.apache.pinot.segment.local.io.readerwriter.PinotDataBufferMemoryManager;
+import org.apache.pinot.segment.local.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentConfig;
+import org.apache.pinot.segment.local.recordtransformer.CompositeTransformer;
+import org.apache.pinot.segment.local.recordtransformer.RecordTransformer;
+import org.apache.pinot.segment.local.segment.creator.impl.V1Constants;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.store.SegmentDirectoryPaths;
+import org.apache.pinot.segment.local.upsert.PartitionUpsertMetadataManager;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
import org.apache.pinot.segment.spi.MutableSegment;
import org.apache.pinot.segment.spi.creator.SegmentVersion;
import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
@@ -85,6 +84,7 @@ import org.apache.pinot.spi.stream.StreamMetadataProvider;
import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
import org.apache.pinot.spi.stream.TransientConsumerException;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.Realtime.CompletionMode;
import org.apache.pinot.spi.utils.IngestionConfigUtils;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/PeerSchemeSplitSegmentCommitter.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/PeerSchemeSplitSegmentCommitter.java
index 118f0ff..796c00e 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/PeerSchemeSplitSegmentCommitter.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/PeerSchemeSplitSegmentCommitter.java
@@ -21,10 +21,10 @@ package org.apache.pinot.core.data.manager.realtime;
import java.io.File;
import java.net.URI;
import org.apache.pinot.common.protocols.SegmentCompletionProtocol;
-import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.LLCSegmentName;
import org.apache.pinot.common.utils.StringUtil;
import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.slf4j.Logger;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java
index 44e1161..aae38c1 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java
@@ -21,9 +21,9 @@ package org.apache.pinot.core.data.manager.realtime;
import java.util.Map;
import org.apache.pinot.common.metrics.ServerMetrics;
import org.apache.pinot.core.data.manager.SegmentDataManager;
-import org.apache.pinot.core.io.readerwriter.PinotDataBufferMemoryManager;
-import org.apache.pinot.core.io.writer.impl.DirectMemoryManager;
-import org.apache.pinot.core.io.writer.impl.MmapMemoryManager;
+import org.apache.pinot.segment.local.io.readerwriter.PinotDataBufferMemoryManager;
+import org.apache.pinot.segment.local.io.writer.impl.DirectMemoryManager;
+import org.apache.pinot.segment.local.io.writer.impl.MmapMemoryManager;
import org.apache.pinot.segment.spi.MutableSegment;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
index 1b79d8e..5d4d43d 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
@@ -42,8 +42,6 @@ import org.apache.pinot.common.metadata.instance.InstanceZKMetadata;
import org.apache.pinot.common.metadata.segment.LLCRealtimeSegmentZKMetadata;
import org.apache.pinot.common.metadata.segment.RealtimeSegmentZKMetadata;
import org.apache.pinot.common.metrics.ServerGauge;
-import org.apache.pinot.common.utils.CommonConstants;
-import org.apache.pinot.common.utils.CommonConstants.Segment.Realtime.Status;
import org.apache.pinot.common.utils.LLCSegmentName;
import org.apache.pinot.common.utils.NamedThreadFactory;
import org.apache.pinot.common.utils.SegmentName;
@@ -52,21 +50,20 @@ import org.apache.pinot.common.utils.TarGzCompressionUtils;
import org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory;
import org.apache.pinot.core.data.manager.BaseTableDataManager;
import org.apache.pinot.core.data.manager.SegmentDataManager;
-import org.apache.pinot.core.data.readers.PinotSegmentColumnReader;
-import org.apache.pinot.core.indexsegment.immutable.ImmutableSegmentImpl;
-import org.apache.pinot.core.indexsegment.immutable.ImmutableSegmentLoader;
-import org.apache.pinot.core.realtime.impl.RealtimeSegmentStatsHistory;
-import org.apache.pinot.core.realtime.impl.ThreadSafeMutableRoaringBitmap;
-import org.apache.pinot.core.segment.index.loader.IndexLoadingConfig;
-import org.apache.pinot.core.segment.index.loader.LoaderUtils;
-import org.apache.pinot.core.segment.index.loader.V3RemoveIndexException;
-import org.apache.pinot.core.segment.virtualcolumn.VirtualColumnProviderFactory;
-import org.apache.pinot.core.upsert.PartitionUpsertMetadataManager;
-import org.apache.pinot.core.upsert.PartitionUpsertMetadataManager.RecordInfo;
import org.apache.pinot.core.upsert.TableUpsertMetadataManager;
-import org.apache.pinot.core.util.IngestionUtils;
import org.apache.pinot.core.util.PeerServerSegmentFinder;
-import org.apache.pinot.core.util.SchemaUtils;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentImpl;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.realtime.impl.RealtimeSegmentStatsHistory;
+import org.apache.pinot.segment.local.realtime.impl.ThreadSafeMutableRoaringBitmap;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.LoaderUtils;
+import org.apache.pinot.segment.local.segment.index.loader.V3RemoveIndexException;
+import org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReader;
+import org.apache.pinot.segment.local.segment.virtualcolumn.VirtualColumnProviderFactory;
+import org.apache.pinot.segment.local.upsert.PartitionUpsertMetadataManager;
+import org.apache.pinot.segment.local.utils.IngestionUtils;
+import org.apache.pinot.segment.local.utils.SchemaUtils;
import org.apache.pinot.segment.spi.ImmutableSegment;
import org.apache.pinot.spi.config.table.IndexingConfig;
import org.apache.pinot.spi.config.table.TableConfig;
@@ -75,8 +72,10 @@ import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.spi.data.readers.PrimaryKey;
import org.apache.pinot.spi.utils.ByteArray;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.Realtime.Status;
-import static org.apache.pinot.common.utils.CommonConstants.Segment.METADATA_URI_FOR_PEER_DOWNLOAD;
+import static org.apache.pinot.spi.utils.CommonConstants.Segment.METADATA_URI_FOR_PEER_DOWNLOAD;
@ThreadSafe
@@ -329,8 +328,8 @@ public class RealtimeTableDataManager extends BaseTableDataManager {
int partitionGroupId = llcSegmentName.getPartitionGroupId();
Semaphore semaphore = _partitionGroupIdToSemaphoreMap.computeIfAbsent(partitionGroupId, k -> new Semaphore(1));
PartitionUpsertMetadataManager partitionUpsertMetadataManager =
- _tableUpsertMetadataManager != null ? _tableUpsertMetadataManager.getOrCreatePartitionManager(partitionGroupId)
- : null;
+ _tableUpsertMetadataManager != null ? _tableUpsertMetadataManager
+ .getOrCreatePartitionManager(partitionGroupId) : null;
segmentDataManager =
new LLRealtimeSegmentDataManager(realtimeSegmentZKMetadata, tableConfig, this, _indexDir.getAbsolutePath(),
indexLoadingConfig, schema, llcSegmentName, semaphore, _serverMetrics, partitionUpsertMetadataManager);
@@ -367,31 +366,32 @@ public class RealtimeTableDataManager extends BaseTableDataManager {
PartitionUpsertMetadataManager partitionUpsertMetadataManager =
_tableUpsertMetadataManager.getOrCreatePartitionManager(partitionGroupId);
int numPrimaryKeyColumns = _primaryKeyColumns.size();
- Iterator<RecordInfo> recordInfoIterator = new Iterator<RecordInfo>() {
- private int _docId = 0;
+ Iterator<PartitionUpsertMetadataManager.RecordInfo> recordInfoIterator =
+ new Iterator<PartitionUpsertMetadataManager.RecordInfo>() {
+ private int _docId = 0;
- @Override
- public boolean hasNext() {
- return _docId < numTotalDocs;
- }
+ @Override
+ public boolean hasNext() {
+ return _docId < numTotalDocs;
+ }
- @Override
- public RecordInfo next() {
- Object[] values = new Object[numPrimaryKeyColumns];
- for (int i = 0; i < numPrimaryKeyColumns; i++) {
- Object value = columnToReaderMap.get(_primaryKeyColumns.get(i)).getValue(_docId);
- if (value instanceof byte[]) {
- value = new ByteArray((byte[]) value);
+ @Override
+ public PartitionUpsertMetadataManager.RecordInfo next() {
+ Object[] values = new Object[numPrimaryKeyColumns];
+ for (int i = 0; i < numPrimaryKeyColumns; i++) {
+ Object value = columnToReaderMap.get(_primaryKeyColumns.get(i)).getValue(_docId);
+ if (value instanceof byte[]) {
+ value = new ByteArray((byte[]) value);
+ }
+ values[i] = value;
+ }
+ PrimaryKey primaryKey = new PrimaryKey(values);
+ Object timeValue = columnToReaderMap.get(_timeColumnName).getValue(_docId);
+ Preconditions.checkArgument(timeValue instanceof Comparable, "time column shall be comparable");
+ long timestamp = IngestionUtils.extractTimeValue((Comparable) timeValue);
+ return new PartitionUpsertMetadataManager.RecordInfo(primaryKey, _docId++, timestamp);
}
- values[i] = value;
- }
- PrimaryKey primaryKey = new PrimaryKey(values);
- Object timeValue = columnToReaderMap.get(_timeColumnName).getValue(_docId);
- Preconditions.checkArgument(timeValue instanceof Comparable, "time column shall be comparable");
- long timestamp = IngestionUtils.extractTimeValue((Comparable) timeValue);
- return new RecordInfo(primaryKey, _docId++, timestamp);
- }
- };
+ };
ThreadSafeMutableRoaringBitmap validDocIds =
partitionUpsertMetadataManager.addSegment(segmentName, recordInfoIterator);
immutableSegment.enableUpsert(partitionUpsertMetadataManager, validDocIds);
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentCommitterFactory.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentCommitterFactory.java
index 9cb5704..1361121 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentCommitterFactory.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentCommitterFactory.java
@@ -21,7 +21,7 @@ package org.apache.pinot.core.data.manager.realtime;
import java.net.URISyntaxException;
import org.apache.pinot.common.metrics.ServerMetrics;
import org.apache.pinot.common.protocols.SegmentCompletionProtocol;
-import org.apache.pinot.core.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
import org.apache.pinot.spi.config.table.TableConfig;
import org.slf4j.Logger;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/partition/ByteArrayPartitionFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/data/partition/ByteArrayPartitionFunction.java
deleted file mode 100644
index efec9dc..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/partition/ByteArrayPartitionFunction.java
+++ /dev/null
@@ -1,62 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.partition;
-
-import com.google.common.base.Preconditions;
-import java.util.Arrays;
-import org.apache.pinot.segment.spi.partition.PartitionFunction;
-
-
-/**
- * Implementation of {@link Byte array partitioner}
- *
- */
-public class ByteArrayPartitionFunction implements PartitionFunction {
- private static final String NAME = "ByteArray";
- private final int _numPartitions;
-
- /**
- * Constructor for the class.
- * @param numPartitions Number of partitions
- */
- public ByteArrayPartitionFunction(int numPartitions) {
- Preconditions.checkArgument(numPartitions > 0, "Number of partitions must be > 0, specified", numPartitions);
- _numPartitions = numPartitions;
- }
-
- @Override
- public int getPartition(Object valueIn) {
- return abs(Arrays.hashCode(valueIn.toString().getBytes())) % _numPartitions;
- }
-
- @Override
- public int getNumPartitions() {
- return _numPartitions;
- }
-
- @Override
- public String toString() {
- return NAME;
- }
-
- private int abs(int n) {
- return (n == Integer.MIN_VALUE) ? 0 : Math.abs(n);
- }
-}
-
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/partition/HashCodePartitionFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/data/partition/HashCodePartitionFunction.java
deleted file mode 100644
index 44ea29b..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/partition/HashCodePartitionFunction.java
+++ /dev/null
@@ -1,54 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.partition;
-
-import com.google.common.base.Preconditions;
-import org.apache.pinot.segment.spi.partition.PartitionFunction;
-
-
-/**
- * Hash code partition function, where:
- * <ul>
- * <li> partitionId = value.hashCode() % {@link #_numPartitions}</li>
- * </ul>
- */
-public class HashCodePartitionFunction implements PartitionFunction {
- private static final String NAME = "HashCode";
- private final int _numPartitions;
-
- public HashCodePartitionFunction(int numPartitions) {
- Preconditions.checkArgument(numPartitions > 0, "Number of partitions must be > 0, specified", numPartitions);
- _numPartitions = numPartitions;
- }
-
- @Override
- public int getPartition(Object value) {
- return Math.abs(value.hashCode()) % _numPartitions;
- }
-
- @Override
- public int getNumPartitions() {
- return _numPartitions;
- }
-
- @Override
- public String toString() {
- return NAME;
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/partition/ModuloPartitionFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/data/partition/ModuloPartitionFunction.java
deleted file mode 100644
index e0c5d40..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/partition/ModuloPartitionFunction.java
+++ /dev/null
@@ -1,73 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.partition;
-
-import com.google.common.base.Preconditions;
-import org.apache.pinot.segment.spi.partition.PartitionFunction;
-
-
-/**
- * Modulo operation based partition function, where:
- * <ul>
- * <li> partitionId = value % {@link #_numPartitions}</li>
- * </ul>
- *
- */
-public class ModuloPartitionFunction implements PartitionFunction {
- private static final String NAME = "Modulo";
- private final int _numPartitions;
-
- /**
- * Constructor for the class.
- * @param numPartitions Number of partitions.
- */
- public ModuloPartitionFunction(int numPartitions) {
- Preconditions.checkArgument(numPartitions > 0, "Number of partitions must be > 0, specified", numPartitions);
- _numPartitions = numPartitions;
- }
-
- /**
- * Returns partition id for a given value. Assumes that the passed in object
- * is either an Integer, or a string representation of an Integer.
- *
- * @param value Value for which to determine the partition id.
- * @return Partition id for the given value.
- */
- @Override
- public int getPartition(Object value) {
- if (value instanceof Integer) {
- return ((Integer) value) % _numPartitions;
- } else if (value instanceof String) {
- return ((Integer.parseInt((String) value)) % _numPartitions);
- } else {
- throw new IllegalArgumentException(
- "Illegal argument for partitioning, expected Integer, got: " + value.getClass().getName());
- }
- }
-
- @Override
- public int getNumPartitions() {
- return _numPartitions;
- }
-
- @Override
- public String toString() {
- return NAME;
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/partition/MurmurPartitionFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/data/partition/MurmurPartitionFunction.java
deleted file mode 100644
index 0f2d7ea..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/partition/MurmurPartitionFunction.java
+++ /dev/null
@@ -1,71 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.partition;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Preconditions;
-import org.apache.pinot.common.utils.HashUtil;
-import org.apache.pinot.common.utils.StringUtil;
-import org.apache.pinot.segment.spi.partition.PartitionFunction;
-
-
-/**
- * Implementation of {@link PartitionFunction} which partitions based on 32 bit murmur hash
- */
-public class MurmurPartitionFunction implements PartitionFunction {
- private static final String NAME = "Murmur";
- private final int _numPartitions;
-
- /**
- * Constructor for the class.
- * @param numPartitions Number of partitions.
- */
- public MurmurPartitionFunction(int numPartitions) {
- Preconditions.checkArgument(numPartitions > 0, "Number of partitions must be > 0");
- _numPartitions = numPartitions;
- }
-
- @Override
- public int getPartition(Object valueIn) {
- String value = (valueIn instanceof String) ? (String) valueIn : valueIn.toString();
- return (murmur2(StringUtil.encodeUtf8(value)) & 0x7fffffff) % _numPartitions;
- }
-
- @Override
- public int getNumPartitions() {
- return _numPartitions;
- }
-
- @Override
- public String toString() {
- return NAME;
- }
-
- /**
- * NOTE: This code has been copied over from org.apache.kafka.common.utils.Utils::murmur2
- *
- * Generates 32 bit murmur2 hash from byte array
- * @param data byte array to hash
- * @return 32 bit hash of the given array
- */
- @VisibleForTesting
- int murmur2(final byte[] data) {
- return HashUtil.murmur2(data);
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/partition/PartitionFunctionFactory.java b/pinot-core/src/main/java/org/apache/pinot/core/data/partition/PartitionFunctionFactory.java
deleted file mode 100644
index 045623e..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/partition/PartitionFunctionFactory.java
+++ /dev/null
@@ -1,89 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.partition;
-
-import java.util.HashMap;
-import java.util.Map;
-import javax.annotation.Nonnull;
-import org.apache.pinot.segment.spi.partition.PartitionFunction;
-
-
-/**
- * Factory to build instances of {@link PartitionFunction}.
- */
-public class PartitionFunctionFactory {
- // Enum for various partition functions to be added.
- public enum PartitionFunctionType {
- Modulo, Murmur, ByteArray, HashCode;
- // Add more functions here.
-
- private static final Map<String, PartitionFunctionType> VALUE_MAP = new HashMap<>();
-
- static {
- for (PartitionFunctionType functionType : PartitionFunctionType.values()) {
- VALUE_MAP.put(functionType.name().toLowerCase(), functionType);
- }
- }
-
- public static PartitionFunctionType fromString(String name) {
- PartitionFunctionType functionType = VALUE_MAP.get(name.toLowerCase());
-
- if (functionType == null) {
- throw new IllegalArgumentException("No enum constant for: " + name);
- }
- return functionType;
- }
- }
-
- /**
- * Private constructor so that the class cannot be instantiated.
- */
- private PartitionFunctionFactory() {
- }
-
- /**
- * This method generates and returns a partition function based on the provided string.
- *
- * @param functionName Name of partition function
- * @param numPartitions Number of partitions.
- * @return Partition function
- */
- // TODO: introduce a way to inject custom partition function
- // a custom partition function could be used in the realtime stream partitioning or offline segment partitioning.
- // The PartitionFunctionFactory should be able to support these default implementations, as well as instantiate based on config
- public static PartitionFunction getPartitionFunction(@Nonnull String functionName, int numPartitions) {
- PartitionFunctionType function = PartitionFunctionType.fromString(functionName);
- switch (function) {
- case Modulo:
- return new ModuloPartitionFunction(numPartitions);
-
- case Murmur:
- return new MurmurPartitionFunction(numPartitions);
-
- case ByteArray:
- return new ByteArrayPartitionFunction(numPartitions);
-
- case HashCode:
- return new HashCodePartitionFunction(numPartitions);
-
- default:
- throw new IllegalArgumentException("Illegal partition function name: " + functionName);
- }
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/GenericRowRecordReader.java b/pinot-core/src/main/java/org/apache/pinot/core/data/readers/GenericRowRecordReader.java
deleted file mode 100644
index f3f6969..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/GenericRowRecordReader.java
+++ /dev/null
@@ -1,72 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.readers;
-
-import java.io.File;
-import java.util.List;
-import java.util.Set;
-import javax.annotation.Nullable;
-import org.apache.pinot.spi.data.readers.GenericRow;
-import org.apache.pinot.spi.data.readers.RecordReader;
-import org.apache.pinot.spi.data.readers.RecordReaderConfig;
-
-
-/**
- * Record reader for list of {@link GenericRow}.
- */
-public class GenericRowRecordReader implements RecordReader {
- private final List<GenericRow> _rows;
- private final int _numRows;
-
- private int _nextRowId = 0;
-
- public GenericRowRecordReader(List<GenericRow> rows) {
- _rows = rows;
- _numRows = rows.size();
- }
-
- @Override
- public void init(File dataFile, Set<String> fieldsToRead, @Nullable RecordReaderConfig recordReaderConfig) {
- }
-
- @Override
- public boolean hasNext() {
- return _nextRowId < _numRows;
- }
-
- @Override
- public GenericRow next() {
- return next(new GenericRow());
- }
-
- @Override
- public GenericRow next(GenericRow reuse) {
- reuse.init(_rows.get(_nextRowId++));
- return reuse;
- }
-
- @Override
- public void rewind() {
- _nextRowId = 0;
- }
-
- @Override
- public void close() {
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/IntermediateSegmentRecordReader.java b/pinot-core/src/main/java/org/apache/pinot/core/data/readers/IntermediateSegmentRecordReader.java
deleted file mode 100644
index 4eb1237..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/IntermediateSegmentRecordReader.java
+++ /dev/null
@@ -1,80 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.readers;
-
-import java.io.File;
-import java.io.IOException;
-import java.util.Set;
-import javax.annotation.Nullable;
-import org.apache.pinot.core.indexsegment.mutable.IntermediateSegment;
-import org.apache.pinot.spi.data.readers.GenericRow;
-import org.apache.pinot.spi.data.readers.RecordReader;
-import org.apache.pinot.spi.data.readers.RecordReaderConfig;
-
-
-public class IntermediateSegmentRecordReader implements RecordReader {
- private final IntermediateSegment _intermediateSegment;
- private final int _numDocs;
-
- private int _nextDocId = 0;
-
- public IntermediateSegmentRecordReader(IntermediateSegment intermediateSegment) {
- _intermediateSegment = intermediateSegment;
- _numDocs = intermediateSegment.getNumDocsIndexed();
- // TODO: support sorted column like _sortedDocIdIterationOrder in {@link RealtimeSegmentRecordReader}
- }
-
- @Override
- public void init(File dataFile, Set<String> fieldsToRead, @Nullable RecordReaderConfig recordReaderConfig)
- throws IOException {
- }
-
- @Override
- public boolean hasNext() {
- return _nextDocId < _numDocs;
- }
-
- @Override
- public GenericRow next()
- throws IOException {
- return next(new GenericRow());
- }
-
- @Override
- public GenericRow next(GenericRow reuse)
- throws IOException {
- return _intermediateSegment.getRecord(_nextDocId++, reuse);
- }
-
- @Override
- public void rewind()
- throws IOException {
- _nextDocId = 0;
- }
-
- @Override
- public void close()
- throws IOException {
- _intermediateSegment.destroy();
- }
-
- public IntermediateSegment getIntermediateSegment() {
- return _intermediateSegment;
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/MultiplePinotSegmentRecordReader.java b/pinot-core/src/main/java/org/apache/pinot/core/data/readers/MultiplePinotSegmentRecordReader.java
deleted file mode 100644
index f40a58a..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/MultiplePinotSegmentRecordReader.java
+++ /dev/null
@@ -1,280 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.readers;
-
-import java.io.File;
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.HashSet;
-import java.util.List;
-import java.util.PriorityQueue;
-import java.util.Set;
-import javax.annotation.Nullable;
-import org.apache.pinot.spi.data.FieldSpec;
-import org.apache.pinot.spi.data.Schema;
-import org.apache.pinot.spi.data.readers.GenericRow;
-import org.apache.pinot.spi.data.readers.RecordReader;
-import org.apache.pinot.spi.data.readers.RecordReaderConfig;
-
-
-/**
- * Record reader for multiple pinot segments.
- */
-public class MultiplePinotSegmentRecordReader implements RecordReader {
- private List<PinotSegmentRecordReader> _pinotSegmentRecordReaders;
- private PriorityQueue<GenericRowWithReader> _priorityQueue;
- private Schema _schema;
- private List<String> _sortOrder;
- private int _currentReaderId;
-
- /**
- * Read records using the passed in schema from multiple pinot segments.
- * <p>Passed in schema must be a subset of the segment schema.
- *
- * @param indexDirs a list of input segment directory paths
- */
- public MultiplePinotSegmentRecordReader(List<File> indexDirs)
- throws Exception {
- this(indexDirs, null, null);
- }
-
- /**
- * Read records using the passed in schema and in the order of sorted column from multiple pinot segments.
- * <p>Passed in schema must be a subset of the segment schema.
- * <p>If sort order is not specified, it will not attempt to sort the segments and simply concatenate the records from
- * muiltiple segments.
- *
- * @param indexDirs a list of input paths for the segment indices
- * @param schema input schema that is a subset of the segment schema
- * @param sortOrder a list of column names that represent the sorting order
- */
- public MultiplePinotSegmentRecordReader(List<File> indexDirs, @Nullable Schema schema,
- @Nullable List<String> sortOrder)
- throws Exception {
- // Initialize pinot segment record readers
- _pinotSegmentRecordReaders = new ArrayList<>(indexDirs.size());
- for (File file : indexDirs) {
- _pinotSegmentRecordReaders.add(new PinotSegmentRecordReader(file, schema, sortOrder));
- }
-
- // Initialize schema
- if (schema == null) {
- // Validate that segment schemas from all segments are the same if the schema is not passed.
- Set<Schema> schemas = new HashSet<>();
- for (PinotSegmentRecordReader pinotSegmentRecordReader : _pinotSegmentRecordReaders) {
- schemas.add(pinotSegmentRecordReader.getSchema());
- }
- if (schemas.size() == 1) {
- _schema = schemas.iterator().next();
- } else {
- throw new IllegalStateException("Schemas from input segments are not the same");
- }
- } else {
- // If the schema is given, the passed schema can directly be used.
- _schema = schema;
- }
-
- // Initialize sort order and priority queue
- _sortOrder = sortOrder;
- if (isSortedSegment()) {
- _priorityQueue = new PriorityQueue<>(_pinotSegmentRecordReaders.size());
- for (PinotSegmentRecordReader recordReader : _pinotSegmentRecordReaders) {
- if (recordReader.hasNext()) {
- _priorityQueue.add(new GenericRowWithReader(recordReader.next(), recordReader, _sortOrder, _schema));
- }
- }
- }
- }
-
- public Schema getSchema() {
- return _schema;
- }
-
- /**
- * Indicate whether the segment should be sorted or not
- */
- private boolean isSortedSegment() {
- return _sortOrder != null && !_sortOrder.isEmpty();
- }
-
- @Override
- public void init(File dataFile, Set<String> fieldsToRead, @Nullable RecordReaderConfig recordReaderConfig) {
- }
-
- @Override
- public boolean hasNext() {
- if (isSortedSegment()) {
- return _priorityQueue.size() > 0;
- } else {
- boolean hasNext = false;
- for (PinotSegmentRecordReader recordReader : _pinotSegmentRecordReaders) {
- if (recordReader.hasNext()) {
- hasNext = true;
- break;
- }
- }
- return hasNext;
- }
- }
-
- @Override
- public GenericRow next() {
- return next(new GenericRow());
- }
-
- @Override
- public GenericRow next(GenericRow reuse) {
- if (isSortedSegment()) {
- GenericRowWithReader genericRowComparable = _priorityQueue.poll();
- GenericRow currentRow = genericRowComparable.getRow();
-
- // Fill reuse with the information from the currentRow
- reuse.init(currentRow);
-
- // If the record reader has more rows left, put back the next minimum value to the queue
- PinotSegmentRecordReader recordReader = genericRowComparable.getRecordReader();
- if (recordReader.hasNext()) {
- currentRow.clear();
- genericRowComparable.setRow(recordReader.next(currentRow));
- genericRowComparable.setRecordReader(recordReader);
- _priorityQueue.add(genericRowComparable);
- }
- return reuse;
- } else {
- // If there is no sorted column specified, simply concatenate the segments
- for (int i = 0; i < _pinotSegmentRecordReaders.size();
- i++, _currentReaderId = (_currentReaderId + 1) % _pinotSegmentRecordReaders.size()) {
- PinotSegmentRecordReader currentReader = _pinotSegmentRecordReaders.get(_currentReaderId);
- if (currentReader.hasNext()) {
- return currentReader.next(reuse);
- }
- }
- throw new RuntimeException("next is called after reading all data");
- }
- }
-
- @Override
- public void rewind() {
- for (PinotSegmentRecordReader recordReader : _pinotSegmentRecordReaders) {
- recordReader.rewind();
- }
- // If the segment is sorted, we need to re-initialize the priority queue
- if (isSortedSegment()) {
- _priorityQueue.clear();
- for (PinotSegmentRecordReader recordReader : _pinotSegmentRecordReaders) {
- if (recordReader.hasNext()) {
- _priorityQueue.add(new GenericRowWithReader(recordReader.next(), recordReader, _sortOrder, _schema));
- }
- }
- } else {
- _currentReaderId = 0;
- }
- }
-
- @Override
- public void close()
- throws IOException {
- for (PinotSegmentRecordReader recordReader : _pinotSegmentRecordReaders) {
- recordReader.close();
- }
- }
-
- /**
- * Wrapper for generic row and record reader along with sorted column. Comparison of this object is based on
- * the value of sorted column.
- *
- * TODO: add the support for multi value columns
- */
- class GenericRowWithReader implements Comparable<GenericRowWithReader> {
- private GenericRow _row;
- private PinotSegmentRecordReader _recordReader;
- private List<String> _sortOrder;
- private Schema _schema;
-
- public GenericRowWithReader(GenericRow row, PinotSegmentRecordReader recordReader, List<String> sortOrder,
- Schema schema) {
- _row = row;
- _recordReader = recordReader;
- _sortOrder = sortOrder;
- _schema = schema;
- }
-
- @Override
- public int compareTo(GenericRowWithReader o) {
- int compare = 0;
- for (String column : _sortOrder) {
- FieldSpec fieldSpec = _schema.getFieldSpecFor(column);
- Object otherVal = o.getRow().getValue(column);
- Object thisVal = _row.getValue(column);
- if (fieldSpec.isSingleValueField()) {
- switch (fieldSpec.getDataType()) {
- case INT:
- compare = ((Integer) thisVal).compareTo((Integer) otherVal);
- break;
- case LONG:
- compare = ((Long) thisVal).compareTo((Long) otherVal);
- break;
- case FLOAT:
- compare = ((Float) thisVal).compareTo((Float) otherVal);
- break;
- case DOUBLE:
- compare = ((Double) thisVal).compareTo((Double) otherVal);
- break;
- case STRING:
- compare = ((String) thisVal).compareTo((String) otherVal);
- break;
- default:
- throw new IllegalStateException("Unsupported column value type");
- }
- } else {
- throw new IllegalStateException("Multi value column is not supported");
- }
-
- if (compare != 0) {
- return compare;
- }
- }
- return compare;
- }
-
- public GenericRow getRow() {
- return _row;
- }
-
- public void setRow(GenericRow row) {
- _row = row;
- }
-
- public PinotSegmentRecordReader getRecordReader() {
- return _recordReader;
- }
-
- public void setRecordReader(PinotSegmentRecordReader recordReader) {
- _recordReader = recordReader;
- }
-
- public Schema getSchema() {
- return _schema;
- }
-
- public void setSchema(Schema schema) {
- _schema = schema;
- }
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/PinotSegmentColumnReader.java b/pinot-core/src/main/java/org/apache/pinot/core/data/readers/PinotSegmentColumnReader.java
deleted file mode 100644
index 310e9d7..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/PinotSegmentColumnReader.java
+++ /dev/null
@@ -1,112 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.readers;
-
-import java.io.Closeable;
-import java.io.IOException;
-import javax.annotation.Nullable;
-import org.apache.pinot.segment.spi.ImmutableSegment;
-import org.apache.pinot.segment.spi.datasource.DataSource;
-import org.apache.pinot.segment.spi.index.reader.Dictionary;
-import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
-import org.apache.pinot.segment.spi.index.reader.ForwardIndexReaderContext;
-
-
-@SuppressWarnings({"rawtypes", "unchecked"})
-public class PinotSegmentColumnReader implements Closeable {
- private final ForwardIndexReader _reader;
- private final ForwardIndexReaderContext _readerContext;
- private final Dictionary _dictionary;
- private final int[] _dictIdBuffer;
-
- public PinotSegmentColumnReader(ImmutableSegment immutableSegment, String column) {
- DataSource dataSource = immutableSegment.getDataSource(column);
- _reader = dataSource.getForwardIndex();
- _readerContext = _reader.createContext();
- _dictionary = dataSource.getDictionary();
- if (_reader.isSingleValue()) {
- _dictIdBuffer = null;
- } else {
- _dictIdBuffer = new int[dataSource.getDataSourceMetadata().getMaxNumValuesPerMVEntry()];
- }
- }
-
- public PinotSegmentColumnReader(ForwardIndexReader reader, @Nullable Dictionary dictionary,
- int maxNumValuesPerMVEntry) {
- _reader = reader;
- _readerContext = _reader.createContext();
- _dictionary = dictionary;
- if (_reader.isSingleValue()) {
- _dictIdBuffer = null;
- } else {
- _dictIdBuffer = new int[maxNumValuesPerMVEntry];
- }
- }
-
- public boolean hasDictionary() {
- return _dictionary != null;
- }
-
- public int getDictId(int docId) {
- return _reader.getDictId(docId, _readerContext);
- }
-
- public Object getValue(int docId) {
- if (_dictionary != null) {
- if (_reader.isSingleValue()) {
- return _dictionary.get(_reader.getDictId(docId, _readerContext));
- } else {
- int numValues = _reader.getDictIdMV(docId, _dictIdBuffer, _readerContext);
- Object[] values = new Object[numValues];
- for (int i = 0; i < numValues; i++) {
- values[i] = _dictionary.get(_dictIdBuffer[i]);
- }
- return values;
- }
- } else {
- // NOTE: Only support single-value raw index
- assert _reader.isSingleValue();
-
- switch (_reader.getValueType()) {
- case INT:
- return _reader.getInt(docId, _readerContext);
- case LONG:
- return _reader.getLong(docId, _readerContext);
- case FLOAT:
- return _reader.getFloat(docId, _readerContext);
- case DOUBLE:
- return _reader.getDouble(docId, _readerContext);
- case STRING:
- return _reader.getString(docId, _readerContext);
- case BYTES:
- return _reader.getBytes(docId, _readerContext);
- default:
- throw new IllegalStateException();
- }
- }
- }
-
- @Override
- public void close()
- throws IOException {
- if (_readerContext != null) {
- _readerContext.close();
- }
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/PinotSegmentRecordReader.java b/pinot-core/src/main/java/org/apache/pinot/core/data/readers/PinotSegmentRecordReader.java
deleted file mode 100644
index d6259d0..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/PinotSegmentRecordReader.java
+++ /dev/null
@@ -1,171 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.readers;
-
-import com.google.common.base.Preconditions;
-import java.io.File;
-import java.io.IOException;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import javax.annotation.Nullable;
-import org.apache.pinot.common.segment.ReadMode;
-import org.apache.pinot.core.data.readers.sort.PinotSegmentSorter;
-import org.apache.pinot.core.indexsegment.immutable.ImmutableSegmentLoader;
-import org.apache.pinot.core.segment.index.metadata.SegmentMetadataImpl;
-import org.apache.pinot.segment.spi.ImmutableSegment;
-import org.apache.pinot.segment.spi.SegmentMetadata;
-import org.apache.pinot.spi.data.FieldSpec;
-import org.apache.pinot.spi.data.Schema;
-import org.apache.pinot.spi.data.readers.GenericRow;
-import org.apache.pinot.spi.data.readers.RecordReader;
-import org.apache.pinot.spi.data.readers.RecordReaderConfig;
-
-
-/**
- * Record reader for Pinot segment.
- */
-public class PinotSegmentRecordReader implements RecordReader {
- private final ImmutableSegment _immutableSegment;
- private final int _numDocs;
- private final Schema _schema;
- private final Map<String, PinotSegmentColumnReader> _columnReaderMap;
- private final int[] _docIdsInSortedColumnOrder;
-
- private int _nextDocId = 0;
-
- /**
- * Read records using the segment schema
- * @param indexDir input path for the segment index
- */
- public PinotSegmentRecordReader(File indexDir)
- throws Exception {
- this(indexDir, null, null);
- }
-
- /**
- * Read records using the segment schema with the given schema and sort order
- * <p>Passed in schema must be a subset of the segment schema.
- *
- * @param indexDir input path for the segment index
- * @param schema input schema that is a subset of the segment schema
- * @param sortOrder a list of column names that represent the sorting order
- */
- public PinotSegmentRecordReader(File indexDir, @Nullable Schema schema, @Nullable List<String> sortOrder)
- throws Exception {
- _immutableSegment = ImmutableSegmentLoader.load(indexDir, ReadMode.mmap);
- try {
- SegmentMetadata segmentMetadata = _immutableSegment.getSegmentMetadata();
- _numDocs = segmentMetadata.getTotalDocs();
- // In order not to expose virtual columns to client, schema shouldn't be fetched from segmentMetadata;
- // otherwise the original metadata will be modified. Hence, initialize a new schema.
- _schema = schema == null ? new SegmentMetadataImpl(indexDir).getSchema() : schema;
- if (_numDocs > 0) {
- _columnReaderMap = new HashMap<>();
- if (schema == null) {
- Collection<String> columnNames = _schema.getColumnNames();
- for (String columnName : columnNames) {
- _columnReaderMap.put(columnName, new PinotSegmentColumnReader(_immutableSegment, columnName));
- }
- } else {
- Schema segmentSchema = segmentMetadata.getSchema();
- Collection<FieldSpec> fieldSpecs = _schema.getAllFieldSpecs();
- for (FieldSpec fieldSpec : fieldSpecs) {
- String columnName = fieldSpec.getName();
- FieldSpec segmentFieldSpec = segmentSchema.getFieldSpecFor(columnName);
- Preconditions.checkState(fieldSpec.equals(segmentFieldSpec),
- "Field spec mismatch for column: %s, in the given schema: %s, in the segment schema: %s", columnName,
- fieldSpec, segmentFieldSpec);
- _columnReaderMap.put(columnName, new PinotSegmentColumnReader(_immutableSegment, columnName));
- }
- }
- // Initialize sorted doc ids
- if (sortOrder != null && !sortOrder.isEmpty()) {
- _docIdsInSortedColumnOrder =
- new PinotSegmentSorter(_numDocs, _schema, _columnReaderMap).getSortedDocIds(sortOrder);
- } else {
- _docIdsInSortedColumnOrder = null;
- }
- } else {
- _columnReaderMap = Collections.emptyMap();
- _docIdsInSortedColumnOrder = null;
- }
- } catch (Exception e) {
- _immutableSegment.destroy();
- throw e;
- }
- }
-
- public Schema getSchema() {
- return _schema;
- }
-
- @Override
- public void init(File dataFile, Set<String> fieldsToRead, @Nullable RecordReaderConfig recordReaderConfig) {
- }
-
- @Override
- public boolean hasNext() {
- return _nextDocId < _numDocs;
- }
-
- @Override
- public GenericRow next() {
- return next(new GenericRow());
- }
-
- @Override
- public GenericRow next(GenericRow reuse) {
- if (_docIdsInSortedColumnOrder == null) {
- reuse = getRecord(reuse, _nextDocId);
- } else {
- reuse = getRecord(reuse, _docIdsInSortedColumnOrder[_nextDocId]);
- }
- _nextDocId++;
- return reuse;
- }
-
- /**
- * Return the row given a docId
- */
- private GenericRow getRecord(GenericRow reuse, int docId) {
- for (FieldSpec fieldSpec : _schema.getAllFieldSpecs()) {
- String fieldName = fieldSpec.getName();
- reuse.putValue(fieldName, _columnReaderMap.get(fieldName).getValue(docId));
- }
- return reuse;
- }
-
- @Override
- public void rewind() {
- _nextDocId = 0;
- }
-
- @Override
- public void close()
- throws IOException {
- for (PinotSegmentColumnReader columnReader : _columnReaderMap.values()) {
- columnReader.close();
- }
- _immutableSegment.destroy();
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/sort/PinotSegmentSorter.java b/pinot-core/src/main/java/org/apache/pinot/core/data/readers/sort/PinotSegmentSorter.java
deleted file mode 100644
index 3dd283b..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/sort/PinotSegmentSorter.java
+++ /dev/null
@@ -1,121 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.readers.sort;
-
-import it.unimi.dsi.fastutil.Arrays;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Map;
-import org.apache.pinot.core.data.readers.PinotSegmentColumnReader;
-import org.apache.pinot.spi.data.FieldSpec;
-import org.apache.pinot.spi.data.Schema;
-
-
-/**
- * Sorter implementation for pinot segments
- */
-public class PinotSegmentSorter implements SegmentSorter {
-
- private int _numDocs;
- private Schema _schema;
- private Map<String, PinotSegmentColumnReader> _columnReaderMap;
- private int[] _sortOrder;
- private List<String> _dimensionNames;
- int _numDimensions;
-
- public PinotSegmentSorter(int numDocs, Schema schema, Map<String, PinotSegmentColumnReader> columnReaderMap) {
- _numDocs = numDocs;
- _schema = schema;
- _columnReaderMap = columnReaderMap;
- _dimensionNames = new ArrayList<>();
- for (FieldSpec fieldSpec : _schema.getAllFieldSpecs()) {
- // Count all fields that are not metrics as dimensions
- if (fieldSpec.getFieldType() != FieldSpec.FieldType.METRIC) {
- String dimensionName = fieldSpec.getName();
- _numDimensions++;
- _dimensionNames.add(dimensionName);
- }
- }
- }
-
- /**
- * Sort the segment by the sort order columns. For PinotSegmentSorter, orderings are computed by comparing
- * dictionary ids.
- *
- * TODO: add the support for no-dictionary and multi-value columns.
- *
- * @param sortOrder a list of column names that represent the sorting order
- * @return an array of sorted docIds
- */
- @Override
- public int[] getSortedDocIds(final List<String> sortOrder) {
- _sortOrder = new int[sortOrder.size()];
- int index = 0;
- for (String dimension : sortOrder) {
- int dimensionId = _dimensionNames.indexOf(dimension);
- if (dimensionId != -1) {
- _sortOrder[index++] = dimensionId;
- } else {
- throw new IllegalStateException(
- "Passed dimension in the sorting order does not exist in the schema: " + dimension);
- }
- }
-
- final int[] sortedDocIds = new int[_numDocs];
- for (int i = 0; i < _numDocs; i++) {
- sortedDocIds[i] = i;
- }
-
- Arrays.quickSort(0, _numDocs, (i1, i2) -> {
- int docId1 = sortedDocIds[i1];
- int docId2 = sortedDocIds[i2];
-
- int compare = 0;
- for (int sortIndex : _sortOrder) {
- String dimensionName = _dimensionNames.get(sortIndex);
- FieldSpec fieldSpec = _schema.getFieldSpecFor(dimensionName);
- PinotSegmentColumnReader columnReader = _columnReaderMap.get(dimensionName);
-
- // Multi value column or no dictionary column is not supported
- boolean isMultiValueColumn = !fieldSpec.isSingleValueField();
- boolean isNoDictionaryColumn = !columnReader.hasDictionary();
- if (isMultiValueColumn || isNoDictionaryColumn) {
- throw new IllegalStateException(
- "Multi value column or no dictionary column is not supported. ( column name: " + dimensionName
- + ", multi value column: " + isMultiValueColumn + ", no dictionary column: " + isNoDictionaryColumn
- + " )");
- }
-
- // Compute the order
- compare = columnReader.getDictId(docId1) - columnReader.getDictId(docId2);
-
- if (compare != 0) {
- return compare;
- }
- }
- return compare;
- }, (i, j) -> {
- int temp = sortedDocIds[i];
- sortedDocIds[i] = sortedDocIds[j];
- sortedDocIds[j] = temp;
- });
-
- return sortedDocIds;
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/sort/SegmentSorter.java b/pinot-core/src/main/java/org/apache/pinot/core/data/readers/sort/SegmentSorter.java
deleted file mode 100644
index 2886542..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/sort/SegmentSorter.java
+++ /dev/null
@@ -1,35 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.readers.sort;
-
-import java.util.List;
-
-
-/**
- * Interface for segment sorters
- */
-public interface SegmentSorter {
-
- /**
- * Compute the sorted docIds given a list of sort order column names
- * @param sortOrder a list of column names that represent the sorting order
- * @return an array of sorted docIds
- */
- int[] getSortedDocIds(final List<String> sortOrder);
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/recordtransformer/CompositeTransformer.java b/pinot-core/src/main/java/org/apache/pinot/core/data/recordtransformer/CompositeTransformer.java
deleted file mode 100644
index b5e236d..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/recordtransformer/CompositeTransformer.java
+++ /dev/null
@@ -1,89 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.recordtransformer;
-
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.List;
-import javax.annotation.Nullable;
-import org.apache.pinot.spi.config.table.TableConfig;
-import org.apache.pinot.spi.data.Schema;
-import org.apache.pinot.spi.data.readers.GenericRow;
-
-
-/**
- * The {@code CompositeTransformer} class performs multiple transforms based on the inner {@link RecordTransformer}s.
- */
-public class CompositeTransformer implements RecordTransformer {
- private final List<RecordTransformer> _transformers;
-
- /**
- * Returns a record transformer that performs null value handling, time/expression/data-type transformation and record
- * sanitization.
- * <p>NOTE: DO NOT CHANGE THE ORDER OF THE RECORD TRANSFORMERS
- * <ul>
- * <li>
- * {@link ExpressionTransformer} before everyone else, so that we get the real columns for other transformers to
- * work on
- * </li>
- * <li>
- * {@link FilterTransformer} after ExpressionTransformer, so that we have source as well as destination columns
- * </li>
- * <li>
- * {@link DataTypeTransformer} after FilterTransformer to convert values to comply with the schema
- * </li>
- * <li>
- * {@link NullValueTransformer} after {@link DataTypeTransformer} because empty Collection/Map/Object[] will be
- * replaced with null in DataTypeTransformer
- * </li>
- * <li>
- * {@link SanitizationTransformer} after {@link NullValueTransformer} so that before sanitation, all values are
- * non-null and follow the data types defined in the schema
- * </li>
- * </ul>
- */
- public static CompositeTransformer getDefaultTransformer(TableConfig tableConfig, Schema schema) {
- return new CompositeTransformer(Arrays
- .asList(new ExpressionTransformer(tableConfig, schema), new FilterTransformer(tableConfig),
- new DataTypeTransformer(schema), new NullValueTransformer(schema), new SanitizationTransformer(schema)));
- }
-
- /**
- * Returns a pass through record transformer that does not transform the record.
- */
- public static CompositeTransformer getPassThroughTransformer() {
- return new CompositeTransformer(Collections.emptyList());
- }
-
- public CompositeTransformer(List<RecordTransformer> transformers) {
- _transformers = transformers;
- }
-
- @Nullable
- @Override
- public GenericRow transform(GenericRow record) {
- for (RecordTransformer transformer : _transformers) {
- record = transformer.transform(record);
- if (record == null) {
- return null;
- }
- }
- return record;
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/recordtransformer/DataTypeTransformer.java b/pinot-core/src/main/java/org/apache/pinot/core/data/recordtransformer/DataTypeTransformer.java
deleted file mode 100644
index be4b1b6..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/recordtransformer/DataTypeTransformer.java
+++ /dev/null
@@ -1,200 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.recordtransformer;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Preconditions;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import javax.annotation.Nullable;
-import org.apache.pinot.common.utils.PinotDataType;
-import org.apache.pinot.spi.data.FieldSpec;
-import org.apache.pinot.spi.data.Schema;
-import org.apache.pinot.spi.data.readers.GenericRow;
-
-
-/**
- * The {@code DataTypeTransformer} class will convert the values to follow the data types in {@link FieldSpec}.
- * <p>NOTE: should put this after all the values has been generated by other transformers (such as
- * {@link NullValueTransformer} and {@link ExpressionTransformer}). After this, all values should be of the desired data
- * types.
- */
-@SuppressWarnings("rawtypes")
-public class DataTypeTransformer implements RecordTransformer {
- private static final Map<Class, PinotDataType> SINGLE_VALUE_TYPE_MAP = new HashMap<>();
- private static final Map<Class, PinotDataType> MULTI_VALUE_TYPE_MAP = new HashMap<>();
-
- static {
- SINGLE_VALUE_TYPE_MAP.put(Boolean.class, PinotDataType.BOOLEAN);
- SINGLE_VALUE_TYPE_MAP.put(Byte.class, PinotDataType.BYTE);
- SINGLE_VALUE_TYPE_MAP.put(Character.class, PinotDataType.CHARACTER);
- SINGLE_VALUE_TYPE_MAP.put(Short.class, PinotDataType.SHORT);
- SINGLE_VALUE_TYPE_MAP.put(Integer.class, PinotDataType.INTEGER);
- SINGLE_VALUE_TYPE_MAP.put(Long.class, PinotDataType.LONG);
- SINGLE_VALUE_TYPE_MAP.put(Float.class, PinotDataType.FLOAT);
- SINGLE_VALUE_TYPE_MAP.put(Double.class, PinotDataType.DOUBLE);
- SINGLE_VALUE_TYPE_MAP.put(String.class, PinotDataType.STRING);
- SINGLE_VALUE_TYPE_MAP.put(byte[].class, PinotDataType.BYTES);
-
- MULTI_VALUE_TYPE_MAP.put(Byte.class, PinotDataType.BYTE_ARRAY);
- MULTI_VALUE_TYPE_MAP.put(Character.class, PinotDataType.CHARACTER_ARRAY);
- MULTI_VALUE_TYPE_MAP.put(Short.class, PinotDataType.SHORT_ARRAY);
- MULTI_VALUE_TYPE_MAP.put(Integer.class, PinotDataType.INTEGER_ARRAY);
- MULTI_VALUE_TYPE_MAP.put(Long.class, PinotDataType.LONG_ARRAY);
- MULTI_VALUE_TYPE_MAP.put(Float.class, PinotDataType.FLOAT_ARRAY);
- MULTI_VALUE_TYPE_MAP.put(Double.class, PinotDataType.DOUBLE_ARRAY);
- MULTI_VALUE_TYPE_MAP.put(String.class, PinotDataType.STRING_ARRAY);
- }
-
- private final Map<String, PinotDataType> _dataTypes = new HashMap<>();
-
- public DataTypeTransformer(Schema schema) {
- for (FieldSpec fieldSpec : schema.getAllFieldSpecs()) {
- if (!fieldSpec.isVirtualColumn()) {
- _dataTypes.put(fieldSpec.getName(), PinotDataType.getPinotDataTypeForIngestion(fieldSpec));
- }
- }
- }
-
- @Override
- public GenericRow transform(GenericRow record) {
- for (Map.Entry<String, PinotDataType> entry : _dataTypes.entrySet()) {
- String column = entry.getKey();
- try {
- Object value = record.getValue(column);
- if (value == null) {
- continue;
- }
- PinotDataType dest = entry.getValue();
- value = standardize(column, value, dest.isSingleValue());
- // NOTE: The standardized value could be null for empty Collection/Map/Object[].
- if (value == null) {
- record.putValue(column, null);
- continue;
- }
-
- // Convert data type if necessary
- PinotDataType source;
- if (value instanceof Object[]) {
- // Multi-value column
- Object[] values = (Object[]) value;
- source = MULTI_VALUE_TYPE_MAP.get(values[0].getClass());
- if (source == null) {
- source = PinotDataType.OBJECT_ARRAY;
- }
- } else {
- // Single-value column
- source = SINGLE_VALUE_TYPE_MAP.get(value.getClass());
- if (source == null) {
- source = PinotDataType.OBJECT;
- }
- }
- if (source != dest) {
- value = dest.convert(value, source);
- }
-
- record.putValue(column, value);
- } catch (Exception e) {
- throw new RuntimeException("Caught exception while transforming data type for column: " + column, e);
- }
- }
- return record;
- }
-
- /**
- * Standardize the value into supported types.
- * <ul>
- * <li>Empty Collection/Map/Object[] will be standardized to null</li>
- * <li>Single-entry Collection/Map/Object[] will be standardized to single value (map key is ignored)</li>
- * <li>Multi-entries Collection/Map/Object[] will be standardized to Object[] (map key is ignored)</li>
- * </ul>
- */
- @VisibleForTesting
- @Nullable
- static Object standardize(String column, @Nullable Object value, boolean isSingleValue) {
- if (value == null) {
- return null;
- }
- if (value instanceof Collection) {
- return standardizeCollection(column, (Collection) value, isSingleValue);
- }
- if (value instanceof Map) {
- return standardizeCollection(column, ((Map) value).values(), isSingleValue);
- }
- if (value instanceof Object[]) {
- Object[] values = (Object[]) value;
- int numValues = values.length;
- if (numValues == 0) {
- return null;
- }
- if (numValues == 1) {
- return standardize(column, values[0], isSingleValue);
- }
- List<Object> standardizedValues = new ArrayList<>(numValues);
- for (Object singleValue : values) {
- Object standardizedValue = standardize(column, singleValue, true);
- if (standardizedValue != null) {
- standardizedValues.add(standardizedValue);
- }
- }
- int numStandardizedValues = standardizedValues.size();
- if (numStandardizedValues == 0) {
- return null;
- }
- if (numStandardizedValues == 1) {
- return standardizedValues.get(0);
- }
- Preconditions.checkState(!isSingleValue, "Cannot read single-value from Object[]: %s for column: %s",
- Arrays.toString(values), column);
- return standardizedValues.toArray();
- }
- return value;
- }
-
- private static Object standardizeCollection(String column, Collection collection, boolean isSingleValue) {
- int numValues = collection.size();
- if (numValues == 0) {
- return null;
- }
- if (numValues == 1) {
- return standardize(column, collection.iterator().next(), isSingleValue);
- }
- List<Object> standardizedValues = new ArrayList<>(numValues);
- for (Object singleValue : collection) {
- Object standardizedValue = standardize(column, singleValue, true);
- if (standardizedValue != null) {
- standardizedValues.add(standardizedValue);
- }
- }
- int numStandardizedValues = standardizedValues.size();
- if (numStandardizedValues == 0) {
- return null;
- }
- if (numStandardizedValues == 1) {
- return standardizedValues.get(0);
- }
- Preconditions
- .checkState(!isSingleValue, "Cannot read single-value from Collection: %s for column: %s", collection, column);
- return standardizedValues.toArray();
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/recordtransformer/ExpressionTransformer.java b/pinot-core/src/main/java/org/apache/pinot/core/data/recordtransformer/ExpressionTransformer.java
deleted file mode 100644
index 52ded3b..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/recordtransformer/ExpressionTransformer.java
+++ /dev/null
@@ -1,101 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.data.recordtransformer;
-
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.LinkedHashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import org.apache.pinot.core.data.function.FunctionEvaluator;
-import org.apache.pinot.core.data.function.FunctionEvaluatorFactory;
-import org.apache.pinot.spi.config.table.TableConfig;
-import org.apache.pinot.spi.config.table.ingestion.TransformConfig;
-import org.apache.pinot.spi.data.FieldSpec;
-import org.apache.pinot.spi.data.Schema;
-import org.apache.pinot.spi.data.readers.GenericRow;
-
-
-/**
- * The {@code ExpressionTransformer} class will evaluate the function expressions.
- * <p>NOTE: should put this before the {@link DataTypeTransformer}. After this, transformed column can be treated as
- * regular column for other record transformers.
- */
-public class ExpressionTransformer implements RecordTransformer {
-
- private final LinkedHashMap<String, FunctionEvaluator> _expressionEvaluators = new LinkedHashMap<>();
-
- public ExpressionTransformer(TableConfig tableConfig, Schema schema) {
- Map<String, FunctionEvaluator> expressionEvaluators = new HashMap<>();
- if (tableConfig.getIngestionConfig() != null && tableConfig.getIngestionConfig().getTransformConfigs() != null) {
... 157953 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org