You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by ja...@apache.org on 2019/07/30 18:51:44 UTC
[incubator-pinot] branch rebalance_llc updated (82cbaca -> 004e4cf)
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a change to branch rebalance_llc
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
discard 82cbaca Take OFFLINE segment into account for real-time rebalancer
add 9c49b40 Only set Helix cluster properties when creating the cluster (#4339)
add 43c70c9 Remove jboss and chibi repository from pom file (#4340)
add e604eaf [TE] frontend - harleyjj/preview - remove day from graph x-axis and set default time range to 48 hours before (#4328)
add 2377037 [TE] frontend - harleyjj/alert-overview - show bounds on Alert Overview (#4314)
add 84b8143 Add support for an aggregation function returning serialized hyperloglogs. (#4293)
add 5fc684d [TE] frontend - harleyjj/timeseries-chart - style legend for current, predicted, and baseline (#4342)
add 0c339d9 [TE] detection health status (#4322)
add 5b34223 [TE] Entity Yaml (Composite Alert) Translator (#4333)
add 6005bd2 [TE] Reset null password to empty (#4343)
add 27c2736 [TE] cleaned up error messages when running in demo mode (#4344)
add cd823b7 Refresh schema when reloading segments (#4335)
add d64aa98 [TE] Authorize service account to prevent config modification (#4341)
add 8ed5200 [TE] Resolve runtime jackson conflicts - Upgrade and pin jackson version (#4347)
add 4d3d02a Within a partition, only allow querying the first CONSUMING segment for the real-time table routing (#4338)
add 17847db Refresh schema when reloading all segments (#4349)
add 3eef6ea Update quick_start.rst (#4319)
add 9497a75 [TE] Fix doc - add import sql url, disable pinot build (#4357)
add 81f433f Raise the threshold of using MMAP for inverted index creation from 100M to 2G (#4358)
add 859c68c [TE] Tuning of Composite alert - Store metric refs in component spec (#4359)
add b75a4c0 Adding support for udfs in Selection (#4346)
add ab6dd9f [TE] Trigger expression based grouping of anomalies - AND, OR and combinations (#4354)
add 4ce688e [TE] update detection health (#4364)
add 866e29a Adding util methods for controller that will be used in offline push (#4360)
add 39811df [TE] frontend - harleyjj/alert-overview - dont use anomalies from bounds endpoint in Alert Overview (#4366)
add 49902ae [TE] set detection window based on detection granularity (#4352)
add 583548c [TE] Bug Fix - Inject the correct metric ref in component spec (#4369)
add db7cd63 [TE] Update name for entity anomalies (#4370)
add 89a3449 Add segment pre-processing Hadoop job (#4253)
add cc83174 Disable Hadoop Pre-process (#4375)
add 3be84f2 Include timestamp in logs (#4379)
add 5e2a810 upgrade fastutil to 8.2.3 (#4325)
add 33756ef [TE] Support Anomaly Hierarchy for Composite alerts (#4371)
add 29bad86 [TE] Inject entity name into grouper to enrich the grouped anomalies (#4378)
add dc63643 Adding support for udf/expression in filter predicates (#4365)
add e28eeb3 [TE] Log the registry components (#4384)
add 33f583b Offheap String Dictionary Benchmark (#4381)
add 732a7b9 Add more key/value pairs into LOG2M_TO_SIZE_IN_BYTES in HllSizeUtils (#4398)
add 49d8fa7 #4317 Feature/variable length bytes offline dictionary for indexing bytes and string dicts. (#4321)
add 02f6181 CompletionConfig for realtime tables (#4367)
add ae1ecef [TE] add a thread pool to run preview tasks (#4405)
add de62d8e [TE] Make email template and subject pluggable under email alert scheme (#4409)
add 7df1a43 [TE] Add Merger after metric grouper; other minor clean up (#4410)
add e71ac1b [TE] Do not authorize when auth is disabled (#4414)
add 0b8af63 Fix codecov link from old url to new (#4412)
add 07e2c72 Move tests for API resources to a different package (#4415)
add 52f69d1 [TE] Added support for Vertica as a data source (#4404)
add a7c419a Adding Calcite SQL Parser and make another entry point to query Pinot (#4387)
add e041ec5 [TE] frontend - harleyjj/detection-health - UI for model performance (#4413)
add 6bec451 [TE] Inject the port from config into Commons mail SSL Smtp port (#4420)
add dac9ae2 prompt on fail schema (#4395)
add b6fe4e6 [TE] detection health ui adjustments (#4421)
add bca6756 make kafka version number controlled by config number (#4396)
add 0993974 [TE] Fix exception handling - Propagate and display the error message/exception on frontind (#4419)
add aadcd36 In ClusterChangeMediator, stop enqueue/process changes if already stopped (#4422)
add 52f9b08 Add pinot community inviter (#4424)
add 2f691fc [TE] detection health - coverage fix (#4428)
add 079b86d Config for overriding initial rows threshold in segment size auto tuning (#4376)
add 50aaaf9 [TE] added ThirdEye configuration documents (#4429)
add f671072 Add URIUtils class to handle URI/URL encoding/decoding (#4426)
add 626c43a Misc fix for controller tests (#4431)
add 12e77bd #4401 Reuse a ThreadLocal byte[] when reading String elements from the variable length value reader. (#4432)
add e720448 Remove the massive console print and log from CalciteSqlCompilerTest (#4434)
add 67afc71 Cleanup un-necessary setups in HelixSetupUtils (#4430)
add b1d28fd Fix potential resource leak in the way we close a collection of closeables (#4418)
add d2b2981 add download excel (#4403)
add 7b1715c 3891: Check for validity of segment start/end time during segment generation (#4368)
add 625482a [TE] Renamed some classes to making it accurate and consistent (#4439)
add 31d916f [TE] Adding Entity GroupKey Anomalies Email Reporter (#4433)
add db5923c Clean up the ServerInstance class and remove the un-necessary ip address stored (#4438)
add 5492217 Fix the potential resource leak from NettyServer (#4440)
add e897271 [TE] add ml auto config into YAML template (#4423)
add 16f028e Fix ServerInstanceTest (#4442)
add 2e07b0b [TE] Improvements to the Entity GroupKey Template (#4444)
add cd579b1 Add comments and fix typo for the SegmentCompletionTest. (#4447)
add 85b07f3 [TE] frontend - harleyjj/preview - provide custom time range before previewing (#4437)
add 365ee85 [TE] Detection and alert config formatters (#4443)
add 9232bb5 Moving handling of NULL values from RecordReaders to NullValueTransfo… (#4399)
add eaecd64 Refactor pinot-connectors to break the dependencies from kafka 0.9 (#4445)
add 024da51 Enable SegmentCompleteionIntegrationTest (#4451)
add 25e7be1 [TE] Bug Fix - Parsing of multiple alert schemes (#4456)
add d18fab9 [TE] anomaly filters potential inconsistency issue fix (#4448)
add 2d0b247 [TE] Update Swagger Resource Package (#4458)
add 6dc820a Issue #4450: Support parsing of `Order By` clause in PQL compiler. (#4453)
add 0fded63 [TE] Inject Sub Entity Name into the anomaly properties (#4452)
add 523f697 Add resource config for lead controller resource (#4449)
add a1fff55 Ensuring partition config is valid during Hadoop preprocess (#4463)
add 14ef543 Remove the usage of guava deprecated methods (#4457)
add 1642f00 [TE] logs and precondition checks for pipeline re-tuning (#4465)
add eca1d2e [TE] frontend - set analysis range using alertDetailsDefaultWindowSize from endpoint (#4460)
add 1d45f87 Enable periodic rebalance as a temporary work-around for the Helix issue (#4459)
add cac4f24 Allow customized metrics prefix in pinot controller/broker/server (#4392)
add c01b852 Add table name parameter for segment push (#4454)
add c81adda [TE] Include alert description in entity email alert (#4471)
add 29bc069 [TE] Include admin recipients in every email alert (cc address) (#4470)
add 9d64d8a Refactor HelixBrokerStarterTest (#4473)
add 8717e35 Adding Support for Kafka 2.0 Consumer (#4397)
add 551b5c3 Time-aware resizing (#4462)
add 9316610 Clean up test utils, disconnect fake instances (#4476)
add c768df8 When initializing mutable dictionary, preserve 10% buffer for cardinality to reduce the chance of re-sizing the dictionary (#4363)
add ad6ff6b Delete extra refreshed segments after segment push (#4464)
add 08be56e minor fix double close (#4478)
add 004e4cf Take OFFLINE segment into account for real-time rebalancer
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (82cbaca)
\
N -- N -- N refs/heads/rebalance_llc (004e4cf)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.travis.yml | 6 +
.travis_install.sh | 7 +-
.travis_test.sh | 15 +-
README.md | 4 +-
docs/pluggable_streams.rst | 48 +
docs/pql_examples.rst | 2 +
.../java/org/apache/pinot/client/Connection.java | 44 +-
.../client/JsonAsyncHttpPinotClientTransport.java | 30 +-
.../apache/pinot/client/PinotClientTransport.java | 6 +
.../{PinotClientTransport.java => Request.java} | 34 +-
.../apache/pinot/client/PreparedStatementTest.java | 40 +-
.../apache/pinot/client/ResultSetGroupTest.java | 38 +-
pinot-api/src/test/resources/log4j2.xml | 2 +-
.../pinot/broker/broker/BrokerServerBuilder.java | 3 +-
.../broker/broker/helix/ClusterChangeMediator.java | 12 +-
.../broker/helix/LiveInstanceChangeHandler.java | 6 +-
.../requesthandler/BaseBrokerRequestHandler.java | 28 +-
...enNestedPredicatesFilterQueryTreeOptimizer.java | 3 +-
.../requesthandler/PinotQueryParserFactory.java | 49 +
.../broker/requesthandler/PinotQueryRequest.java | 33 +-
.../pinot/broker/routing/CfgBasedRouting.java | 64 -
.../LowLevelConsumerRoutingTableBuilder.java | 6 +-
.../builder/LowLevelRoutingTableBuilderUtil.java | 83 +-
.../EmptyBrokerOnlineOfflineStateModelFactory.java | 70 -
...EmptySegmentOnlineOfflineStateModelFactory.java | 71 -
.../broker/broker/HelixBrokerStarterTest.java | 357 ++---
.../request/PqlAndCalciteSqlCompatibilityTest.java | 123 ++
.../LowLevelConsumerRoutingTableBuilderTest.java | 74 +-
pinot-broker/src/test/resources/log4j2.xml | 2 +-
pinot-common/pom.xml | 8 +-
.../pinot/common/config/CompletionConfig.java | 63 +
.../apache/pinot/common/config/IndexingConfig.java | 20 +-
.../SegmentsValidationAndRetentionConfig.java | 30 +-
.../apache/pinot/common/config/TableConfig.java | 8 +-
.../common}/function/AggregationFunctionType.java | 7 +-
.../function/FunctionDefinitionRegistry.java} | 22 +-
.../apache/pinot/common/metrics/BrokerMetrics.java | 12 +-
.../pinot/common/metrics/ControllerMetrics.java | 9 +-
.../apache/pinot/common/metrics/ServerMetrics.java | 24 +-
.../pinot/common/request/AggregationInfo.java | 384 +++--
.../apache/pinot/common/request/BrokerRequest.java | 1533 ++++++++++----------
.../apache/pinot/common/request/DataSource.java | 216 ++-
.../apache/pinot/common/request/Expression.java | 421 +++---
.../pinot/common/request/ExpressionType.java | 11 +-
.../pinot/common/request/FilterOperator.java | 18 +-
.../apache/pinot/common/request/FilterQuery.java | 561 ++++---
.../pinot/common/request/FilterQueryMap.java | 260 ++--
.../org/apache/pinot/common/request/Function.java | 339 ++---
.../org/apache/pinot/common/request/GroupBy.java | 420 +++---
.../pinot/common/request/HavingFilterQuery.java | 562 ++++---
.../pinot/common/request/HavingFilterQueryMap.java | 259 ++--
.../apache/pinot/common/request/Identifier.java | 220 ++-
.../pinot/common/request/InstanceRequest.java | 542 ++++---
.../org/apache/pinot/common/request/Literal.java | 364 ++---
.../apache/pinot/common/request/PinotQuery.java | 1048 ++++++-------
.../apache/pinot/common/request/QuerySource.java | 218 ++-
.../org/apache/pinot/common/request/QueryType.java | 485 +++----
.../org/apache/pinot/common/request/Selection.java | 486 +++----
.../apache/pinot/common/request/SelectionSort.java | 285 ++--
.../pinot/common/response/ServerInstance.java | 180 +--
.../apache/pinot/common/utils/CommonConstants.java | 27 +-
.../common/utils/FileUploadDownloadClient.java | 84 +-
.../org/apache/pinot/common/utils/FileUtils.java | 59 +-
.../org/apache/pinot/common/utils/JsonUtils.java | 6 +-
.../org/apache/pinot/common/utils/StringUtil.java | 4 +-
.../org/apache/pinot/common/utils/URIUtils.java | 88 ++
.../common/utils/request/FilterQueryTree.java | 11 +
.../pinot/common/utils/request/RequestUtils.java | 62 +-
.../apache/pinot/common/utils/time/TimeUtils.java | 15 +
.../org/apache/pinot/filesystem/LocalPinotFS.java | 91 +-
.../pinot/{pql => }/parsers/AbstractCompiler.java | 7 +-
.../utils/BrokerRequestComparisonUtils.java | 235 +++
.../parsers/PinotQuery2BrokerRequestConverter.java | 22 +-
.../org/apache/pinot/pql/parsers/Pql2Compiler.java | 63 +-
.../parsers/pql2/ast/BetweenPredicateAstNode.java | 2 +-
.../pql2/ast/ComparisonPredicateAstNode.java | 22 +-
.../pql/parsers/pql2/ast/InPredicateAstNode.java | 2 +-
.../pinot/pql/parsers/pql2/ast/OrderByAstNode.java | 16 +-
.../pql/parsers/pql2/ast/OutputColumnAstNode.java | 22 +-
.../pql/parsers/pql2/ast/PredicateListAstNode.java | 2 +-
.../pql2/ast/RegexpLikePredicateAstNode.java | 2 +-
.../parsers/CalciteSqlCompiler.java} | 18 +-
.../apache/pinot/sql/parsers/CalciteSqlParser.java | 273 ++++
.../parsers/SqlCompilationException.java} | 19 +-
.../apache/pinot/startree/hll/HllSizeUtils.java | 14 +-
.../pinot/common/config/IndexingConfigTest.java | 3 +
.../pinot/common/config/TableConfigTest.java | 28 +
.../apache/pinot/common/utils/URIUtilsTest.java | 68 +
.../apache/pinot/pql/parsers/Pql2CompilerTest.java | 56 +-
.../request/BrokerRequestSerializationTest.java | 10 +-
.../pinot/sql/parsers/CalciteSqlCompilerTest.java | 425 ++++++
pinot-common/src/test/resources/log4j2.xml | 2 +-
pinot-common/src/test/resources/pql_queries.list | 5 +
.../{pql_queries.list => sql_queries.list} | 247 ++--
pinot-common/src/thrift/request.thrift | 1 +
pinot-connectors/pinot-connector-kafka-0.9/pom.xml | 16 +-
.../realtime/impl/kafka/KafkaStarterUtils.java | 171 ---
.../impl/kafka/server/KafkaDataProducer.java | 57 +
.../kafka/server/KafkaDataServerStartable.java | 109 ++
...inot.core.realtime.stream.StreamConsumerFactory | 19 +
.../kafka/KafkaPartitionLevelConsumerTest.java | 2 +-
.../pinot-connector-kafka-2.0/README.md | 57 +
.../pom.xml | 50 +-
.../realtime/impl/kafka2/KafkaConsumerFactory.java | 51 +
.../realtime/impl/kafka2/KafkaMessageBatch.java | 63 +
.../KafkaPartitionLevelConnectionHandler.java | 73 +
.../impl/kafka2/KafkaPartitionLevelConsumer.java | 66 +
.../kafka2/KafkaPartitionLevelStreamConfig.java | 147 ++
.../impl/kafka2/KafkaStreamLevelConsumer.java | 179 +++
.../kafka2/KafkaStreamLevelConsumerManager.java | 194 +++
.../impl/kafka2/KafkaStreamLevelStreamConfig.java | 136 ++
.../impl/kafka2/KafkaStreamMetadataProvider.java | 67 +
.../impl/kafka2/server/KafkaDataProducer.java | 77 +
.../kafka2/server/KafkaDataServerStartable.java | 98 ++
...inot.core.realtime.stream.StreamConsumerFactory | 19 +
.../kafka2/KafkaPartitionLevelConsumerTest.java | 293 ++++
.../KafkaPartitionLevelStreamConfigTest.java | 162 +++
.../impl/kafka2/utils/EmbeddedZooKeeper.java | 60 +
.../impl/kafka2/utils/MiniKafkaCluster.java | 175 +++
.../pinot-connector-kafka-base/README.md | 18 +-
.../{ => pinot-connector-kafka-base}/pom.xml | 43 +-
.../impl/kafka/KafkaAvroMessageDecoder.java | 0
.../impl/kafka/KafkaJSONMessageDecoder.java | 0
.../realtime/impl/kafka/KafkaStarterUtils.java | 118 ++
.../impl/kafka/KafkaStreamConfigProperties.java | 19 +-
.../core/realtime/impl/kafka/MessageAndOffset.java | 39 +-
pinot-connectors/pom.xml | 15 +
.../apache/pinot/controller/ControllerConf.java | 53 +-
.../apache/pinot/controller/ControllerStarter.java | 22 +-
.../api/resources/FileUploadPathProvider.java | 5 +-
.../resources/LLCSegmentCompletionHandlers.java | 25 +-
.../api/resources/PinotControllerHealthCheck.java | 4 +-
.../api/resources/PinotSegmentRestletResource.java | 20 +-
.../PinotSegmentUploadRestletResource.java | 89 +-
.../resources/PinotTableConfigRestletResource.java | 2 +
.../api/resources/PinotTableRestletResource.java | 10 +-
.../pinot/controller/api/resources/TableViews.java | 4 +-
.../controller/api/upload/SegmentValidator.java | 2 +-
.../helix/ControllerRequestBuilderUtil.java | 152 --
.../helix/ControllerRequestURLBuilder.java | 21 +-
.../EmptyBrokerOnlineOfflineStateModelFactory.java | 70 -
...EmptySegmentOnlineOfflineStateModelFactory.java | 81 --
.../helix/core/PinotHelixResourceManager.java | 34 +-
.../helix/core/PinotTableIdealStateBuilder.java | 24 -
.../helix/core/SegmentDeletionManager.java | 34 +-
...ealtimeBalanceNumSegmentAssignmentStrategy.java | 3 +-
...ltimeReplicaGroupSegmentAssignmentStrategy.java | 3 +-
.../realtime/PinotLLCRealtimeSegmentManager.java | 34 +-
.../segment/FlushThresholdUpdateManager.java | 11 +-
.../SegmentSizeBasedFlushThresholdUpdater.java | 14 +-
.../helix/core/util/HelixSetupUtils.java | 293 ++--
.../controller/validation/StorageQuotaChecker.java | 32 +-
.../src/main/resources/static/css/pinot.css | 16 +-
.../src/main/resources/static/js/init.js | 12 +
.../src/main/resources/static/query/index.html | 8 +-
.../api/{resources => }/AccessControlTest.java | 10 +-
.../FileUploadPathProviderTest.java | 3 +-
.../api/{resources => }/PinotFileUploadTest.java | 41 +-
.../PinotInstanceRestletResourceTest.java | 6 +-
.../PinotSchemaRestletResourceTest.java | 2 +-
.../PinotSegmentRestletResourceTest.java | 31 +-
.../PinotTableRestletResourceTest.java | 13 +-
.../PinotTenantRestletResourceTest.java | 14 +-
.../SegmentCompletionProtocolDeserTest.java | 2 +-
.../SegmentCompletionUtilsTest.java | 2 +-
.../{resources => }/ServerTableSizeReaderTest.java | 3 +-
.../api/{resources => }/TableSizeReaderTest.java | 2 +-
.../api/{resources => }/TableViewsTest.java | 15 +-
.../helix/ControllerInstanceToggleTest.java | 33 +-
.../helix/ControllerPeriodicTaskStarterTest.java | 38 +-
.../controller/helix/ControllerSentinelTestV2.java | 45 +-
.../controller/helix/ControllerTenantTest.java | 111 +-
.../pinot/controller/helix/ControllerTest.java | 336 ++++-
.../pinot/controller/helix/HelixHelperTest.java | 6 +-
.../controller/helix/PinotControllerModeTest.java | 275 ++--
.../controller/helix/PinotResourceManagerTest.java | 43 +-
.../helix/core/PinotHelixResourceManagerTest.java | 139 +-
.../helix/core/realtime/SegmentCompletionTest.java | 41 +-
.../segment/FlushThresholdUpdaterTest.java | 28 +-
.../ReplicaGroupRebalanceStrategyTest.java | 33 +-
.../sharding/SegmentAssignmentStrategyTest.java | 133 +-
.../validation/ValidationManagerTest.java | 49 +-
pinot-controller/src/test/resources/log4j2.xml | 2 +-
pinot-core/pom.xml | 4 -
.../core/data/aggregator/AvgValueAggregator.java | 2 +-
.../core/data/aggregator/CountValueAggregator.java | 2 +-
.../DistinctCountHLLValueAggregator.java | 2 +-
.../core/data/aggregator/MaxValueAggregator.java | 2 +-
.../aggregator/MinMaxRangeValueAggregator.java | 2 +-
.../core/data/aggregator/MinValueAggregator.java | 2 +-
.../aggregator/PercentileEstValueAggregator.java | 2 +-
.../PercentileTDigestValueAggregator.java | 2 +-
.../core/data/aggregator/SumValueAggregator.java | 2 +-
.../core/data/aggregator/ValueAggregator.java | 2 +-
.../data/aggregator/ValueAggregatorFactory.java | 4 +-
.../realtime/HLRealtimeSegmentDataManager.java | 9 +-
.../realtime/LLRealtimeSegmentDataManager.java | 95 +-
.../data/partition/MurmurPartitionFunction.java | 2 +-
.../pinot/core/data/readers/RecordReaderUtils.java | 14 +-
.../core/data/readers/sort/PinotSegmentSorter.java | 75 +-
...dTransformer.java => CompositeTransformer.java} | 20 +-
.../recordtransformer/NullValueTransformer.java | 53 +
...ansformer.java => SanitizationTransformer.java} | 6 +-
.../generator/SegmentGeneratorConfig.java | 23 +
.../indexsegment/mutable/MutableSegmentImpl.java | 18 +-
.../impl/FixedByteSingleValueMultiColReader.java | 16 +-
.../core/io/util/FixedBitIntReaderWriter.java | 16 +-
.../core/io/util/FixedByteValueReaderWriter.java | 16 +-
.../org/apache/pinot/core/io/util/ValueReader.java | 3 +-
.../io/util/VarLengthBytesValueReaderWriter.java | 278 ++++
.../impl/FixedByteSingleValueMultiColWriter.java | 23 +-
.../pinot/core/minion/BackfillDateTimeColumn.java | 4 +-
.../apache/pinot/core/minion/SegmentConverter.java | 18 +-
.../pinot/core/operator/blocks/TransformBlock.java | 4 +
.../RangelessBitmapDocIdIterator.java | 2 +-
.../dociditerators/SVScanDocIdIterator.java | 8 +-
.../operator/filter/ExpressionFilterOperator.java | 381 +++++
.../core/operator/filter/FilterOperatorUtils.java | 3 +
.../predicate/PredicateEvaluatorProvider.java | 7 +-
.../query/DictionaryBasedAggregationOperator.java | 2 +-
.../query/MetadataBasedAggregationOperator.java | 2 +-
.../core/operator/query/SelectionOnlyOperator.java | 109 --
.../core/operator/query/SelectionOperator.java | 282 ++++
.../operator/query/SelectionOrderByOperator.java | 116 --
.../transform/TransformBlockDataFetcher.java | 340 +++++
.../core/operator/transform/TransformOperator.java | 13 +-
.../org/apache/pinot/core/plan/FilterPlanNode.java | 24 +-
.../plan/MetadataBasedAggregationPlanNode.java | 2 +-
.../apache/pinot/core/plan/SelectionPlanNode.java | 38 +-
.../apache/pinot/core/plan/TransformPlanNode.java | 50 +-
.../core/plan/maker/InstancePlanMakerImplV2.java | 2 +-
.../aggregation/DefaultAggregationExecutor.java | 2 +-
.../aggregation/function/AggregationFunction.java | 1 +
.../function/AggregationFunctionFactory.java | 5 +
.../function/AggregationFunctionUtils.java | 1 +
.../function/AvgAggregationFunction.java | 1 +
.../function/AvgMVAggregationFunction.java | 1 +
.../function/CountAggregationFunction.java | 1 +
.../function/CountMVAggregationFunction.java | 1 +
.../function/DistinctCountAggregationFunction.java | 1 +
.../DistinctCountHLLAggregationFunction.java | 1 +
.../DistinctCountHLLMVAggregationFunction.java | 1 +
.../DistinctCountMVAggregationFunction.java | 1 +
...=> DistinctCountRawHLLAggregationFunction.java} | 63 +-
... DistinctCountRawHLLMVAggregationFunction.java} | 30 +-
.../function/FastHLLAggregationFunction.java | 1 +
.../function/MaxAggregationFunction.java | 1 +
.../function/MaxMVAggregationFunction.java | 1 +
.../function/MinAggregationFunction.java | 1 +
.../function/MinMVAggregationFunction.java | 1 +
.../function/MinMaxRangeAggregationFunction.java | 1 +
.../function/MinMaxRangeMVAggregationFunction.java | 1 +
.../function/PercentileAggregationFunction.java | 1 +
.../function/PercentileEstAggregationFunction.java | 1 +
.../PercentileEstMVAggregationFunction.java | 1 +
.../function/PercentileMVAggregationFunction.java | 1 +
.../PercentileTDigestAggregationFunction.java | 1 +
.../PercentileTDigestMVAggregationFunction.java | 1 +
.../function/SumAggregationFunction.java | 1 +
.../function/SumMVAggregationFunction.java | 1 +
.../function/customobject/SerializedHLL.java | 37 +-
.../groupby/DefaultGroupByExecutor.java | 2 +-
.../aggregation/groupby/utils/DoubleToIdMap.java | 2 +-
.../aggregation/groupby/utils/FloatToIdMap.java | 2 +-
.../aggregation/groupby/utils/IntToIdMap.java | 2 +-
.../aggregation/groupby/utils/LongToIdMap.java | 2 +-
.../query/pruner/ColumnValueSegmentPruner.java | 5 +
.../core/query/pruner/PartitionSegmentPruner.java | 5 +
.../core/query/request/ServerQueryRequest.java | 18 +-
.../query/selection/SelectionOperatorUtils.java | 4 +-
.../converter/RealtimeSegmentConverter.java | 21 +-
.../core/realtime/impl/RealtimeSegmentConfig.java | 20 +-
.../pinot/core/realtime/stream/StreamConfig.java | 55 +-
.../realtime/stream/StreamConfigProperties.java | 8 +-
.../core/realtime/stream/StreamDataProducer.java | 18 +-
.../core/realtime/stream/StreamDataProvider.java | 46 +
.../stream/StreamDataServerStartable.java} | 52 +-
.../segment/creator/ColumnIndexCreationInfo.java | 10 +-
.../RecordReaderSegmentCreationDataSource.java | 4 +-
.../creator/impl/SegmentColumnarIndexCreator.java | 147 +-
.../creator/impl/SegmentDictionaryCreator.java | 72 +-
.../impl/SegmentIndexCreationDriverImpl.java | 10 +-
.../fwd/SingleValueSortedForwardIndexCreator.java | 10 +-
.../inv/OffHeapBitmapInvertedIndexCreator.java | 30 +-
.../segment/index/loader/IndexLoadingConfig.java | 11 +
.../defaultcolumn/BaseDefaultColumnHandler.java | 8 +-
.../index/readers/ImmutableDictionaryReader.java | 18 +-
.../name/NormalizedDateSegmentNameGenerator.java | 18 +-
.../segment/store/SegmentLocalFSDirectory.java | 2 +-
.../pinot/core/startree/StarTreeDataTable.java | 35 +-
.../executor/StarTreeAggregationExecutor.java | 2 +-
.../startree/executor/StarTreeGroupByExecutor.java | 2 +-
.../startree/operator/StarTreeFilterOperator.java | 5 +-
.../startree/plan/StarTreeTransformPlanNode.java | 3 +-
.../startree/v2/AggregationFunctionColumnPair.java | 2 +-
.../startree/v2/builder/BaseSingleTreeBuilder.java | 2 +-
.../v2/builder/OffHeapSingleTreeBuilder.java | 53 +-
.../startree/v2/store/StarTreeLoaderUtils.java | 2 +-
.../function/FunctionExpressionEvaluatorTest.java | 1 -
.../pinot/core/data/readers/PinotSegmentUtil.java | 74 +-
.../data/readers/RecordReaderSampleDataTest.java | 8 +-
.../pinot/core/data/readers/RecordReaderTest.java | 8 +-
.../recordtransformer/RecordTransformerTest.java | 6 +-
.../MutableSegmentImplAggregateMetricsTest.java | 1 +
.../mutable/MutableSegmentImplTest.java | 3 +-
.../mutable/MutableSegmentImplTestUtils.java | 4 +-
.../util/VarLengthBytesValueReaderWriterTest.java | 170 +++
.../pinot/core/minion/SegmentConverterTest.java | 16 +-
...adataAndDictionaryAggregationPlanMakerTest.java | 12 +
.../function/AggregationFunctionFactoryTest.java | 11 +
.../function/AggregationFunctionTypeTest.java | 5 +
.../core/segment/index/ColumnMetadataTest.java | 6 +
.../segment/index/SegmentMetadataImplTest.java | 6 +
.../SegmentV1V2ToV3FormatConverterTest.java | 6 +
.../SegmentGenerationWithTimeColumnTest.java | 36 +-
.../index/loader/SegmentPreProcessorTest.java | 6 +
.../startree/StarTreeIndexTestSegmentHelper.java | 6 +
.../pinot/core/startree/TestStarTreeMetadata.java | 6 +
.../pinot/core/startree/hll/HllFieldSizeTest.java | 2 +-
.../hll/SegmentWithHllIndexCreateHelper.java | 6 +
.../v2/AggregationFunctionColumnPairTest.java | 11 +-
.../pinot/core/startree/v2/BaseStarTreeV2Test.java | 2 +-
.../org/apache/pinot/core/util/CrcUtilsTest.java | 6 +
.../pinot/queries/BaseMultiValueQueriesTest.java | 6 +
.../pinot/queries/BaseSingleValueQueriesTest.java | 6 +
.../apache/pinot/queries/FastHllQueriesTest.java | 10 +-
...InnerSegmentSelectionMultiValueQueriesTest.java | 123 +-
...nnerSegmentSelectionSingleValueQueriesTest.java | 146 +-
...terSegmentAggregationMultiValueQueriesTest.java | 29 +-
...erSegmentAggregationSingleValueQueriesTest.java | 33 +-
.../org/apache/pinot/queries/QueriesTestUtils.java | 18 +-
.../DefaultAggregationExecutorTest.java | 2 +-
.../segments/v1/creator/DictionariesTest.java | 6 +
.../pinot/segments/v1/creator/IntArraysTest.java | 6 +
.../util/IntDoubleIndexedPriorityQueueTest.java | 2 +-
.../test/java/org/apache/pinot/util/TestUtils.java | 96 +-
pinot-core/src/test/resources/log4j2.xml | 2 +-
pinot-distribution/pinot-assembly.xml | 5 +
pinot-hadoop/pom.xml | 5 +
.../pinot/hadoop/PinotHadoopJobLauncher.java | 7 +-
.../pinot/hadoop/io/CombineAvroKeyInputFormat.java | 57 +
.../apache/pinot/hadoop/job/ControllerRestApi.java | 4 +
.../pinot/hadoop/job/DefaultControllerRestApi.java | 54 +-
.../pinot/hadoop/job/InternalConfigConstants.java | 39 +
.../pinot/hadoop/job/JobConfigConstants.java | 9 +
.../pinot/hadoop/job/SegmentCreationJob.java | 33 +-
.../pinot/hadoop/job/SegmentPreprocessingJob.java | 524 +++++++
.../apache/pinot/hadoop/job/SegmentTarPushJob.java | 61 +-
.../apache/pinot/hadoop/job/SegmentUriPushJob.java | 4 +-
.../{mapper => mappers}/SegmentCreationMapper.java | 4 +-
.../job/mappers/SegmentPreprocessingMapper.java | 124 ++
.../job/partitioners/GenericPartitioner.java | 65 +
.../job/partitioners/PartitionFunctionFactory.java | 98 ++
.../job/reducers/SegmentPreprocessingReducer.java | 84 ++
.../pinot/hadoop/utils/JobPreparationHelper.java | 58 +
.../hadoop/io/DeleteExtraPushedSegmentsTest.java | 102 ++
pinot-integration-tests/pom.xml | 4 +-
.../tests/BaseClusterIntegrationTest.java | 81 +-
.../tests/BaseClusterIntegrationTestSet.java | 111 +-
.../tests/ClusterIntegrationTestUtils.java | 84 +-
.../pinot/integration/tests/ClusterTest.java | 63 +-
.../ControllerPeriodicTasksIntegrationTests.java | 89 +-
.../DeleteAPIHybridClusterIntegrationTest.java | 49 +-
...lakyConsumerRealtimeClusterIntegrationTest.java | 14 +-
...mentBuildPushOfflineClusterIntegrationTest.java | 80 +-
.../tests/HybridClusterIntegrationTest.java | 11 +-
...ridClusterIntegrationTestCommandLineRunner.java | 18 +-
...onaryAggregationPlanClusterIntegrationTest.java | 2 +-
.../MultiNodesOfflineClusterIntegrationTest.java | 7 +
.../tests/OfflineClusterIntegrationTest.java | 172 ++-
.../tests/PinotURIUploadIntegrationTest.java | 15 +-
.../tests/RealtimeClusterIntegrationTest.java | 33 +-
....java => SegmentCompletionIntegrationTest.java} | 70 +-
.../tests/SimpleMinionClusterIntegrationTest.java | 6 +-
.../tests/StarTreeClusterIntegrationTest.java | 2 +-
.../tests/StarTreeV2ClusterIntegrationTest.java | 4 +-
.../tests/UploadRefreshDeleteIntegrationTest.java | 2 +-
.../src/test/resources/log4j2.xml | 2 +-
.../org/apache/pinot/minion/MinionStarter.java | 4 +-
.../BaseMultipleSegmentsConversionExecutor.java | 17 +-
.../BaseSingleSegmentConversionExecutor.java | 13 +-
.../apache/pinot/minion/metrics/MinionMetrics.java | 6 +-
pinot-minion/src/test/resources/log4j2.xml | 2 +-
pinot-perf/pom.xml | 8 +-
.../pinot/perf/BenchmarkDictionaryCreation.java | 12 +
.../apache/pinot/perf/BenchmarkQueryEngine.java | 1 -
.../perf/BenchmarkRealtimeConsumptionSpeed.java | 8 +-
.../pinot/perf/BenchmarkStringDictionary.java | 118 +-
.../org/apache/pinot/perf/RealtimeStressTest.java | 8 +-
.../pinot/perf/StringDictionaryPerfTest.java | 86 +-
.../org/apache/pinot/server/conf/ServerConf.java | 7 +
.../server/request/ScheduledRequestHandler.java | 18 +-
.../apache/pinot/server/starter/ServerBuilder.java | 4 +-
.../starter/helix/HelixInstanceDataManager.java | 12 +-
.../server/starter/helix/HelixServerStarter.java | 6 +-
.../api/{resources => }/BaseResourceTest.java | 2 +-
.../api/{resources => }/TableSizeResourceTest.java | 2 +-
.../api/{resources => }/TablesResourceTest.java | 2 +-
.../request/ScheduledRequestHandlerTest.java | 42 +-
pinot-tools/pom.xml | 34 +-
.../org/apache/pinot/tools/HybridQuickstart.java | 39 +-
.../org/apache/pinot/tools/RealtimeQuickStart.java | 29 +-
.../command/BackfillDateTimeColumnCommand.java | 2 +-
.../tools/admin/command/StartKafkaCommand.java | 11 +-
.../admin/command/StreamAvroIntoKafkaCommand.java | 20 +-
.../tools/admin/command/UploadSegmentCommand.java | 6 +-
.../pinot/tools/backfill/BackfillSegmentUtils.java | 7 +-
.../pinot/tools/perf/PerfBenchmarkDriver.java | 19 -
.../pinot/tools/perf/PerfBenchmarkDriverConf.java | 63 -
.../tools/query/comparison/ClusterStarter.java | 4 +-
.../realtime/provisioning/MemoryEstimator.java | 18 +-
.../pinot/tools/streams/AirlineDataStream.java | 17 +-
.../pinot/tools/streams/MeetupRsvpStream.java | 28 +-
.../airlineStats_realtime_table_config.json | 0
.../meetupRsvp_realtime_table_config.json | 2 +-
.../airlineStats_realtime_table_config.json | 38 +
.../meetupRsvp_realtime_table_config.json | 5 +-
.../meetupRsvp_realtime_table_config.json | 2 +-
.../apache/pinot/transport/netty/NettyServer.java | 59 +-
.../pinot/transport/common/ServerInstanceTest.java | 62 +-
.../pinot/transport/netty/NettyTestUtils.java | 8 +-
.../transport/perf/ScatterGatherPerfClient.java | 5 +-
.../transport/perf/ScatterGatherPerfServer.java | 8 +-
.../pool/AsyncPoolResourceManagerAdapterTest.java | 14 +-
.../pinot/transport/pool/KeyedPoolImplTest.java | 25 +-
.../transport/scattergather/ScatterGatherTest.java | 24 +-
pom.xml | 138 +-
thirdeye/README.md | 3 +
thirdeye/docs/configuration.rst | 210 +++
thirdeye/docs/import_sql_metric.rst | 7 +-
thirdeye/docs/index.rst | 2 +-
thirdeye/docs/production.rst | 27 -
thirdeye/docs/quick_start.rst | 18 +-
thirdeye/install.sh | 10 +-
thirdeye/pom.xml | 19 +-
.../app/pods/components/alert-details/component.js | 97 +-
.../app/pods/components/alert-details/template.hbs | 105 +-
.../pods/components/anomaly-summary/component.js | 2 +-
.../pods/components/detection-health/component.js | 197 +++
.../pods/components/detection-health/template.hbs | 84 ++
.../pods/components/rootcause-chart/component.js | 2 +-
.../app/pods/components/stats-cards/component.js | 42 +-
.../app/pods/components/stats-cards/template.hbs | 61 +-
.../pods/components/timeseries-chart/component.js | 12 +-
.../app/pods/components/yaml-editor/template.hbs | 2 +-
.../app/pods/manage/explore/route.js | 52 +-
.../app/pods/manage/explore/template.hbs | 3 +
.../app/pods/manage/yaml/route.js | 50 +-
.../app/pods/manage/yaml/template.hbs | 4 +-
.../app/pods/screenshot/controller.js | 2 +-
.../app/pods/self-serve/create-alert/template.hbs | 2 +-
.../app/styles/components/timeseries-chart.scss | 16 +-
.../app/styles/shared/_styles.scss | 118 ++
thirdeye/thirdeye-frontend/app/utils/anomaly.js | 22 +-
.../thirdeye-frontend/app/utils/api/anomaly.js | 18 +-
thirdeye/thirdeye-frontend/app/utils/constants.js | 1 +
.../app/utils/manage-alert-utils.js | 4 +-
thirdeye/thirdeye-frontend/app/utils/utils.js | 14 +-
.../pods/components/yaml-editor/component-test.js | 2 +-
thirdeye/thirdeye-pinot/config/dashboard.yml | 3 +-
.../config/data-sources/data-sources-config.yml | 2 +-
thirdeye/thirdeye-pinot/config/h2db.mv.db | Bin 0 -> 4947968 bytes
thirdeye/thirdeye-pinot/pom.xml | 15 +
.../alert/content/BaseEmailContentFormatter.java | 95 +-
.../content/EntityGroupKeyContentFormatter.java | 151 ++
...HierarchicalAnomaliesEmailContentFormatter.java | 5 +-
...a => MetricAnomaliesEmailContentFormatter.java} | 17 +-
...nboardingNotificationEmailContentFormatter.java | 1 +
.../anomaly/alert/util/AnomalyReportGenerator.java | 2 +-
.../anomaly/alert/v2/AlertTaskRunnerV2.java | 2 +-
.../anomaly/merge/AnomalyTimeBasedSummarizer.java | 2 +
.../anomaly/utils/ThirdeyeMetricsUtil.java | 3 +
.../dashboard/DetectionPreviewConfiguration.java | 50 +
.../dashboard/ThirdEyeDashboardApplication.java | 2 +-
.../dashboard/ThirdEyeDashboardConfiguration.java | 9 +
.../dashboard/resources/v2/AnomaliesResource.java | 24 +-
.../dashboard/resources/v2/pojo/SearchFilters.java | 2 -
.../bao/jdbc/MergedAnomalyResultManagerImpl.java | 104 +-
.../datalayer/pojo/DetectionAlertConfigBean.java | 9 +
.../datalayer/pojo/DetectionConfigBean.java | 9 +
.../datasource/mock/AutoOnboardMockDataSource.java | 24 +-
.../datasource/sql/SqlResponseCacheLoader.java | 49 +-
.../pinot/thirdeye/datasource/sql/SqlUtils.java | 3 +
.../thirdeye/detection/DefaultDataProvider.java | 2 +-
.../thirdeye/detection/DetectionPipeline.java | 37 +-
.../detection/DetectionPipelineLoader.java | 10 +-
.../detection/DetectionPipelineTaskRunner.java | 11 +-
.../thirdeye/detection/DetectionResource.java | 52 +-
.../pinot/thirdeye/detection/DetectionUtils.java | 48 +
.../pinot/thirdeye/detection/ModelRetuneFlow.java | 13 +-
.../detection/alert/DetectionAlertTaskFactory.java | 13 +-
.../alert/scheme/DetectionEmailAlerter.java | 89 +-
.../annotation/registry/DetectionRegistry.java | 5 +
.../AbsoluteChangeRuleAnomalyFilter.java | 3 +-
.../thirdeye/detection/components/MockGrouper.java | 0
.../PercentageChangeRuleAnomalyFilter.java | 10 +-
.../SitewideImpactRuleAnomalyFilter.java | 5 +-
.../components/ThresholdRuleAnomalyFilter.java | 14 +-
.../components/TriggerConditionGrouper.java | 203 +++
.../detection/health/AnomalyCoverageStatus.java | 73 +
.../thirdeye/detection/health/DetectionHealth.java | 263 ++++
.../detection/health/DetectionTaskStatus.java | 127 ++
.../thirdeye/detection/health/HealthStatus.java | 28 +
.../detection/health/RegressionStatus.java | 101 ++
.../thirdeye/detection/spec/MockGrouperSpec.java | 0
.../spec/TriggerConditionGrouperSpec.java} | 40 +-
.../validators/DetectionConfigValidator.java | 23 +-
.../detection/wrapper/AnomalyDetectorWrapper.java | 6 +
.../wrapper/ChildKeepingMergeWrapper.java | 16 +-
.../thirdeye/detection/wrapper/GrouperWrapper.java | 39 +-
.../detection/yaml/DetectionConfigTuner.java | 41 +-
.../thirdeye/detection/yaml/YamlResource.java | 154 +-
.../yaml/translator/ConfigTranslator.java | 13 +
.../yaml/translator/DetectionConfigTranslator.java | 232 ++-
.../translator/DetectionMetricAttributeHolder.java | 97 ++
.../yaml/translator/DetectionMetricProperties.java | 62 +
.../translator/SubscriptionConfigTranslator.java | 14 +-
.../pinot/thirdeye/formatter/DTOFormatter.java | 38 +
.../formatter/DetectionAlertConfigFormatter.java | 46 +
.../formatter/DetectionConfigFormatter.java | 189 +++
.../apache/pinot/thirdeye/util/ThirdEyeUtils.java | 5 +-
.../detector/entity-groupkey-anomaly-report.ftl | 99 ++
...ly-report.ftl => metric-anomalies-template.ftl} | 0
.../commons/TestEmailContentFormatterFactory.java | 6 +-
.../alert/content/ContentFormatterUtils.java | 48 +
...> TestEntityGroupKeyEmailContentFormatter.java} | 138 +-
...HierarchicalAnomaliesEmailContentFormatter.java | 22 +-
... TestMetricAnomaliesEmailContentFormatter.java} | 26 +-
...TestOnboardingNotificationContentFormatter.java | 20 +-
.../pinot/thirdeye/datalayer/DaoTestUtils.java | 13 +
.../bao/TestGroupedAnomalyResultsManager.java | 6 +-
.../bao/TestMergedAnomalyResultManager.java | 46 +-
.../pinot/thirdeye/detection/DataProviderTest.java | 2 +
.../thirdeye/detection/DetectionTestUtils.java | 9 +
.../thirdeye/detection/alert/SendAlertTest.java | 2 +-
.../AbsoluteChangeRuleAnomalyFilterTest.java | 25 +-
.../PercentageChangeRuleAnomalyFilterTest.java | 60 +-
.../SitewideImpactRuleAnomalyFilterTest.java | 20 +-
.../components/ThresholdRuleAnomalyFilterTest.java | 56 +-
.../components/TriggerConditionGrouperTest.java | 231 +++
.../detection/health/DetectionHealthTest.java | 130 ++
.../wrapper/AnomalyDetectorWrapperTest.java | 4 +-
.../thirdeye/detection/yaml/YamlResourceTest.java | 27 +-
.../translator/DetectionConfigTranslatorTest.java | 18 +
.../formatter/DetectionConfigFormatterTest.java | 82 ++
.../tools/RunAdhocDatabaseQueriesTool.java | 31 +-
.../validators/entity-pipeline-config-2.yaml | 2 +-
.../detection/yaml/alertconfig/alert-config-5.yaml | 3 +
.../compositePipelineTranslatorTestResult-1.json | 93 +-
.../compositePipelineTranslatorTestResult-2.json | 1 +
.../compositePipelineTranslatorTestResult-4.json | 32 +
.../compositePipelineTranslatorTestResult-5.json | 87 ++
.../yaml/translator/pipeline-config-4.yaml | 26 +
.../translator/pipeline-config-5.yaml} | 17 +-
...st-entity-groupby-email-content-formatter.html} | 84 +-
...st-hierarchical-metric-anomalies-template.html} | 0
...er.html => test-metric-anomalies-template.html} | 0
557 files changed, 20012 insertions(+), 9976 deletions(-)
copy pinot-api/src/main/java/org/apache/pinot/client/{PinotClientTransport.java => Request.java} (61%)
create mode 100644 pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/PinotQueryParserFactory.java
copy pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/ControllerRestApi.java => pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/PinotQueryRequest.java (60%)
delete mode 100644 pinot-broker/src/main/java/org/apache/pinot/broker/routing/CfgBasedRouting.java
delete mode 100644 pinot-broker/src/test/java/org/apache/pinot/broker/broker/EmptyBrokerOnlineOfflineStateModelFactory.java
delete mode 100644 pinot-broker/src/test/java/org/apache/pinot/broker/broker/EmptySegmentOnlineOfflineStateModelFactory.java
create mode 100644 pinot-broker/src/test/java/org/apache/pinot/broker/request/PqlAndCalciteSqlCompatibilityTest.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/config/CompletionConfig.java
rename {pinot-core/src/main/java/org/apache/pinot/core/query/aggregation => pinot-common/src/main/java/org/apache/pinot/common}/function/AggregationFunctionType.java (93%)
copy pinot-common/src/main/java/org/apache/pinot/{pql/parsers/AbstractCompiler.java => common/function/FunctionDefinitionRegistry.java} (69%)
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/utils/URIUtils.java
copy pinot-common/src/main/java/org/apache/pinot/{pql => }/parsers/AbstractCompiler.java (81%)
create mode 100644 pinot-common/src/main/java/org/apache/pinot/parsers/utils/BrokerRequestComparisonUtils.java
copy pinot-common/src/main/java/org/apache/pinot/{pql/parsers/AbstractCompiler.java => sql/parsers/CalciteSqlCompiler.java} (61%)
create mode 100644 pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
rename pinot-common/src/main/java/org/apache/pinot/{pql/parsers/AbstractCompiler.java => sql/parsers/SqlCompilationException.java} (69%)
create mode 100644 pinot-common/src/test/java/org/apache/pinot/common/utils/URIUtilsTest.java
create mode 100644 pinot-common/src/test/java/org/apache/pinot/sql/parsers/CalciteSqlCompilerTest.java
copy pinot-common/src/test/resources/{pql_queries.list => sql_queries.list} (87%)
delete mode 100644 pinot-connectors/pinot-connector-kafka-0.9/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaStarterUtils.java
create mode 100644 pinot-connectors/pinot-connector-kafka-0.9/src/main/java/org/apache/pinot/core/realtime/impl/kafka/server/KafkaDataProducer.java
create mode 100644 pinot-connectors/pinot-connector-kafka-0.9/src/main/java/org/apache/pinot/core/realtime/impl/kafka/server/KafkaDataServerStartable.java
create mode 100644 pinot-connectors/pinot-connector-kafka-0.9/src/main/resources/META-INF/services/org.apache.pinot.core.realtime.stream.StreamConsumerFactory
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/README.md
copy pinot-connectors/{pinot-connector-kafka-0.9 => pinot-connector-kafka-2.0}/pom.xml (59%)
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaConsumerFactory.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaMessageBatch.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaPartitionLevelConnectionHandler.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaPartitionLevelConsumer.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaPartitionLevelStreamConfig.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaStreamLevelConsumer.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaStreamLevelConsumerManager.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaStreamLevelStreamConfig.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaStreamMetadataProvider.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/server/KafkaDataProducer.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/server/KafkaDataServerStartable.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/resources/META-INF/services/org.apache.pinot.core.realtime.stream.StreamConsumerFactory
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/test/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaPartitionLevelConsumerTest.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/test/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaPartitionLevelStreamConfigTest.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/test/java/org/apache/pinot/core/realtime/impl/kafka2/utils/EmbeddedZooKeeper.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/test/java/org/apache/pinot/core/realtime/impl/kafka2/utils/MiniKafkaCluster.java
copy pinot-common/src/test/resources/log4j2.xml => pinot-connectors/pinot-connector-kafka-base/README.md (68%)
copy pinot-connectors/{ => pinot-connector-kafka-base}/pom.xml (60%)
rename pinot-connectors/{pinot-connector-kafka-0.9 => pinot-connector-kafka-base}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaAvroMessageDecoder.java (100%)
rename pinot-connectors/{pinot-connector-kafka-0.9 => pinot-connector-kafka-base}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaJSONMessageDecoder.java (100%)
create mode 100644 pinot-connectors/pinot-connector-kafka-base/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaStarterUtils.java
rename pinot-connectors/{pinot-connector-kafka-0.9 => pinot-connector-kafka-base}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaStreamConfigProperties.java (96%)
copy pinot-core/src/main/java/org/apache/pinot/core/io/util/ValueReader.java => pinot-connectors/pinot-connector-kafka-base/src/main/java/org/apache/pinot/core/realtime/impl/kafka/MessageAndOffset.java (56%)
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/ControllerRequestBuilderUtil.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/EmptyBrokerOnlineOfflineStateModelFactory.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/EmptySegmentOnlineOfflineStateModelFactory.java
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/AccessControlTest.java (87%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/FileUploadPathProviderTest.java (95%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/PinotFileUploadTest.java (79%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/PinotInstanceRestletResourceTest.java (96%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/PinotSchemaRestletResourceTest.java (98%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/PinotSegmentRestletResourceTest.java (85%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/PinotTableRestletResourceTest.java (96%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/PinotTenantRestletResourceTest.java (89%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/SegmentCompletionProtocolDeserTest.java (99%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/SegmentCompletionUtilsTest.java (96%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/ServerTableSizeReaderTest.java (98%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/TableSizeReaderTest.java (99%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/TableViewsTest.java (93%)
rename pinot-core/src/main/java/org/apache/pinot/core/data/recordtransformer/{CompoundTransformer.java => CompositeTransformer.java} (70%)
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/data/recordtransformer/NullValueTransformer.java
rename pinot-core/src/main/java/org/apache/pinot/core/data/recordtransformer/{SanitationTransformer.java => SanitizationTransformer.java} (92%)
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/io/util/VarLengthBytesValueReaderWriter.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/filter/ExpressionFilterOperator.java
delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/query/SelectionOnlyOperator.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/query/SelectionOperator.java
delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/query/SelectionOrderByOperator.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/transform/TransformBlockDataFetcher.java
copy pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/{SumAggregationFunction.java => DistinctCountRawHLLAggregationFunction.java} (53%)
copy pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/{CountMVAggregationFunction.java => DistinctCountRawHLLMVAggregationFunction.java} (64%)
copy pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerMetrics.java => pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/customobject/SerializedHLL.java (51%)
copy pinot-api/src/main/java/org/apache/pinot/client/PinotClientTransport.java => pinot-core/src/main/java/org/apache/pinot/core/realtime/stream/StreamDataProducer.java (67%)
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/realtime/stream/StreamDataProvider.java
copy pinot-core/src/main/java/org/apache/pinot/core/{io/util/ValueReader.java => realtime/stream/StreamDataServerStartable.java} (53%)
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/io/util/VarLengthBytesValueReaderWriterTest.java
create mode 100644 pinot-hadoop/src/main/java/org/apache/pinot/hadoop/io/CombineAvroKeyInputFormat.java
create mode 100644 pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/InternalConfigConstants.java
create mode 100644 pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java
rename pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/{mapper => mappers}/SegmentCreationMapper.java (99%)
create mode 100644 pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/mappers/SegmentPreprocessingMapper.java
create mode 100644 pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/partitioners/GenericPartitioner.java
create mode 100644 pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/partitioners/PartitionFunctionFactory.java
create mode 100644 pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/reducers/SegmentPreprocessingReducer.java
create mode 100644 pinot-hadoop/src/main/java/org/apache/pinot/hadoop/utils/JobPreparationHelper.java
create mode 100644 pinot-hadoop/src/test/java/org/apache/pinot/hadoop/io/DeleteExtraPushedSegmentsTest.java
rename pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/{SegmentCompletionIntegrationTests.java => SegmentCompletionIntegrationTest.java} (88%)
rename pinot-server/src/test/java/org/apache/pinot/server/api/{resources => }/BaseResourceTest.java (99%)
rename pinot-server/src/test/java/org/apache/pinot/server/api/{resources => }/TableSizeResourceTest.java (98%)
rename pinot-server/src/test/java/org/apache/pinot/server/api/{resources => }/TablesResourceTest.java (99%)
copy pinot-tools/src/main/resources/sample_data/{ => kafka_0.9}/airlineStats_realtime_table_config.json (100%)
copy pinot-tools/src/main/resources/sample_data/{ => kafka_0.9}/meetupRsvp_realtime_table_config.json (94%)
create mode 100644 pinot-tools/src/main/resources/sample_data/kafka_2.0/airlineStats_realtime_table_config.json
copy pinot-tools/src/main/resources/sample_data/{ => kafka_2.0}/meetupRsvp_realtime_table_config.json (82%)
create mode 100644 thirdeye/docs/configuration.rst
delete mode 100644 thirdeye/docs/production.rst
create mode 100644 thirdeye/thirdeye-frontend/app/pods/components/detection-health/component.js
create mode 100644 thirdeye/thirdeye-frontend/app/pods/components/detection-health/template.hbs
create mode 100644 thirdeye/thirdeye-pinot/config/h2db.mv.db
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/alert/content/EntityGroupKeyContentFormatter.java
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/alert/content/{MultipleAnomaliesEmailContentFormatter.java => MetricAnomaliesEmailContentFormatter.java} (92%)
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/DetectionPreviewConfiguration.java
rename thirdeye/thirdeye-pinot/src/{test => main}/java/org/apache/pinot/thirdeye/detection/components/MockGrouper.java (100%)
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/TriggerConditionGrouper.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/health/AnomalyCoverageStatus.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/health/DetectionHealth.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/health/DetectionTaskStatus.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/health/HealthStatus.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/health/RegressionStatus.java
copy thirdeye/thirdeye-pinot/src/{test => main}/java/org/apache/pinot/thirdeye/detection/spec/MockGrouperSpec.java (100%)
rename thirdeye/thirdeye-pinot/src/{test/java/org/apache/pinot/thirdeye/detection/spec/MockGrouperSpec.java => main/java/org/apache/pinot/thirdeye/detection/spec/TriggerConditionGrouperSpec.java} (50%)
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/translator/DetectionMetricAttributeHolder.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/translator/DetectionMetricProperties.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/formatter/DTOFormatter.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/formatter/DetectionAlertConfigFormatter.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/formatter/DetectionConfigFormatter.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/resources/org/apache/pinot/thirdeye/detector/entity-groupkey-anomaly-report.ftl
rename thirdeye/thirdeye-pinot/src/main/resources/org/apache/pinot/thirdeye/detector/{holiday-anomaly-report.ftl => metric-anomalies-template.ftl} (100%)
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/alert/content/ContentFormatterUtils.java
copy thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/alert/content/{TestMultipleAnomaliesEmailContentFormatter.java => TestEntityGroupKeyEmailContentFormatter.java} (58%)
rename thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/alert/content/{TestMultipleAnomaliesEmailContentFormatter.java => TestMetricAnomaliesEmailContentFormatter.java} (86%)
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/components/TriggerConditionGrouperTest.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/health/DetectionHealthTest.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/formatter/DetectionConfigFormatterTest.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/yaml/translator/compositePipelineTranslatorTestResult-4.json
create mode 100644 thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/yaml/translator/compositePipelineTranslatorTestResult-5.json
create mode 100644 thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/yaml/translator/pipeline-config-4.yaml
copy thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/{validators/entity-pipeline-config-2.yaml => yaml/translator/pipeline-config-5.yaml} (80%)
copy thirdeye/thirdeye-pinot/src/test/resources/{test-multiple-anomalies-email-content-formatter.html => test-entity-groupby-email-content-formatter.html} (58%)
rename thirdeye/thirdeye-pinot/src/test/resources/{test-hierarchical-anomalies-email-content-formatter.html => test-hierarchical-metric-anomalies-template.html} (100%)
rename thirdeye/thirdeye-pinot/src/test/resources/{test-multiple-anomalies-email-content-formatter.html => test-metric-anomalies-template.html} (100%)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org