You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by ja...@apache.org on 2019/09/24 00:45:52 UTC
[incubator-pinot] branch offheap_var_length_store updated (a342d57
-> 5ee259e)
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a change to branch offheap_var_length_store
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
discard a342d57 Implement OffHeapMutableBytesStore for real-time var-length bytes store
add 732a7b9 Add more key/value pairs into LOG2M_TO_SIZE_IN_BYTES in HllSizeUtils (#4398)
add 49d8fa7 #4317 Feature/variable length bytes offline dictionary for indexing bytes and string dicts. (#4321)
add 02f6181 CompletionConfig for realtime tables (#4367)
add ae1ecef [TE] add a thread pool to run preview tasks (#4405)
add de62d8e [TE] Make email template and subject pluggable under email alert scheme (#4409)
add 7df1a43 [TE] Add Merger after metric grouper; other minor clean up (#4410)
add e71ac1b [TE] Do not authorize when auth is disabled (#4414)
add 0b8af63 Fix codecov link from old url to new (#4412)
add 07e2c72 Move tests for API resources to a different package (#4415)
add 52f69d1 [TE] Added support for Vertica as a data source (#4404)
add a7c419a Adding Calcite SQL Parser and make another entry point to query Pinot (#4387)
add e041ec5 [TE] frontend - harleyjj/detection-health - UI for model performance (#4413)
add 6bec451 [TE] Inject the port from config into Commons mail SSL Smtp port (#4420)
add dac9ae2 prompt on fail schema (#4395)
add b6fe4e6 [TE] detection health ui adjustments (#4421)
add bca6756 make kafka version number controlled by config number (#4396)
add 0993974 [TE] Fix exception handling - Propagate and display the error message/exception on frontind (#4419)
add aadcd36 In ClusterChangeMediator, stop enqueue/process changes if already stopped (#4422)
add 52f9b08 Add pinot community inviter (#4424)
add 2f691fc [TE] detection health - coverage fix (#4428)
add 079b86d Config for overriding initial rows threshold in segment size auto tuning (#4376)
add 50aaaf9 [TE] added ThirdEye configuration documents (#4429)
add f671072 Add URIUtils class to handle URI/URL encoding/decoding (#4426)
add 626c43a Misc fix for controller tests (#4431)
add 12e77bd #4401 Reuse a ThreadLocal byte[] when reading String elements from the variable length value reader. (#4432)
add e720448 Remove the massive console print and log from CalciteSqlCompilerTest (#4434)
add 67afc71 Cleanup un-necessary setups in HelixSetupUtils (#4430)
add b1d28fd Fix potential resource leak in the way we close a collection of closeables (#4418)
add d2b2981 add download excel (#4403)
add 7b1715c 3891: Check for validity of segment start/end time during segment generation (#4368)
add 625482a [TE] Renamed some classes to making it accurate and consistent (#4439)
add 31d916f [TE] Adding Entity GroupKey Anomalies Email Reporter (#4433)
add db5923c Clean up the ServerInstance class and remove the un-necessary ip address stored (#4438)
add 5492217 Fix the potential resource leak from NettyServer (#4440)
add e897271 [TE] add ml auto config into YAML template (#4423)
add 16f028e Fix ServerInstanceTest (#4442)
add 2e07b0b [TE] Improvements to the Entity GroupKey Template (#4444)
add cd579b1 Add comments and fix typo for the SegmentCompletionTest. (#4447)
add 85b07f3 [TE] frontend - harleyjj/preview - provide custom time range before previewing (#4437)
add 365ee85 [TE] Detection and alert config formatters (#4443)
add 9232bb5 Moving handling of NULL values from RecordReaders to NullValueTransfo… (#4399)
add eaecd64 Refactor pinot-connectors to break the dependencies from kafka 0.9 (#4445)
add 024da51 Enable SegmentCompleteionIntegrationTest (#4451)
add 25e7be1 [TE] Bug Fix - Parsing of multiple alert schemes (#4456)
add d18fab9 [TE] anomaly filters potential inconsistency issue fix (#4448)
add 2d0b247 [TE] Update Swagger Resource Package (#4458)
add 6dc820a Issue #4450: Support parsing of `Order By` clause in PQL compiler. (#4453)
add 0fded63 [TE] Inject Sub Entity Name into the anomaly properties (#4452)
add 523f697 Add resource config for lead controller resource (#4449)
add a1fff55 Ensuring partition config is valid during Hadoop preprocess (#4463)
add 14ef543 Remove the usage of guava deprecated methods (#4457)
add 1642f00 [TE] logs and precondition checks for pipeline re-tuning (#4465)
add eca1d2e [TE] frontend - set analysis range using alertDetailsDefaultWindowSize from endpoint (#4460)
add 1d45f87 Enable periodic rebalance as a temporary work-around for the Helix issue (#4459)
add cac4f24 Allow customized metrics prefix in pinot controller/broker/server (#4392)
add c01b852 Add table name parameter for segment push (#4454)
add c81adda [TE] Include alert description in entity email alert (#4471)
add 29bc069 [TE] Include admin recipients in every email alert (cc address) (#4470)
add 9d64d8a Refactor HelixBrokerStarterTest (#4473)
add 8717e35 Adding Support for Kafka 2.0 Consumer (#4397)
add 551b5c3 Time-aware resizing (#4462)
add 9316610 Clean up test utils, disconnect fake instances (#4476)
add c768df8 When initializing mutable dictionary, preserve 10% buffer for cardinality to reduce the chance of re-sizing the dictionary (#4363)
add ad6ff6b Delete extra refreshed segments after segment push (#4464)
add 08be56e minor fix double close (#4478)
add 1c81dfb Take OFFLINE segment into account for real-time rebalancer (#4337)
add 8d8bf3b [TE] Make the dashboard report summary to show the value for correct time ranges (#4479)
add bc268f1 [TE] Remove hard-coded AdLibAnomalyDetector and AdLibAlertFilter (#4477)
add fcc6994 Fix the flakiness of PinotControllerModeTest by enabling the periodic rebalance (#4487)
add 02c6db6 Fix num entries scanned in filter for MultiValue scanner (#4455)
add 85d606d [TE] Display dimensions in anomalies tab (#4481)
add dfdf443 Add support in the rebalancer for the user to provide minimum number of serving replicas (#4446)
add 90e55d3 [TE] Check empty address to fix Email Exception (#4486)
add 7fde375 [TE] Adding Mean Variance Rule Detector (#4466)
add 6cf8c60 Clean up tenant config validation to correctly handle tag override (#4488)
add 4fea0d4 An auto recommendation for inverted index (#4474)
add eac11ec Disable state transition stats verification in rebalancer cluster integration test (#4490)
add bae8c1f [Instance Assignment] Add configs for instance assignment (#4489)
add 8c15047 Adding kafka 2.0 doc for using simple consumer (#4493)
add c88845a [TE] frontend - harleyjj/jsyaml - add jsyaml package to parse valid yaml as secondary to yamljs (#4480)
add 4f66513 [TE] Flatten Anomaly Results to a list of Map (#4491)
add cd75e12 [TE] Remove the offline suffix for offline pinot table. (#4492)
add 6c9486e Add a thread in ControllerLeadershipManager to periodically fetch controller leadership info (#4497)
add ac1f387 Rename metric to reflect what's being counted: segments-processed vs segments-queried (#4499)
add 2bbdfab [TE] Remove deprecated stage classes and tests (#4496)
add b73427d [TE] [Composite-Alert] Adding Entity GroupKey with Whitelist Template and Formatter (#4495)
add d1180d9 Down-grade Helix to version 0.8.4 (#4502)
add aea8fca [TE] anomalies table endpoint (#4501)
add 8e02f48 [Instance Assignment] Add implementations for instance assignment (#4504)
add 6741396 Fix the toggle table state API (#4509)
add af75788 Adding support for Map type fields (#4388)
add fa615df Put Helix manager out of resource manager (#4506)
add a174d81 [TE] Update AnomalyFlattenResource (#4505)
add 61b2a07 [TE] frontend - harleyjj/aiavailability - implements AI availability table UI (#4510)
add 38fbae9 Fixed the url for report (#4513)
add d340627 Simplify ControllerRequestURLBuilder, remove invalid URLs (#4512)
add c920824 [TE] fix for availability report (#4515)
add 3236e40 Adding config to use let controller/broker/server to set hostname (#4517)
add ca2444f Make MAX_PLAN_THREADS >= 1 (#4518)
add 115cab9 The external view check in rebalancer should also check for no extra servers (#4498)
add 1d7b4a5 [Instance Assignment] Enhance Instance class to include the pool config (#4522)
add 3299444 Updating license and notice binary files for aea8fca7e2535ce2e8a445754cf5c25bef0cb542 (#4524)
add 0a13d98 Fixing minor bug with duplicate column names in the query (#4523)
add 6e209e4 Handle NPE from getting live instance (#4528)
add 84464af brag about Star-Tree Index support (#4530)
add 4f8ffc7 [Controller Separation] Add logic for lead controller resource (#4323)
add 4e41bc3 [Instance Assignment] Plug in instance assignment (#4516)
add d120f4d [TE] frontend - harleyjj/create-alert - add link for onboarding mysql and presto metrics (#4483)
add 0d69302 [Instance Assignment] De-couple assignment strategy from SegmentAssignment (#4533)
add 9f276fc Fix the record reader for multi-value with null element (#4541)
add f3893c9 [Instance Assignment] Rename instance level replica to replica-group (#4536)
add 3d9bd65 Fixing metrics aggregation logic for Float and Double (#4537)
add 83cc88d Expose a method to determine if a QueryExceptionErrorCode represents a client-side error (#4532)
add d261a76 Expose table-name via StreamConfig (#4543)
add d62f467 [TE] Make HTTPS redirection configurable (#4539)
add c0dbbfc Set the consumption end time of a segment based on its creation time (#4546)
add d3c2695 [TE] Skip creating notification task when there is no anomaly to notify (#4544)
add cf980cb [Instance Assignment] Plug in segment assignment for offline segment assignment and table rebalance (#4549)
add 66d31d6 Fixing bytes data type merge logic (#4534)
add cf9974c Fixing the issue of selection order-by won't give correct ordering (#4540)
add f688776 Cleanup preprocess (#4552)
add 7c7b6cc Support transform functions with AVG aggregation function (#4557)
add 9b5775e [Issue #4551] Remove memory allocation for virtual columns in consumi… (#4555)
add d8d0bb5 Add metrics on Helix and Pinot controller leadership (#4561)
add 38c73aa Fix the issue of scheduling task on all tables (#4563)
add 23ffd67 Handle the virtual column logic in FieldSpec instead of Schema (#4562)
add c2f6b25 Indexed table (#4547)
add 9edb2cc [TE] frontend - harleyjj/alert-details - add rule and modifier columns to anomaly table (#4564)
add 77446e7 [TE] support entity group expression (#4377)
add e98efcb [TE] fix multi-day granularity alignment issue (#4566)
add 9180dac fixed "StartTree" typo (#4571)
add 16cd4fa fixed hyperlink to PinotFS (#4570)
add c30ef17 [TE] only store evaluations for top dimensions (#4572)
add 3216159 Support aws s3 with Parquet in pinot-tools (#4556)
add fbc28d8 [TE] Add timeout for all tasks and clean up RUNNING states if timeout (#4554)
add 9df877d Use docker compose for pinot quickstart (#4575)
add 584368d Merge LLC related integration tests (#4573)
add f3aa4db Retrieve routing table by exact table name instead of by prefix (#3037)
add a1c5af7 [TE] Return the anomaly details to render snapshot (#4576)
add 05609e4 Use Apache Pinot License (#4584)
add 76e0c62 Adding Predicate supports for Bytes Column without dictionary (#4586)
add 68092ab Adding integration test for Hadoop pre-process job (#4577)
add 3c2f0b3 Adding example for kubernetes deployment on GKE (#4581)
add dbcbd2e [TE] Empty Entity report bug fix (#4598)
add 5e525d0 [TE] Exclude template/blacklist emails before sending out alert (#4601)
add 0a32309 [TE] fix create alert error message NPE (#4594)
add e70a781 Add ApacheCon Slide to README file (#4604)
add 67ea9bb [TE] Merger fix to consider anomalies generated earlier (#4565)
add f5017be [TE] Redesign to support multiple Alert Content formatters (#4599)
add add0f07 [TE] Update AI Availability table (#4606)
add fe7fff6 [TE] Add anomaly type for merged anomaly result (#4605)
add 8f9bb5d [TE] Remove the logic of reading stderr from phantomjs to avoid zombie processes (#4611)
add 45205e8 [TE] frontend - harleyjj/alert-details - handle reported anomaly for anomaly table (#4587)
add fe29287 [TE] frontend - harleyjj/auth - handle expired token and 401 errors plus redirect after login (#4580)
add 76fcc86 [TE] frontend - harleyjj/ai - fix small bug in AI Availability (#4600)
add 09207f9 [TE] consider anomaly type during merge (#4613)
add 8d593f3 [TE] Bug fix to initialize the configDAO (#4616)
add c7b647a [TE][Composite-Alert] Entity Anomaly Merger (#4609)
add c87b110 [TE] Add data set display name and show it on UI (#4619)
add 239237e [TE] the last time stamp filter out the anomalies (#4621)
add 44d1ce9 Remove import of sun.misc.Cleaner to make code compatible with jdk11 (#4624)
add 04c4dc2 Revert "Remove import of sun.misc.Cleaner to make code compatible with jdk11 (#4624)" (#4625)
add eda4176 [TE] [Notification] Refactor email filters to a generalized alert filter (#4620)
add 3668c79 Documentation update for realtime Pinot (#4630)
add f8c2cf3 Put order-by expressions in front of selection expreesions in selection DataSchema (#4629)
add 5ee259e Implement OffHeapMutableBytesStore for real-time var-length bytes store
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (a342d57)
\
N -- N -- N refs/heads/offheap_var_length_store (5ee259e)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.travis.yml | 6 +
.travis_install.sh | 7 +-
.travis_test.sh | 15 +-
LICENSE-binary | 18 +-
NOTICE-binary | 125 +-
README.md | 13 +-
docker/Dockerfile | 5 +-
docker/README.md | 33 +-
docker/docker-build.sh | 11 +-
docker/docker-compose.yml | 82 ++
docs/architecture.rst | 68 +-
docs/code_modules.rst | 5 +-
docs/contribution_guidelines.rst | 3 +-
docs/multitenancy.rst | 358 -----
docs/pluggable_storage.rst | 2 +-
docs/pluggable_streams.rst | 96 ++
docs/pql_examples.rst | 9 +-
docs/tableconfig_schema.rst | 168 ++-
docs/tuning_pinot.rst | 1 +
docs/tuning_realtime_performance.rst | 77 +
kubernetes/README.md | 27 +
kubernetes/examples/gke/skaffold/README.md | 119 ++
.../examples/gke/skaffold/cleanup.sh | 34 +-
.../gke/skaffold/gke-storageclass-kafka-pd.yml | 37 +-
.../gke-storageclass-pinot-controller-pd.yml | 37 +-
.../skaffold/gke-storageclass-pinot-server-pd.yml | 37 +-
.../gke/skaffold/gke-storageclass-zk-pd.yml | 37 +-
kubernetes/examples/gke/skaffold/kafka.yml | 408 ++++++
kubernetes/examples/gke/skaffold/pinot-broker.yml | 88 ++
.../examples/gke/skaffold/pinot-controller.yml | 106 ++
.../examples/gke/skaffold/pinot-example-loader.yml | 43 +
kubernetes/examples/gke/skaffold/pinot-server.yml | 101 ++
.../examples/gke/skaffold/query-pinot-data.sh | 35 +-
kubernetes/examples/gke/skaffold/setup.sh | 51 +
.../examples/gke/skaffold/skaffold.yaml | 44 +-
kubernetes/examples/gke/skaffold/zookeeper.yml | 80 +
.../java/org/apache/pinot/client/Connection.java | 44 +-
.../client/JsonAsyncHttpPinotClientTransport.java | 30 +-
.../apache/pinot/client/PinotClientTransport.java | 6 +
.../{PinotClientTransport.java => Request.java} | 34 +-
.../apache/pinot/client/PreparedStatementTest.java | 40 +-
.../apache/pinot/client/ResultSetGroupTest.java | 38 +-
.../apache/pinot/broker/api/RequestStatistics.java | 9 +-
.../pinot/broker/broker/BrokerServerBuilder.java | 3 +-
.../broker/broker/helix/ClusterChangeMediator.java | 12 +-
.../broker/broker/helix/HelixBrokerStarter.java | 5 +-
.../broker/helix/LiveInstanceChangeHandler.java | 6 +-
.../requesthandler/BaseBrokerRequestHandler.java | 28 +-
...enNestedPredicatesFilterQueryTreeOptimizer.java | 3 +-
.../requesthandler/PinotQueryParserFactory.java | 49 +
.../broker/requesthandler/PinotQueryRequest.java | 31 +-
.../pinot/broker/routing/CfgBasedRouting.java | 64 -
.../routing/HelixExternalViewBasedRouting.java | 4 +-
.../broker/routing/RoutingTableBuilderFactory.java | 3 +-
.../EmptyBrokerOnlineOfflineStateModelFactory.java | 70 -
...EmptySegmentOnlineOfflineStateModelFactory.java | 71 -
.../broker/broker/HelixBrokerStarterTest.java | 357 ++---
.../request/PqlAndCalciteSqlCompatibilityTest.java | 123 ++
pinot-common/pom.xml | 8 +-
.../common/assignment/InstancePartitions.java | 136 ++
.../InstancePartitionsType.java | 12 +-
.../common/assignment/InstancePartitionsUtils.java | 133 ++
.../pinot/common/config/CompletionConfig.java | 63 +
.../apache/pinot/common/config/IndexingConfig.java | 20 +-
.../org/apache/pinot/common/config/Instance.java | 145 ++
.../common/config/SegmentPartitionConfig.java | 2 +-
.../SegmentsValidationAndRetentionConfig.java | 30 +-
.../apache/pinot/common/config/TableConfig.java | 109 +-
.../apache/pinot/common/config/TagNameUtils.java | 25 +-
.../config/instance/InstanceAssignmentConfig.java | 75 +
...InstanceAssignmentConfigMapChildKeyHandler.java | 19 +-
.../instance/InstanceAssignmentConfigUtils.java | 110 ++
.../config/instance/InstanceConstraintConfig.java | 35 +-
.../InstanceReplicaGroupPartitionConfig.java | 113 ++
.../config/instance/InstanceTagPoolConfig.java | 86 ++
.../org/apache/pinot/common/data/FieldSpec.java | 11 +
.../java/org/apache/pinot/common/data/Schema.java | 17 +-
.../pinot/common/exception/QueryException.java | 24 +
.../pinot/common/metadata/ZKMetadataProvider.java | 8 -
.../apache/pinot/common/metrics/BrokerMetrics.java | 12 +-
.../pinot/common/metrics/ControllerGauge.java | 6 +
.../pinot/common/metrics/ControllerMeter.java | 1 +
.../pinot/common/metrics/ControllerMetrics.java | 9 +-
.../apache/pinot/common/metrics/ServerMetrics.java | 24 +-
.../apache/pinot/common/query/ReduceService.java | 4 +-
.../pinot/common/request/AggregationInfo.java | 384 +++--
.../apache/pinot/common/request/BrokerRequest.java | 1533 ++++++++++----------
.../apache/pinot/common/request/DataSource.java | 216 ++-
.../apache/pinot/common/request/Expression.java | 421 +++---
.../pinot/common/request/ExpressionType.java | 11 +-
.../pinot/common/request/FilterOperator.java | 18 +-
.../apache/pinot/common/request/FilterQuery.java | 561 ++++---
.../pinot/common/request/FilterQueryMap.java | 260 ++--
.../org/apache/pinot/common/request/Function.java | 339 ++---
.../org/apache/pinot/common/request/GroupBy.java | 420 +++---
.../pinot/common/request/HavingFilterQuery.java | 562 ++++---
.../pinot/common/request/HavingFilterQueryMap.java | 259 ++--
.../apache/pinot/common/request/Identifier.java | 220 ++-
.../pinot/common/request/InstanceRequest.java | 542 ++++---
.../org/apache/pinot/common/request/Literal.java | 364 ++---
.../apache/pinot/common/request/PinotQuery.java | 1048 ++++++-------
.../apache/pinot/common/request/QuerySource.java | 218 ++-
.../org/apache/pinot/common/request/QueryType.java | 485 +++----
.../org/apache/pinot/common/request/Selection.java | 486 +++----
.../apache/pinot/common/request/SelectionSort.java | 285 ++--
.../pinot/common/response/ServerInstance.java | 180 +--
.../common/restlet/resources/RebalanceResult.java | 67 -
.../apache/pinot/common/utils/CommonConstants.java | 64 +-
.../org/apache/pinot/common/utils/DataSchema.java | 25 +-
.../org/apache/pinot/common/utils/DataTable.java | 14 +-
.../common/utils/FileUploadDownloadClient.java | 59 +-
.../org/apache/pinot/common/utils/FileUtils.java | 59 +-
.../org/apache/pinot/common/utils/HashUtil.java | 47 +
.../org/apache/pinot/common/utils/JsonUtils.java | 36 +-
.../apache/pinot/common/utils/ServiceStatus.java | 15 +-
.../org/apache/pinot/common/utils/StringUtil.java | 4 +-
.../org/apache/pinot/common/utils/URIUtils.java | 88 ++
.../pinot/common/utils/helix/HelixHelper.java | 47 +-
.../common/utils/helix/LeadControllerUtils.java | 61 +
.../pinot/common/utils/request/RequestUtils.java | 55 +-
.../apache/pinot/common/utils/time/TimeUtils.java | 15 +
.../org/apache/pinot/filesystem/LocalPinotFS.java | 91 +-
.../pinot/{pql => }/parsers/AbstractCompiler.java | 7 +-
.../utils/BrokerRequestComparisonUtils.java | 235 +++
.../parsers/PinotQuery2BrokerRequestConverter.java | 12 +-
.../apache/pinot/pql/parsers/Pql2AstListener.java | 2 +-
.../org/apache/pinot/pql/parsers/Pql2Compiler.java | 63 +-
.../parsers/pql2/ast/BetweenPredicateAstNode.java | 2 +-
.../pql2/ast/ComparisonPredicateAstNode.java | 20 +-
.../pql/parsers/pql2/ast/InPredicateAstNode.java | 2 +-
.../pinot/pql/parsers/pql2/ast/OrderByAstNode.java | 56 +-
.../pql/parsers/pql2/ast/OutputColumnAstNode.java | 2 +-
.../pql/parsers/pql2/ast/PredicateListAstNode.java | 2 +-
.../pql2/ast/RegexpLikePredicateAstNode.java | 2 +-
.../parsers/CalciteSqlCompiler.java} | 18 +-
.../apache/pinot/sql/parsers/CalciteSqlParser.java | 273 ++++
.../pinot/sql/parsers/SqlCompilationException.java | 20 +-
.../apache/pinot/startree/hll/HllSizeUtils.java | 14 +-
.../pinot/common/config/IndexingConfigTest.java | 3 +
.../pinot/common/config/TableConfigTest.java | 90 +-
.../apache/pinot/common/utils/URIUtilsTest.java | 68 +
.../apache/pinot/pql/parsers/Pql2CompilerTest.java | 62 +-
.../request/BrokerRequestSerializationTest.java | 17 +-
.../pinot/sql/parsers/CalciteSqlCompilerTest.java | 425 ++++++
pinot-common/src/test/resources/pql_queries.list | 5 +
.../{pql_queries.list => sql_queries.list} | 247 ++--
pinot-common/src/thrift/request.thrift | 1 +
pinot-connectors/pinot-connector-kafka-0.9/pom.xml | 16 +-
.../realtime/impl/kafka/KafkaStarterUtils.java | 171 ---
.../impl/kafka/server/KafkaDataProducer.java | 57 +
.../kafka/server/KafkaDataServerStartable.java | 109 ++
...inot.core.realtime.stream.StreamConsumerFactory | 30 +-
.../impl/kafka/KafkaLowLevelStreamConfigTest.java | 2 +-
.../kafka/KafkaPartitionLevelConsumerTest.java | 16 +-
.../pinot-connector-kafka-2.0/README.md | 57 +
.../pom.xml | 50 +-
.../realtime/impl/kafka2/KafkaConsumerFactory.java | 51 +
.../realtime/impl/kafka2/KafkaMessageBatch.java | 63 +
.../KafkaPartitionLevelConnectionHandler.java | 73 +
.../impl/kafka2/KafkaPartitionLevelConsumer.java | 66 +
.../kafka2/KafkaPartitionLevelStreamConfig.java | 147 ++
.../impl/kafka2/KafkaStreamLevelConsumer.java | 179 +++
.../kafka2/KafkaStreamLevelConsumerManager.java | 194 +++
.../impl/kafka2/KafkaStreamLevelStreamConfig.java | 136 ++
.../impl/kafka2/KafkaStreamMetadataProvider.java | 67 +
.../impl/kafka2/server/KafkaDataProducer.java | 77 +
.../kafka2/server/KafkaDataServerStartable.java | 98 ++
...inot.core.realtime.stream.StreamConsumerFactory | 30 +-
.../kafka2/KafkaPartitionLevelConsumerTest.java | 298 ++++
.../KafkaPartitionLevelStreamConfigTest.java} | 28 +-
.../impl/kafka2/utils/EmbeddedZooKeeper.java | 60 +
.../impl/kafka2/utils/MiniKafkaCluster.java | 175 +++
.../pinot-connector-kafka-base/README.md | 23 +
.../{ => pinot-connector-kafka-base}/pom.xml | 43 +-
.../impl/kafka/KafkaAvroMessageDecoder.java | 0
.../impl/kafka/KafkaJSONMessageDecoder.java | 0
.../realtime/impl/kafka/KafkaStarterUtils.java | 118 ++
.../impl/kafka/KafkaStreamConfigProperties.java | 19 +-
.../core/realtime/impl/kafka/MessageAndOffset.java | 39 +-
pinot-connectors/pom.xml | 15 +
.../apache/pinot/controller/ControllerConf.java | 53 +-
.../controller/ControllerLeadershipManager.java | 117 --
.../apache/pinot/controller/ControllerStarter.java | 195 ++-
.../pinot/controller/LeadControllerManager.java | 239 +++
.../pinot/controller/api/pojos/Instance.java | 135 --
.../api/resources/FileUploadPathProvider.java | 5 +-
.../resources/LLCSegmentCompletionHandlers.java | 25 +-
.../api/resources/PinotControllerHealthCheck.java | 4 +-
.../PinotInstanceAssignmentRestletResource.java | 301 ++++
.../resources/PinotInstanceRestletResource.java | 8 +-
.../api/resources/PinotSegmentRestletResource.java | 20 +-
.../PinotSegmentUploadRestletResource.java | 163 +--
.../resources/PinotTableConfigRestletResource.java | 37 +-
.../api/resources/PinotTableRestletResource.java | 162 +--
.../pinot/controller/api/resources/TableViews.java | 4 +-
.../controller/api/upload/SegmentValidator.java | 37 +-
.../api/upload/SegmentValidatorResponse.java | 55 -
.../pinot/controller/api/upload/ZKOperator.java | 17 +-
.../helix/ControllerRequestBuilderUtil.java | 152 --
.../helix/ControllerRequestURLBuilder.java | 223 +--
.../EmptyBrokerOnlineOfflineStateModelFactory.java | 70 -
...EmptySegmentOnlineOfflineStateModelFactory.java | 81 --
.../controller/helix/SegmentStatusChecker.java | 8 +-
.../helix/core/PinotHelixResourceManager.java | 670 ++++-----
.../helix/core/PinotTableIdealStateBuilder.java | 26 +-
.../helix/core/SegmentDeletionManager.java | 34 +-
.../controller/helix/core/TableRebalancer.java | 303 ----
.../helix/core/assignment/InstancePartitions.java | 99 --
.../core/assignment/InstancePartitionsUtils.java | 97 --
.../HashBasedRotateInstanceConstraintApplier.java | 60 +
.../instance/InstanceAssignmentDriver.java | 89 ++
.../instance/InstanceConstraintApplier.java | 25 +-
.../InstanceReplicaGroupPartitionSelector.java | 193 +++
.../instance/InstanceTagPoolSelector.java | 133 ++
...OfflineBalanceNumSegmentAssignmentStrategy.java | 96 --
...flineReplicaGroupSegmentAssignmentStrategy.java | 200 ---
.../segment/OfflineSegmentAssignment.java | 264 ++++
...ealtimeBalanceNumSegmentAssignmentStrategy.java | 155 --
...ltimeReplicaGroupSegmentAssignmentStrategy.java | 159 --
.../segment/RealtimeSegmentAssignment.java | 221 +++
...ignmentStrategy.java => SegmentAssignment.java} | 27 +-
.../segment/SegmentAssignmentFactory.java} | 32 +-
.../segment/SegmentAssignmentStrategyFactory.java | 57 -
.../assignment/segment/SegmentAssignmentUtils.java | 75 +-
.../core/minion/PinotHelixTaskResourceManager.java | 4 +-
.../helix/core/minion/PinotTaskManager.java | 19 +-
.../core/periodictask/ControllerPeriodicTask.java | 16 +-
.../ControllerPeriodicTaskScheduler.java | 60 -
.../realtime/PinotLLCRealtimeSegmentManager.java | 77 +-
.../core/realtime/PinotRealtimeSegmentManager.java | 48 +-
.../core/realtime/SegmentCompletionManager.java | 54 +-
.../segment/FlushThresholdUpdateManager.java | 13 +-
.../SegmentSizeBasedFlushThresholdUpdater.java | 14 +-
.../rebalance/DefaultRebalanceSegmentStrategy.java | 350 -----
...onstants.java => RebalanceConfigConstants.java} | 24 +-
.../helix/core/rebalance/RebalanceResult.java | 71 +
.../core/rebalance/RebalanceSegmentStrategy.java | 55 -
.../rebalance/RebalanceSegmentStrategyFactory.java | 50 -
.../ReplicaGroupRebalanceSegmentStrategy.java | 532 -------
.../helix/core/rebalance/TableRebalancer.java | 405 ++++++
.../core/relocation/RealtimeSegmentRelocator.java | 255 +---
.../helix/core/retention/RetentionManager.java | 8 +-
.../BalanceNumSegmentAssignmentStrategy.java | 98 --
.../core/sharding/BucketizedSegmentStrategy.java | 66 -
.../core/sharding/RandomAssignmentStrategy.java | 64 -
.../ReplicaGroupSegmentAssignmentStrategy.java | 105 --
.../sharding/SegmentAssignmentStrategyFactory.java | 46 -
...rollerResourceMasterSlaveStateModelFactory.java | 84 ++
.../helix/core/util/HelixSetupUtils.java | 255 ++--
.../helix/core/util/ZKMetadataUtils.java | 4 +-
.../BrokerResourceValidationManager.java | 5 +-
.../validation/OfflineSegmentIntervalChecker.java | 13 +-
.../RealtimeSegmentValidationManager.java | 12 +-
.../controller/validation/StorageQuotaChecker.java | 47 +-
.../src/main/resources/static/css/pinot.css | 16 +-
.../src/main/resources/static/js/init.js | 12 +
.../src/main/resources/static/query/index.html | 8 +-
.../controller/LeadControllerManagerTest.java | 125 ++
.../api/{resources => }/AccessControlTest.java | 10 +-
.../FileUploadPathProviderTest.java | 3 +-
.../api/{resources => }/PinotFileUploadTest.java | 41 +-
...PinotInstanceAssignmentRestletResourceTest.java | 339 +++++
.../api/PinotInstanceRestletResourceTest.java | 163 +++
.../PinotSchemaRestletResourceTest.java | 2 +-
.../PinotSegmentRestletResourceTest.java | 31 +-
.../PinotTableRestletResourceTest.java | 72 +-
.../api/PinotTenantRestletResourceTest.java | 71 +
.../SegmentCompletionProtocolDeserTest.java | 2 +-
.../SegmentCompletionUtilsTest.java | 2 +-
.../{resources => }/ServerTableSizeReaderTest.java | 3 +-
.../api/{resources => }/TableSizeReaderTest.java | 2 +-
.../api/{resources => }/TableViewsTest.java | 15 +-
.../PinotInstanceRestletResourceTest.java | 161 --
.../resources/PinotTenantRestletResourceTest.java | 111 --
.../helix/ControllerInstanceToggleTest.java | 33 +-
.../helix/ControllerPeriodicTaskStarterTest.java | 38 +-
.../controller/helix/ControllerSentinelTestV2.java | 45 +-
.../controller/helix/ControllerTenantTest.java | 111 +-
.../pinot/controller/helix/ControllerTest.java | 350 ++++-
.../pinot/controller/helix/HelixHelperTest.java | 6 +-
.../controller/helix/PinotControllerModeTest.java | 308 ++--
.../controller/helix/PinotResourceManagerTest.java | 43 +-
.../controller/helix/SegmentStatusCheckerTest.java | 73 +-
.../helix/core/PinotHelixResourceManagerTest.java | 321 ++--
.../instance/InstanceAssignmentTest.java | 465 ++++++
...flineNonReplicaGroupSegmentAssignmentTest.java} | 47 +-
... OfflineReplicaGroupSegmentAssignmentTest.java} | 205 +--
...ltimeNonReplicaGroupSegmentAssignmentTest.java} | 99 +-
...RealtimeReplicaGroupSegmentAssignmentTest.java} | 135 +-
.../segment/SegmentAssignmentUtilsTest.java | 104 +-
.../periodictask/ControllerPeriodicTaskTest.java | 7 +-
.../PinotLLCRealtimeSegmentManagerTest.java | 7 +-
.../helix/core/realtime/SegmentCompletionTest.java | 304 ++--
.../segment/FlushThresholdUpdaterTest.java | 28 +-
.../rebalance/DefaultRebalanceStrategyTest.java | 639 --------
.../ReplicaGroupRebalanceStrategyTest.java | 327 -----
.../core/rebalance/TableRebalancerClusterTest.java | 322 ++++
.../helix/core/rebalance/TableRebalancerTest.java | 187 +++
.../relocation/RealtimeSegmentRelocatorTest.java | 286 ----
.../helix/core/retention/RetentionManagerTest.java | 12 +-
.../sharding/SegmentAssignmentStrategyTest.java | 439 ------
.../helix/core/util/TableRebalancerTest.java | 120 --
.../controller/utils/ReplicaGroupTestUtils.java | 140 --
.../validation/StorageQuotaCheckerTest.java | 36 +-
.../validation/ValidationManagerTest.java | 49 +-
pinot-core/pom.xml | 4 -
.../core/common/datatable/DataTableImplV2.java | 21 +-
.../realtime/HLRealtimeSegmentDataManager.java | 11 +-
.../realtime/LLRealtimeSegmentDataManager.java | 120 +-
.../manager/realtime/RealtimeTableDataManager.java | 2 +-
.../realtime/SegmentBuildTimeLeaseExtender.java | 8 +-
.../apache/pinot/core/data/order/OrderByUtils.java | 295 ++++
.../data/partition/MurmurPartitionFunction.java | 41 +-
.../pinot/core/data/readers/AvroRecordReader.java | 3 +-
.../apache/pinot/core/data/readers/FileFormat.java | 2 +-
.../pinot/core/data/readers/JSONRecordReader.java | 23 +-
.../pinot/core/data/readers/RecordReaderUtils.java | 44 +-
...dTransformer.java => CompositeTransformer.java} | 20 +-
.../recordtransformer/NullValueTransformer.java | 49 +
...ansformer.java => SanitizationTransformer.java} | 6 +-
.../core/data/table/ConcurrentIndexedTable.java | 137 ++
.../apache/pinot/core/data/table/IndexedTable.java | 69 +
.../java/org/apache/pinot/core/data/table/Key.java | 39 +-
.../org/apache/pinot/core/data/table/Record.java | 34 +-
.../pinot/core/data/table/SimpleIndexedTable.java | 126 ++
.../org/apache/pinot/core/data/table/Table.java | 68 +
.../pinot/core/indexsegment/IndexSegment.java | 2 +-
.../generator/SegmentGeneratorConfig.java | 31 +-
.../immutable/ImmutableSegmentImpl.java | 27 +-
.../immutable/ImmutableSegmentLoader.java | 6 +-
.../indexsegment/mutable/MutableSegmentImpl.java | 126 +-
.../impl/FixedByteSingleValueMultiColReader.java | 16 +-
.../core/io/util/FixedBitIntReaderWriter.java | 16 +-
.../core/io/util/FixedByteValueReaderWriter.java | 16 +-
.../org/apache/pinot/core/io/util/ValueReader.java | 3 +-
.../io/util/VarLengthBytesValueReaderWriter.java | 278 ++++
.../impl/FixedByteSingleValueMultiColWriter.java | 23 +-
.../pinot/core/minion/BackfillDateTimeColumn.java | 4 +-
.../apache/pinot/core/minion/SegmentConverter.java | 18 +-
.../dociditerators/MVScanDocIdIterator.java | 14 +-
.../dociditerators/SVScanDocIdIterator.java | 11 +
.../operator/docvalsets/TransformBlockValSet.java | 6 +
.../operator/filter/ExpressionFilterOperator.java | 21 +-
.../BaseDictionaryBasedPredicateEvaluator.java | 19 +-
.../BaseRawValueBasedPredicateEvaluator.java | 34 +-
.../predicate/EqualsPredicateEvaluatorFactory.java | 22 +
.../predicate/InPredicateEvaluatorFactory.java | 26 +
.../NotEqualsPredicateEvaluatorFactory.java | 22 +
.../predicate/NotInPredicateEvaluatorFactory.java | 27 +
.../filter/predicate/PredicateEvaluator.java | 20 +-
.../predicate/RangePredicateEvaluatorFactory.java | 51 +
.../operator/query/EmptySelectionOperator.java | 26 +-
.../core/operator/query/SelectionOnlyOperator.java | 126 ++
.../core/operator/query/SelectionOperator.java | 282 ----
.../operator/query/SelectionOrderByOperator.java | 179 +++
.../transform/TransformBlockDataFetcher.java | 5 +-
.../core/operator/transform/TransformOperator.java | 21 +-
.../function/MapValueTransformFunction.java | 95 ++
.../function/TransformFunctionFactory.java | 2 +-
.../apache/pinot/core/plan/CombinePlanNode.java | 7 +-
.../apache/pinot/core/plan/SelectionPlanNode.java | 33 +-
.../apache/pinot/core/plan/TransformPlanNode.java | 68 +-
.../query/executor/ServerQueryExecutorV1Impl.java | 12 +-
.../core/query/reduce/BrokerReduceService.java | 274 +++-
.../pinot/core/query/scheduler/QueryScheduler.java | 2 +-
.../core/query/selection/SelectionFetcher.java | 5 +-
.../query/selection/SelectionOperatorService.java | 247 +---
.../query/selection/SelectionOperatorUtils.java | 378 ++---
...ator.java => BytesSelectionColumnIterator.java} | 20 +-
...SelectionSingleValueColumnWithDictIterator.java | 5 -
.../iterator/StringSelectionColumnIterator.java | 10 +-
.../converter/RealtimeSegmentConverter.java | 21 +-
.../core/realtime/impl/RealtimeSegmentConfig.java | 20 +-
.../stream/AvroRecordToPinotRowGenerator.java | 6 +-
.../stream/PartitionLevelStreamConfig.java | 5 +-
.../pinot/core/realtime/stream/StreamConfig.java | 70 +-
.../realtime/stream/StreamConfigProperties.java | 8 +-
.../core/realtime/stream/StreamDataProducer.java | 17 +-
.../core/realtime/stream/StreamDataProvider.java | 46 +
.../realtime/stream/StreamDataServerStartable.java | 35 +-
.../segment/creator/ColumnIndexCreationInfo.java | 10 +-
.../RecordReaderSegmentCreationDataSource.java | 4 +-
.../creator/impl/SegmentColumnarIndexCreator.java | 152 +-
.../creator/impl/SegmentDictionaryCreator.java | 74 +-
.../impl/SegmentIndexCreationDriverImpl.java | 26 +-
.../fwd/SingleValueSortedForwardIndexCreator.java | 10 +-
.../inv/OffHeapBitmapInvertedIndexCreator.java | 26 +-
.../segment/index/loader/IndexLoadingConfig.java | 11 +
.../defaultcolumn/BaseDefaultColumnHandler.java | 8 +-
.../index/readers/ImmutableDictionaryReader.java | 18 +-
.../name/NormalizedDateSegmentNameGenerator.java | 14 +-
.../segment/store/SegmentLocalFSDirectory.java | 2 +-
.../VirtualColumnProviderFactory.java | 6 +-
.../startree/plan/StarTreeTransformPlanNode.java | 3 +-
.../java/org/apache/pinot/core/util/AvroUtils.java | 35 +
.../apache/pinot/core/util/ReplicationUtils.java | 42 +-
.../server/realtime/ControllerLeaderLocator.java | 153 +-
.../ServerSegmentCompletionProtocolHandler.java | 10 +-
.../function/FunctionExpressionEvaluatorTest.java | 1 -
.../realtime/LLRealtimeSegmentDataManagerTest.java | 3 +-
.../pinot/core/data/order/OrderByUtilsTest.java | 265 ++++
.../pinot/core/data/readers/PinotSegmentUtil.java | 74 +-
.../data/readers/RecordReaderSampleDataTest.java | 8 +-
.../pinot/core/data/readers/RecordReaderTest.java | 8 +-
.../core/data/readers/RecordReaderUtilsTest.java | 36 +-
.../recordtransformer/RecordTransformerTest.java | 6 +-
.../pinot/core/data/table/IndexedTableTest.java | 275 ++++
.../MutableSegmentImplAggregateMetricsTest.java | 33 +-
.../mutable/MutableSegmentImplTest.java | 3 +-
.../mutable/MutableSegmentImplTestUtils.java | 4 +-
.../util/VarLengthBytesValueReaderWriterTest.java | 170 +++
.../pinot/core/minion/SegmentConverterTest.java | 16 +-
.../function/ValueInTransformFunctionTest.java | 2 -
...adataAndDictionaryAggregationPlanMakerTest.java | 12 +
.../NoDictionaryEqualsPredicateEvaluatorsTest.java | 46 +-
.../NoDictionaryInPredicateEvaluatorTest.java | 45 +-
.../NoDictionaryRangePredicateEvaluatorTest.java | 57 +
.../predicate/PredicateEvaluatorTestUtils.java | 7 +
...ngeOfflineDictionaryPredicateEvaluatorTest.java | 29 +-
.../impl/fakestream/FakeStreamConfigUtils.java | 5 +-
.../core/realtime/stream/StreamConfigTest.java | 67 +-
.../core/segment/index/ColumnMetadataTest.java | 6 +
.../segment/index/SegmentMetadataImplTest.java | 6 +
.../SegmentV1V2ToV3FormatConverterTest.java | 6 +
.../SegmentGenerationWithTimeColumnTest.java | 36 +-
.../index/loader/SegmentPreProcessorTest.java | 6 +
.../startree/StarTreeIndexTestSegmentHelper.java | 6 +
.../pinot/core/startree/TestStarTreeMetadata.java | 6 +
.../pinot/core/startree/hll/HllFieldSizeTest.java | 2 +-
.../hll/SegmentWithHllIndexCreateHelper.java | 6 +
.../org/apache/pinot/core/util/CrcUtilsTest.java | 6 +
.../pinot/queries/BaseMultiValueQueriesTest.java | 6 +
.../pinot/queries/BaseSingleValueQueriesTest.java | 6 +
.../apache/pinot/queries/FastHllQueriesTest.java | 6 +
...nerSegmentAggregationMultiValueQueriesTest.java | 24 +-
...InnerSegmentSelectionMultiValueQueriesTest.java | 54 +-
...nnerSegmentSelectionSingleValueQueriesTest.java | 60 +-
...terSegmentAggregationMultiValueQueriesTest.java | 42 +-
.../apache/pinot/queries/TransformQueriesTest.java | 267 ++++
.../DefaultAggregationExecutorTest.java | 2 +-
.../selection/SelectionOperatorServiceTest.java | 239 +--
.../segments/v1/creator/DictionariesTest.java | 6 +
.../pinot/segments/v1/creator/IntArraysTest.java | 6 +
.../realtime/ControllerLeaderLocatorTest.java | 99 +-
.../test/java/org/apache/pinot/util/TestUtils.java | 96 +-
pinot-distribution/pinot-assembly.xml | 5 +
.../pinot/hadoop/job/DefaultControllerRestApi.java | 4 +-
.../pinot/hadoop/job/InternalConfigConstants.java | 46 +
.../pinot/hadoop/job/JobConfigConstants.java | 8 +-
.../pinot/hadoop/job/SegmentCreationJob.java | 17 +-
.../pinot/hadoop/job/SegmentPreprocessingJob.java | 323 +++--
.../apache/pinot/hadoop/job/SegmentTarPushJob.java | 61 +-
.../apache/pinot/hadoop/job/SegmentUriPushJob.java | 4 +-
.../hadoop/job/mappers/SegmentCreationMapper.java | 2 +-
.../job/mappers/SegmentPreprocessingMapper.java | 51 +-
.../job/partitioners/GenericPartitioner.java | 8 +-
.../job/reducers/SegmentPreprocessingReducer.java | 4 +-
.../pinot/hadoop/utils/JobPreparationHelper.java | 11 +
.../hadoop/io/DeleteExtraPushedSegmentsTest.java | 102 ++
pinot-integration-tests/pom.xml | 6 +-
...umSegmentAssignmentStrategyIntegrationTest.java | 138 --
.../tests/BaseClusterIntegrationTest.java | 71 +-
.../tests/BaseClusterIntegrationTestSet.java | 111 +-
.../tests/ClusterIntegrationTestUtils.java | 130 +-
.../pinot/integration/tests/ClusterTest.java | 64 +-
.../ControllerPeriodicTasksIntegrationTests.java | 87 +-
.../DeleteAPIHybridClusterIntegrationTest.java | 26 +-
...lakyConsumerRealtimeClusterIntegrationTest.java | 14 +-
...mentBuildPushOfflineClusterIntegrationTest.java | 91 +-
.../tests/HybridClusterIntegrationTest.java | 9 +-
...ridClusterIntegrationTestCommandLineRunner.java | 16 +-
.../tests/LLCRealtimeClusterIntegrationTest.java | 119 +-
...CRealtimeClusterSplitCommitIntegrationTest.java | 44 -
.../tests/MapTypeClusterIntegrationTest.java | 223 +++
...onaryAggregationPlanClusterIntegrationTest.java | 2 +-
.../MultiNodesOfflineClusterIntegrationTest.java | 7 +
.../tests/OfflineClusterIntegrationTest.java | 80 +-
.../tests/PinotURIUploadIntegrationTest.java | 15 +-
.../tests/RealtimeClusterIntegrationTest.java | 33 +-
....java => SegmentCompletionIntegrationTest.java} | 72 +-
.../tests/StarTreeClusterIntegrationTest.java | 2 +-
.../tests/StarTreeV2ClusterIntegrationTest.java | 2 +-
.../tests/UploadRefreshDeleteIntegrationTest.java | 2 +-
...e_Performance_2014_100k_subset.test_queries_10K | 2 +-
.../org/apache/pinot/minion/MinionStarter.java | 10 +-
.../BaseMultipleSegmentsConversionExecutor.java | 17 +-
.../BaseSingleSegmentConversionExecutor.java | 13 +-
.../apache/pinot/minion/metrics/MinionMetrics.java | 6 +-
pinot-perf/pom.xml | 4 +-
.../pinot/perf/BenchmarkDictionaryCreation.java | 12 +
.../apache/pinot/perf/BenchmarkIndexedTable.java | 218 +++
.../apache/pinot/perf/BenchmarkQueryEngine.java | 1 -
.../perf/BenchmarkRealtimeConsumptionSpeed.java | 8 +-
.../org/apache/pinot/perf/RealtimeStressTest.java | 8 +-
.../pinot/perf/StringDictionaryPerfTest.java | 86 +-
.../org/apache/pinot/server/conf/ServerConf.java | 7 +
.../server/request/ScheduledRequestHandler.java | 18 +-
.../apache/pinot/server/starter/ServerBuilder.java | 4 +-
.../server/starter/helix/HelixServerStarter.java | 12 +-
.../api/{resources => }/BaseResourceTest.java | 2 +-
.../api/{resources => }/TableSizeResourceTest.java | 2 +-
.../api/{resources => }/TablesResourceTest.java | 2 +-
.../request/ScheduledRequestHandlerTest.java | 42 +-
pinot-tools/pom.xml | 58 +-
.../apache/pinot/tools/AbstractBaseCommand.java | 5 +
.../org/apache/pinot/tools/HybridQuickstart.java | 39 +-
.../apache/pinot/tools/PinotSegmentRebalancer.java | 242 ---
.../apache/pinot/tools/PinotTableRebalancer.java | 53 +
.../org/apache/pinot/tools/RealtimeQuickStart.java | 29 +-
.../pinot/tools/admin/PinotAdministrator.java | 43 +-
.../command/BackfillDateTimeColumnCommand.java | 2 +-
.../tools/admin/command/CreateSegmentCommand.java | 74 +-
.../tools/admin/command/RebalanceTableCommand.java | 101 +-
.../tools/admin/command/StartKafkaCommand.java | 11 +-
.../admin/command/StreamAvroIntoKafkaCommand.java | 20 +-
.../tools/admin/command/UploadSegmentCommand.java | 6 +-
.../pinot/tools/backfill/BackfillSegmentUtils.java | 7 +-
.../pinot/tools/perf/PerfBenchmarkDriver.java | 49 +-
.../pinot/tools/perf/PerfBenchmarkDriverConf.java | 63 -
.../tools/query/comparison/ClusterStarter.java | 4 +-
.../query/comparison/SegmentInfoProvider.java | 43 +-
.../realtime/provisioning/MemoryEstimator.java | 18 +-
.../pinot/tools/streams/AirlineDataStream.java | 17 +-
.../pinot/tools/streams/MeetupRsvpStream.java | 28 +-
.../tools/tuner/CollectMetadataForIndexTuning.java | 97 ++
.../tools/tuner/EntriesScannedQuantileReport.java | 89 ++
.../pinot/tools/tuner/IndexTunerCommand.java | 199 +++
.../pinot/tools/tuner/driver/TunerDriver.java | 209 +++
.../meta/manager/JsonFileMetaManagerImpl.java | 247 ++++
.../tools/tuner/meta/manager/MetaManager.java | 59 +
.../manager/collector/ColStatsAccumulatorObj.java | 156 ++
.../manager/collector/CompressedFilePathIter.java | 123 ++
.../tuner/meta/manager/collector/PathWrapper.java | 74 +
.../collector/SegmentMetadataCollector.java | 272 ++++
.../pinot/tools/tuner/query/src/InputIterator.java | 22 +-
.../tuner/query/src/LogInputIteratorImpl.java | 165 +++
.../query/src/parser/BrokerLogParserImpl.java | 76 +
.../tools/tuner/query/src/parser/QueryParser.java | 24 +-
.../src/stats/wrapper/AbstractQueryStats.java | 16 +-
.../stats/wrapper/IndexSuggestQueryStatsImpl.java | 117 ++
.../tools/tuner/strategy/AbstractAccumulator.java | 64 +
.../tools/tuner/strategy/FrequencyAccumulator.java | 36 +-
.../pinot/tools/tuner/strategy/FrequencyImpl.java | 200 +++
.../tuner/strategy/ParseBasedAccumulator.java | 62 +
.../tools/tuner/strategy/ParserBasedImpl.java | 453 ++++++
.../strategy/QuantileAnalysisAccumulator.java | 88 ++
.../tools/tuner/strategy/QuantileAnalysisImpl.java | 351 +++++
.../pinot/tools/tuner/strategy/TuningStrategy.java | 60 +
.../airlineStats_offline_table_config.json | 0
.../docker/airlineStats_realtime_table_config.json | 38 +
.../airlineStats_realtime_table_config.json | 0
.../meetupRsvp_realtime_table_config.json | 2 +-
.../airlineStats_realtime_table_config.json | 38 +
.../meetupRsvp_realtime_table_config.json | 5 +-
.../meetupRsvp_realtime_table_config.json | 2 +-
.../apache/pinot/transport/netty/NettyServer.java | 59 +-
.../pinot/transport/common/ServerInstanceTest.java | 62 +-
.../pinot/transport/netty/NettyTestUtils.java | 8 +-
.../transport/perf/ScatterGatherPerfClient.java | 5 +-
.../transport/perf/ScatterGatherPerfServer.java | 8 +-
.../pool/AsyncPoolResourceManagerAdapterTest.java | 14 +-
.../pinot/transport/pool/KeyedPoolImplTest.java | 25 +-
.../transport/scattergather/ScatterGatherTest.java | 24 +-
pom.xml | 117 +-
thirdeye/README.md | 3 +
thirdeye/docs/configuration.rst | 210 +++
thirdeye/docs/index.rst | 2 +-
thirdeye/docs/production.rst | 27 -
thirdeye/docs/quick_start.rst | 8 +-
.../app/mirage/factories/alert.js | 6 +-
.../app/pods/aiavailability/controller.js | 102 ++
.../app/pods/aiavailability/route.js | 198 +++
.../app/pods/aiavailability/template.hbs | 60 +
.../app/pods/anomalies/controller.js | 14 +-
.../app/pods/application/route.js | 12 +-
.../app/pods/components/alert-details/component.js | 50 +-
.../app/pods/components/alert-details/template.hbs | 127 +-
.../pods/components/anomaly-summary/component.js | 1 +
.../pods/components/anomaly-summary/template.hbs | 11 +-
.../pods/components/detection-health/component.js | 197 +++
.../pods/components/detection-health/template.hbs | 84 ++
.../pods/components/detection-yaml/component.js | 11 +-
.../pods/components/detection-yaml/template.hbs | 5 +-
.../app/pods/components/stats-cards/component.js | 42 +-
.../app/pods/components/stats-cards/template.hbs | 61 +-
.../app/pods/components/yaml-editor/template.hbs | 2 +-
.../thirdeye-frontend/app/pods/home/index/route.js | 13 +
.../app/pods/home/share-dashboard/route.js | 6 +-
thirdeye/thirdeye-frontend/app/pods/login/route.js | 25 +-
.../app/pods/manage/alerts/index/route.js | 22 +-
.../app/pods/manage/alerts/route.js | 13 +-
.../app/pods/manage/explore/route.js | 91 +-
.../app/pods/manage/explore/template.hbs | 2 +
.../thirdeye-frontend/app/pods/manage/route.js | 13 +-
.../app/pods/manage/yaml/route.js | 85 +-
.../app/pods/manage/yaml/template.hbs | 4 +-
.../thirdeye-frontend/app/pods/rootcause/route.js | 13 +-
.../app/pods/self-serve/create-alert/template.hbs | 2 +-
.../self-serve/import-sql-metric/controller.js | 4 +-
.../app/pods/services/query-cache/service.js | 8 +-
thirdeye/thirdeye-frontend/app/router.js | 2 +
.../styles/components/range-pill-selectors.scss | 4 +
.../app/styles/shared/_styles.scss | 118 ++
thirdeye/thirdeye-frontend/app/utils/anomaly.js | 24 +-
.../thirdeye-frontend/app/utils/api/anomaly.js | 13 +-
thirdeye/thirdeye-frontend/app/utils/constants.js | 1 +
.../app/utils/manage-alert-utils.js | 21 +-
thirdeye/thirdeye-frontend/app/utils/utils.js | 5 +-
thirdeye/thirdeye-frontend/config/environment.js | 2 +
thirdeye/thirdeye-frontend/ember-cli-build.js | 7 +
thirdeye/thirdeye-frontend/package.json | 1 +
.../acceptance/self-serve-alert-tuning-test.js | 2 -
.../acceptance/self-serve-import-metric-test.js | 3 +-
.../pods/components/yaml-editor/component-test.js | 2 +-
thirdeye/thirdeye-frontend/yarn.lock | 8 +
thirdeye/thirdeye-pinot/config/dashboard.yml | 3 +-
.../thirdeye/alert/commons/AnomalyFeedConfig.java | 6 +-
.../alert/content/EmailContentFormatter.java | 62 -
.../EmailContentFormatterConfiguration.java | 128 --
.../thirdeye/alert/fetcher/BaseAnomalyFetcher.java | 4 +-
.../apache/pinot/thirdeye/anomaly/AnomalyType.java | 22 +-
.../thirdeye/anomaly/alert/AlertJobContext.java | 1 +
.../thirdeye/anomaly/alert/AlertTaskInfo.java | 1 +
...nshotHelper.java => AlertScreenshotHelper.java} | 56 +-
.../anomaly/alert/util/AnomalyReportGenerator.java | 4 +-
.../thirdeye/anomaly/alert/util/EmailHelper.java | 29 +-
.../anomaly/alert/v2/AlertTaskRunnerV2.java | 40 +-
.../anomaly/monitor/MonitorTaskRunner.java | 23 +-
.../tasks/FunctionCreationOnboardingTask.java | 7 +-
.../onboard/tasks/NotificationOnboardingTask.java | 34 +-
.../pinot/thirdeye/anomaly/task/TaskDriver.java | 47 +-
.../anomaly/task/TaskDriverConfiguration.java | 9 +
.../anomaly/utils/ThirdeyeMetricsUtil.java | 3 +
.../AbstractModularizedAnomalyFunction.java | 2 +-
.../dashboard/DetectionPreviewConfiguration.java | 50 +
.../dashboard/ThirdEyeDashboardApplication.java | 5 +-
.../dashboard/ThirdEyeDashboardConfiguration.java | 9 +
.../resources/AnomalyFlattenResource.java | 228 +++
.../dashboard/resources/v2/AnomaliesResource.java | 24 +-
.../dashboard/resources/v2/DataResource.java | 13 +-
.../resources/v2/RootCauseMetricResource.java | 4 +-
.../dashboard/resources/v2/pojo/SearchFilters.java | 2 -
.../v2/rootcause/MetricEntityFormatter.java | 4 +-
.../datalayer/bao/MergedAnomalyResultManager.java | 2 +
.../pinot/thirdeye/datalayer/bao/TaskManager.java | 2 +
.../bao/jdbc/MergedAnomalyResultManagerImpl.java | 9 +
.../datalayer/bao/jdbc/TaskManagerImpl.java | 7 +
.../thirdeye/datalayer/dto/AnomalyFunctionDTO.java | 7 +-
.../thirdeye/datalayer/dto/DatasetConfigDTO.java | 9 +
.../datalayer/entity/DatasetConfigIndex.java | 10 +
.../thirdeye/datalayer/pojo/DatasetConfigBean.java | 58 +-
.../datalayer/pojo/MergedAnomalyResultBean.java | 14 +
.../{StringUtils.java => ThirdEyeStringUtils.java} | 2 +-
.../datasource/pinot/PinotDataSourceMaxTime.java | 7 +-
.../datasource/pinot/PinotThirdEyeDataSource.java | 6 +-
.../pinot/thirdeye/datasource/pinot/PqlUtils.java | 9 +-
.../datasource/sql/SqlResponseCacheLoader.java | 37 +
.../pinot/thirdeye/datasource/sql/SqlUtils.java | 3 +
.../pinot/thirdeye/detection/ConfigUtils.java | 29 +
.../pinot/thirdeye/detection/DataProvider.java | 5 +
.../thirdeye/detection/DefaultDataProvider.java | 45 +-
.../thirdeye/detection/DetectionPipeline.java | 35 +-
.../detection/DetectionPipelineLoader.java | 10 +-
.../detection/DetectionPipelineTaskRunner.java | 11 +-
.../thirdeye/detection/DetectionResource.java | 15 +-
.../pinot/thirdeye/detection/DetectionUtils.java | 15 -
.../pinot/thirdeye/detection/ExpressionParser.java | 141 ++
.../pinot/thirdeye/detection/ModelRetuneFlow.java | 13 +-
.../alert/DetectionAlertFilterNotification.java | 62 +
.../alert/DetectionAlertFilterResult.java | 16 +-
.../detection/alert/DetectionAlertJob.java | 30 +-
.../detection/alert/DetectionAlertTaskFactory.java | 13 +-
.../alert/StatefulDetectionAlertFilter.java | 40 +
.../filter/DimensionDetectionAlertFilter.java | 25 +-
.../alert/filter/PerUserDimensionAlertFilter.java | 19 +-
.../ToAllRecipientsDetectionAlertFilter.java | 36 +-
.../alert/scheme/DetectionEmailAlerter.java | 189 ++-
.../detection/algorithm/DimensionWrapper.java | 19 +-
.../thirdeye/detection/algorithm/MergeWrapper.java | 126 +-
.../algorithm/stage/AnomalyDetectionStage.java | 38 -
.../stage/AnomalyDetectionStageWrapper.java | 217 ---
.../algorithm/stage/AnomalyFilterStage.java | 38 -
.../algorithm/stage/AnomalyFilterStageWrapper.java | 105 --
.../stage/BaselineRuleDetectionStage.java | 144 --
.../algorithm/stage/BaselineRuleFilterStage.java | 119 --
.../algorithm/stage/GrouperStageWrapper.java | 89 --
.../stage/StaticAnomalyDetectionStage.java | 169 ---
.../algorithm/stage/StaticAnomalyFilterStage.java | 62 -
.../stage/ThresholdRuleDetectionStage.java | 107 --
.../algorithm/stage/ThresholdRuleFilterStage.java | 90 --
.../annotation/registry/DetectionRegistry.java | 7 +
.../AbsoluteChangeRuleAnomalyFilter.java | 3 +-
.../components/MeanVarianceRuleDetector.java | 290 ++++
.../PercentageChangeRuleAnomalyFilter.java | 10 +-
.../SitewideImpactRuleAnomalyFilter.java | 5 +-
.../components/ThresholdRuleAnomalyFilter.java | 14 +-
.../components/TriggerConditionGrouper.java | 29 +-
.../thirdeye/detection/health/DetectionHealth.java | 4 +-
.../detection/health/DetectionTaskStatus.java | 14 +
.../spec/MeanVarianceRuleDetectorSpec.java | 60 +
.../thirdeye/detection/spi/model/AnomalySlice.java | 52 +-
.../validators/DetectionConfigValidator.java | 43 +-
.../detection/wrapper/AnomalyDetectorWrapper.java | 10 +-
.../wrapper/BaselineFillingMergeWrapper.java | 19 +-
.../wrapper/ChildKeepingMergeWrapper.java | 17 +-
.../wrapper/EntityAnomalyMergeWrapper.java | 77 +
.../thirdeye/detection/wrapper/GrouperWrapper.java | 16 +-
.../detection/yaml/DetectionConfigTuner.java | 8 -
.../thirdeye/detection/yaml/YamlResource.java | 85 +-
.../yaml/translator/DetectionConfigTranslator.java | 59 +-
.../translator/DetectionMetricAttributeHolder.java | 13 +-
.../translator/SubscriptionConfigTranslator.java | 12 +-
.../pinot/thirdeye/formatter/DTOFormatter.java | 38 +
.../formatter/DetectionAlertConfigFormatter.java | 46 +
.../formatter/DetectionConfigFormatter.java | 256 ++++
.../commons/EmailEntity.java | 2 +-
.../content/BaseNotificationContent.java} | 420 ++----
.../notification/content/NotificationContent.java | 55 +
.../content/templates/EntityGroupKeyContent.java | 202 +++
.../templates/HierarchicalAnomaliesContent.java} | 96 +-
.../content/templates/MetricAnomaliesContent.java} | 60 +-
.../templates/OnboardingNotificationContent.java} | 43 +-
.../formatter/ADContentFormatterContext.java} | 8 +-
.../formatter/AlertContentFormatterFactory.java} | 14 +-
.../formatter/channels/EmailContentFormatter.java | 125 ++
.../apache/pinot/thirdeye/util/ThirdEyeUtils.java | 21 +-
.../detector/entity-groupkey-anomaly-report.ftl | 152 ++
...ly-report.ftl => metric-anomalies-template.ftl} | 0
.../src/main/resources/schema/create-schema.sql | 2 +
.../commons/TestEmailContentFormatterFactory.java | 39 -
.../alert/fetcher/TestContinuumAnomalyFetcher.java | 7 +-
.../transform/testAnomalyRemovalFunction.java | 2 +-
.../pinot/thirdeye/datalayer/DaoTestUtils.java | 53 +
.../bao/TestMergedAnomalyResultManager.java | 66 +
...UtilsTest.java => ThirdEyeStringUtilsTest.java} | 4 +-
.../thirdeye/datasource/pinot/PqlUtilsTest.java | 4 +-
.../resources/TestAnomalyFlattenResource.java | 144 ++
.../pinot/thirdeye/detection/DataProviderTest.java | 2 +-
.../thirdeye/detection/DetectionTestUtils.java | 9 +
.../thirdeye/detection/alert/SendAlertTest.java | 2 +-
.../detection/alert/filter/AlertFilterUtils.java | 65 +
.../filter/DimensionDetectionAlertFilterTest.java | 33 +-
.../filter/PerUserDimensionAlertFilterTest.java | 38 +-
.../ToAllRecipientsDetectionAlertFilterTest.java | 43 +-
.../DetectionTimeWindowSuppressorTest.java | 14 +-
.../detection/algorithm/DimensionWrapperTest.java | 32 +-
.../algorithm/LegacyDimensionWrapperTest.java | 12 +-
.../detection/algorithm/MergeWrapperTest.java | 58 +-
.../stage/AnomalyDetectionStageWrapperTest.java | 87 --
.../stage/BaselineRuleDetectionStageTest.java | 141 --
.../algorithm/stage/BaselineRuleFilterTest.java | 173 ---
.../stage/ThresholdRuleDetectionStageTest.java | 93 --
.../algorithm/stage/ThresholdRuleFilterTest.java | 167 ---
.../AbsoluteChangeRuleAnomalyFilterTest.java | 25 +-
.../detection/components/ExpressionParserTest.java | 90 ++
.../components/MeanVarianceRuleDetectorTest.java | 175 +++
.../PercentageChangeRuleAnomalyFilterTest.java | 60 +-
.../SitewideImpactRuleAnomalyFilterTest.java | 20 +-
.../components/ThresholdRuleAnomalyFilterTest.java | 56 +-
.../components/TriggerConditionGrouperTest.java | 40 +-
.../wrapper/AnomalyDetectorWrapperTest.java | 4 +-
.../wrapper/EntityAnomalyMergeWrapperTest.java | 136 ++
.../thirdeye/detection/yaml/YamlResourceTest.java | 6 +-
.../formatter/DetectionConfigFormatterTest.java | 86 ++
.../integration/NotificationTaskSchedulerTest.java | 183 +++
.../notification/ContentFormatterUtils.java | 48 +
.../templates/TestEntityGroupKeyContent.java | 293 ++++
.../TestHierarchicalAnomaliesContent.java} | 42 +-
.../templates/TestMetricAnomaliesContent.java} | 41 +-
.../TestOnboardingNotificationContent.java} | 39 +-
.../TestAlertContentFormatterFactory.java | 41 +
.../tools/RunAdhocDatabaseQueriesTool.java | 32 +-
.../thirdeye/detection/algorithm/timeseries-2y.csv | 107 ++
.../detection/yaml/alertconfig/alert-config-5.yaml | 3 +
.../compositePipelineTranslatorTestResult-1.json | 108 +-
.../compositePipelineTranslatorTestResult-2.json | 1 +
.../compositePipelineTranslatorTestResult-4.json | 1 +
.../compositePipelineTranslatorTestResult-5.json | 4 +
.../src/test/resources/sample-alert-config.yml | 38 +
.../src/test/resources/sample-detection-config.yml | 19 +
...st-entity-groupby-email-content-formatter.html} | 86 +-
...by-with-whitelist-email-content-formatter.html} | 104 +-
...st-hierarchical-metric-anomalies-template.html} | 0
...er.html => test-metric-anomalies-template.html} | 0
784 files changed, 35240 insertions(+), 21789 deletions(-)
create mode 100644 docker/docker-compose.yml
delete mode 100644 docs/multitenancy.rst
create mode 100644 docs/tuning_realtime_performance.rst
create mode 100644 kubernetes/README.md
create mode 100644 kubernetes/examples/gke/skaffold/README.md
copy docker/docker-build.sh => kubernetes/examples/gke/skaffold/cleanup.sh (53%)
copy docker/docker-build.sh => kubernetes/examples/gke/skaffold/gke-storageclass-kafka-pd.yml (52%)
mode change 100755 => 100644
copy docker/docker-build.sh => kubernetes/examples/gke/skaffold/gke-storageclass-pinot-controller-pd.yml (52%)
mode change 100755 => 100644
copy docker/docker-build.sh => kubernetes/examples/gke/skaffold/gke-storageclass-pinot-server-pd.yml (52%)
mode change 100755 => 100644
copy docker/docker-build.sh => kubernetes/examples/gke/skaffold/gke-storageclass-zk-pd.yml (52%)
mode change 100755 => 100644
create mode 100644 kubernetes/examples/gke/skaffold/kafka.yml
create mode 100644 kubernetes/examples/gke/skaffold/pinot-broker.yml
create mode 100644 kubernetes/examples/gke/skaffold/pinot-controller.yml
create mode 100644 kubernetes/examples/gke/skaffold/pinot-example-loader.yml
create mode 100644 kubernetes/examples/gke/skaffold/pinot-server.yml
copy docker/docker-build.sh => kubernetes/examples/gke/skaffold/query-pinot-data.sh (52%)
create mode 100755 kubernetes/examples/gke/skaffold/setup.sh
copy docker/docker-build.sh => kubernetes/examples/gke/skaffold/skaffold.yaml (52%)
mode change 100755 => 100644
create mode 100644 kubernetes/examples/gke/skaffold/zookeeper.yml
copy pinot-api/src/main/java/org/apache/pinot/client/{PinotClientTransport.java => Request.java} (61%)
create mode 100644 pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/PinotQueryParserFactory.java
copy pinot-common/src/main/java/org/apache/pinot/pql/parsers/AbstractCompiler.java => pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/PinotQueryRequest.java (60%)
delete mode 100644 pinot-broker/src/main/java/org/apache/pinot/broker/routing/CfgBasedRouting.java
delete mode 100644 pinot-broker/src/test/java/org/apache/pinot/broker/broker/EmptyBrokerOnlineOfflineStateModelFactory.java
delete mode 100644 pinot-broker/src/test/java/org/apache/pinot/broker/broker/EmptySegmentOnlineOfflineStateModelFactory.java
create mode 100644 pinot-broker/src/test/java/org/apache/pinot/broker/request/PqlAndCalciteSqlCompatibilityTest.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/assignment/InstancePartitions.java
rename pinot-common/src/main/java/org/apache/pinot/common/{utils => assignment}/InstancePartitionsType.java (75%)
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/assignment/InstancePartitionsUtils.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/config/CompletionConfig.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/config/Instance.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/config/instance/InstanceAssignmentConfig.java
copy pinot-controller/src/main/java/org/apache/pinot/controller/LeadershipChangeSubscriber.java => pinot-common/src/main/java/org/apache/pinot/common/config/instance/InstanceAssignmentConfigMapChildKeyHandler.java (70%)
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/config/instance/InstanceAssignmentConfigUtils.java
copy pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/sharding/SegmentAssignmentStrategy.java => pinot-common/src/main/java/org/apache/pinot/common/config/instance/InstanceConstraintConfig.java (54%)
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/config/instance/InstanceReplicaGroupPartitionConfig.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/config/instance/InstanceTagPoolConfig.java
delete mode 100644 pinot-common/src/main/java/org/apache/pinot/common/restlet/resources/RebalanceResult.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/utils/URIUtils.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/utils/helix/LeadControllerUtils.java
copy pinot-common/src/main/java/org/apache/pinot/{pql => }/parsers/AbstractCompiler.java (81%)
create mode 100644 pinot-common/src/main/java/org/apache/pinot/parsers/utils/BrokerRequestComparisonUtils.java
copy pinot-common/src/main/java/org/apache/pinot/{pql/parsers/AbstractCompiler.java => sql/parsers/CalciteSqlCompiler.java} (61%)
create mode 100644 pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
copy pinot-controller/src/main/java/org/apache/pinot/controller/LeadershipChangeSubscriber.java => pinot-common/src/main/java/org/apache/pinot/sql/parsers/SqlCompilationException.java (71%)
create mode 100644 pinot-common/src/test/java/org/apache/pinot/common/utils/URIUtilsTest.java
create mode 100644 pinot-common/src/test/java/org/apache/pinot/sql/parsers/CalciteSqlCompilerTest.java
copy pinot-common/src/test/resources/{pql_queries.list => sql_queries.list} (87%)
delete mode 100644 pinot-connectors/pinot-connector-kafka-0.9/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaStarterUtils.java
create mode 100644 pinot-connectors/pinot-connector-kafka-0.9/src/main/java/org/apache/pinot/core/realtime/impl/kafka/server/KafkaDataProducer.java
create mode 100644 pinot-connectors/pinot-connector-kafka-0.9/src/main/java/org/apache/pinot/core/realtime/impl/kafka/server/KafkaDataServerStartable.java
copy docker/docker-build.sh => pinot-connectors/pinot-connector-kafka-0.9/src/main/resources/META-INF/services/org.apache.pinot.core.realtime.stream.StreamConsumerFactory (52%)
mode change 100755 => 100644
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/README.md
copy pinot-connectors/{pinot-connector-kafka-0.9 => pinot-connector-kafka-2.0}/pom.xml (59%)
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaConsumerFactory.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaMessageBatch.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaPartitionLevelConnectionHandler.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaPartitionLevelConsumer.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaPartitionLevelStreamConfig.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaStreamLevelConsumer.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaStreamLevelConsumerManager.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaStreamLevelStreamConfig.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaStreamMetadataProvider.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/server/KafkaDataProducer.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/main/java/org/apache/pinot/core/realtime/impl/kafka2/server/KafkaDataServerStartable.java
copy docker/docker-build.sh => pinot-connectors/pinot-connector-kafka-2.0/src/main/resources/META-INF/services/org.apache.pinot.core.realtime.stream.StreamConsumerFactory (52%)
mode change 100755 => 100644
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/test/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaPartitionLevelConsumerTest.java
copy pinot-connectors/{pinot-connector-kafka-0.9/src/test/java/org/apache/pinot/core/realtime/impl/kafka/KafkaLowLevelStreamConfigTest.java => pinot-connector-kafka-2.0/src/test/java/org/apache/pinot/core/realtime/impl/kafka2/KafkaPartitionLevelStreamConfigTest.java} (82%)
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/test/java/org/apache/pinot/core/realtime/impl/kafka2/utils/EmbeddedZooKeeper.java
create mode 100644 pinot-connectors/pinot-connector-kafka-2.0/src/test/java/org/apache/pinot/core/realtime/impl/kafka2/utils/MiniKafkaCluster.java
create mode 100644 pinot-connectors/pinot-connector-kafka-base/README.md
copy pinot-connectors/{ => pinot-connector-kafka-base}/pom.xml (60%)
rename pinot-connectors/{pinot-connector-kafka-0.9 => pinot-connector-kafka-base}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaAvroMessageDecoder.java (100%)
rename pinot-connectors/{pinot-connector-kafka-0.9 => pinot-connector-kafka-base}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaJSONMessageDecoder.java (100%)
create mode 100644 pinot-connectors/pinot-connector-kafka-base/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaStarterUtils.java
rename pinot-connectors/{pinot-connector-kafka-0.9 => pinot-connector-kafka-base}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaStreamConfigProperties.java (96%)
copy pinot-core/src/main/java/org/apache/pinot/core/io/util/ValueReader.java => pinot-connectors/pinot-connector-kafka-base/src/main/java/org/apache/pinot/core/realtime/impl/kafka/MessageAndOffset.java (56%)
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/ControllerLeadershipManager.java
create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/LeadControllerManager.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/api/pojos/Instance.java
create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotInstanceAssignmentRestletResource.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/api/upload/SegmentValidatorResponse.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/ControllerRequestBuilderUtil.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/EmptyBrokerOnlineOfflineStateModelFactory.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/EmptySegmentOnlineOfflineStateModelFactory.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/TableRebalancer.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/InstancePartitions.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/InstancePartitionsUtils.java
create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/HashBasedRotateInstanceConstraintApplier.java
create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/GrouperStage.java => pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceConstraintApplier.java (57%)
create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceTagPoolSelector.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineBalanceNumSegmentAssignmentStrategy.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineReplicaGroupSegmentAssignmentStrategy.java
create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/OfflineSegmentAssignment.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeBalanceNumSegmentAssignmentStrategy.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeReplicaGroupSegmentAssignmentStrategy.java
create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/RealtimeSegmentAssignment.java
rename pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/{SegmentAssignmentStrategy.java => SegmentAssignment.java} (59%)
rename pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/{sharding/SegmentAssignmentStrategy.java => assignment/segment/SegmentAssignmentFactory.java} (53%)
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentStrategyFactory.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/periodictask/ControllerPeriodicTaskScheduler.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/DefaultRebalanceSegmentStrategy.java
rename pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/{RebalanceUserConfigConstants.java => RebalanceConfigConstants.java} (59%)
create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/RebalanceResult.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/RebalanceSegmentStrategy.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/RebalanceSegmentStrategyFactory.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ReplicaGroupRebalanceSegmentStrategy.java
create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/sharding/BalanceNumSegmentAssignmentStrategy.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/sharding/BucketizedSegmentStrategy.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/sharding/RandomAssignmentStrategy.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/sharding/ReplicaGroupSegmentAssignmentStrategy.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/sharding/SegmentAssignmentStrategyFactory.java
create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/statemodel/LeadControllerResourceMasterSlaveStateModelFactory.java
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/LeadControllerManagerTest.java
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/AccessControlTest.java (87%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/FileUploadPathProviderTest.java (95%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/PinotFileUploadTest.java (79%)
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotInstanceAssignmentRestletResourceTest.java
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotInstanceRestletResourceTest.java
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/PinotSchemaRestletResourceTest.java (98%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/PinotSegmentRestletResourceTest.java (85%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/PinotTableRestletResourceTest.java (84%)
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotTenantRestletResourceTest.java
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/SegmentCompletionProtocolDeserTest.java (99%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/SegmentCompletionUtilsTest.java (96%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/ServerTableSizeReaderTest.java (98%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/TableSizeReaderTest.java (99%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{resources => }/TableViewsTest.java (93%)
delete mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotInstanceRestletResourceTest.java
delete mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotTenantRestletResourceTest.java
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentTest.java
rename pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/{OfflineBalanceNumSegmentAssignmentStrategyTest.java => OfflineNonReplicaGroupSegmentAssignmentTest.java} (74%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/{OfflineReplicaGroupSegmentAssignmentStrategyTest.java => OfflineReplicaGroupSegmentAssignmentTest.java} (63%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/{RealtimeBalanceNumSegmentAssignmentStrategyTest.java => RealtimeNonReplicaGroupSegmentAssignmentTest.java} (73%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/{RealtimeReplicaGroupSegmentAssignmentStrategyTest.java => RealtimeReplicaGroupSegmentAssignmentTest.java} (61%)
delete mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/rebalance/DefaultRebalanceStrategyTest.java
delete mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/rebalance/ReplicaGroupRebalanceStrategyTest.java
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancerClusterTest.java
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancerTest.java
delete mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/relocation/RealtimeSegmentRelocatorTest.java
delete mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/sharding/SegmentAssignmentStrategyTest.java
delete mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/util/TableRebalancerTest.java
delete mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/utils/ReplicaGroupTestUtils.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/data/order/OrderByUtils.java
rename pinot-core/src/main/java/org/apache/pinot/core/data/recordtransformer/{CompoundTransformer.java => CompositeTransformer.java} (70%)
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/data/recordtransformer/NullValueTransformer.java
rename pinot-core/src/main/java/org/apache/pinot/core/data/recordtransformer/{SanitationTransformer.java => SanitizationTransformer.java} (92%)
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/data/table/ConcurrentIndexedTable.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/data/table/IndexedTable.java
copy pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerMetrics.java => pinot-core/src/main/java/org/apache/pinot/core/data/table/Key.java (56%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/BaseDetectionStage.java => pinot-core/src/main/java/org/apache/pinot/core/data/table/Record.java (64%)
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/data/table/SimpleIndexedTable.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/data/table/Table.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/io/util/VarLengthBytesValueReaderWriter.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/query/SelectionOnlyOperator.java
delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/query/SelectionOperator.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/query/SelectionOrderByOperator.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/MapValueTransformFunction.java
copy pinot-core/src/main/java/org/apache/pinot/core/query/selection/iterator/{StringSelectionColumnIterator.java => BytesSelectionColumnIterator.java} (69%)
rename pinot-common/src/main/java/org/apache/pinot/pql/parsers/AbstractCompiler.java => pinot-core/src/main/java/org/apache/pinot/core/realtime/stream/StreamDataProducer.java (67%)
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/realtime/stream/StreamDataProvider.java
copy pinot-controller/src/main/java/org/apache/pinot/controller/LeadershipChangeSubscriber.java => pinot-core/src/main/java/org/apache/pinot/core/realtime/stream/StreamDataServerStartable.java (55%)
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/data/order/OrderByUtilsTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/data/table/IndexedTableTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/io/util/VarLengthBytesValueReaderWriterTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/queries/TransformQueriesTest.java
create mode 100644 pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/InternalConfigConstants.java
create mode 100644 pinot-hadoop/src/test/java/org/apache/pinot/hadoop/io/DeleteExtraPushedSegmentsTest.java
delete mode 100644 pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BalanceNumSegmentAssignmentStrategyIntegrationTest.java
delete mode 100644 pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/LLCRealtimeClusterSplitCommitIntegrationTest.java
create mode 100644 pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MapTypeClusterIntegrationTest.java
rename pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/{SegmentCompletionIntegrationTests.java => SegmentCompletionIntegrationTest.java} (87%)
create mode 100644 pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkIndexedTable.java
rename pinot-server/src/test/java/org/apache/pinot/server/api/{resources => }/BaseResourceTest.java (99%)
rename pinot-server/src/test/java/org/apache/pinot/server/api/{resources => }/TableSizeResourceTest.java (98%)
rename pinot-server/src/test/java/org/apache/pinot/server/api/{resources => }/TablesResourceTest.java (99%)
delete mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/PinotSegmentRebalancer.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/PinotTableRebalancer.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/CollectMetadataForIndexTuning.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/EntriesScannedQuantileReport.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/IndexTunerCommand.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/driver/TunerDriver.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/meta/manager/JsonFileMetaManagerImpl.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/meta/manager/MetaManager.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/meta/manager/collector/ColStatsAccumulatorObj.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/meta/manager/collector/CompressedFilePathIter.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/meta/manager/collector/PathWrapper.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/meta/manager/collector/SegmentMetadataCollector.java
copy pinot-controller/src/main/java/org/apache/pinot/controller/LeadershipChangeSubscriber.java => pinot-tools/src/main/java/org/apache/pinot/tools/tuner/query/src/InputIterator.java (63%)
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/query/src/LogInputIteratorImpl.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/query/src/parser/BrokerLogParserImpl.java
rename pinot-controller/src/main/java/org/apache/pinot/controller/LeadershipChangeSubscriber.java => pinot-tools/src/main/java/org/apache/pinot/tools/tuner/query/src/parser/QueryParser.java (63%)
copy pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/sharding/SegmentAssignmentStrategyEnum.java => pinot-tools/src/main/java/org/apache/pinot/tools/tuner/query/src/stats/wrapper/AbstractQueryStats.java (73%)
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/query/src/stats/wrapper/IndexSuggestQueryStatsImpl.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/strategy/AbstractAccumulator.java
copy pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerMetrics.java => pinot-tools/src/main/java/org/apache/pinot/tools/tuner/strategy/FrequencyAccumulator.java (53%)
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/strategy/FrequencyImpl.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/strategy/ParseBasedAccumulator.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/strategy/ParserBasedImpl.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/strategy/QuantileAnalysisAccumulator.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/strategy/QuantileAnalysisImpl.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/tuner/strategy/TuningStrategy.java
copy pinot-tools/src/main/resources/sample_data/{ => docker}/airlineStats_offline_table_config.json (100%)
create mode 100644 pinot-tools/src/main/resources/sample_data/docker/airlineStats_realtime_table_config.json
copy pinot-tools/src/main/resources/sample_data/{ => kafka_0.9}/airlineStats_realtime_table_config.json (100%)
copy pinot-tools/src/main/resources/sample_data/{ => kafka_0.9}/meetupRsvp_realtime_table_config.json (94%)
create mode 100644 pinot-tools/src/main/resources/sample_data/kafka_2.0/airlineStats_realtime_table_config.json
copy pinot-tools/src/main/resources/sample_data/{ => kafka_2.0}/meetupRsvp_realtime_table_config.json (82%)
create mode 100644 thirdeye/docs/configuration.rst
delete mode 100644 thirdeye/docs/production.rst
create mode 100644 thirdeye/thirdeye-frontend/app/pods/aiavailability/controller.js
create mode 100644 thirdeye/thirdeye-frontend/app/pods/aiavailability/route.js
create mode 100644 thirdeye/thirdeye-frontend/app/pods/aiavailability/template.hbs
create mode 100644 thirdeye/thirdeye-frontend/app/pods/components/detection-health/component.js
create mode 100644 thirdeye/thirdeye-frontend/app/pods/components/detection-health/template.hbs
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/alert/content/EmailContentFormatter.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/alert/content/EmailContentFormatterConfiguration.java
rename pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/sharding/SegmentAssignmentStrategyEnum.java => thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/AnomalyType.java (71%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/alert/util/{EmailScreenshotHelper.java => AlertScreenshotHelper.java} (56%)
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/DetectionPreviewConfiguration.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/resources/AnomalyFlattenResource.java
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/util/{StringUtils.java => ThirdEyeStringUtils.java} (98%)
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/ExpressionParser.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/alert/DetectionAlertFilterNotification.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/AnomalyDetectionStage.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/AnomalyDetectionStageWrapper.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/AnomalyFilterStage.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/AnomalyFilterStageWrapper.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/BaselineRuleDetectionStage.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/BaselineRuleFilterStage.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/GrouperStageWrapper.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/StaticAnomalyDetectionStage.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/StaticAnomalyFilterStage.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/ThresholdRuleDetectionStage.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/stage/ThresholdRuleFilterStage.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/MeanVarianceRuleDetector.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/spec/MeanVarianceRuleDetectorSpec.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/wrapper/EntityAnomalyMergeWrapper.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/formatter/DTOFormatter.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/formatter/DetectionAlertConfigFormatter.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/formatter/DetectionConfigFormatter.java
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/{alert => notification}/commons/EmailEntity.java (97%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/{alert/content/BaseEmailContentFormatter.java => notification/content/BaseNotificationContent.java} (68%)
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/notification/content/NotificationContent.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/notification/content/templates/EntityGroupKeyContent.java
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/{alert/content/HierarchicalAnomaliesEmailContentFormatter.java => notification/content/templates/HierarchicalAnomaliesContent.java} (66%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/{alert/content/MultipleAnomaliesEmailContentFormatter.java => notification/content/templates/MetricAnomaliesContent.java} (79%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/{alert/content/OnboardingNotificationEmailContentFormatter.java => notification/content/templates/OnboardingNotificationContent.java} (75%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/{alert/content/EmailContentFormatterContext.java => notification/formatter/ADContentFormatterContext.java} (88%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/{alert/commons/EmailContentFormatterFactory.java => notification/formatter/AlertContentFormatterFactory.java} (65%)
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/notification/formatter/channels/EmailContentFormatter.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/resources/org/apache/pinot/thirdeye/detector/entity-groupkey-anomaly-report.ftl
rename thirdeye/thirdeye-pinot/src/main/resources/org/apache/pinot/thirdeye/detector/{holiday-anomaly-report.ftl => metric-anomalies-template.ftl} (100%)
delete mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/alert/commons/TestEmailContentFormatterFactory.java
rename thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/datalayer/util/{StringUtilsTest.java => ThirdEyeStringUtilsTest.java} (75%)
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/datasource/resources/TestAnomalyFlattenResource.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/alert/filter/AlertFilterUtils.java
delete mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/algorithm/stage/AnomalyDetectionStageWrapperTest.java
delete mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/algorithm/stage/BaselineRuleDetectionStageTest.java
delete mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/algorithm/stage/BaselineRuleFilterTest.java
delete mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/algorithm/stage/ThresholdRuleDetectionStageTest.java
delete mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/algorithm/stage/ThresholdRuleFilterTest.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/components/ExpressionParserTest.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/components/MeanVarianceRuleDetectorTest.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/wrapper/EntityAnomalyMergeWrapperTest.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/formatter/DetectionConfigFormatterTest.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/integration/NotificationTaskSchedulerTest.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/notification/ContentFormatterUtils.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/notification/content/templates/TestEntityGroupKeyContent.java
rename thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/{alert/content/TestHierarchicalAnomaliesEmailContentFormatter.java => notification/content/templates/TestHierarchicalAnomaliesContent.java} (83%)
rename thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/{alert/content/TestMultipleAnomaliesEmailContentFormatter.java => notification/content/templates/TestMetricAnomaliesContent.java} (81%)
rename thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/{alert/content/TestOnboardingNotificationContentFormatter.java => notification/content/templates/TestOnboardingNotificationContent.java} (81%)
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/notification/formatter/TestAlertContentFormatterFactory.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/algorithm/timeseries-2y.csv
create mode 100644 thirdeye/thirdeye-pinot/src/test/resources/sample-alert-config.yml
create mode 100644 thirdeye/thirdeye-pinot/src/test/resources/sample-detection-config.yml
copy thirdeye/thirdeye-pinot/src/test/resources/{test-multiple-anomalies-email-content-formatter.html => test-entity-groupby-email-content-formatter.html} (58%)
copy thirdeye/thirdeye-pinot/src/test/resources/{test-multiple-anomalies-email-content-formatter.html => test-entity-groupby-with-whitelist-email-content-formatter.html} (51%)
rename thirdeye/thirdeye-pinot/src/test/resources/{test-hierarchical-anomalies-email-content-formatter.html => test-hierarchical-metric-anomalies-template.html} (100%)
rename thirdeye/thirdeye-pinot/src/test/resources/{test-multiple-anomalies-email-content-formatter.html => test-metric-anomalies-template.html} (100%)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org