You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by kh...@apache.org on 2021/10/10 04:52:32 UTC
[pinot] branch threshold_updater_patch updated (78a8b86 -> 3385a48)
This is an automated email from the ASF dual-hosted git repository.
kharekartik pushed a change to branch threshold_updater_patch
in repository https://gitbox.apache.org/repos/asf/pinot.git.
discard 78a8b86 Merge branch 'master' into threshold_updater_patch
add 1ea4260 adding json-bigInt package to parse long values in json (#7120)
add 3c033a6 Cleanup PinotHelixResourceManager and move the default tag handling to the starter (#7123)
add 0eb0f89 Fix UpsertConfig JSON deserialization. (#7125)
add e656e16 Fix custom instance id for controller/broker/minion (#7127)
add 1956c07 Add timestamp datatype support in JDBC (#7117)
add b99e804 Fixing Controller/Broker/Server/Minion zk and cluster name configs in PinotServiceMananger when start using config file (#7131)
add 4fdf01b Fixing init controller conf for StartControllerCommand (#7132)
add a251c42 Adding ability to check ingestion status for Offline Pinot table (#7070)
add 4e88da6 Add datetime function with 2 arguments (#7116)
add 0102227 Fix package for ReplicationUtils. (#7135)
add c42aef6 Validate parsing multi value delimiter as unicode (#7133)
add 8b5c0ac Release helm pinot 0.2.4 (#7136)
add 88bff60 Add support to skip kinesis test if docker is not available (#7140)
add 1a12510 Move minion metrics definition to pinot-common package (#7107)
add 7fb65f0 Update ExecutionStats.java (#7146)
add f0675af Switch to using 'authority' instead of 'host' when building parent URI in PinotFS::move (#7037)
add 0500c12 Fixing pinot-admin script to parse env var PLUGINS_INCLUDE (#7145)
add 52d733c Provide HTTP client to kinesis builder (#7148)
add 7233e2c Support Dictionary Based Plan For DISTINCT (#7141)
add 0447e74 Fix the exception message for segment reset (#7154)
add f04078e Upgrade avro-mapred to 1.9.2 so that all avro dependencies are 1.9.2. (#7111)
add e17029f Wire TaskMetricsEmitter to controller starter (#7126)
add 1ef50f7 Add controller/server API to fetch aggregated segment metadata (#7102)
add db5dfd7 Fix race condition in MinionInstancesCleanupTask (#7122)
add de2f0e0 Enable parallel builds for compat checker (#7149)
add 34e8257 Add systemResourceInfo to getInstance API (#7160)
add 6d0ab91 Fix typo in SegmentMetadataFetcher. (#7162)
add dbaac00 Fix errors happening due to parallel builds (#7172)
add 480395f Add Apache Pulsar low level and high level connector (#7026)
add a766c65 Updated CONTRIBUTING.md with Code Formatting Guidelines (#7170)
add db5561a Improve the segment aggregated metadata controller/server API (#7157)
add 738a584 Extract time handling for SegmentProcessorFramework (#7158)
add b6907fc add groovy unit test for sha256 data masking (#7179)
add fe83e95 Added a query param to /tables for getting sorted table names based on time metadata (#7142)
add d6462cd Bug fix: JDBC driver throws null pointer exception when metric columns are not present in Schema (#7175)
add febdcbc Use resourceAsStream to fetch resources in tests (#7181)
add e0163f4 Integrate enhanced SegmentProcessorFramework into MergeRollupTaskExecutor (#7180)
add a0ad49d Make some things public (#7186)
add 4c04e1f [7156] Human Readable Controller Configs (#7173)
add 9e81326 Fix the config key for TASK_MANAGER_FREQUENCY_PERIOD (#7198)
add 039bd7b Move pinot url away from incubator (#7199)
add 2f3774b Reduce the disk usage for segment conversion task (#7193)
add d09c0ad Refactoring SegmentGenerationAndPushTask for extensibility (#7188)
add ac7e5f9 Fixed pinot java client to add zkClient close (#7196)
add c6c407d Change build status badge to use github-action (#7201)
add 7e6cd9e Add the support of geoToH3 function (#7182)
add 25d14c2 Added ParquetNativeRecordReader in config's doc (#7204)
add 87ee80a Updating maven binary download link (#7209)
add 717940b Introduce OR Predicate Execution On Star Tree Index (#7184)
add c2c887a allow to extract values from array of objects with jsonPathArray (#7208)
add 069bae0 Bump up the "grpc-protobuf-lite" version to 1.19.0 in Apache Pulsar plugin (#7220)
add 5af6078 Fix the shaded issue with Apache Pulsar (#7223)
add 573651b Update license, notice for 0.8.0 release (#7225)
add 5e0dcff 7228 /sql REST call now compatible with auth (#7230)
add df9f5ef moving the map of pinot data type and java type to common util (#7231)
add 95a6ae8 add cli args for auth tokens and user/password to ingestion job command (#7233)
add 78b08e4 convert types per value for array with mixing types (#7234)
add 06ac649 ignore query json parse errors (#7165)
add 2831223 Support get metadata and indexes API for REALTIME tables. (#7169)
add 29a6ba9 Make group trim size configurable for better accuracy (#7241)
add c1ac8a1 Do not shade jar for pinot native plug-in components (#7245)
add 1f24630 Cleanup: Make `jobType` case insensitive. (#7244)
add 6274e86 Allow any column as the upsert comparison column (#7200)
add 4da1dae Fix pulsar connector issue with `NoSuchMethodError` (#7247)
add 066c793 Adding shutdownhook for PinotServiceManager cmd (#7251)
add f99a2e6 move PinotSM shutdownhook to cleanup method (#7253)
add 329a419 Add missing `SegmentMetadataPushJobRunner` config in all the example yaml files. (#7258)
add 0d96c7f Treat STRING to BOOLEAN data type change as backward compatible schema change (#7259)
add 82fe4fa Replace gcp hardcoded values with generic annotations (#6985)
add d7e08b2 Split tests to multiple test set for github action (#7257)
add 337c58c Fix segment conversion executor for in-place conversion (#7265)
add 5227266 extend Reload rest API with option to force download segment (#7249)
add e4d391c Merge/Rollup task scheduler for offline tables. (#7178)
add 7add772 Add option to store the hash of the upsert primary key (#7246)
add c2637d1 Remove pinot-pulsar module from the binary release (#7272)
add d563611 Reformat tests (#7278)
add 6ef7f94 Enable hit counter and max hit rate tracker (#7236)
add 44f31ac Clean up the old SegmentZKMetadataPruner (#7281)
add 385b3cf Update reflection lib from 0.9.11 to 0.9.9 (#7277)
add 6fb51d2 Fix status return from compat checker (#7279)
add d40e2d3 update Slack Pinot community invite link (#7284)
add b0acde0 Minor improvement and auto-reformat for MergeRollupTaskGenerator (#7288)
add 46a4cf7 Fix maven settings file collision (#7287)
add 0ea617f Introduce LIKE Operator (#7214)
add dc90161 quick clean to reduce size of later PRs (#7292)
add dbaf658 Unify segment ZK metadata (#7255)
add fca88a2 fixing npe issue during cleanup pinot-admin (#7291)
add 7b79388 Fix MergeRollupTask uploading segments not updating their metadata (#7289)
add 3b1c1f0 Adding pinot dropwizard metrics plugin (#7263)
add 50884b7 Add github action workflow for ad-hoc compatibility test (#7296)
add 7a4eaa9 Show uncompressed znRecord from zk api (#7304)
add 43ffa03 Add TimeUnit to metrics in MinionQueryPhase (#7307)
add 12e5bcd add method to get column indices created in segment folder (#7297)
add 761c0e7 Enable maven checkstyle plugins by default for all new pinot components and fix pinot-spi styling (#7308)
add 107255a Acquire and release index buffers in Segment level operator (#7295)
add f034e28 Null handling for time column (#7269)
add d0b8d88 MergeRollupTask integration tests (#7283)
add c152494 add underscore to obj vars (#7316)
add 5ebabad Add debug endpoint to get minion task status. (#7300)
add a539958 Fix pinot-common code styling (#7311)
add ae83527 Fixing code style for pinot-plugins (#7314)
add cf68e22 [7187] Validate CSV Header For Configured Delimiter (#7237)
add f11b5d6 Only emit 0 for LLC_PARTITION_CONSUMING if LLRealtimeSegmentDataManager isn't stopped (#7322)
add bfe64b5 reformat pinot-java-client and pinot-jdbc-client (#7326)
add aa5f318 add mode aggregation function (#7318)
add 94b9ca3 cleanup segment preprocessor a bit to make next RP smaller (#7294)
add 892a0c1 Cleanup: Replace `Incubator Pinot` reference in the UI with `Apache Pinot`. (#7339)
add f3ce66f Code style pinot-segment-spi and pinot-segment-local (#7317)
add 612803e Add an optional flag to disable swagger in Pinot servers. (#7341)
add 52a2e40 Fix the issue with concurrent modification for segment lineage (#7343)
add a074cde mark and sweep indices for V3 segment format (#7301)
add b71f38a Infer the data type for LiteralTransformFunction (#7332)
add 5417cbb Delete metadata properly on table deletion (#7329)
add 9116710 7090 segmentnamegenerator accept input file parameter (#7222)
add 48c1fc0 Fix TableNotFound error message in PinotHelixResourceManager.getExistingTableNamesWithType() function. (#7340)
add 888c961 Add segmentNameGeneratorType config to IndexingConfig (#7346)
add 7e9ca6a Reformat code base with line length 120 (#7342)
add bb34406 reformat pinot-controller (#7331)
add 2fddf28 fixing code styling for pinot-broker (#7333)
add 4b8ca72 reformat pinot-server (#7334)
add 7a44f4a Manually trigger PeriodicTask (#7174)
add 58d86a6 reformat pinot-perf/pinot-tools/pinot-integration-tests/pinot-minion (#7335)
add ce2c367 fixing code styling for pinot-core (#7325)
add 94c695f Remove prefetch query option, since the whole prefetch-acquire-release is already withing an instance option (#7352)
add c907dca fixing permission check case issue in BasicAuthPrincipal (#7354)
add 3bbc2d0 Prepare for next development (0.9.0-SNAPSHOT) (#7356)
add f951cce Support ID_SET aggregation on MV columns (#7355)
add 767a149 Add endpoint to check minion task status for a single task. (#7353)
add c419c2c Showing partial status of segment and counting CONSUMING state as good segment status (#7327)
add 16e4e3f Allow logical identifier in filter (#7347)
add 2d1f7dc Add functions to return raw results for Percentile TDigest and Est (#7226)
add a5f8cf4 fix upload LLC segment endpoint truncated download URL (#7361)
add 7016d03 Cmd+enter shortcut to run query in query console (#7359)
add 70efd03 Fix task scheduling on table update (#7362)
add 93f2efb Fix metric method for ONLINE_MINION_INSTANCES metric (#7363)
add 004c8af Make JsonToPinotSchema consistent with AvroSchemaToPinotSchema (#7366)
add 0681e4b Bug fix: make currentOffset in consuming segment volatile (#7365)
add c5e8ddf Fix misleading error msg for missing URI (#7367)
add 3892812 Add "num rows in segments" and "num segments queried per host" to the output of Realtime Provisioning Rule (#7282)
add c62a1fb make getColumnIndices accurate (#7370)
add 21767c6 Increase code coverage for pinot-java-client - ConnectionFactory (#7350)
add 2302bd2 check for less than zero start and end time (#7372)
add d412ecf Fix SegmentZKMetadta time handling (#7375)
add f5f072c Optimize IndexedTable (#7373)
add 29f27f8 Add cleanup method for HadoopSegmentCreationJob (#7371)
add 71b6890 Check schema backward-compatibility when updating schema through addSchema with override (#7374)
add de4ee0f Remove obsolete indices during segment reloading for V3 format (wrap up) (#7286)
add 27b4da7 showing tooltip in SQL Editor (#7387)
add 8fab3be Make the SQL Editor box expandable (#7381)
add 8106b69 add more integ test cases for index removal and stronger assertion (#7385)
add 5976922 Optimize TableResizer (#7392)
add fe14d60 fixing pinot-presto docker image build script (#7401)
add 6294d1b Remove unused constant FIELD_VALUE (#7396)
add e7901ee Missing Prometheus metric name (#7406)
add 58fcd5b simplify association between Java Class and PinotDataType for faster mapping (#7402)
add c27ae81 Cache ~/.m2/repository to speed up CI speed (#7408)
add 91679c0 Adding presto-pinot-driver module (#7384)
add 3d28c0e skip .factorypath from rat (#7419)
add dc80d67 avoid creating stateless ParseContextImpl once per jsonpath evaluation, avoid varargs allocation (#7412)
add 514f469 Obfuscator base implementation (#7407)
add ffcf9b9 fixing compatibility check for presto-pinot-driver (#7423)
add c1cc61e Replace MINUS(strColum1, strColumn2) with STRCMP(strColumn1,strColumn2) (#7394)
add 1fa6eda By default skip maven-deploy-plugin, so mvn deploy command won't fail (#7426)
add fe03999 Fix retention for cleaning up segment lineage (#7424)
add f3068bc Introduce resultSize in IndexedTable (#7420)
add 0e66d68 remove unused exceptions (#7429)
add 0ce2546 modify segment generator to not return illegal filenames (#7085)
add 11f8550 adding in source/target version (#7431)
add ad2e063 Remove the flaky checks in MergeRollupMinionClusterIntegrationTest (#7432)
add a5f3dc5 Move segment download and segment reload into TableManager (#7319)
add 69a91ac Add aws sts dependency to enable auth using web identity token. (#7017)
add 0ee33e8 add a artifact upload step to compatibility verifier job (#7425)
add 6605bc0 upload missing LLC segment to segment store by controller periodic task (#6778)
add 2dbc982 Fix parsing error messages returned to FileUploadDownloadClient (#7428)
add 6a7cbf1 Add aws sts dependency to enable auth using web identity token (#7445)
add dc78c79 refactor RangeIndexReader to make it easier to evolve its implementation (#7435)
add b23a1f3 FetchContext to hold configs in acquire/release calls (#7447)
add 31ca84e ban prefix increment (#7442)
add 0069fc3 Fix checkstyle violation and limit the prefix inc/dec check in for look (#7451)
add 9c2a333 Mask credentials in debug endpoint /appconfigs (#7452)
add 53cfdf3 Offset based realtime consumption status checker (#7267)
add b990c57 Support REGEXP_EXTRACT (#7114)
add fabda2b create pinot-integration-test-base module test-jar (#7399)
add 2c74f20 exclude healthcheck endpoint from auth (#7455)
add 9224590 Fix classpath with plugins java 8 (#7400)
add 0f42295 fix manifest scan which drives /version endpoint (#7456)
add a3df342 Add additional override in SegmentWriter and SegmentUploader (#7458)
add 6976697 Update some usage of BatchConfig (#7459)
add 4972110 Unify CombineOperator multi-threading logic (#7450)
add 5b15f08 add SingleSpaceSeparator rule and fix recent violations (#7467)
add c22886a add causes to stack trace return (#7460)
add 72d05a7 fix auth token injection in SegmentGenerationAndPushTaskExecutor (#7464)
add 34d987d add a linter checker (#7466)
add fcae6e0 hotfix for #7459 (#7471)
add b2bac68 Fix missing rate limiter if brokerResourceEV becomes null due to ZK connection (#7470)
add c31d1a1 add license checker to linter and fail fast (#7475)
add 8aeca69 Replace emptiness checks with `isEmpty()` (#7468)
add 043c658 Fix testNoBrokerResource test (#7478)
add 48e5aaf Revert back to use BatchIngestionConfig parameter (#7473)
add 4000568 Fix race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks: (#7427)
add f67666a define and enforce static import code style (#7483)
add 8a05ad8 fix segemnt uploader property in auth quickstart (#7479)
add cd7cf77 do not allow qualification of member variables with this (#7484)
add d99eb3c bit-sliced range index for int, long, float, double, dictionarized SV columns (#7454)
add f381abe use MethodHandle to access vectorized unsigned comparison on JDK9+ (#7487)
add 240bcb8 Fix pql double quote checker exception (#7485)
add bd5bff1 create controller resource packages config key (#7488)
add debb132 Add option to limit thread usage per query (#7492)
add d638444 Move acquire/release to right before/after execution call (#7493)
add 1fdc2eb [issue-7495] Fix minion metrics exporter config (#7496)
add 70374e2 remove unused dependency (#7498)
add 9b42278 fix pinot-controller/broker recent fork timeout (#7490)
add cccc1e3 fix volatile stopped flag (#7508)
add 57614d1 Follow up to improve restlet API package config key (#7497)
add 76023ae remove unnecessary wait during broker test (#7505)
add 30f5490 1. Catch timeout exception to wait segments become online for endReplaceSegments(), so it can be retried. (#7509)
add c9e36dd [7264] Add Exception to Broker Response When Not All Segments Are Available (Partial Response) (#7397)
add 517a0dc test xmls are not updated with latest test classes (#7474)
add 02c4998 Add mergeRollupTask delay metrics (#7368)
add f306460 Do not allow query to override the execution threads over the instance-level limit (#7515)
add af90f79 Improved range queries (#7513)
add 9410aef adding favicon to the pinot controller ui (#7518)
add 63f7ad9 Enhance TableCache to support schema name different from table name (#7525)
add 7be21c8 Faster bitmap scans (#7530)
add 19f66fd fix segment generation commands (#7527)
add acb775d add validation for realtimeToOffline task (#7523)
add 1be1b48 Fix TableCacheTest by using unique schema and table name (#7538)
add 8f41621 Optimize EmptySegmentPruner to skip pruning when there is no empty segments (#7531)
add c8aa3e3 In TableRebalancer, allow increasing from 1 replica without downtime (#7532)
add 57a1f3f Support JSON queries with top-level array path expression. (#7511)
add 12b106b Fix SegmentGenerationWithTimeColumnTest (#7537)
add 6969b38 Introduce MinionConf, move END_REPLACE_SEGMENTS_TIMEOUT_MS to minion config instead of task config. (#7516)
add c36d2b9 Cleanup sys out (#7547)
add 3385a48 Resolve merge conflicts
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (78a8b86)
\
N -- N -- N refs/heads/threshold_updater_patch (3385a48)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/pinot_compatibility_tests.yml | 64 +
.github/workflows/pinot_tests.yml | 104 +-
.../scripts/.pinot_compatibility_verifier.sh | 57 +
.github/workflows/scripts/.pinot_linter.sh | 33 +
.github/workflows/scripts/.pinot_test.sh | 53 +-
CONTRIBUTING.md | 5 +
DISCLAIMER | 10 -
LICENSE-binary | 124 +-
NOTICE | 2 +-
NOTICE-binary | 9 +-
README.md | 17 +-
compatibility-verifier/checkoutAndBuild.sh | 234 +-
compatibility-verifier/compCheck.sh | 36 +-
config/checkstyle.xml | 93 +-
config/codestyle-intellij.xml | 13 +-
config/suppressions.xml | 16 +
.../pinotdruidbenchmark/PinotThroughput.java | 1 +
docker/images/pinot-presto/Dockerfile | 6 +-
docker/images/pinot/Dockerfile | 2 +-
.../images/pinot/etc/conf/pinot-broker-log4j2.xml | 8 +-
.../pinot/etc/conf/pinot-controller-log4j2.xml | 8 +-
.../images/pinot/etc/conf/pinot-server-log4j2.xml | 8 +-
.../etc/jmx_prometheus_javaagent/configs/pinot.yml | 8 +-
.../docker/ingestion-job-specs/airlineStats.yaml | 12 +-
.../docker/ingestion-job-specs/baseballStats.yaml | 12 +-
kubernetes/helm/README-dev.md | 4 +
kubernetes/helm/index.yaml | 28 +
kubernetes/helm/pinot-0.2.4.tgz | Bin 0 -> 26536 bytes
kubernetes/helm/pinot/Chart.yaml | 4 +-
.../pinot/templates/broker/service-external.yaml | 5 +-
.../helm/pinot/templates/broker/service.yaml | 5 +-
.../templates/controller/service-external.yaml | 5 +-
.../helm/pinot/templates/controller/service.yaml | 5 +-
.../helm/pinot/templates/minion/service.yaml | 5 +-
.../helm/pinot/templates/server/service.yaml | 5 +-
kubernetes/helm/pinot/values.yaml | 9 +-
kubernetes/helm/presto/Chart.yaml | 4 +-
licenses-binary/LICENSE-antlr4-runtime.txt | 28 +
licenses-binary/LICENSE-bouncycastle.txt | 7 +
licenses-binary/LICENSE-google-api.txt | 27 +
licenses-binary/LICENSE-jcip.txt | 7 +
pinot-broker/pom.xml | 2 +-
.../apache/pinot/broker/api/RequestStatistics.java | 4 +-
.../broker/api/resources/PinotBrokerDebug.java | 26 +-
.../api/resources/PinotBrokerHealthCheck.java | 11 +-
.../broker/api/resources/PinotBrokerRouting.java | 15 +-
.../broker/api/resources/PinotClientRequest.java | 44 +-
.../broker/broker/helix/BaseBrokerStarter.java | 40 +-
.../broker/broker/helix/ClusterChangeMediator.java | 5 +-
.../broker/broker/helix/HelixBrokerStarter.java | 6 +-
.../broker/pruner/PartitionZKMetadataPruner.java | 138 -
.../pinot/broker/pruner/SegmentPrunerContext.java | 48 -
.../broker/pruner/SegmentZKMetadataPruner.java | 38 -
.../pruner/SegmentZKMetadataPrunerProvider.java | 56 -
.../pruner/SegmentZKMetadataPrunerService.java | 67 -
.../HelixExternalViewBasedQueryQuotaManager.java | 123 +-
.../apache/pinot/broker/queryquota/HitCounter.java | 8 +-
.../pinot/broker/queryquota/MaxHitRateTracker.java | 10 +-
.../pinot/broker/queryquota/QueryQuotaEntity.java | 4 +
.../requesthandler/BaseBrokerRequestHandler.java | 41 +-
.../SingleConnectionBrokerRequestHandler.java | 13 +-
.../pinot/broker/routing/RoutingManager.java | 20 +-
.../instanceselector/BaseInstanceSelector.java | 11 +-
.../SegmentLineageBasedSegmentPreSelector.java | 14 +-
.../routing/segmentpruner/EmptySegmentPruner.java | 4 +
.../segmentpruner/PartitionSegmentPruner.java | 1 +
.../segmentpruner/SegmentPrunerFactory.java | 11 +-
.../routing/segmentpruner/TimeSegmentPruner.java | 19 +-
.../segmentpruner/interval/IntervalTree.java | 33 +-
.../segmentselector/RealtimeSegmentSelector.java | 23 +-
.../segmentselector/SegmentSelectorFactory.java | 1 +
.../broker/HelixBrokerStarterHostnamePortTest.java | 21 +-
.../broker/broker/HelixBrokerStarterTest.java | 14 +-
.../broker/pruner/SegmentZKMetadataPrunerTest.java | 64 -
...elixExternalViewBasedQueryQuotaManagerTest.java | 35 +-
.../broker/queryquota/MaxHitRateTrackerTest.java | 3 +-
.../request/PqlAndCalciteSqlCompatibilityTest.java | 51 +-
.../LiteralOnlyBrokerRequestTest.java | 18 +-
.../broker/requesthandler/QueryValidationTest.java | 26 +-
.../broker/routing/IntervalST/IntervalTest.java | 57 -
.../routing/IntervalST/IntervalTreeTest.java | 89 -
.../broker/routing/intervalst/IntervalTest.java | 57 +
.../routing/intervalst/IntervalTreeTest.java | 89 +
.../routing/segmentpruner/SegmentPrunerTest.java | 108 +-
.../timeboundary/TimeBoundaryManagerTest.java | 39 +-
.../src/test/resources/SampleExternalView.json | 2652 ++++++++--------
pinot-broker/src/test/resources/log4j2.xml | 2 +-
pinot-clients/pinot-java-client/pom.xml | 2 +-
.../org/apache/pinot/client/AbstractResultSet.java | 2 +-
.../org/apache/pinot/client/BrokerResponse.java | 2 +-
.../org/apache/pinot/client/BrokerSelector.java | 12 +
.../java/org/apache/pinot/client/Connection.java | 5 +-
.../org/apache/pinot/client/ConnectionFactory.java | 9 +-
.../apache/pinot/client/DynamicBrokerSelector.java | 62 +-
.../org/apache/pinot/client/ExecutionStats.java | 52 +-
.../apache/pinot/client/ExternalViewReader.java | 15 +-
.../apache/pinot/client/ResultTableResultSet.java | 3 +-
.../apache/pinot/client/SimpleBrokerSelector.java | 14 +-
.../java/org/apache/pinot/client/TextTable.java | 29 +-
.../pinot/client/AggregationResultSetTest.java | 32 +-
.../apache/pinot/client/ConnectionFactoryTest.java | 48 +-
.../pinot/client/DynamicBrokerSelectorTest.java | 101 +-
.../apache/pinot/client/ExecutionStatsTest.java | 48 +-
.../pinot/client/ExternalViewReaderTest.java | 40 +-
.../apache/pinot/client/GroupByResultSetTest.java | 33 +-
.../pinot/client/ResultTableResultSetTest.java | 37 +-
.../pinot/client/SelectionResultSetTest.java | 31 +-
.../org/apache/pinot/client/TextTableTest.java | 23 +-
.../src/test/resources/log4j2.xml | 2 +-
.../org/apache/pinot/client/aggregation.json | 15 +-
.../apache/pinot/client/aggregationGroupBy.json | 79 +-
.../org/apache/pinot/client/exception.json | 12 +-
.../org/apache/pinot/client/selection.json | 3302 +++++++++++++++++++-
pinot-clients/pinot-jdbc-client/pom.xml | 37 +-
.../pinot/client/PinotConnectionMetaData.java | 30 +-
.../java/org/apache/pinot/client/PinotDriver.java | 6 +-
.../pinot/client/PinotPreparedStatement.java | 35 +-
.../apache/pinot/client/PinotResultMetadata.java | 6 +-
.../org/apache/pinot/client/PinotStatement.java | 30 +
.../controller/PinotControllerTransport.java | 6 +-
.../client/controller/request/TableRequest.java | 1 -
.../response/ControllerTenantBrokerResponse.java | 10 +-
.../client/controller/response/SchemaResponse.java | 12 +-
.../client/controller/response/TableResponse.java | 2 +-
.../org/apache/pinot/client/utils/Constants.java | 25 +-
.../apache/pinot/client/utils/DateTimeUtils.java | 29 +-
.../org/apache/pinot/client/utils/DriverUtils.java | 18 +
.../client/DummyPinotClientTransportFactory.java | 2 +-
.../org/apache/pinot/client/PinotDriverTest.java | 11 +-
.../pinot/client/PinotPreparedStatementTest.java | 33 +-
.../src/test/resources/result_table.json | 72 +-
.../src/test/resources/selection.json | 3302 +++++++++++++++++++-
pinot-clients/pom.xml | 2 +-
pinot-common/pom.xml | 45 +-
.../main/java/org/apache/pinot/common/Utils.java | 46 +-
.../common/assignment/InstancePartitions.java | 3 +-
.../pinot/common/exception/QueryException.java | 48 +-
.../exception/SchemaAlreadyExistsException.java | 26 +
.../common/function/DateTimePatternHandler.java | 3 +
.../pinot/common/function/DateTimeUtils.java | 5 +-
.../function/FunctionDefinitionRegistry.java | 2 +
.../apache/pinot/common/function/TimeZoneKey.java | 47 +-
.../common/function/TransformFunctionType.java | 9 +-
.../scalar/DataTypeConversionFunctions.java | 6 +-
.../common/function/scalar/DateTimeFunctions.java | 17 +-
.../common/function/scalar/HashFunctions.java | 1 +
.../common/function/scalar/JsonFunctions.java | 67 +-
.../common/function/scalar/StringFunctions.java | 54 +
.../apache/pinot/common/lineage/LineageEntry.java | 4 +-
.../pinot/common/lineage/SegmentLineage.java | 3 +-
.../common/lineage/SegmentLineageAccessHelper.java | 26 +-
.../pinot/common/lineage/SegmentLineageUtils.java | 18 +
.../messages/RoutingTableRebuildMessage.java | 1 +
.../common/messages/RunPeriodicTaskMessage.java | 66 +
.../common/messages/SegmentReloadMessage.java | 12 +-
.../pinot/common/metadata/ZKMetadataProvider.java | 161 +-
.../segment/LLCRealtimeSegmentZKMetadata.java | 151 -
.../metadata/segment/OfflineSegmentZKMetadata.java | 135 -
.../segment/RealtimeSegmentZKMetadata.java | 144 -
.../common/metadata/segment/SegmentZKMetadata.java | 494 +--
.../SegmentZKMetadataCustomMapModifier.java | 2 +-
.../pinot/common/metrics/AbstractMetrics.java | 15 +-
.../apache/pinot/common/metrics/BrokerGauge.java | 18 +-
.../apache/pinot/common/metrics/BrokerMeter.java | 18 +-
.../pinot/common/metrics/BrokerQueryPhase.java | 6 +-
.../apache/pinot/common/metrics/BrokerTimer.java | 19 +-
.../pinot/common/metrics/ControllerGauge.java | 23 +-
.../pinot/common/metrics/ControllerMeter.java | 22 +-
.../pinot/common/metrics/ControllerMetrics.java | 3 +-
.../pinot/common/metrics/ControllerTimer.java | 12 +-
.../apache/pinot/common/metrics/MinionGauge.java | 51 +
.../apache/pinot/common/metrics/MinionMeter.java | 58 +
.../apache/pinot/common/metrics/MinionMetrics.java | 49 +
.../pinot/common/metrics/MinionQueryPhase.java | 37 +
.../apache/pinot/common/metrics/MinionTimer.java | 47 +
.../pinot/common/metrics/PinotMetricUtils.java | 30 +-
.../apache/pinot/common/metrics/ServerGauge.java | 18 +-
.../apache/pinot/common/metrics/ServerMeter.java | 18 +-
.../pinot/common/metrics/ServerQueryPhase.java | 6 +-
.../apache/pinot/common/metrics/ServerTimer.java | 12 +-
.../pinot/common/metrics/ValidationMetrics.java | 18 +-
.../apache/pinot/common/minion/Granularity.java | 27 -
.../common/minion/MergeRollupTaskMetadata.java | 22 +-
.../apache/pinot/common/minion/MinionClient.java | 5 +-
.../common/minion/MinionTaskMetadataUtils.java | 45 +-
.../RealtimeToOfflineSegmentsTaskMetadata.java | 3 +-
.../protocols/SegmentCompletionProtocol.java | 46 +-
.../request/context/RequestContextUtils.java | 5 +
.../response/broker/BrokerResponseNative.java | 15 +-
.../common/restlet/resources/ResourceUtils.java | 4 +-
.../common/restlet/resources/SegmentErrorInfo.java | 24 +-
.../restlet/resources/SegmentServerDebugInfo.java | 1 -
.../common/restlet/resources/SegmentSizeInfo.java | 28 +-
.../resources/StartReplaceSegmentsRequest.java | 7 +-
.../restlet/resources/SystemResourceInfo.java | 6 +-
.../restlet/resources/TableMetadataInfo.java | 81 +
.../common/restlet/resources/TableSegments.java | 10 +-
.../common/restlet/resources/TableSizeInfo.java | 28 +-
.../pinot/common/restlet/resources/TablesList.java | 2 +-
.../segment/generation/SegmentGenerationUtils.java | 2 +
.../pinot/common/tier/TierSegmentSelector.java | 1 -
.../common/tier/TimeBasedTierSegmentSelector.java | 15 +-
.../common/utils/ClientSSLContextGenerator.java | 30 +-
.../org/apache/pinot/common/utils/DataTable.java | 6 +-
.../apache/pinot/common/utils/DateTimeUtils.java | 47 +
.../common/utils/FileUploadDownloadClient.java | 49 +-
.../apache/pinot/common/utils/HLCSegmentName.java | 5 +-
.../org/apache/pinot/common/utils/HashUtil.java | 11 +-
.../apache/pinot/common/utils/LLCSegmentName.java | 13 +-
.../org/apache/pinot/common/utils/LLCUtils.java | 3 +
.../LikeToRegexpLikePatternConverterUtils.java | 52 +
.../pinot/common/utils/NamedThreadFactory.java | 19 +-
.../apache/pinot/common/utils/PinotAppConfigs.java | 7 +-
.../apache/pinot/common/utils/PinotDataType.java | 199 +-
.../pinot/common/utils/PrimitiveArrayUtils.java | 3 +
.../apache/pinot/common/utils/SegmentUtils.java | 32 +-
.../apache/pinot/common/utils/ServiceStatus.java | 83 +-
.../org/apache/pinot/common/utils/StringUtil.java | 9 +-
.../pinot/common/utils/TarGzCompressionUtils.java | 14 +-
.../org/apache/pinot/common/utils/ZkStarter.java | 3 +
.../pinot/common/utils/config/TierConfigUtils.java | 3 +-
.../common/utils/fetcher/HttpSegmentFetcher.java | 7 +-
.../common/utils/fetcher/HttpsSegmentFetcher.java | 2 +-
.../utils/fetcher/SegmentFetcherFactory.java | 1 +
.../pinot/common/utils/grpc/GrpcQueryClient.java | 40 +-
.../pinot/common/utils/helix/HelixHelper.java | 6 +-
.../common/utils/helix/LeadControllerUtils.java | 18 +-
.../PinotHelixPropertyStoreZnRecordProvider.java | 20 +-
.../pinot/common/utils/helix/TableCache.java | 76 +-
.../common/utils/request/FilterQueryTree.java | 32 +-
.../common/utils/webhdfs/WebHdfsV1Client.java | 3 +-
.../utils/BrokerRequestComparisonUtils.java | 4 +-
.../parsers/PinotQuery2BrokerRequestConverter.java | 22 +-
.../org/apache/pinot/pql/parsers/Pql2Compiler.java | 30 +-
.../pinot/pql/parsers/pql2/ast/BaseAstNode.java | 2 +-
.../parsers/pql2/ast/BooleanOperatorAstNode.java | 2 +-
.../pql2/ast/ComparisonPredicateAstNode.java | 2 +-
.../pinot/pql/parsers/pql2/ast/FilterKind.java | 1 +
.../pql2/ast/FloatingPointLiteralAstNode.java | 10 +-
.../pql/parsers/pql2/ast/FunctionCallAstNode.java | 6 +-
.../pinot/pql/parsers/pql2/ast/OrderByAstNode.java | 3 +-
.../pinot/pql/parsers/pql2/ast/TopAstNode.java | 2 +-
.../org/apache/pinot/pql/parsers/utils/Pair.java | 16 +-
.../apache/pinot/sql/parsers/CalciteSqlParser.java | 32 +-
.../pinot/common/data/DateTimeFormatSpecTest.java | 187 +-
.../apache/pinot/common/data/FieldSpecTest.java | 51 +-
.../org/apache/pinot/common/data/SchemaTest.java | 48 +
.../pinot/common/exception/QueryExceptionTest.java | 53 +
.../function/AggregationFunctionTypeTest.java | 9 +
.../pinot/common/function/JsonFunctionsTest.java | 130 +-
.../common/function/TransformFunctionTypeTest.java | 3 +-
.../scalar/ArrayAwareJacksonJsonProviderTest.java | 114 +
.../pinot/common/http/MultiGetRequestTest.java | 49 +-
.../pinot/common/lineage/SegmentLineageTest.java | 6 +
.../common/metadata/InstanceZKMetadataTest.java | 4 +-
.../metadata/MetadataEqualsHashCodeTest.java | 31 +-
.../pinot/common/metadata/MetadataUtils.java | 5 +-
.../RealtimeToOfflineSegmentsTaskMetadataTest.java | 2 +-
.../common/metadata/SegmentZKMetadataTest.java | 92 +-
.../pinot/common/metrics/PinotMetricUtilsTest.java | 16 +-
.../generation/SegmentGenerationUtilsTest.java | 3 +-
.../pinot/common/tier/TierSegmentSelectorTest.java | 11 +-
.../apache/pinot/common/utils/DataSchemaTest.java | 14 +-
.../pinot/common/utils/EqualityUtilsTest.java | 9 +-
.../common/utils/FileUploadDownloadClientTest.java | 14 +-
.../LikeToRegexpLikePatternConverterUtilsTest.java | 77 +
.../pinot/common/utils/PinotDataTypeTest.java | 141 +-
.../pinot/common/utils/SegmentNameBuilderTest.java | 8 +-
.../pinot/common/utils/ServiceStatusTest.java | 18 +-
.../common/utils/config/TableConfigSerDeTest.java | 18 +-
.../pinot/common/utils/helix/HelixHelperTest.java | 2 +-
.../apache/pinot/pql/parsers/Pql2CompilerTest.java | 20 +-
.../request/BrokerRequestSerializationTest.java | 40 +-
.../pinot/request/BrokerResponseNativeTest.java | 2 -
.../pinot/sql/parsers/CalciteSqlCompilerTest.java | 126 +-
.../test/java/org/apache/pinot/util/TestUtils.java | 3 +
pinot-connectors/pinot-spark-connector/pom.xml | 491 +--
pinot-connectors/pom.xml | 39 +-
pinot-connectors/presto-pinot-driver/pom.xml | 738 +++++
.../presto/PinotScatterGatherQueryClient.java | 334 ++
.../presto/grpc/PinotStreamingQueryClient.java | 65 +
.../apache/pinot/connector/presto/grpc/Utils.java | 54 +
pinot-controller/pom.xml | 2 +-
.../pinot/controller/BaseControllerStarter.java | 55 +-
.../apache/pinot/controller/ControllerConf.java | 310 +-
...ControllerUserDefinedMessageHandlerFactory.java | 134 +
.../pinot/controller/LeadControllerManager.java | 34 +-
.../pinot/controller/api/CallableHttpGet.java | 8 +-
.../api/ControllerAdminApiApplication.java | 12 +-
.../pinot/controller/api/access/AccessControl.java | 8 +-
.../api/access/AuthenticationFilter.java | 2 +-
.../api/access/BasicAuthAccessControlFactory.java | 5 +-
.../pinot/controller/api/debug/TableDebugInfo.java | 18 +-
.../api/events/DefaultMetadataEventNotifier.java | 1 +
.../api/events/MetadataEventNotifier.java | 1 +
.../pinot/controller/api/resources/Constants.java | 4 +
.../api/resources/ControllerFilePathProvider.java | 16 +-
.../api/resources/ControllerResponseFilter.java | 7 +-
.../resources/LLCSegmentCompletionHandlers.java | 69 +-
.../api/resources/PinotBrokerRestletResource.java | 35 +-
.../api/resources/PinotClusterConfigs.java | 36 +-
.../api/resources/PinotControllerAuthResource.java | 5 +-
.../api/resources/PinotControllerHealthCheck.java | 10 +-
...PinotControllerPeriodicTaskRestletResource.java | 123 +
.../resources/PinotIngestionRestletResource.java | 30 +-
.../PinotInstanceAssignmentRestletResource.java | 18 +-
.../resources/PinotInstanceRestletResource.java | 98 +-
.../PinotLeadControllerRestletResource.java | 6 +-
.../api/resources/PinotQueryResource.java | 4 +-
.../api/resources/PinotSchemaRestletResource.java | 78 +-
.../api/resources/PinotSegmentRestletResource.java | 199 +-
.../PinotSegmentUploadDownloadRestletResource.java | 84 +-
.../api/resources/PinotTableIndexingConfigs.java | 12 +-
.../api/resources/PinotTableInstances.java | 27 +-
.../api/resources/PinotTableMetadataConfigs.java | 12 +-
.../api/resources/PinotTableRestletResource.java | 199 +-
.../controller/api/resources/PinotTableSchema.java | 9 +-
.../api/resources/PinotTableSegmentConfigs.java | 18 +-
.../api/resources/PinotTableTenantConfigs.java | 11 +-
.../api/resources/PinotTaskRestletResource.java | 50 +-
.../api/resources/PinotTenantRestletResource.java | 166 +-
.../controller/api/resources/ResourceUtils.java | 45 +
.../api/resources/ServerTableSizeReader.java | 9 +-
.../api/resources/TableConfigsRestletResource.java | 31 +-
.../api/resources/TableDebugResource.java | 49 +-
.../pinot/controller/api/resources/TableSize.java | 17 +-
.../pinot/controller/api/resources/TableViews.java | 14 +-
.../resources/WebApplicationExceptionMapper.java | 10 +-
.../api/resources/ZookeeperResource.java | 124 +-
.../controller/api/upload/SegmentValidator.java | 7 +-
.../pinot/controller/api/upload/ZKOperator.java | 85 +-
.../helix/ControllerRequestURLBuilder.java | 23 +-
.../controller/helix/SegmentStatusChecker.java | 19 +-
...erResourceOnlineOfflineStateModelGenerator.java | 2 +
.../helix/core/PinotHelixResourceManager.java | 318 +-
...lixSegmentOnlineOfflineStateModelGenerator.java | 2 +
.../helix/core/PinotTableIdealStateBuilder.java | 20 +-
.../helix/core/SegmentDeletionManager.java | 4 +-
.../InstanceReplicaGroupPartitionSelector.java | 4 +-
.../segment/OfflineDimTableSegmentAssignment.java | 3 +-
.../segment/OfflineSegmentAssignment.java | 27 +-
.../segment/RealtimeSegmentAssignment.java | 19 +-
.../core/assignment/segment/SegmentAssignment.java | 5 +-
.../segment/SegmentAssignmentFactory.java | 3 +-
.../assignment/segment/SegmentAssignmentUtils.java | 3 +-
.../helix/core/minion/ClusterInfoAccessor.java | 89 +-
.../core/minion/MinionInstancesCleanupTask.java | 42 +-
.../core/minion/PinotHelixTaskResourceManager.java | 349 ++-
.../helix/core/minion/PinotTaskManager.java | 166 +-
.../helix/core/minion/TaskMetricsEmitter.java | 35 +-
.../core/minion/generator/TaskGeneratorUtils.java | 3 +
.../core/periodictask/ControllerPeriodicTask.java | 26 +-
.../realtime/PinotLLCRealtimeSegmentManager.java | 240 +-
.../core/realtime/PinotRealtimeSegmentManager.java | 52 +-
.../core/realtime/SegmentCompletionManager.java | 242 +-
.../segment/CommittingSegmentDescriptor.java | 6 +-
.../segment/DefaultFlushThresholdUpdater.java | 7 +-
.../realtime/segment/FlushThresholdUpdater.java | 6 +-
.../SegmentSizeBasedFlushThresholdUpdater.java | 12 +-
.../core/rebalance/RebalanceConfigConstants.java | 2 +
.../helix/core/rebalance/TableRebalancer.java | 50 +-
.../helix/core/relocation/SegmentRelocator.java | 1 +
.../helix/core/retention/RetentionManager.java | 53 +-
.../retention/strategy/TimeRetentionStrategy.java | 4 +-
.../helix/core/util/HelixSetupUtils.java | 18 +-
.../helix/core/util/ZKMetadataUtils.java | 3 +-
.../controller/helix/starter/HelixConfig.java | 3 +
.../controller/recommender/RecommenderDriver.java | 12 +-
.../recommender/data/generator/DataGenerator.java | 81 +-
.../data/generator/DataGeneratorSpec.java | 112 +-
.../data/generator/GeneratorFactory.java | 7 +-
.../data/generator/MultiValueGeneratorHelper.java | 2 +
.../data/generator/NumberGenerator.java | 42 +-
.../data/generator/PatternMixtureGenerator.java | 86 +-
.../data/generator/PatternSeasonalGenerator.java | 113 +-
.../data/generator/PatternSequenceGenerator.java | 47 +-
.../data/generator/PatternSpikeGenerator.java | 123 +-
.../data/generator/PatternStringGenerator.java | 39 +-
.../recommender/data/generator/PatternType.java | 6 +-
.../data/generator/SchemaAnnotation.java | 1 +
.../data/generator/StringGenerator.java | 42 +-
.../exceptions/InvalidInputException.java | 4 +-
.../controller/recommender/io/ConfigManager.java | 3 +-
.../controller/recommender/io/InputManager.java | 14 +-
.../recommender/io/metadata/FieldMetadata.java | 14 +-
.../io/metadata/SchemaWithMetaData.java | 10 +-
.../realtime/provisioning/MemoryEstimator.java | 116 +-
.../controller/recommender/rules/AbstractRule.java | 4 +-
.../recommender/rules/RulesToExecute.java | 13 +-
.../rules/impl/AggregateMetricsRule.java | 5 +-
.../recommender/rules/impl/BloomFilterRule.java | 15 +-
.../recommender/rules/impl/FlagQueryRule.java | 14 +-
.../rules/impl/InvertedSortedIndexJointRule.java | 19 +-
.../recommender/rules/impl/KafkaPartitionRule.java | 24 +-
.../NoDictionaryOnHeapDictionaryJointRule.java | 36 +-
.../rules/impl/PinotTablePartitionRule.java | 61 +-
.../recommender/rules/impl/RangeIndexRule.java | 15 +-
.../rules/impl/RealtimeProvisioningRule.java | 46 +-
.../recommender/rules/impl/SegmentSizeRule.java | 35 +-
.../rules/impl/VariedLengthDictionaryRule.java | 14 +-
.../recommender/rules/io/FlaggedQueries.java | 16 +-
.../recommender/rules/io/configs/IndexConfig.java | 196 +-
.../rules/io/configs/PartitionConfig.java | 18 +-
.../io/configs/SegmentSizeRecommendations.java | 32 +-
.../rules/io/params/BloomFilterRuleParams.java | 27 +-
.../rules/io/params/FlagQueryRuleParams.java | 11 +-
.../params/InvertedSortedIndexJointRuleParams.java | 111 +-
...oDictionaryOnHeapDictionaryJointRuleParams.java | 80 +-
.../rules/io/params/PartitionRuleParams.java | 56 +-
.../rules/io/params/RangeIndexRuleParams.java | 12 +-
.../io/params/RealtimeProvisioningRuleParams.java | 51 +-
.../rules/io/params/RecommenderConstants.java | 17 +-
.../rules/io/params/SegmentSizeRuleParams.java | 25 +-
.../recommender/rules/utils/FixedLenBitset.java | 69 +-
.../rules/utils/PredicateParseResult.java | 30 +-
.../utils/QueryInvertedSortedIndexRecommender.java | 159 +-
.../controller/tuner/TableConfigTunerRegistry.java | 10 +-
.../controller/tuner/TableConfigTunerUtils.java | 5 +-
.../controller/util/AutoAddInvertedIndex.java | 6 +-
.../util/ConsumingSegmentInfoReader.java | 20 +-
.../pinot/controller/util/FileIngestionHelper.java | 34 +-
.../controller/util/SegmentCompletionUtils.java | 5 +-
.../controller/util/SegmentIntervalUtils.java | 2 +
.../util/ServerSegmentMetadataReader.java | 95 +-
.../util/TableIngestionStatusHelper.java | 94 +
.../pinot/controller/util/TableMetadataReader.java | 21 +
.../controller/util/TableRetentionValidator.java | 46 +-
.../pinot/controller/util/TableSizeReader.java | 105 +-
.../BrokerResourceValidationManager.java | 3 +-
.../validation/OfflineSegmentIntervalChecker.java | 31 +-
.../RealtimeSegmentValidationManager.java | 45 +-
.../controller/validation/StorageQuotaChecker.java | 72 +-
.../src/main/resources/app/components/SideBar.tsx | 2 +-
.../src/main/resources/app/components/Table.tsx | 9 +
.../main/resources/app/components/TableToolbar.tsx | 16 +-
pinot-controller/src/main/resources/app/index.html | 6 +-
.../src/main/resources/app/pages/Query.tsx | 54 +-
.../src/main/resources/app/requests/index.ts | 6 +-
.../src/main/resources/app/styles/styles.css | 2 +-
.../main/resources/app/utils/PinotMethodUtils.ts | 45 +-
.../src/main/resources/app/utils/axios-config.ts | 6 +-
pinot-controller/src/main/resources/favicon.ico | Bin 0 -> 15406 bytes
.../src/main/resources/package-lock.json | 166 +
pinot-controller/src/main/resources/package.json | 3 +
.../src/main/resources/webpack.config.js | 8 +
.../pinot/controller/ControllerConfTest.java | 205 ++
.../controller/ControllerStarterStatelessTest.java | 110 +
.../pinot/controller/ControllerStarterTest.java | 84 -
.../pinot/controller/ControllerTestSetup.java | 3 +-
.../pinot/controller/ControllerTestUtils.java | 106 +-
.../controller/LeadControllerManagerTest.java | 17 +-
.../pinot/controller/api/AccessControlTest.java | 3 +-
.../ConsumingSegmentInfoReaderStatelessTest.java | 354 +++
.../api/ConsumingSegmentInfoReaderTest.java | 363 ---
.../api/ControllerFilePathProviderTest.java | 6 +-
.../api/PinotControllerAppConfigsTest.java | 12 +-
.../pinot/controller/api/PinotFileUploadTest.java | 6 +-
...PinotIngestionRestletResourceStatelessTest.java | 125 +
.../api/PinotIngestionRestletResourceTest.java | 125 -
...PinotInstanceAssignmentRestletResourceTest.java | 25 +-
.../api/PinotInstanceRestletResourceTest.java | 4 +-
.../api/PinotSchemaRestletResourceTest.java | 139 +-
.../api/PinotSegmentRestletResourceTest.java | 16 +-
.../controller/api/PinotSegmentsMetadataTest.java | 126 +-
.../api/PinotTableRestletResourceTest.java | 3 +-
.../api/PinotTenantRestletResourceTest.java | 12 +-
.../api/SegmentCompletionProtocolDeserTest.java | 14 +-
.../controller/api/ServerTableSizeReaderTest.java | 107 +-
.../api/TableConfigsRestletResourceTest.java | 15 +-
.../pinot/controller/api/TableSizeReaderTest.java | 182 +-
.../pinot/controller/api/TableViewsTest.java | 85 +-
.../api/access/AuthenticationFilterTest.java | 16 +-
.../PinotDummyExtraRestletResourceTest.java | 60 +
...otSegmentUploadDownloadRestletResourceTest.java | 8 +-
.../PinotDummyExtraRestletResource.java | 45 +
.../controller/api/upload/ZKOperatorTest.java | 39 +-
.../helix/ControllerInstanceToggleTest.java | 39 +-
...ControllerPeriodicTaskStarterStatelessTest.java | 2 +-
.../controller/helix/ControllerSentinelTestV2.java | 38 +-
.../helix/ControllerTenantStatelessTest.java | 27 +-
.../pinot/controller/helix/ControllerTest.java | 124 +-
.../pinot/controller/helix/HelixHelperTest.java | 49 +-
.../helix/PinotControllerModeStatelessTest.java | 6 +-
.../controller/helix/PinotResourceManagerTest.java | 71 +-
.../controller/helix/SegmentStatusCheckerTest.java | 427 +--
.../pinot/controller/helix/TableCacheTest.java | 139 +-
.../PinotHelixResourceManagerStatelessTest.java | 1 +
.../helix/core/PinotHelixResourceManagerTest.java | 254 +-
.../OfflineDimTableSegmentAssignmentTest.java | 2 +-
...fflineNonReplicaGroupSegmentAssignmentTest.java | 3 +-
...NonReplicaGroupTieredSegmentAssignmentTest.java | 3 +-
.../OfflineReplicaGroupSegmentAssignmentTest.java | 11 +-
...altimeNonReplicaGroupSegmentAssignmentTest.java | 6 +-
...NonReplicaGroupTieredSegmentAssignmentTest.java | 9 +-
.../segment/SegmentAssignmentUtilsTest.java | 17 +-
.../MinionInstancesCleanupTaskStatelessTest.java | 81 +
.../minion/MinionInstancesCleanupTaskTest.java | 66 -
.../core/minion/PinotTaskManagerStatelessTest.java | 158 +
.../helix/core/minion/PinotTaskManagerTest.java | 146 -
.../PinotLLCRealtimeSegmentManagerTest.java | 229 +-
.../helix/core/realtime/SegmentCompletionTest.java | 1444 ++++-----
.../segment/FlushThresholdUpdaterTest.java | 36 +-
.../TableRebalancerClusterStatelessTest.java | 2 +-
.../helix/core/rebalance/TableRebalancerTest.java | 27 +
.../helix/core/retention/RetentionManagerTest.java | 72 +-
.../core/retention/SegmentLineageCleanupTest.java | 38 +-
.../strategy/TimeRetentionStrategyTest.java | 29 +-
.../core/util/SegmentDeletionManagerTest.java | 44 +-
.../controller/recommender/TestConfigEngine.java | 104 +-
.../generator/MultiValueGeneratorHelperTest.java | 5 +-
.../data/generator/NumberGeneratorTest.java | 12 +-
.../data/generator/TimeGeneratorTest.java | 3 +-
.../realtime/provisioning/MemoryEstimatorTest.java | 12 +-
.../rules/impl/AggregateMetricsRuleTest.java | 26 +-
.../rules/impl/PinotTablePartitionRuleTest.java | 16 +-
.../rules/impl/SegmentSizeRuleTest.java | 63 +-
.../tuner/RealTimeAutoIndexTunerTest.java | 23 +-
.../controller/util/ListenerConfigUtilTest.java | 9 +-
.../pinot/controller/utils/FakeHttpServer.java | 55 +
.../controller/utils/SegmentMetadataMockUtils.java | 14 +-
.../validation/StorageQuotaCheckerTest.java | 8 +-
.../validation/ValidationManagerStatelessTest.java | 6 +-
.../validation/ValidationManagerTest.java | 62 +-
...ealtimeProvisioningInput_highIngestionRate.json | 75 +
pinot-controller/testng-statefull.xml | 110 +-
pinot-controller/testng-stateless.xml | 7 +-
pinot-core/pom.xml | 13 +-
.../apache/pinot/core/auth/BasicAuthPrincipal.java | 11 +-
.../org/apache/pinot/core/auth/BasicAuthUtils.java | 3 +-
.../pinot/core/common/BlockDocIdIterator.java | 6 +
.../apache/pinot/core/common/MinionConstants.java | 65 +-
.../apache/pinot/core/common/ObjectSerDeUtils.java | 152 +-
.../core/common/datatable/DataTableImplV3.java | 58 +-
.../core/data/manager/BaseTableDataManager.java | 234 +-
.../core/data/manager/InstanceDataManager.java | 29 +-
.../manager/offline/DimensionTableDataManager.java | 10 +-
.../manager/realtime/DefaultSegmentCommitter.java | 3 +-
.../realtime/HLRealtimeSegmentDataManager.java | 32 +-
.../realtime/LLRealtimeSegmentDataManager.java | 260 +-
.../manager/realtime/PinotFSSegmentUploader.java | 6 +-
.../manager/realtime/RealtimeTableDataManager.java | 92 +-
.../realtime/SegmentBuildTimeLeaseExtender.java | 11 +-
.../manager/realtime/SegmentCommitterFactory.java | 12 +-
.../core/data/manager/realtime/TimerService.java | 5 +-
.../core/data/table/ConcurrentIndexedTable.java | 159 +-
.../apache/pinot/core/data/table/IndexedTable.java | 132 +-
.../pinot/core/data/table/IntermediateRecord.java | 19 +-
.../java/org/apache/pinot/core/data/table/Key.java | 4 +
.../pinot/core/data/table/SimpleIndexedTable.java | 119 +-
.../apache/pinot/core/data/table/TableResizer.java | 269 +-
.../table/UnboundedConcurrentIndexedTable.java | 41 +-
.../transform/function/GeoToH3Function.java | 121 +
.../transform/function/ScalarFunctions.java | 15 +
.../transform/function/StAreaFunction.java | 10 +-
.../pinot/core/minion/RawIndexConverter.java | 2 +-
.../apache/pinot/core/minion/SegmentConverter.java | 4 +-
.../AcquireReleaseColumnsSegmentOperator.java | 84 +
.../core/operator/InstanceResponseOperator.java | 39 +-
.../operator/blocks/IntermediateResultsBlock.java | 2 +-
.../operator/blocks/PassThroughTransformBlock.java | 2 +-
.../combine/AggregationOnlyCombineOperator.java | 4 +-
.../core/operator/combine/BaseCombineOperator.java | 81 +-
.../operator/combine/CombineOperatorUtils.java | 11 +-
.../operator/combine/DistinctCombineOperator.java | 4 +-
.../operator/combine/GroupByCombineOperator.java | 46 +-
.../combine/GroupByOrderByCombineOperator.java | 179 +-
...xValueBasedSelectionOrderByCombineOperator.java | 69 +-
.../combine/SelectionOnlyCombineOperator.java | 12 +-
.../combine/SelectionOrderByCombineOperator.java | 14 +-
.../dociditerators/MVScanDocIdIterator.java | 29 +-
.../dociditerators/SVScanDocIdIterator.java | 27 +-
.../filter/RangeIndexBasedFilterOperator.java | 106 +-
.../BaseRawValueBasedPredicateEvaluator.java | 1 -
.../FSTBasedRegexpPredicateEvaluatorFactory.java | 209 +-
.../operator/filter/predicate/PredicateUtils.java | 2 +
.../predicate/RangePredicateEvaluatorFactory.java | 3 +-
.../query/AggregationGroupByOrderByOperator.java | 58 +-
.../query/DictionaryBasedDistinctOperator.java | 148 +
.../StreamingSelectionOnlyCombineOperator.java | 39 +-
.../transform/PassThroughTransformOperator.java | 4 +-
.../function/AndOperatorTransformFunction.java | 4 +-
.../transform/function/BaseTransformFunction.java | 1 +
.../function/BinaryOperatorTransformFunction.java | 90 +-
.../transform/function/CaseTransformFunction.java | 121 +-
.../DateTimeConversionTransformFunction.java | 3 +-
.../function/DateTruncTransformFunction.java | 21 +-
.../function/GroovyTransformFunction.java | 12 +-
.../function/JsonExtractKeyTransformFunction.java | 11 +-
.../JsonExtractScalarTransformFunction.java | 19 +-
.../function/LiteralTransformFunction.java | 68 +-
.../function/LogicalOperatorTransformFunction.java | 2 +-
.../function/LookupTransformFunction.java | 3 +-
.../function/MapValueTransformFunction.java | 3 +-
.../function/OrOperatorTransformFunction.java | 4 +-
.../function/RegexpExtractTransformFunction.java | 116 +
.../function/TimeConversionTransformFunction.java | 3 +-
.../function/TransformFunctionFactory.java | 44 +-
.../datetime/BaseDateTimeTransformer.java | 1 +
.../pinot/core/periodictask/BasePeriodicTask.java | 98 +-
.../pinot/core/periodictask/PeriodicTask.java | 11 +
.../core/periodictask/PeriodicTaskScheduler.java | 69 +-
.../plan/AcquireReleaseColumnsSegmentPlanNode.java | 57 +
.../plan/AggregationGroupByOrderByPlanNode.java | 25 +-
.../core/plan/AggregationGroupByPlanNode.java | 4 +-
.../pinot/core/plan/AggregationPlanNode.java | 5 +-
.../apache/pinot/core/plan/CombinePlanNode.java | 55 +-
.../core/plan/DictionaryBasedDistinctPlanNode.java | 67 +
.../apache/pinot/core/plan/DocIdSetPlanNode.java | 2 +-
.../pinot/core/plan/InstanceResponsePlanNode.java | 12 +-
.../apache/pinot/core/plan/TransformPlanNode.java | 6 +-
.../core/plan/maker/InstancePlanMakerImplV2.java | 192 +-
.../aggregation/function/AggregationFunction.java | 3 +-
.../function/AggregationFunctionFactory.java | 34 +
.../function/CountAggregationFunction.java | 6 +-
.../DistinctCountBitmapMVAggregationFunction.java | 1 +
.../DistinctCountMVAggregationFunction.java | 1 +
.../DistinctCountRawHLLAggregationFunction.java | 3 +-
...istinctCountThetaSketchAggregationFunction.java | 9 +-
.../function/IdSetAggregationFunction.java | 428 ++-
.../function/ModeAggregationFunction.java | 691 ++++
.../PercentileRawEstAggregationFunction.java | 141 +
.../PercentileRawEstMVAggregationFunction.java | 43 +
.../PercentileRawTDigestAggregationFunction.java | 142 +
.../PercentileRawTDigestMVAggregationFunction.java | 43 +
.../groupby/DefaultGroupByExecutor.java | 2 +-
.../groupby/DictionaryBasedGroupKeyGenerator.java | 26 +-
.../core/query/config/QueryExecutorConfig.java | 3 +-
.../core/query/config/QueryPlannerConfig.java | 2 +-
.../pinot/core/query/distinct/DistinctTable.java | 7 +
...ryBasedSingleColumnDistinctOrderByExecutor.java | 3 +-
.../query/executor/ServerQueryExecutorV1Impl.java | 64 +-
.../pinot/core/query/optimizer/QueryOptimizer.java | 5 +-
.../optimizer/filter/NumericalFilterOptimizer.java | 41 +-
.../statement/JsonStatementOptimizer.java | 80 +-
.../statement/StringPredicateFilterOptimizer.java | 119 +
.../query/pruner/ColumnValueSegmentPruner.java | 9 +-
.../core/query/reduce/BrokerReduceService.java | 9 +-
.../query/reduce/DistinctDataTableReducer.java | 2 +-
.../core/query/reduce/GroupByDataTableReducer.java | 23 +-
.../core/query/reduce/ResultReducerFactory.java | 2 +
.../query/reduce/SelectionDataTableReducer.java | 4 +-
.../core/query/request/context/QueryContext.java | 9 +-
.../core/query/request/context/ThreadTimer.java | 8 +-
.../query/scheduler/AbstractSchedulerGroup.java | 44 +-
.../query/scheduler/MultiLevelPriorityQueue.java | 84 +-
.../core/query/scheduler/PriorityScheduler.java | 66 +-
.../pinot/core/query/scheduler/QueryScheduler.java | 81 +-
.../query/scheduler/QuerySchedulerFactory.java | 5 +-
.../query/scheduler/SchedulerQueryContext.java | 23 +-
.../query/scheduler/fcfs/FCFSQueryScheduler.java | 6 +-
.../resources/BoundedAccountingExecutor.java | 50 +-
.../resources/PolicyBasedResourceManager.java | 14 +-
.../scheduler/resources/ResourceLimitPolicy.java | 26 +-
.../query/scheduler/resources/ResourceManager.java | 36 +-
.../resources/UnboundedResourceManager.java | 6 +-
.../scheduler/tokenbucket/TokenSchedulerGroup.java | 69 +-
.../query/selection/SelectionOperatorUtils.java | 3 +-
.../apache/pinot/core/query/utils/TrieNode.java | 6 +-
.../requesthandler/PinotQueryParserFactory.java | 3 +
.../processing/filter/RecordFilterFactory.java | 2 +
.../processing/framework/SegmentConfig.java | 2 +-
.../framework/SegmentProcessorConfig.java | 100 +-
.../framework/SegmentProcessorFramework.java | 134 +-
.../segment/processing/mapper/SegmentMapper.java | 52 +-
.../partitioner/ColumnValuePartitioner.java | 1 -
.../processing/partitioner/NoOpPartitioner.java | 1 -
.../processing/partitioner/Partitioner.java | 33 +
.../processing/partitioner/PartitionerConfig.java | 3 +-
.../processing/partitioner/PartitionerFactory.java | 4 +-
.../partitioner/RoundRobinPartitioner.java | 1 -
.../partitioner/TableConfigPartitioner.java | 1 -
.../partitioner/TransformFunctionPartitioner.java | 1 -
.../processing/timehandler/EpochTimeHandler.java | 69 +
.../processing/timehandler/NoOpTimeHandler.java | 32 +
.../processing/timehandler/TimeHandler.java | 45 +
.../processing/timehandler/TimeHandlerConfig.java | 106 +
.../processing/timehandler/TimeHandlerFactory.java | 57 +
.../TransformFunctionRecordTransformer.java | 3 +-
.../core/startree/CompositePredicateEvaluator.java | 57 +
.../apache/pinot/core/startree/StarTreeUtils.java | 155 +-
.../startree/operator/StarTreeFilterOperator.java | 117 +-
.../startree/plan/StarTreeDocIdSetPlanNode.java | 7 +-
.../core/startree/plan/StarTreeFilterPlanNode.java | 9 +-
.../startree/plan/StarTreeProjectionPlanNode.java | 4 +-
.../startree/plan/StarTreeTransformPlanNode.java | 5 +-
.../pinot/core/transport/AsyncQueryResponse.java | 4 +-
.../core/transport/InstanceRequestHandler.java | 24 +-
.../pinot/core/transport/ListenerConfig.java | 10 +-
.../apache/pinot/core/transport/QueryServer.java | 1 -
.../pinot/core/transport/ServerChannels.java | 6 +-
.../pinot/core/transport/ServerInstance.java | 13 +-
.../org/apache/pinot/core/util/GroupByUtils.java | 28 +-
.../apache/pinot/core/util/ListenerConfigUtil.java | 53 +-
.../pinot/core/util/PeerServerSegmentFinder.java | 14 +-
.../org/apache/pinot/core/util/QueryOptions.java | 44 +-
.../core/util/SegmentCompletionProtocolUtils.java | 5 +
.../pinot/core/util/SegmentProcessorAvroUtils.java | 3 +-
.../java/org/apache/pinot/core/util/SizeUtil.java | 5 +-
.../pinot/core/util/SortedRangeIntersection.java | 25 +-
.../java/org/apache/pinot/core/util/TlsUtils.java | 16 +-
.../server/realtime/ControllerLeaderLocator.java | 79 +-
.../ServerSegmentCompletionProtocolHandler.java | 8 +-
.../org/apache/pinot/core/auth/BasicAuthTest.java | 51 +
.../pinot/core/common/ObjectSerDeUtilsTest.java | 68 +
.../core/common/datatable/DataTableSerDeTest.java | 4 +
.../core/common/datatable/DataTableUtilsTest.java | 5 +-
.../core/data/function/ArrayFunctionsTest.java | 159 +-
.../core/data/function/DateTimeFunctionsTest.java | 352 ++-
.../data/function/GroovyFunctionEvaluatorTest.java | 36 +-
.../function/InbuiltFunctionEvaluatorTest.java | 36 +
.../core/data/function/JsonFunctionsTest.java | 41 +-
.../BaseTableDataManagerAcquireSegmentTest.java | 447 +++
.../data/manager/BaseTableDataManagerTest.java | 672 ++--
.../offline/DimensionTableDataManagerTest.java | 3 +-
.../realtime/LLRealtimeSegmentDataManagerTest.java | 115 +-
.../realtime/PinotFSSegmentUploaderTest.java | 17 +-
.../realtime/RealtimeTableDataManagerTest.java | 56 +
.../pinot/core/data/table/IndexedTableTest.java | 50 +-
.../pinot/core/data/table/TableResizerTest.java | 177 +-
.../core/geospatial/serde/GeometrySerdeTest.java | 14 +-
.../core/geospatial/transform/GeoFunctionTest.java | 33 +-
.../geospatial/transform/GeoToH3FunctionTest.java | 45 +
.../geospatial/transform/StAreaFunctionTest.java | 6 +-
.../geospatial/transform/StPointFunctionTest.java | 10 +-
.../operator/ThreadCpuTimeMeasurementTest.java | 46 +-
.../operator/combine/CombineSlowOperatorsTest.java | 11 +-
.../combine/SelectionCombineOperatorTest.java | 4 +-
.../dociditerators/AndDocIdIteratorTest.java | 7 +-
.../dociditerators/OrDocIdIteratorTest.java | 7 +-
.../predicate/PredicateEvaluatorTestUtils.java | 3 +
...ngeOfflineDictionaryPredicateEvaluatorTest.java | 23 +-
.../function/AdditionTransformFunctionTest.java | 7 +-
.../function/AndOperatorTransformFunctionTest.java | 4 +-
.../function/ArrayBaseTransformFunctionTest.java | 12 +-
.../BinaryOperatorTransformFunctionTest.java | 10 +-
.../function/CaseTransformFunctionTest.java | 68 +-
.../DateTimeConversionTransformFunctionTest.java | 18 +-
.../function/DivisionTransformFunctionTest.java | 10 +-
.../function/GroovyTransformFunctionTest.java | 49 +-
.../JsonExtractScalarTransformFunctionTest.java | 9 +-
.../function/LiteralTransformFunctionTest.java | 20 +-
.../LogicalOperatorTransformFunctionTest.java | 2 +-
.../function/LookupTransformFunctionTest.java | 80 +-
.../function/ModuloTransformFunctionTest.java | 10 +-
.../MultiplicationTransformFunctionTest.java | 7 +-
.../function/OrOperatorTransformFunctionTest.java | 4 +-
.../function/RegexpTransformFunctionTest.java | 74 +
.../function/SubtractionTransformFunctionTest.java | 10 +-
.../TimeConversionTransformFunctionTest.java | 29 +-
.../function/ValueInTransformFunctionTest.java | 17 +-
.../datetime/DateTimeConverterTest.java | 257 +-
.../periodictask/PeriodicTaskSchedulerTest.java | 87 +-
.../pinot/core/plan/CombinePlanNodeTest.java | 16 +-
...adataAndDictionaryAggregationPlanMakerTest.java | 107 +-
.../DoubleAggregationResultHolderTest.java | 6 +-
.../function/AggregationFunctionFactoryTest.java | 49 +
.../DictionaryBasedGroupKeyGeneratorTest.java | 3 +-
.../groupby/GroupByInSegmentTrimTest.java | 298 --
.../query/aggregation/groupby/GroupByTrimTest.java | 282 ++
.../executor/QueryExecutorExceptionsTest.java | 188 ++
.../core/query/optimizer/QueryOptimizerTest.java | 26 +-
.../filter/NumericalFilterOptimizerTest.java | 106 +-
.../filter/TimePredicateFilterOptimizerTest.java | 34 +-
.../statement/JsonStatementOptimizerTest.java | 206 +-
.../StringPredicateFilterOptimizerTest.java | 66 +
.../core/query/optimizer/statement/TestHelper.java | 59 +
.../pruner/SelectionQuerySegmentPrunerTest.java | 2 +-
.../core/query/reduce/HavingFilterHandlerTest.java | 13 +-
.../query/reduce/PostAggregationHandlerTest.java | 7 +-
.../BrokerRequestToQueryContextConverterTest.java | 21 +-
.../scheduler/MultiLevelPriorityQueueTest.java | 133 +-
.../query/scheduler/PrioritySchedulerTest.java | 90 +-
.../pinot/core/query/scheduler/TestHelper.java | 2 +
.../core/query/scheduler/TestSchedulerGroup.java | 6 +-
.../query/scheduler/TestSchedulerGroupFactory.java | 14 +-
.../scheduler/fcfs/FCFSSchedulerGroupTest.java | 6 +-
.../resources/BoundedAccountingExecutorTest.java | 36 +-
.../tokenbucket/TokenSchedulerGroupTest.java | 44 +-
.../selection/SelectionOperatorServiceTest.java | 120 +-
.../fakestream/FakePartitionLevelConsumer.java | 27 +-
.../impl/fakestream/FakeStreamConfigUtils.java | 3 +
.../impl/fakestream/FakeStreamConsumerFactory.java | 3 +-
.../impl/fakestream/FakeStreamLevelConsumer.java | 6 +-
.../fakestream/FakeStreamMetadataProvider.java | 9 +-
.../core/realtime/stream/StreamConfigTest.java | 8 +-
.../processing/framework/PartitionerTest.java | 2 +-
.../segment/processing/framework/ReducerTest.java | 24 +-
.../processing/framework/SegmentMapperTest.java | 140 +-
.../framework/SegmentProcessorFrameworkTest.java | 823 ++---
.../pinot/core/startree/v2/BaseStarTreeV2Test.java | 28 +-
.../org/apache/pinot/core/util/CrcUtilsTest.java | 5 +-
.../pinot/core/util/DoubleComparisonUtil.java | 4 +-
.../core/util/PeerServerSegmentFinderTest.java | 12 +-
.../apache/pinot/core/util/SchemaUtilsTest.java | 53 +-
.../core/util/SortedRangeIntersectionTest.java | 2 +-
.../apache/pinot/queries/BooleanQueriesTest.java | 5 +-
.../queries/DistinctCountBitmapQueriesTest.java | 14 +-
.../pinot/queries/DistinctCountQueriesTest.java | 7 +-
.../DistinctCountThetaSketchQueriesTest.java | 6 +-
.../apache/pinot/queries/DistinctQueriesTest.java | 146 +-
.../apache/pinot/queries/ExpectedQueryResult.java | 56 +
.../queries/FSTBasedRegexpLikeQueriesTest.java | 84 +-
.../apache/pinot/queries/H3IndexQueriesTest.java | 7 +-
.../org/apache/pinot/queries/IdSetQueriesTest.java | 155 +-
...erSegmentAggregationSingleValueQueriesTest.java | 6 +-
...nnerSegmentSelectionSingleValueQueriesTest.java | 2 +-
...terSegmentAggregationMultiValueQueriesTest.java | 133 +-
...erSegmentAggregationSingleValueQueriesTest.java | 129 +-
.../InterSegmentOrderByMultiValueQueriesTest.java | 17 +-
.../InterSegmentOrderBySingleValueQueriesTest.java | 290 +-
...terSegmentResultTableMultiValueQueriesTest.java | 386 ++-
...erSegmentResultTableSingleValueQueriesTest.java | 85 +-
.../pinot/queries/JsonDataTypeQueriesTest.java | 76 +-
.../apache/pinot/queries/JsonPathQueriesTest.java | 103 +-
.../org/apache/pinot/queries/ModeQueriesTest.java | 977 ++++++
.../NoDictionaryCompressionQueriesTest.java | 141 +-
.../queries/PercentileTDigestQueriesTest.java | 3 +-
.../org/apache/pinot/queries/QueriesTestUtils.java | 108 +-
.../RangePredicateWithSortedInvertedIndexTest.java | 4 +-
...SegmentPartitionedDistinctCountQueriesTest.java | 10 +-
.../queries/SegmentWithNullValueVectorTest.java | 15 +-
.../apache/pinot/queries/StUnionQueriesTest.java | 20 +-
.../pinot/queries/SumPrecisionQueriesTest.java | 41 +-
.../pinot/queries/TextSearchQueriesTest.java | 1198 ++++---
.../apache/pinot/queries/TimestampQueriesTest.java | 10 +-
.../apache/pinot/queries/TransformQueriesTest.java | 6 +-
.../converter/RealtimeSegmentConverterTest.java | 14 +-
.../realtime/ControllerLeaderLocatorTest.java | 73 +-
.../resources/data/dimBaseballTeams_schema.json | 4 +-
pinot-core/src/test/resources/log4j2.xml | 2 +-
pinot-distribution/pinot-assembly.xml | 142 +-
pinot-distribution/pom.xml | 6 +-
pinot-integration-test-base/pom.xml | 225 ++
.../tests/BaseClusterIntegrationTest.java | 590 ++++
.../integration/tests/BasicAuthTestUtils.java | 0
.../tests/ClusterIntegrationTestUtils.java | 1193 +++++++
.../pinot/integration/tests/ClusterTest.java | 494 +++
...ridClusterIntegrationTestCommandLineRunner.java | 441 +++
.../pinot/integration/tests/QueryGenerator.java | 1105 +++++++
pinot-integration-tests/pom.xml | 81 +-
.../java/org/apache/pinot/compat/tests/BaseOp.java | 12 +-
.../pinot/compat/tests/ClusterDescriptor.java | 22 +-
.../pinot/compat/tests/CompatibilityOpsRunner.java | 6 +-
.../org/apache/pinot/compat/tests/QueryOp.java | 11 +-
.../org/apache/pinot/compat/tests/SegmentOp.java | 38 +-
.../pinot/compat/tests/SqlResultComparator.java | 14 +-
.../org/apache/pinot/compat/tests/StreamOp.java | 38 +-
.../org/apache/pinot/compat/tests/TableOp.java | 25 +-
.../java/org/apache/pinot/compat/tests/Utils.java | 2 +
.../tests/AdminConsoleIntegrationTest.java | 32 +-
.../tests/BaseClusterIntegrationTest.java | 589 ----
.../tests/BaseClusterIntegrationTestSet.java | 119 +-
.../tests/ChaosMonkeyIntegrationTest.java | 36 +-
.../tests/ClusterIntegrationTestUtils.java | 1190 -------
.../pinot/integration/tests/ClusterTest.java | 493 ---
.../ControllerPeriodicTasksIntegrationTest.java | 26 +-
...vertToRawIndexMinionClusterIntegrationTest.java | 7 +-
...eTimeFieldSpecHybridClusterIntegrationTest.java | 4 +-
...ridClusterIntegrationTestCommandLineRunner.java | 440 ---
.../IngestionConfigHybridIntegrationTest.java | 31 +-
.../tests/JsonPathClusterIntegrationTest.java | 81 +-
.../tests/LLCRealtimeClusterIntegrationTest.java | 19 +-
.../MergeRollupMinionClusterIntegrationTest.java | 593 ++++
...onaryAggregationPlanClusterIntegrationTest.java | 6 +-
.../MultiNodesOfflineClusterIntegrationTest.java | 14 -
.../tests/NullHandlingIntegrationTest.java | 4 +-
.../tests/OfflineClusterIntegrationTest.java | 423 ++-
...rDownloadLLCRealtimeClusterIntegrationTest.java | 68 +-
.../pinot/integration/tests/QueryGenerator.java | 1101 -------
.../tests/RealtimeKinesisIntegrationTest.java | 93 +-
...fflineSegmentsMinionClusterIntegrationTest.java | 80 +-
.../tests/SegmentCompletionIntegrationTest.java | 5 +-
...PartitionLLCRealtimeClusterIntegrationTest.java | 23 +-
.../tests/ServerStarterIntegrationTest.java | 25 +-
.../tests/SimpleMinionClusterIntegrationTest.java | 2 +-
.../tests/StarTreeClusterIntegrationTest.java | 3 +-
.../tests/ThetaSketchIntegrationTest.java | 9 +-
.../UpsertTableSegmentUploadIntegrationTest.java | 6 +-
.../minion/tasks/TestTaskExecutorFactory.java | 5 +
.../ControllerLeaderLocatorIntegrationTest.java | 16 +-
pinot-minion/pom.xml | 2 +-
.../org/apache/pinot/minion/BaseMinionStarter.java | 32 +-
.../java/org/apache/pinot/minion/MinionConf.java | 66 +
.../org/apache/pinot/minion/MinionContext.java | 2 +-
.../executor/MinionTaskZkMetadataManager.java | 8 +-
.../minion/executor/PinotTaskExecutorFactory.java | 9 +
.../executor/TaskExecutorFactoryRegistry.java | 5 +-
.../apache/pinot/minion/metrics/MinionGauge.java | 53 -
.../apache/pinot/minion/metrics/MinionMeter.java | 59 -
.../apache/pinot/minion/metrics/MinionMetrics.java | 50 -
.../pinot/minion/metrics/MinionQueryPhase.java | 39 -
.../apache/pinot/minion/metrics/MinionTimer.java | 45 -
.../minion/taskfactory/TaskFactoryRegistry.java | 16 +-
pinot-minion/src/test/resources/log4j2.xml | 2 +-
pinot-perf/pom.xml | 37 +-
.../apache/pinot/perf/BenchmarkCombineGroupBy.java | 25 +-
.../pinot/perf/BenchmarkDictionaryCreation.java | 4 +-
.../pinot/perf/BenchmarkFixedBitIntReader.java | 5 +-
.../BenchmarkFixedBitSVForwardIndexReader.java | 5 +-
.../perf/BenchmarkFixedIntArrayOffHeapIdMap.java | 6 +-
.../pinot/perf/BenchmarkGeospatialSerde.java | 185 +-
.../apache/pinot/perf/BenchmarkIndexedTable.java | 22 +-
.../BenchmarkNoDictionaryIntegerCompression.java | 47 +-
.../perf/BenchmarkNoDictionaryLongCompression.java | 46 +-
.../BenchmarkNoDictionaryStringCompression.java | 46 +-
...BenchmarkOffheapBitmapInvertedIndexCreator.java | 56 +-
.../apache/pinot/perf/BenchmarkQueryEngine.java | 13 +-
.../org/apache/pinot/perf/BenchmarkRangeIndex.java | 497 +++
.../pinot/perf/BenchmarkRoaringBitmapCreation.java | 217 ++
.../pinot/perf/BenchmarkScanDocIdIterators.java | 246 ++
.../org/apache/pinot/perf/DictionaryDumper.java | 2 +
.../java/org/apache/pinot/perf/Distribution.java | 89 +
.../pinot/perf/ForwardIndexWriterBenchmark.java | 5 +-
.../org/apache/pinot/perf/RawIndexBenchmark.java | 5 +-
.../pinot/perf/StringDictionaryPerfTest.java | 19 +-
.../pinot-batch-ingestion-common/pom.xml | 2 +-
.../batch/common/SegmentGenerationJobUtils.java | 3 +
.../batch/common/SegmentGenerationTaskRunner.java | 16 +-
.../src/test/resources/log4j2.xml | 2 +-
.../pinot-batch-ingestion-hadoop/pom.xml | 6 +-
.../batch/hadoop/HadoopSegmentCreationMapper.java | 6 +-
.../segmentCreationAndTarPushJobSpec.yaml | 1 +
.../segmentCreationAndUriPushJobSpec.yaml | 1 +
.../src/main/resources/segmentCreationJobSpec.yaml | 1 +
.../src/main/resources/segmentTarPushJobSpec.yaml | 1 +
.../src/main/resources/segmentUriPushJobSpec.yaml | 1 +
.../src/test/resources/log4j2.xml | 2 +-
.../pinot-batch-ingestion-spark/pom.xml | 2 +-
.../spark/SparkSegmentGenerationJobRunner.java | 3 +-
.../spark/SparkSegmentMetadataPushJobRunner.java | 3 +-
.../batch/spark/SparkSegmentTarPushJobRunner.java | 3 +-
.../batch/spark/SparkSegmentUriPushJobRunner.java | 3 +-
.../segmentCreationAndTarPushJobSpec.yaml | 1 +
.../segmentCreationAndUriPushJobSpec.yaml | 1 +
.../src/main/resources/segmentCreationJobSpec.yaml | 1 +
.../src/main/resources/segmentTarPushJobSpec.yaml | 1 +
.../src/main/resources/segmentUriPushJobSpec.yaml | 1 +
.../pinot-batch-ingestion-standalone/pom.xml | 6 +-
.../ingestion/batch/standalone/JobUtils.java | 2 +
.../segmentCreationAndTarPushJobSpec.yaml | 1 +
.../segmentCreationAndUriPushJobSpec.yaml | 1 +
.../src/main/resources/segmentCreationJobSpec.yaml | 1 +
.../src/main/resources/segmentTarPushJobSpec.yaml | 1 +
.../src/main/resources/segmentUriPushJobSpec.yaml | 1 +
.../standalone/SegmentGenerationJobRunnerTest.java | 2 +-
.../src/test/resources/log4j2.xml | 2 +-
pinot-plugins/pinot-batch-ingestion/pom.xml | 2 +-
.../v0_deprecated/pinot-hadoop/pom.xml | 3 +-
.../pinot/hadoop/PinotHadoopJobLauncher.java | 2 +
.../pinot/hadoop/job/HadoopSegmentCreationJob.java | 14 +-
.../hadoop/job/HadoopSegmentPreprocessingJob.java | 13 +-
.../pinot/hadoop/job/InternalConfigConstants.java | 3 +
.../job/mappers/AvroDataPreprocessingMapper.java | 3 +-
.../hadoop/job/mappers/SegmentCreationMapper.java | 12 +-
.../job/mappers/SegmentPreprocessingMapper.java | 3 +-
.../AvroDataPreprocessingPartitioner.java | 9 +-
.../job/partitioners/GenericPartitioner.java | 13 +-
.../OrcDataPreprocessingPartitioner.java | 13 +-
.../job/preprocess/DataPreprocessingHelper.java | 3 +-
.../preprocess/DataPreprocessingHelperFactory.java | 10 +-
.../job/preprocess/OrcDataPreprocessingHelper.java | 5 +-
.../job/reducers/AvroDataPreprocessingReducer.java | 3 +-
.../utils/PinotHadoopJobPreparationHelper.java | 3 +
.../utils/preprocess/DataPreprocessingUtils.java | 4 +-
.../hadoop/data/IngestionSchemaValidatorTest.java | 75 +-
.../v0_deprecated/pinot-ingestion-common/pom.xml | 2 +-
.../pinot/ingestion/common/JobConfigConstants.java | 3 +
.../ingestion/jobs/SegmentPreprocessingJob.java | 3 +-
.../ingestion/utils/JobPreparationHelper.java | 3 +
.../v0_deprecated/pinot-spark/pom.xml | 2 +-
.../apache/pinot/spark/PinotSparkJobLauncher.java | 2 +
.../spark/jobs/SparkSegmentCreationFunction.java | 11 +-
.../pinot/spark/jobs/SparkSegmentTarPushJob.java | 12 +-
.../pinot/spark/jobs/SparkSegmentUriPushJob.java | 8 +-
.../utils/PinotSparkJobPreparationHelper.java | 3 +
.../src/test/java/SegmentCreationSparkTest.java | 131 -
.../pinot/spark/SegmentCreationSparkTest.java | 132 +
.../pinot-batch-ingestion/v0_deprecated/pom.xml | 2 +-
.../pinot-environment/pinot-azure/pom.xml | 56 +-
.../plugin/provider/AzureEnvironmentProvider.java | 41 +-
.../provider/AzureEnvironmentProviderTest.java | 40 +-
pinot-plugins/pinot-environment/pom.xml | 4 +-
pinot-plugins/pinot-file-system/pinot-adls/pom.xml | 2 +-
.../pinot/plugin/filesystem/ADLSGen2PinotFS.java | 80 +-
.../pinot/plugin/filesystem/AzurePinotFSUtil.java | 6 +-
.../filesystem/test/ADLSGen2PinotFSTest.java | 42 +-
.../filesystem/test/AzurePinotFSUtilTest.java | 15 +-
.../pinot-adls/src/test/resources/log4j2.xml | 2 +-
pinot-plugins/pinot-file-system/pinot-gcs/pom.xml | 2 +-
.../apache/pinot/plugin/filesystem/GcsPinotFS.java | 55 +-
.../org/apache/pinot/plugin/filesystem/GcsUri.java | 17 +-
.../pinot/plugin/filesystem/TestGcsPinotFS.java | 318 +-
.../apache/pinot/plugin/filesystem/TestGcsUri.java | 73 +-
pinot-plugins/pinot-file-system/pinot-hdfs/pom.xml | 2 +-
.../pinot/plugin/filesystem/HadoopPinotFS.java | 6 +-
.../pinot-hdfs/src/test/resources/log4j2.xml | 2 +-
pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 43 +-
.../apache/pinot/plugin/filesystem/S3PinotFS.java | 2 +-
.../pinot/plugin/filesystem/S3PinotFSTest.java | 24 +-
.../pinot/plugin/filesystem/S3TestUtils.java | 2 +
pinot-plugins/pinot-file-system/pom.xml | 4 +-
.../pinot-input-format/pinot-avro-base/pom.xml | 9 +-
.../avro/AvroIngestionSchemaValidator.java | 32 +-
.../plugin/inputformat/avro/AvroSchemaUtil.java | 3 +
.../pinot/plugin/inputformat/avro/AvroUtils.java | 25 +-
.../avro/AvroRecordExtractorComplexTypesTest.java | 76 +-
.../inputformat/avro/AvroRecordExtractorTest.java | 3 +-
.../plugin/inputformat/avro/AvroUtilsTest.java | 58 +-
.../pinot-avro-base/src/test/resources/log4j2.xml | 2 +-
.../pinot-input-format/pinot-avro/pom.xml | 12 +-
.../inputformat/avro/KafkaAvroMessageDecoder.java | 32 +-
.../pinot-confluent-avro/pom.xml | 2 +-
...aConfluentSchemaRegistryAvroMessageDecoder.java | 16 +-
pinot-plugins/pinot-input-format/pinot-csv/pom.xml | 12 +-
.../plugin/inputformat/csv/CSVRecordExtractor.java | 6 +-
.../inputformat/csv/CSVRecordExtractorConfig.java | 5 +-
.../plugin/inputformat/csv/CSVRecordReader.java | 20 +
.../inputformat/csv/CSVRecordExtractorTest.java | 6 +-
.../inputformat/csv/CSVRecordReaderTest.java | 72 +-
.../pinot-csv/src/test/resources/log4j2.xml | 2 +-
.../pinot-input-format/pinot-json/pom.xml | 12 +-
.../inputformat/json/JSONRecordExtractorTest.java | 6 +-
.../pinot-json/src/test/resources/log4j2.xml | 2 +-
pinot-plugins/pinot-input-format/pinot-orc/pom.xml | 2 +-
.../plugin/inputformat/orc/ORCRecordReader.java | 18 +-
.../inputformat/orc/ORCRecordExtractorTest.java | 91 +-
.../inputformat/orc/ORCRecordReaderTest.java | 5 +-
.../pinot-orc/src/test/resources/log4j2.xml | 2 +-
.../pinot-input-format/pinot-parquet/pom.xml | 2 +-
.../parquet/ParquetNativeRecordExtractor.java | 4 +
.../parquet/ParquetNativeRecordReader.java | 3 +-
.../pinot-parquet/src/test/resources/log4j2.xml | 2 +-
.../pinot-input-format/pinot-protobuf/pom.xml | 2 +-
.../inputformat/protobuf/ProtoBufFieldInfo.java | 2 +-
.../protobuf/ProtoBufRecordExtractorTest.java | 43 +-
.../pinot-protobuf/src/test/resources/log4j2.xml | 2 +-
.../pinot-input-format/pinot-thrift/pom.xml | 2 +-
.../inputformat/thrift/ThriftRecordExtractor.java | 2 +-
.../thrift/ThriftRecordExtractorTest.java | 62 +-
.../pinot-thrift/src/test/resources/log4j2.xml | 2 +-
pinot-plugins/pinot-input-format/pom.xml | 4 +-
.../pinot-metrics/pinot-dropwizard/pom.xml | 73 +
.../metrics/dropwizard/DropWizardHistogram.java | 41 +
.../metrics/dropwizard/DropwizardCounter.java | 43 +
.../plugin/metrics/dropwizard/DropwizardGauge.java | 52 +
.../metrics/dropwizard/DropwizardJmxReporter.java | 38 +
.../plugin/metrics/dropwizard/DropwizardMeter.java | 52 +
.../metrics/dropwizard/DropwizardMetered.java | 77 +
.../metrics/dropwizard/DropwizardMetric.java | 36 +
.../metrics/dropwizard/DropwizardMetricName.java | 67 +
.../dropwizard/DropwizardMetricsFactory.java | 67 +
.../dropwizard/DropwizardMetricsRegistry.java | 101 +
.../DropwizardMetricsRegistryListener.java | 36 +
.../plugin/metrics/dropwizard/DropwizardTimer.java | 87 +
pinot-plugins/pinot-metrics/pinot-yammer/pom.xml | 11 +-
.../metrics/yammer/YammerMetricsRegistry.java | 5 +-
pinot-plugins/pinot-metrics/pom.xml | 3 +-
.../pinot-minion-builtin-tasks/pom.xml | 4 +-
.../BaseMultipleSegmentsConversionExecutor.java | 51 +-
.../tasks/BaseSingleSegmentConversionExecutor.java | 28 +-
.../plugin/minion/tasks/BaseTaskExecutor.java | 9 +-
.../pinot/plugin/minion/tasks/MergeTaskUtils.java | 147 +
.../minion/tasks/SegmentConversionUtils.java | 8 +-
.../ConvertToRawIndexTaskExecutor.java | 55 -
.../ConvertToRawIndexTaskExecutorFactory.java | 44 -
.../ConvertToRawIndexTaskGenerator.java | 132 -
.../ConvertToRawIndexTaskExecutor.java | 55 +
.../ConvertToRawIndexTaskExecutorFactory.java | 49 +
.../ConvertToRawIndexTaskGenerator.java | 148 +
.../minion/tasks/merge_rollup/MergeProperties.java | 50 -
.../merge_rollup/MergeRollupTaskExecutor.java | 127 -
.../MergeRollupTaskExecutorFactory.java | 44 -
.../tasks/merge_rollup/MergeRollupTaskUtils.java | 117 -
.../tasks/mergerollup/MergeRollupTaskExecutor.java | 125 +
.../MergeRollupTaskExecutorFactory.java | 51 +
.../mergerollup/MergeRollupTaskGenerator.java | 628 ++++
.../tasks/mergerollup/MergeRollupTaskUtils.java | 58 +
.../minion/tasks/purge/PurgeTaskExecutor.java | 4 +-
.../tasks/purge/PurgeTaskExecutorFactory.java | 5 +
.../RealtimeToOfflineSegmentsTaskExecutor.java | 329 --
...altimeToOfflineSegmentsTaskExecutorFactory.java | 49 -
.../RealtimeToOfflineSegmentsTaskGenerator.java | 311 --
.../RealtimeToOfflineSegmentsTaskExecutor.java | 204 ++
...altimeToOfflineSegmentsTaskExecutorFactory.java | 57 +
.../RealtimeToOfflineSegmentsTaskGenerator.java | 318 ++
.../SegmentGenerationAndPushResult.java | 89 -
.../SegmentGenerationAndPushTaskExecutor.java | 304 --
...egmentGenerationAndPushTaskExecutorFactory.java | 44 -
.../SegmentGenerationAndPushTaskGenerator.java | 336 --
.../SegmentGenerationAndPushTaskUtils.java | 75 -
.../SegmentGenerationAndPushResult.java | 89 +
.../SegmentGenerationAndPushTaskExecutor.java | 315 ++
...egmentGenerationAndPushTaskExecutorFactory.java | 49 +
.../SegmentGenerationAndPushTaskGenerator.java | 342 ++
.../SegmentGenerationAndPushTaskUtils.java | 77 +
.../plugin/minion/tasks/MergeTaskUtilsTest.java | 172 +
.../plugin/minion/tasks/TaskRegistryTest.java | 18 +-
.../merge_rollup/MergeRollupTaskExecutorTest.java | 129 -
.../merge_rollup/MergeRollupTaskUtilsTest.java | 83 -
.../mergerollup/MergeRollupTaskExecutorTest.java | 129 +
.../mergerollup/MergeRollupTaskGeneratorTest.java | 663 ++++
.../mergerollup/MergeRollupTaskUtilsTest.java | 63 +
.../RealtimeToOfflineSegmentsTaskExecutorTest.java | 450 ---
...RealtimeToOfflineSegmentsTaskGeneratorTest.java | 457 ---
.../RealtimeToOfflineSegmentsTaskExecutorTest.java | 451 +++
...RealtimeToOfflineSegmentsTaskGeneratorTest.java | 454 +++
.../SegmentGenerationAndPushTaskGeneratorTest.java | 81 -
.../SegmentGenerationAndPushTaskGeneratorTest.java | 81 +
pinot-plugins/pinot-minion-tasks/pom.xml | 2 +-
.../pinot-segment-uploader-default/pom.xml | 4 +-
.../segmentuploader/SegmentUploaderDefault.java | 16 +-
pinot-plugins/pinot-segment-uploader/pom.xml | 2 +-
.../pinot-segment-writer-file-based/pom.xml | 4 +-
.../filebased/FileBasedSegmentWriter.java | 14 +-
.../filebased/FileBasedSegmentWriterTest.java | 10 +-
pinot-plugins/pinot-segment-writer/pom.xml | 2 +-
.../pinot-stream-ingestion/pinot-kafka-0.9/pom.xml | 2 +-
.../plugin/stream/kafka09/ConsumerAndIterator.java | 5 +-
.../plugin/stream/kafka09/KafkaBrokerWrapper.java | 2 +-
.../stream/kafka09/KafkaConnectionHandler.java | 14 +-
.../stream/kafka09/KafkaConsumerManager.java | 2 +
.../stream/kafka09/KafkaHighLevelStreamConfig.java | 20 +-
.../stream/kafka09/KafkaLowLevelStreamConfig.java | 4 +-
.../kafka09/KafkaPartitionLevelConsumer.java | 3 +-
.../stream/kafka09/KafkaStreamLevelConsumer.java | 57 +-
.../kafka09/KafkaStreamMetadataProvider.java | 17 +-
.../stream/kafka09/SimpleConsumerFactory.java | 3 +-
.../stream/kafka09/SimpleConsumerMessageBatch.java | 14 +-
.../stream/kafka09/server/KafkaDataProducer.java | 8 +-
.../kafka09/server/KafkaDataServerStartable.java | 6 +-
.../kafka09/KafkaLowLevelStreamConfigTest.java | 4 +-
.../kafka09/KafkaPartitionLevelConsumerTest.java | 68 +-
.../pinot-kafka-0.9/src/test/resources/log4j2.xml | 2 +-
.../pinot-stream-ingestion/pinot-kafka-2.0/pom.xml | 2 +-
.../plugin/stream/kafka20/KafkaMessageBatch.java | 14 +-
.../kafka20/KafkaPartitionLevelConsumer.java | 3 +-
.../stream/kafka20/KafkaStreamLevelConsumer.java | 73 +-
.../kafka20/KafkaStreamLevelConsumerManager.java | 8 +-
.../kafka20/KafkaStreamLevelStreamConfig.java | 30 +-
.../kafka20/KafkaStreamMetadataProvider.java | 7 +-
.../stream/kafka20/server/KafkaDataProducer.java | 14 +-
.../kafka20/server/KafkaDataServerStartable.java | 40 +-
.../pinot-kafka-2.0/src/test/resources/log4j2.xml | 2 +-
.../pinot-kafka-base/pom.xml | 2 +-
.../stream/kafka/KafkaJSONMessageDecoder.java | 3 +-
.../stream/kafka/KafkaStreamConfigProperties.java | 3 +
.../stream/kafka/KafkaJSONMessageDecoderTest.java | 5 +-
.../pinot-kafka-base/src/test/resources/log4j2.xml | 2 +-
.../pinot-stream-ingestion/pinot-kinesis/pom.xml | 42 +-
.../pinot/plugin/stream/kinesis/KinesisConfig.java | 4 +-
.../stream/kinesis/KinesisConnectionHandler.java | 3 +-
.../plugin/stream/kinesis/KinesisConsumer.java | 4 +-
.../kinesis/KinesisPartitionGroupOffset.java | 14 +-
.../kinesis/KinesisStreamMetadataProvider.java | 2 +-
.../plugin/stream/kinesis/KinesisConsumerTest.java | 62 +-
.../kinesis/KinesisStreamMetadataProviderTest.java | 73 +-
.../pinot-stream-ingestion/pinot-pulsar/pom.xml | 397 +++
.../stream/pulsar/MessageIdStreamOffset.java | 72 +
.../pulsar/MessageIdStreamOffsetFactory.java | 47 +
.../pinot/plugin/stream/pulsar/PulsarConfig.java | 83 +
.../stream/pulsar/PulsarConsumerFactory.java | 66 +
.../plugin/stream/pulsar/PulsarMessageBatch.java | 88 +
.../PulsarPartitionLevelConnectionHandler.java | 80 +
.../pulsar/PulsarPartitionLevelConsumer.java | 140 +
.../stream/pulsar/PulsarStreamLevelConsumer.java | 113 +
.../pulsar/PulsarStreamLevelConsumerManager.java | 178 ++
.../pulsar/PulsarStreamMetadataProvider.java | 159 +
.../plugin/stream/pulsar/PulsarConsumerTest.java | 191 ++
.../stream/pulsar/PulsarStandaloneCluster.java | 132 +
.../src/test/resources/standalone.properties | 904 ++++++
pinot-plugins/pinot-stream-ingestion/pom.xml | 3 +-
pinot-plugins/pom.xml | 7 +-
pinot-segment-local/pom.xml | 11 +-
.../local/aggregator/ValueAggregatorFactory.java | 4 +
.../segment/local/customobject/QuantileDigest.java | 469 ++-
.../customobject/SerializedQuantileDigest.java | 50 +
.../local/customobject/SerializedTDigest.java | 49 +
.../local/data/manager/TableDataManager.java | 27 +-
.../local/function/FunctionEvaluatorFactory.java | 18 +-
.../local/function/InbuiltFunctionEvaluator.java | 44 +-
.../immutable/ImmutableSegmentImpl.java | 19 +-
.../immutable/ImmutableSegmentLoader.java | 107 +-
.../indexsegment/mutable/IntermediateSegment.java | 3 +-
.../indexsegment/mutable/MutableSegmentImpl.java | 78 +-
.../local/io/compression/LZ4Compressor.java | 1 +
.../local/io/compression/LZ4Decompressor.java | 1 +
.../local/io/compression/ZstandardCompressor.java | 3 +-
.../io/compression/ZstandardDecompressor.java | 1 +
.../RealtimeIndexOffHeapMemoryManager.java | 3 +-
.../local/io/util/FixedBitIntReaderWriterV2.java | 5 +-
.../segment/local/io/util/PinotDataBitSet.java | 33 +-
.../segment/local/io/util/PinotDataBitSetV2.java | 70 +-
.../writer/impl/BaseChunkSVForwardIndexWriter.java | 8 +-
.../writer/impl/FixedBitMVForwardIndexWriter.java | 117 +-
.../impl/FixedByteChunkSVForwardIndexWriter.java | 4 +-
.../writer/impl/MutableOffHeapByteArrayStore.java | 6 +-
.../impl/VarByteChunkSVForwardIndexWriter.java | 7 +-
.../local/loader/LocalSegmentDirectoryLoader.java | 3 +-
.../local/realtime/impl/RealtimeSegmentConfig.java | 50 +-
.../realtime/impl/RealtimeSegmentStatsHistory.java | 24 +-
.../dictionary/BaseOffHeapMutableDictionary.java | 7 +-
.../forward/FixedByteMVMutableForwardIndex.java | 3 +-
.../RealtimeLuceneDocIdCollector.java | 7 +-
.../RealtimeLuceneIndexReaderRefreshThread.java | 3 +-
.../RealtimeLuceneIndexRefreshState.java | 30 +-
.../local/realtime/impl/json/MutableJsonIndex.java | 3 +-
.../recordtransformer/ComplexTypeTransformer.java | 30 +-
.../recordtransformer/CompositeTransformer.java | 3 +-
.../recordtransformer/DataTypeTransformer.java | 51 +-
.../recordtransformer/ExpressionTransformer.java | 3 +-
.../recordtransformer/NullValueTransformer.java | 32 +-
.../creator/impl/SegmentColumnarIndexCreator.java | 107 +-
.../creator/impl/SegmentCreationDriverFactory.java | 7 +-
.../impl/SegmentIndexCreationDriverImpl.java | 152 +-
.../fwd/SingleValueFixedByteRawIndexCreator.java | 8 +-
.../fwd/SingleValueVarByteRawIndexCreator.java | 8 +-
.../impl/inv/BitSlicedRangeIndexCreator.java | 143 +
.../segment/creator/impl/inv/MmapFileWriter.java | 55 +
.../creator/impl/inv/RangeIndexCreator.java | 28 +-
.../impl/inv/geospatial/OffHeapH3IndexCreator.java | 2 +-
.../impl/nullvalue/NullValueVectorCreator.java | 1 +
.../stats/AbstractColumnStatisticsCollector.java | 68 +-
.../stats/BytesColumnPredIndexStatsCollector.java | 2 +-
.../stats/DoubleColumnPreIndexStatsCollector.java | 4 +-
.../stats/FloatColumnPreIndexStatsCollector.java | 4 +-
.../stats/IntColumnPreIndexStatsCollector.java | 4 +-
.../stats/LongColumnPreIndexStatsCollector.java | 4 +-
.../stats/SegmentPreIndexStatsCollectorImpl.java | 37 +-
.../stats/StringColumnPreIndexStatsCollector.java | 4 +-
.../index/column/PhysicalColumnIndexContainer.java | 22 +-
.../segment/index/datasource/BaseDataSource.java | 7 +-
.../index/datasource/MutableDataSource.java | 3 +-
.../local/segment/index/loader/IndexHandler.java | 31 +
.../segment/index/loader/IndexHandlerFactory.java | 62 +
.../segment/index/loader/IndexLoadingConfig.java | 9 +-
.../segment/index/loader/SegmentPreProcessor.java | 160 +-
.../index/loader/V3RemoveIndexException.java | 29 -
.../index/loader/V3UpdateIndexException.java | 29 -
.../loader/bloomfilter/BloomFilterHandler.java | 56 +-
.../ColumnMinMaxValueGenerator.java | 6 +-
.../defaultcolumn/BaseDefaultColumnHandler.java | 28 +-
.../defaultcolumn/V1DefaultColumnHandler.java | 2 +-
.../defaultcolumn/V3DefaultColumnHandler.java | 50 +-
.../index/loader/invertedindex/H3IndexHandler.java | 64 +-
.../loader/invertedindex/InvertedIndexHandler.java | 59 +-
.../loader/invertedindex/JsonIndexHandler.java | 61 +-
.../invertedindex/LuceneFSTIndexHandler.java | 57 +-
.../loader/invertedindex/RangeIndexHandler.java | 77 +-
.../loader/invertedindex/TextIndexHandler.java | 57 +-
.../index/readers/BitSlicedRangeIndexReader.java | 137 +
.../index/readers/BitmapInvertedIndexReader.java | 29 -
.../index/readers/LuceneFSTIndexReader.java | 8 +-
.../index/readers/NullValueVectorReaderImpl.java | 8 +-
.../segment/index/readers/RangeIndexReader.java | 220 --
.../index/readers/RangeIndexReaderImpl.java | 265 ++
.../forward/BaseChunkSVForwardIndexReader.java | 6 +-
.../readers/json/ImmutableJsonIndexReader.java | 4 +-
.../index/readers/text/LuceneDocIdCollector.java | 6 +-
.../readers/MultiplePinotSegmentRecordReader.java | 2 +-
.../local/segment/store/FilePerIndexDirectory.java | 53 +-
.../segment/local/segment/store/IndexEntry.java | 20 +-
.../segment/local/segment/store/IndexKey.java | 30 +-
.../segment/store/SegmentLocalFSDirectory.java | 53 +-
.../segment/store/SingleFileIndexDirectory.java | 201 +-
.../local/segment/store/TextIndexUtils.java | 41 +
.../virtualcolumn/VirtualColumnIndexContainer.java | 3 +-
.../VirtualColumnProviderFactory.java | 3 +
.../startree/v2/builder/BaseSingleTreeBuilder.java | 4 +-
.../startree/v2/builder/StarTreeIndexCombiner.java | 6 +-
.../v2/builder/StarTreeV2BuilderConfig.java | 6 +-
.../startree/v2/store/StarTreeDataSource.java | 5 +-
.../startree/v2/store/StarTreeIndexMapUtils.java | 11 +-
.../upsert/PartitionUpsertMetadataManager.java | 215 +-
.../pinot/segment/local/upsert/RecordLocation.java | 11 +-
.../local/upsert/TableUpsertMetadataManager.java | 8 +-
.../segment/local/upsert/merger/UnionMerger.java | 5 +-
.../apache/pinot/segment/local/utils/CrcUtils.java | 3 +-
.../pinot/segment/local/utils/FPOrdering.java | 78 +
.../segment/local/utils/GeometrySerializer.java | 8 +-
.../pinot/segment/local/utils/GeometryType.java | 8 +-
.../pinot/segment/local/utils/GeometryUtils.java | 12 +-
.../pinot/segment/local/utils/HashUtils.java | 35 +
.../pinot/segment/local/utils/IngestionUtils.java | 33 +-
.../segment/local/utils/ReplicationUtils.java | 12 +-
.../pinot/segment/local/utils/SchemaUtils.java | 17 +-
.../segment/local/utils/SegmentPushUtils.java | 6 +-
.../segment/local/utils/TableConfigUtils.java | 92 +-
.../local/utils/fst/PinotBufferIndexInput.java | 94 +-
.../segment/local/utils/fst/RegexpMatcher.java | 39 +-
.../mutable/MutableSegmentImplTestUtils.java | 8 +-
.../MutableSegmentImplUpsertComparisonColTest.java | 91 +
.../mutable/MutableSegmentImplUpsertTest.java | 31 +-
.../segment/local/io/util/PinotDataBitSetTest.java | 5 +-
.../impl/RealtimeSegmentStatsHistoryTest.java | 3 +-
.../impl/dictionary/MutableDictionaryTest.java | 6 +-
.../recordtransformer/DataTypeTransformerTest.java | 12 +-
.../ExpressionTransformerTest.java | 39 +-
.../ExpressionTransformerTimeTest.java | 24 +-
.../recordtransformer/RecordTransformerTest.java | 122 +-
.../segment/local/segment/creator/BlocksTest.java | 5 +-
.../local/segment/creator/DictionariesTest.java | 24 +-
.../local/segment/creator/SegmentTestUtils.java | 3 +
.../local/segment/index/ColumnMetadataTest.java | 8 +-
.../segment/local/segment/index/JsonIndexTest.java | 27 +-
.../segment/index/SegmentMetadataImplTest.java | 8 +-
.../SegmentV1V2ToV3FormatConverterTest.java | 2 +-
.../index/creator/BitSlicedIndexCreatorTest.java | 521 +++
.../index/creator/RangeIndexCreatorTest.java | 430 ++-
.../SegmentGenerationWithBytesTypeTest.java | 22 +-
.../SegmentGenerationWithFilterRecordsTest.java | 7 +-
.../SegmentGenerationWithNoRecordsTest.java | 21 +-
.../SegmentGenerationWithTimeColumnTest.java | 76 +-
.../index/creator/SegmentPartitionTest.java | 3 +-
.../forward/VarByteChunkSVForwardIndexTest.java | 10 +-
.../index/loader/SegmentPreProcessorTest.java | 629 ++--
.../BaseDefaultColumnHandlerTest.java | 35 +-
.../FixedByteWidthRowColDataFileReaderTest.java | 2 -
...FixedByteWidthRowColForwardIndexWriterTest.java | 13 +-
.../MultiplePinotSegmentRecordReaderTest.java | 24 +-
.../readers/PinotSegmentRecordReaderTest.java | 22 +-
.../local/segment/readers/PinotSegmentUtil.java | 3 +-
.../readers/RecordReaderSampleDataTest.java | 27 +-
.../store/ColumnIndexDirectoryTestHelper.java | 15 +-
.../segment/store/FilePerIndexDirectoryTest.java | 176 +-
.../segment/local/segment/store/IndexKeyTest.java | 43 +
.../segment/store/SegmentLocalFSDirectoryTest.java | 40 +-
.../store/SingleFileIndexDirectoryTest.java | 268 +-
.../upsert/PartitionUpsertMetadataManagerTest.java | 134 +-
.../upsert/merger/PartialUpsertMergerTest.java | 8 +-
.../pinot/segment/local/utils/FPOrderingTest.java | 134 +
.../pinot/segment/local/utils/HashUtilsTest.java | 34 +
.../segment/local/utils/IngestionUtilsTest.java | 6 +-
.../segment/local/utils/TableConfigUtilsTest.java | 143 +-
.../segment/local/utils/fst/FSTBuilderTest.java | 98 +-
.../src/test/resources/data/log4j2.xml | 2 +-
.../src/test/resources/data/newColumnsSchema3.json | 4 -
...nsSchema4.json => newColumnsSchemaWithFST.json} | 0
.../resources/data/newColumnsSchemaWithH3Json.json | 78 +
...xtSchema.json => newColumnsSchemaWithText.json} | 0
.../data/test_upsert_comparison_col_data.json | 26 +
.../data/test_upsert_comparison_col_schema.json | 25 +
pinot-segment-spi/pom.xml | 13 +-
.../pinot/segment/spi/AggregationFunctionType.java | 13 +
.../org/apache/pinot/segment/spi/Constants.java | 2 +
.../org/apache/pinot/segment/spi/FetchContext.java | 60 +
.../org/apache/pinot/segment/spi/IndexSegment.java | 24 +-
.../org/apache/pinot/segment/spi/V1Constants.java | 6 +-
.../segment/spi/compression/ChunkDecompressor.java | 3 +-
.../spi/creator/ColumnIndexCreationInfo.java | 52 +-
.../segment/spi/creator/ColumnStatistics.java | 3 +-
.../spi/creator/SegmentGeneratorConfig.java | 18 +-
.../segment/spi/creator/StatsCollectorConfig.java | 3 +-
.../creator/name/FixedSegmentNameGenerator.java | 4 +-
.../name/InputFileSegmentNameGenerator.java | 89 +
.../name/NormalizedDateSegmentNameGenerator.java | 7 +-
.../spi/creator/name/SegmentNameGenerator.java | 12 +
.../creator/name/SegmentNameGeneratorFactory.java | 72 +
.../creator/name/SimpleSegmentNameGenerator.java | 13 +-
.../pinot/segment/spi/datasource/DataSource.java | 3 +-
.../spi/index/column/ColumnIndexContainer.java | 3 +-
.../creator/CombinedInvertedIndexCreator.java | 23 +
.../spi/index/creator/GeoSpatialIndexCreator.java | 2 +-
.../creator/RawValueBasedInvertedIndexCreator.java | 2 -
.../index/creator/SegmentIndexCreationInfo.java | 6 +-
.../segment/spi/index/creator/TextIndexType.java | 4 +-
.../spi/index/metadata/SegmentMetadataImpl.java | 2 +-
.../spi/index/reader/H3IndexResolution.java | 3 +-
.../spi/index/reader/MutableForwardIndex.java | 2 -
.../segment/spi/index/reader/RangeIndexReader.java | 120 +
.../spi/index/reader/SortedIndexReader.java | 3 +-
.../spi/loader/SegmentDirectoryLoaderRegistry.java | 12 +-
.../pinot/segment/spi/memory/CleanerUtil.java | 85 +-
.../pinot/segment/spi/memory/PinotDataBuffer.java | 7 +-
.../spi/partition/MurmurPartitionFunction.java | 2 +
.../spi/partition/PartitionFunctionFactory.java | 3 +-
.../pinot/segment/spi/partition/Partitioner.java | 33 -
.../segment/spi/store/ColumnIndexDirectory.java | 24 +-
.../pinot/segment/spi/store/ColumnIndexType.java | 6 +-
.../pinot/segment/spi/store/SegmentDirectory.java | 37 +-
.../segment/spi/store/SegmentDirectoryPaths.java | 7 +-
.../name/InputFileSegmentNameGeneratorTest.java | 64 +
.../NormalizedDateSegmentNameGeneratorTest.java | 55 +-
.../name/SimpleSegmentNameGeneratorTest.java | 27 +
.../misc/AggregationFunctionColumnPairTest.java | 18 +
.../spi/partition/PartitionFunctionTest.java | 17 +-
pinot-server/pom.xml | 2 +-
.../server/api/access/AccessControlFactory.java | 2 +-
.../pinot/server/api/resources/DebugResource.java | 3 +-
.../pinot/server/api/resources/ErrorInfo.java | 12 +-
.../pinot/server/api/resources/ErrorResponse.java | 6 +-
.../server/api/resources/HealthCheckResource.java | 5 +-
.../server/api/resources/MmapDebugResource.java | 12 +-
.../api/resources/SegmentMetadataFetcher.java | 124 +-
.../server/api/resources/TableSizeResource.java | 28 +-
.../pinot/server/api/resources/TablesResource.java | 185 +-
.../org/apache/pinot/server/conf/ServerConf.java | 10 +-
.../pinot/server/starter/ServerInstance.java | 3 +-
.../starter/ServerQueriesDisabledTracker.java | 10 +-
.../server/starter/helix/AdminApiApplication.java | 40 +-
.../server/starter/helix/BaseServerStarter.java | 67 +-
.../helix/DefaultHelixStarterServerConfig.java | 3 +
.../starter/helix/HelixInstanceDataManager.java | 101 +-
.../helix/HelixInstanceDataManagerConfig.java | 3 +-
.../server/starter/helix/HelixServerStarter.java | 6 +-
.../helix/OffsetBasedConsumptionStatusChecker.java | 114 +
.../starter/helix/SegmentFetcherAndLoader.java | 241 --
.../pinot/server/starter/helix/SegmentLocks.java | 3 +
.../helix/SegmentMessageHandlerFactory.java | 21 +-
.../SegmentOnlineOfflineStateModelFactory.java | 18 +-
.../apache/pinot/server/api/AccessControlTest.java | 9 +-
.../apache/pinot/server/api/BaseResourceTest.java | 8 +-
.../server/api/PinotServerAppConfigsTest.java | 12 +-
.../pinot/server/api/TableSizeResourceTest.java | 30 +-
.../pinot/server/api/TablesResourceTest.java | 14 +-
.../OffsetBasedConsumptionStatusCheckerTest.java | 251 ++
pinot-server/src/test/resources/log4j2.xml | 2 +-
pinot-spi/pom.xml | 9 +-
.../pinot/spi/annotations/InterfaceStability.java | 2 +-
.../spi/annotations/metrics/MetricsFactory.java | 3 +-
.../annotations/metrics/PinotMetricsFactory.java | 1 +
.../annotations/minion/EventObserverFactory.java | 3 +-
.../annotations/minion/TaskExecutorFactory.java | 3 +-
.../spi/annotations/minion/TaskGenerator.java | 3 +-
.../org/apache/pinot/spi/config/ConfigUtils.java | 5 +-
.../apache/pinot/spi/config/instance/Instance.java | 7 +-
.../pinot/spi/config/table/CompletionConfig.java | 4 +-
.../apache/pinot/spi/config/table/FieldConfig.java | 30 +-
.../pinot/spi/config/table/IndexingConfig.java | 30 +-
.../spi/config/table/SegmentPartitionConfig.java | 4 +-
.../SegmentsValidationAndRetentionConfig.java | 13 +-
.../apache/pinot/spi/config/table/TableConfig.java | 20 +-
.../apache/pinot/spi/config/table/TableStatus.java | 5 +-
.../pinot/spi/config/table/TableTaskConfig.java | 4 +-
.../apache/pinot/spi/config/table/TierConfig.java | 4 +-
.../pinot/spi/config/table/UpsertConfig.java | 36 +-
.../table/assignment/InstanceAssignmentConfig.java | 9 +-
.../InstanceReplicaGroupPartitionConfig.java | 15 +-
.../table/assignment/InstanceTagPoolConfig.java | 8 +-
.../config/table/ingestion/IngestionConfig.java | 3 +-
.../config/table/ingestion/TransformConfig.java | 3 +-
.../pinot/spi/crypt/PinotCrypterFactory.java | 2 +-
.../apache/pinot/spi/data/ComplexFieldSpec.java | 2 +-
.../apache/pinot/spi/data/DateTimeFieldSpec.java | 10 +-
.../pinot/spi/data/DateTimeFormatPatternSpec.java | 10 +-
.../java/org/apache/pinot/spi/data/FieldSpec.java | 5 +-
.../java/org/apache/pinot/spi/data/Schema.java | 48 +-
.../pinot/spi/data/SchemaValidatorFactory.java | 6 +-
.../org/apache/pinot/spi/data/TimeFieldSpec.java | 4 +-
.../apache/pinot/spi/data/TimeGranularitySpec.java | 7 +-
.../apache/pinot/spi/data/readers/GenericRow.java | 9 +-
.../apache/pinot/spi/data/readers/PrimaryKey.java | 5 +
.../spi/data/readers/RecordReaderFactory.java | 12 +-
.../java/org/apache/pinot/spi/env/Environment.java | 1 +
.../apache/pinot/spi/env/PinotConfiguration.java | 152 +-
.../PinotEnvironmentProvider.java | 9 +-
.../PinotEnvironmentProviderFactory.java | 27 +-
.../spi/exception/EarlyTerminationException.java | 1 -
.../org/apache/pinot/spi/filesystem/PinotFS.java | 18 +-
.../spi/ingestion/batch/BatchConfigProperties.java | 3 +
.../spi/ingestion/batch/IngestionJobLauncher.java | 38 +-
.../pinot/spi/ingestion/batch/spec/Constants.java | 3 +
.../batch/spec/SegmentGenerationJobSpec.java | 18 +-
.../batch/spec/SegmentGenerationTaskSpec.java | 7 +
.../pinot/spi/ingestion/batch/spec/TlsSpec.java | 24 +-
.../segment/uploader/SegmentUploader.java | 10 +-
.../ingestion/segment/writer/SegmentWriter.java | 10 +-
.../org/apache/pinot/spi/metrics/PinotMeter.java | 1 -
.../org/apache/pinot/spi/metrics/PinotMetered.java | 1 +
.../apache/pinot/spi/metrics/PinotMetricName.java | 3 +-
.../pinot/spi/metrics/PinotMetricsRegistry.java | 3 +-
.../apache/pinot/spi/plugin/PluginClassLoader.java | 62 +-
.../org/apache/pinot/spi/plugin/PluginManager.java | 101 +-
.../org/apache/pinot/spi/stream/LongMsgOffset.java | 4 +-
.../apache/pinot/spi/stream/OffsetCriteria.java | 4 +-
.../pinot/spi/stream/PartitionGroupConsumer.java | 3 +-
.../stream/PartitionGroupConsumptionStatus.java | 4 +-
.../pinot/spi/stream/PartitionGroupMetadata.java | 3 +-
.../spi/stream/PartitionGroupMetadataFetcher.java | 10 +-
.../pinot/spi/stream/PartitionLevelConsumer.java | 9 +-
.../org/apache/pinot/spi/stream/StreamConfig.java | 9 +-
.../pinot/spi/stream/StreamConfigProperties.java | 9 +-
.../pinot/spi/stream/StreamConsumerFactory.java | 1 -
.../spi/stream/StreamConsumerFactoryProvider.java | 4 +-
.../pinot/spi/stream/StreamDataProvider.java | 5 +-
.../pinot/spi/stream/StreamLevelConsumer.java | 3 +-
.../pinot/spi/stream/StreamMessageDecoder.java | 3 +-
.../pinot/spi/stream/StreamMessageMetadata.java | 1 -
.../pinot/spi/stream/StreamMetadataProvider.java | 19 +-
.../pinot/spi/stream/StreamPartitionMsgOffset.java | 1 +
.../org/apache/pinot/spi/utils/BooleanUtils.java | 2 +
.../java/org/apache/pinot/spi/utils/ByteArray.java | 77 +-
.../apache/pinot/spi/utils/CommonConstants.java | 62 +-
.../org/apache/pinot/spi/utils/DataSizeUtils.java | 10 +-
.../org/apache/pinot/spi/utils/EqualityUtils.java | 20 +-
.../pinot/spi/utils/GroovyTemplateUtils.java | 3 +
.../pinot/spi/utils/IngestionConfigUtils.java | 11 +-
.../java/org/apache/pinot/spi/utils/JsonUtils.java | 13 +-
.../java/org/apache/pinot/spi/utils/NetUtils.java | 3 +
.../org/apache/pinot/spi/utils/Obfuscator.java | 140 +
.../main/java/org/apache/pinot/spi/utils/Pair.java | 16 +-
.../java/org/apache/pinot/spi/utils/Pairs.java | 19 +-
.../pinot/spi/utils/PinotReflectionUtils.java | 6 +-
.../org/apache/pinot/spi/utils/ResourceFinder.java | 2 +
.../java/org/apache/pinot/spi/utils/TimeUtils.java | 3 +
.../org/apache/pinot/spi/utils/TimestampUtils.java | 2 +
.../spi/utils/builder/TableConfigBuilder.java | 7 +
.../pinot/spi/utils/retry/BaseRetryPolicy.java | 2 +-
.../apache/pinot/spi/config/ConfigUtilsTest.java | 17 +
.../pinot/spi/config/table/IndexingConfigTest.java | 2 +
.../pinot/spi/config/table/UpsertConfigTest.java | 10 +-
.../data/readers/AbstractRecordExtractorTest.java | 3 +-
.../spi/data/readers/AbstractRecordReaderTest.java | 3 +-
.../pinot/spi/data/readers/GenericRowTest.java | 1 +
.../pinot/spi/data/readers/PrimaryKeyTest.java | 11 +
.../spi/data/readers/RecordReaderFactoryTest.java | 4 +-
.../pinot/spi/env/PinotConfigurationTest.java | 14 +-
.../PinotEnvironmentProviderFactoryTest.java | 26 +-
.../pinot/spi/filesystem/LocalPinotFSTest.java | 8 +-
.../pinot/spi/filesystem/PinotFSFactoryTest.java | 6 +-
.../apache/pinot/spi/filesystem/PinotFSTest.java | 181 ++
.../pinot/spi/ingestion/batch/BatchConfigTest.java | 8 +-
.../ingestion/batch/IngestionJobLauncherTest.java | 9 +
.../apache/pinot/spi/plugin/PluginManagerTest.java | 32 +-
.../org/apache/pinot/spi/utils/ByteArrayTest.java | 65 +
.../pinot/spi/utils/IngestionConfigUtilsTest.java | 8 +-
.../org/apache/pinot/spi/utils/JsonUtilsTest.java | 21 +-
.../org/apache/pinot/spi/utils/ObfuscatorTest.java | 119 +
.../spi/utils/builder/TableNameBuilderTest.java | 2 +-
.../pinot/spi/utils/retry/RetryPolicyTest.java | 12 +-
pinot-spi/src/test/resources/TestRecordReader.java | 3 +-
.../resources/ingestion_job_spec_template.yaml | 1 +
.../test/resources/ingestion_job_spec_unicode.yaml | 48 +
pinot-spi/src/test/resources/log4j2.xml | 2 +-
pinot-tools/pom.xml | 3 +-
.../org/apache/pinot/tools/AuthQuickstart.java | 5 +-
.../pinot/tools/AutoAddInvertedIndexTool.java | 30 +-
.../org/apache/pinot/tools/BootstrapTableTool.java | 8 +-
.../apache/pinot/tools/ClusterStateVerifier.java | 6 +-
.../apache/pinot/tools/GitHubEventsQuickstart.java | 3 +-
.../org/apache/pinot/tools/HybridQuickstart.java | 14 +-
.../org/apache/pinot/tools/JoinQuickStart.java | 6 +-
.../apache/pinot/tools/JsonIndexQuickStart.java | 3 +-
.../OfflineComplexTypeHandlingQuickStart.java | 6 +-
.../pinot/tools/PartialUpsertQuickStart.java | 12 +-
.../apache/pinot/tools/PinotNumReplicaChanger.java | 12 +-
.../apache/pinot/tools/PinotTableRebalancer.java | 4 +-
.../org/apache/pinot/tools/PinotToolLauncher.java | 3 +-
.../org/apache/pinot/tools/PinotZKChanger.java | 26 +-
.../org/apache/pinot/tools/QuickStartBase.java | 2 +-
.../java/org/apache/pinot/tools/Quickstart.java | 7 +-
.../apache/pinot/tools/QuickstartTableRequest.java | 28 +-
.../RealtimeComplexTypeHandlingQuickStart.java | 6 +-
.../pinot/tools/RealtimeJsonIndexQuickStart.java | 4 +-
.../org/apache/pinot/tools/RealtimeQuickStart.java | 6 +-
.../org/apache/pinot/tools/SegmentDumpTool.java | 4 +-
.../java/org/apache/pinot/tools/SpeedTest.java | 70 +-
.../apache/pinot/tools/StarTreeIndexViewer.java | 6 +-
.../org/apache/pinot/tools/UpdateSegmentState.java | 36 +-
.../apache/pinot/tools/ValidateTableRetention.java | 10 +-
.../pinot/tools/admin/PinotAdministrator.java | 9 +-
.../org/apache/pinot/tools/admin/PinotBroker.java | 3 +
.../apache/pinot/tools/admin/PinotController.java | 3 +
.../org/apache/pinot/tools/admin/PinotMinion.java | 3 +
.../org/apache/pinot/tools/admin/PinotServer.java | 3 +
.../tools/admin/PinotServiceManagerStarter.java | 4 +
.../admin/command/AbstractBaseAdminCommand.java | 11 +-
.../tools/admin/command/AddSchemaCommand.java | 3 +-
.../pinot/tools/admin/command/AddTableCommand.java | 12 +-
.../tools/admin/command/AddTenantCommand.java | 15 +-
.../tools/admin/command/AnonymizeDataCommand.java | 108 +-
.../admin/command/AvroSchemaToPinotSchema.java | 34 +-
.../tools/admin/command/BootstrapTableCommand.java | 9 +-
.../admin/command/ChangeNumReplicasCommand.java | 12 +-
.../tools/admin/command/ChangeTableState.java | 3 +-
.../tools/admin/command/CreateSegmentCommand.java | 10 +-
.../tools/admin/command/DeleteClusterCommand.java | 3 +-
.../tools/admin/command/GenerateDataCommand.java | 14 +-
.../command/GitHubEventsQuickStartCommand.java | 1 +
.../tools/admin/command/ImportDataCommand.java | 14 +-
.../tools/admin/command/JsonToPinotSchema.java | 38 +-
.../command/LaunchDataIngestionJobCommand.java | 23 +-
.../tools/admin/command/MoveReplicaGroup.java | 156 +-
.../OfflineSegmentIntervalCheckerCommand.java | 17 +-
.../admin/command/OperateClusterConfigCommand.java | 9 +-
.../tools/admin/command/PostQueryCommand.java | 3 +-
.../tools/admin/command/QuickStartCommand.java | 9 +-
.../command/RealtimeProvisioningHelperCommand.java | 76 +-
.../tools/admin/command/RebalanceTableCommand.java | 70 +-
.../command/SegmentProcessorFrameworkCommand.java | 64 +-
.../admin/command/ShowClusterInfoCommand.java | 95 +-
.../tools/admin/command/StartBrokerCommand.java | 22 +-
.../admin/command/StartControllerCommand.java | 23 +-
.../tools/admin/command/StartKafkaCommand.java | 3 +-
.../tools/admin/command/StartMinionCommand.java | 14 +-
.../tools/admin/command/StartServerCommand.java | 32 +-
.../admin/command/StartServiceManagerCommand.java | 59 +-
.../tools/admin/command/StartZookeeperCommand.java | 10 +-
.../tools/admin/command/StopProcessCommand.java | 3 +-
.../admin/command/StreamAvroIntoKafkaCommand.java | 10 +-
.../admin/command/StreamGitHubEventsCommand.java | 13 +-
.../tools/admin/command/UploadSegmentCommand.java | 16 +-
.../tools/admin/command/ValidateConfigCommand.java | 13 +-
.../admin/command/VerifyClusterStateCommand.java | 6 +-
.../tools/admin/command/VerifySegmentState.java | 26 +-
.../anonymizer/ArrayBasedGlobalDictionaries.java | 58 +-
.../pinot/tools/anonymizer/GlobalDictionaries.java | 3 +-
.../anonymizer/MapBasedGlobalDictionaries.java | 30 +-
.../anonymizer/PinotDataAndQueryAnonymizer.java | 192 +-
.../pinot/tools/backfill/BackfillSegmentUtils.java | 2 +-
.../config/validator/TableConfigValidator.java | 1 +
.../tools/filesystem/PinotFSBenchmarkDriver.java | 35 +-
.../tools/filesystem/PinotFSBenchmarkRunner.java | 30 +-
.../pinot/tools/perf/PerfBenchmarkDriver.java | 12 +-
.../pinot/tools/perf/PerfBenchmarkDriverConf.java | 157 +-
.../pinot/tools/perf/PerfBenchmarkRunner.java | 30 +-
.../org/apache/pinot/tools/perf/QueryRunner.java | 88 +-
.../tools/query/comparison/ClusterStarter.java | 5 +-
.../tools/query/comparison/QueryComparison.java | 24 +-
.../query/comparison/SegmentInfoProvider.java | 3 +
.../tools/query/comparison/StatsGenerator.java | 7 +-
.../apache/pinot/tools/scan/query/Aggregation.java | 2 +-
.../tools/scan/query/AggregationFuncFactory.java | 3 +
.../apache/pinot/tools/scan/query/AvgFunction.java | 8 +-
.../pinot/tools/scan/query/CountFunction.java | 6 +-
.../tools/scan/query/DistinctCountFunction.java | 6 +-
.../tools/scan/query/EqualsPredicateFilter.java | 2 +-
.../pinot/tools/scan/query/GroupByOperator.java | 2 +-
.../pinot/tools/scan/query/InPredicateFilter.java | 2 +-
.../apache/pinot/tools/scan/query/MaxFunction.java | 4 +-
.../apache/pinot/tools/scan/query/MinFunction.java | 4 +-
.../tools/scan/query/MinMaxRangeFunction.java | 6 +-
.../tools/scan/query/NotInPredicateFilter.java | 2 +-
.../pinot/tools/scan/query/NotPredicateFilter.java | 2 +-
.../apache/pinot/tools/scan/query/Projection.java | 4 +-
.../pinot/tools/scan/query/QueryResponse.java | 8 +-
.../tools/scan/query/RangePredicateFilter.java | 4 +-
.../apache/pinot/tools/scan/query/ResultTable.java | 12 +-
.../tools/scan/query/ScanBasedQueryProcessor.java | 2 +-
.../tools/scan/query/SegmentQueryProcessor.java | 7 +-
.../apache/pinot/tools/scan/query/SumFunction.java | 4 +-
.../org/apache/pinot/tools/scan/query/Utils.java | 2 +
.../converter/PinotSegmentConvertCommand.java | 15 +-
.../segment/converter/SegmentMergeCommand.java | 21 +-
.../processor/SegmentProcessorFrameworkSpec.java | 40 +-
.../pinot/tools/service/PinotServiceManager.java | 23 +-
.../service/PinotServiceManagerStatusCallback.java | 7 +-
.../resources/PinotServiceManagerHealthCheck.java | 15 +-
.../PinotServiceManagerInstanceResource.java | 71 +-
.../pinot/tools/streams/AirlineDataStream.java | 79 +-
.../streams/githubevents/GitHubAPICaller.java | 26 +-
.../githubevents/PullRequestMergedEvent.java | 12 +-
.../PullRequestMergedEventsStream.java | 29 +-
.../pinot/tools/utils/KafkaStarterUtils.java | 5 +-
.../apache/pinot/tools/utils/PinotConfigUtils.java | 49 +-
.../src/main/resources/appAssemblerScriptTemplate | 5 +-
pinot-tools/src/main/resources/conf/log4j2.xml | 4 +-
.../src/main/resources/conf/pinot-admin-log4j2.xml | 2 +-
.../src/main/resources/conf/pinot-tools-log4j2.xml | 2 +-
.../src/main/resources/conf/quickstart-log4j2.xml | 4 +-
.../batch/airlineStats/hadoopIngestionJobSpec.yaml | 12 +-
.../batch/airlineStats/ingestionJobSpec.yaml | 12 +-
.../batch/airlineStats/sparkIngestionJobSpec.yaml | 12 +-
.../batch/baseballStats/ingestionJobSpec.yaml | 13 +-
.../batch/baseballStats/sparkIngestionJobSpec.yaml | 12 +-
.../batch/dimBaseballTeams/ingestionJobSpec.yaml | 12 +-
...bEvents_offline_complexTypeHandling_schema.json | 81 +-
...s_offline_complexTypeHandling_table_config.json | 4 +-
.../ingestionJobComplexTypeHandlingSpec.yaml | 12 +-
.../batch/githubEvents/ingestionJobSpec.yaml | 12 +-
.../batch/githubEvents/sparkIngestionJobSpec.yaml | 12 +-
.../batch/starbucksStores/ingestionJobSpec.yaml | 12 +-
.../starbucksStores_offline_table_config.json | 4 +-
.../starbucksStores/starbucksStores_schema.json | 1 -
...eHandling_meetupRsvp_realtime_table_config.json | 4 +-
.../complexTypeHandling_meetupRsvp_schema.json | 67 +-
.../upsert_meetupRsvp_realtime_table_config.json | 4 +-
.../meetupRsvp/upsert_meetupRsvp_schema.json | 4 +-
...t_partial_meetupRsvp_realtime_table_config.json | 6 +-
.../upsert_partial_meetupRsvp_schema.json | 4 +-
pinot-tools/src/main/resources/generator/README.md | 21 +-
.../resources/generator/complexWebsite_config.json | 22 +-
.../generator/complexWebsite_generator.json | 624 +++-
.../resources/generator/simpleWebsite_config.json | 22 +-
.../generator/simpleWebsite_generator.json | 22 +-
pinot-tools/src/main/resources/log4j2.xml | 57 +-
.../pinot/tools/TestCreateSegmentCommand.java | 21 +-
.../pinot/tools/TestDataAndQueryAnonymizer.java | 21 +-
pinot-tools/src/test/resources/log4j2.xml | 2 +-
.../resources/test_data/test_invalid_data.json | 11 +-
pom.xml | 169 +-
1690 files changed, 64113 insertions(+), 30393 deletions(-)
create mode 100644 .github/workflows/pinot_compatibility_tests.yml
create mode 100755 .github/workflows/scripts/.pinot_compatibility_verifier.sh
create mode 100755 .github/workflows/scripts/.pinot_linter.sh
delete mode 100644 DISCLAIMER
create mode 100644 kubernetes/helm/pinot-0.2.4.tgz
create mode 100644 licenses-binary/LICENSE-antlr4-runtime.txt
create mode 100644 licenses-binary/LICENSE-bouncycastle.txt
create mode 100644 licenses-binary/LICENSE-google-api.txt
create mode 100644 licenses-binary/LICENSE-jcip.txt
delete mode 100644 pinot-broker/src/main/java/org/apache/pinot/broker/pruner/PartitionZKMetadataPruner.java
delete mode 100644 pinot-broker/src/main/java/org/apache/pinot/broker/pruner/SegmentPrunerContext.java
delete mode 100644 pinot-broker/src/main/java/org/apache/pinot/broker/pruner/SegmentZKMetadataPruner.java
delete mode 100644 pinot-broker/src/main/java/org/apache/pinot/broker/pruner/SegmentZKMetadataPrunerProvider.java
delete mode 100644 pinot-broker/src/main/java/org/apache/pinot/broker/pruner/SegmentZKMetadataPrunerService.java
delete mode 100644 pinot-broker/src/test/java/org/apache/pinot/broker/pruner/SegmentZKMetadataPrunerTest.java
delete mode 100644 pinot-broker/src/test/java/org/apache/pinot/broker/routing/IntervalST/IntervalTest.java
delete mode 100644 pinot-broker/src/test/java/org/apache/pinot/broker/routing/IntervalST/IntervalTreeTest.java
create mode 100644 pinot-broker/src/test/java/org/apache/pinot/broker/routing/intervalst/IntervalTest.java
create mode 100644 pinot-broker/src/test/java/org/apache/pinot/broker/routing/intervalst/IntervalTreeTest.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/exception/SchemaAlreadyExistsException.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/messages/RunPeriodicTaskMessage.java
delete mode 100644 pinot-common/src/main/java/org/apache/pinot/common/metadata/segment/LLCRealtimeSegmentZKMetadata.java
delete mode 100644 pinot-common/src/main/java/org/apache/pinot/common/metadata/segment/OfflineSegmentZKMetadata.java
delete mode 100644 pinot-common/src/main/java/org/apache/pinot/common/metadata/segment/RealtimeSegmentZKMetadata.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/metrics/MinionGauge.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/metrics/MinionMeter.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/metrics/MinionMetrics.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/metrics/MinionQueryPhase.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/metrics/MinionTimer.java
delete mode 100644 pinot-common/src/main/java/org/apache/pinot/common/minion/Granularity.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/restlet/resources/TableMetadataInfo.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/utils/DateTimeUtils.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/utils/LikeToRegexpLikePatternConverterUtils.java
create mode 100644 pinot-common/src/test/java/org/apache/pinot/common/exception/QueryExceptionTest.java
create mode 100644 pinot-common/src/test/java/org/apache/pinot/common/function/scalar/ArrayAwareJacksonJsonProviderTest.java
create mode 100644 pinot-common/src/test/java/org/apache/pinot/common/utils/LikeToRegexpLikePatternConverterUtilsTest.java
create mode 100644 pinot-connectors/presto-pinot-driver/pom.xml
create mode 100644 pinot-connectors/presto-pinot-driver/src/main/java/org/apache/pinot/connector/presto/PinotScatterGatherQueryClient.java
create mode 100644 pinot-connectors/presto-pinot-driver/src/main/java/org/apache/pinot/connector/presto/grpc/PinotStreamingQueryClient.java
create mode 100644 pinot-connectors/presto-pinot-driver/src/main/java/org/apache/pinot/connector/presto/grpc/Utils.java
create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/ControllerUserDefinedMessageHandlerFactory.java
create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotControllerPeriodicTaskRestletResource.java
create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ResourceUtils.java
create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/util/TableIngestionStatusHelper.java
create mode 100644 pinot-controller/src/main/resources/favicon.ico
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/ControllerConfTest.java
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/ControllerStarterStatelessTest.java
delete mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/ControllerStarterTest.java
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/api/ConsumingSegmentInfoReaderStatelessTest.java
delete mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/api/ConsumingSegmentInfoReaderTest.java
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotIngestionRestletResourceStatelessTest.java
delete mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotIngestionRestletResourceTest.java
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDummyExtraRestletResourceTest.java
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/extrapackage/PinotDummyExtraRestletResource.java
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/minion/MinionInstancesCleanupTaskStatelessTest.java
delete mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/minion/MinionInstancesCleanupTaskTest.java
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManagerStatelessTest.java
delete mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManagerTest.java
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/utils/FakeHttpServer.java
create mode 100644 pinot-controller/src/test/resources/recommenderInput/RealtimeProvisioningInput_highIngestionRate.json
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/geospatial/transform/function/GeoToH3Function.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/AcquireReleaseColumnsSegmentOperator.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/query/DictionaryBasedDistinctOperator.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/RegexpExtractTransformFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/plan/AcquireReleaseColumnsSegmentPlanNode.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/plan/DictionaryBasedDistinctPlanNode.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/ModeAggregationFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/PercentileRawEstAggregationFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/PercentileRawEstMVAggregationFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/PercentileRawTDigestAggregationFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/PercentileRawTDigestMVAggregationFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/StringPredicateFilterOptimizer.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/segment/processing/partitioner/Partitioner.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/segment/processing/timehandler/EpochTimeHandler.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/segment/processing/timehandler/NoOpTimeHandler.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/segment/processing/timehandler/TimeHandler.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/segment/processing/timehandler/TimeHandlerConfig.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/segment/processing/timehandler/TimeHandlerFactory.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/startree/CompositePredicateEvaluator.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/auth/BasicAuthTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/data/manager/BaseTableDataManagerAcquireSegmentTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManagerTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/geospatial/transform/GeoToH3FunctionTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/RegexpTransformFunctionTest.java
delete mode 100644 pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/groupby/GroupByInSegmentTrimTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/groupby/GroupByTrimTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/query/executor/QueryExecutorExceptionsTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/StringPredicateFilterOptimizerTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/TestHelper.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/queries/ExpectedQueryResult.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/queries/ModeQueriesTest.java
create mode 100644 pinot-integration-test-base/pom.xml
create mode 100644 pinot-integration-test-base/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTest.java
rename {pinot-integration-tests => pinot-integration-test-base}/src/test/java/org/apache/pinot/integration/tests/BasicAuthTestUtils.java (100%)
create mode 100644 pinot-integration-test-base/src/test/java/org/apache/pinot/integration/tests/ClusterIntegrationTestUtils.java
create mode 100644 pinot-integration-test-base/src/test/java/org/apache/pinot/integration/tests/ClusterTest.java
create mode 100644 pinot-integration-test-base/src/test/java/org/apache/pinot/integration/tests/HybridClusterIntegrationTestCommandLineRunner.java
create mode 100644 pinot-integration-test-base/src/test/java/org/apache/pinot/integration/tests/QueryGenerator.java
delete mode 100644 pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTest.java
delete mode 100644 pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ClusterIntegrationTestUtils.java
delete mode 100644 pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ClusterTest.java
delete mode 100644 pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/HybridClusterIntegrationTestCommandLineRunner.java
create mode 100644 pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MergeRollupMinionClusterIntegrationTest.java
delete mode 100644 pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/QueryGenerator.java
create mode 100644 pinot-minion/src/main/java/org/apache/pinot/minion/MinionConf.java
delete mode 100644 pinot-minion/src/main/java/org/apache/pinot/minion/metrics/MinionGauge.java
delete mode 100644 pinot-minion/src/main/java/org/apache/pinot/minion/metrics/MinionMeter.java
delete mode 100644 pinot-minion/src/main/java/org/apache/pinot/minion/metrics/MinionMetrics.java
delete mode 100644 pinot-minion/src/main/java/org/apache/pinot/minion/metrics/MinionQueryPhase.java
delete mode 100644 pinot-minion/src/main/java/org/apache/pinot/minion/metrics/MinionTimer.java
create mode 100644 pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkRangeIndex.java
create mode 100644 pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkRoaringBitmapCreation.java
create mode 100644 pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkScanDocIdIterators.java
create mode 100644 pinot-perf/src/main/java/org/apache/pinot/perf/Distribution.java
delete mode 100644 pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-spark/src/test/java/SegmentCreationSparkTest.java
create mode 100644 pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-spark/src/test/java/org/apache/pinot/spark/SegmentCreationSparkTest.java
create mode 100644 pinot-plugins/pinot-metrics/pinot-dropwizard/pom.xml
create mode 100644 pinot-plugins/pinot-metrics/pinot-dropwizard/src/main/java/org/apache/pinot/plugin/metrics/dropwizard/DropWizardHistogram.java
create mode 100644 pinot-plugins/pinot-metrics/pinot-dropwizard/src/main/java/org/apache/pinot/plugin/metrics/dropwizard/DropwizardCounter.java
create mode 100644 pinot-plugins/pinot-metrics/pinot-dropwizard/src/main/java/org/apache/pinot/plugin/metrics/dropwizard/DropwizardGauge.java
create mode 100644 pinot-plugins/pinot-metrics/pinot-dropwizard/src/main/java/org/apache/pinot/plugin/metrics/dropwizard/DropwizardJmxReporter.java
create mode 100644 pinot-plugins/pinot-metrics/pinot-dropwizard/src/main/java/org/apache/pinot/plugin/metrics/dropwizard/DropwizardMeter.java
create mode 100644 pinot-plugins/pinot-metrics/pinot-dropwizard/src/main/java/org/apache/pinot/plugin/metrics/dropwizard/DropwizardMetered.java
create mode 100644 pinot-plugins/pinot-metrics/pinot-dropwizard/src/main/java/org/apache/pinot/plugin/metrics/dropwizard/DropwizardMetric.java
create mode 100644 pinot-plugins/pinot-metrics/pinot-dropwizard/src/main/java/org/apache/pinot/plugin/metrics/dropwizard/DropwizardMetricName.java
create mode 100644 pinot-plugins/pinot-metrics/pinot-dropwizard/src/main/java/org/apache/pinot/plugin/metrics/dropwizard/DropwizardMetricsFactory.java
create mode 100644 pinot-plugins/pinot-metrics/pinot-dropwizard/src/main/java/org/apache/pinot/plugin/metrics/dropwizard/DropwizardMetricsRegistry.java
create mode 100644 pinot-plugins/pinot-metrics/pinot-dropwizard/src/main/java/org/apache/pinot/plugin/metrics/dropwizard/DropwizardMetricsRegistryListener.java
create mode 100644 pinot-plugins/pinot-metrics/pinot-dropwizard/src/main/java/org/apache/pinot/plugin/metrics/dropwizard/DropwizardTimer.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/MergeTaskUtils.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/convert_to_raw_index/ConvertToRawIndexTaskExecutor.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/convert_to_raw_index/ConvertToRawIndexTaskExecutorFactory.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/convert_to_raw_index/ConvertToRawIndexTaskGenerator.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/converttorawindex/ConvertToRawIndexTaskExecutor.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/converttorawindex/ConvertToRawIndexTaskExecutorFactory.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/converttorawindex/ConvertToRawIndexTaskGenerator.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/merge_rollup/MergeProperties.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/merge_rollup/MergeRollupTaskExecutor.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/merge_rollup/MergeRollupTaskExecutorFactory.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/merge_rollup/MergeRollupTaskUtils.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskExecutor.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskExecutorFactory.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskUtils.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/realtime_to_offline_segments/RealtimeToOfflineSegmentsTaskExecutor.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/realtime_to_offline_segments/RealtimeToOfflineSegmentsTaskExecutorFactory.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/realtime_to_offline_segments/RealtimeToOfflineSegmentsTaskGenerator.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/realtimetoofflinesegments/RealtimeToOfflineSegmentsTaskExecutor.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/realtimetoofflinesegments/RealtimeToOfflineSegmentsTaskExecutorFactory.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/realtimetoofflinesegments/RealtimeToOfflineSegmentsTaskGenerator.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segment_generation_and_push/SegmentGenerationAndPushResult.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segment_generation_and_push/SegmentGenerationAndPushTaskExecutor.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segment_generation_and_push/SegmentGenerationAndPushTaskExecutorFactory.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segment_generation_and_push/SegmentGenerationAndPushTaskGenerator.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segment_generation_and_push/SegmentGenerationAndPushTaskUtils.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segmentgenerationandpush/SegmentGenerationAndPushResult.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segmentgenerationandpush/SegmentGenerationAndPushTaskExecutor.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segmentgenerationandpush/SegmentGenerationAndPushTaskExecutorFactory.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segmentgenerationandpush/SegmentGenerationAndPushTaskGenerator.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segmentgenerationandpush/SegmentGenerationAndPushTaskUtils.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/test/java/org/apache/pinot/plugin/minion/tasks/MergeTaskUtilsTest.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/test/java/org/apache/pinot/plugin/minion/tasks/merge_rollup/MergeRollupTaskExecutorTest.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/test/java/org/apache/pinot/plugin/minion/tasks/merge_rollup/MergeRollupTaskUtilsTest.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/test/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskExecutorTest.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/test/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGeneratorTest.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/test/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskUtilsTest.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/test/java/org/apache/pinot/plugin/minion/tasks/realtime_to_offline_segments/RealtimeToOfflineSegmentsTaskExecutorTest.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/test/java/org/apache/pinot/plugin/minion/tasks/realtime_to_offline_segments/RealtimeToOfflineSegmentsTaskGeneratorTest.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/test/java/org/apache/pinot/plugin/minion/tasks/realtimetoofflinesegments/RealtimeToOfflineSegmentsTaskExecutorTest.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/test/java/org/apache/pinot/plugin/minion/tasks/realtimetoofflinesegments/RealtimeToOfflineSegmentsTaskGeneratorTest.java
delete mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/test/java/org/apache/pinot/plugin/minion/tasks/segment_generation_and_push/SegmentGenerationAndPushTaskGeneratorTest.java
create mode 100644 pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/test/java/org/apache/pinot/plugin/minion/tasks/segmentgenerationandpush/SegmentGenerationAndPushTaskGeneratorTest.java
create mode 100644 pinot-plugins/pinot-stream-ingestion/pinot-pulsar/pom.xml
create mode 100644 pinot-plugins/pinot-stream-ingestion/pinot-pulsar/src/main/java/org/apache/pinot/plugin/stream/pulsar/MessageIdStreamOffset.java
create mode 100644 pinot-plugins/pinot-stream-ingestion/pinot-pulsar/src/main/java/org/apache/pinot/plugin/stream/pulsar/MessageIdStreamOffsetFactory.java
create mode 100644 pinot-plugins/pinot-stream-ingestion/pinot-pulsar/src/main/java/org/apache/pinot/plugin/stream/pulsar/PulsarConfig.java
create mode 100644 pinot-plugins/pinot-stream-ingestion/pinot-pulsar/src/main/java/org/apache/pinot/plugin/stream/pulsar/PulsarConsumerFactory.java
create mode 100644 pinot-plugins/pinot-stream-ingestion/pinot-pulsar/src/main/java/org/apache/pinot/plugin/stream/pulsar/PulsarMessageBatch.java
create mode 100644 pinot-plugins/pinot-stream-ingestion/pinot-pulsar/src/main/java/org/apache/pinot/plugin/stream/pulsar/PulsarPartitionLevelConnectionHandler.java
create mode 100644 pinot-plugins/pinot-stream-ingestion/pinot-pulsar/src/main/java/org/apache/pinot/plugin/stream/pulsar/PulsarPartitionLevelConsumer.java
create mode 100644 pinot-plugins/pinot-stream-ingestion/pinot-pulsar/src/main/java/org/apache/pinot/plugin/stream/pulsar/PulsarStreamLevelConsumer.java
create mode 100644 pinot-plugins/pinot-stream-ingestion/pinot-pulsar/src/main/java/org/apache/pinot/plugin/stream/pulsar/PulsarStreamLevelConsumerManager.java
create mode 100644 pinot-plugins/pinot-stream-ingestion/pinot-pulsar/src/main/java/org/apache/pinot/plugin/stream/pulsar/PulsarStreamMetadataProvider.java
create mode 100644 pinot-plugins/pinot-stream-ingestion/pinot-pulsar/src/test/java/org/apache/pinot/plugin/stream/pulsar/PulsarConsumerTest.java
create mode 100644 pinot-plugins/pinot-stream-ingestion/pinot-pulsar/src/test/java/org/apache/pinot/plugin/stream/pulsar/PulsarStandaloneCluster.java
create mode 100644 pinot-plugins/pinot-stream-ingestion/pinot-pulsar/src/test/resources/standalone.properties
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/SerializedQuantileDigest.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/SerializedTDigest.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/inv/BitSlicedRangeIndexCreator.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/inv/MmapFileWriter.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexHandler.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexHandlerFactory.java
delete mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/V3RemoveIndexException.java
delete mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/V3UpdateIndexException.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/BitSlicedRangeIndexReader.java
delete mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/RangeIndexReader.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/RangeIndexReaderImpl.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/TextIndexUtils.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/FPOrdering.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/HashUtils.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplUpsertComparisonColTest.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/creator/BitSlicedIndexCreatorTest.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/store/IndexKeyTest.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/FPOrderingTest.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/HashUtilsTest.java
rename pinot-segment-local/src/test/resources/data/{newColumnsSchema4.json => newColumnsSchemaWithFST.json} (100%)
create mode 100644 pinot-segment-local/src/test/resources/data/newColumnsSchemaWithH3Json.json
rename pinot-segment-local/src/test/resources/data/{newColumnsWithTextSchema.json => newColumnsSchemaWithText.json} (100%)
create mode 100644 pinot-segment-local/src/test/resources/data/test_upsert_comparison_col_data.json
create mode 100644 pinot-segment-local/src/test/resources/data/test_upsert_comparison_col_schema.json
create mode 100644 pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/FetchContext.java
create mode 100644 pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/name/InputFileSegmentNameGenerator.java
create mode 100644 pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/name/SegmentNameGeneratorFactory.java
create mode 100644 pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/CombinedInvertedIndexCreator.java
create mode 100644 pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/RangeIndexReader.java
delete mode 100644 pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/Partitioner.java
create mode 100644 pinot-segment-spi/src/test/java/org/apache/pinot/segment/spi/creator/name/InputFileSegmentNameGeneratorTest.java
create mode 100644 pinot-server/src/main/java/org/apache/pinot/server/starter/helix/OffsetBasedConsumptionStatusChecker.java
delete mode 100644 pinot-server/src/main/java/org/apache/pinot/server/starter/helix/SegmentFetcherAndLoader.java
create mode 100644 pinot-server/src/test/java/org/apache/pinot/server/starter/helix/OffsetBasedConsumptionStatusCheckerTest.java
create mode 100644 pinot-spi/src/main/java/org/apache/pinot/spi/utils/Obfuscator.java
create mode 100644 pinot-spi/src/test/java/org/apache/pinot/spi/filesystem/PinotFSTest.java
create mode 100644 pinot-spi/src/test/java/org/apache/pinot/spi/utils/ByteArrayTest.java
create mode 100644 pinot-spi/src/test/java/org/apache/pinot/spi/utils/ObfuscatorTest.java
create mode 100644 pinot-spi/src/test/resources/ingestion_job_spec_unicode.yaml
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org