You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by jl...@apache.org on 2021/11/17 00:25:35 UTC
[pinot] branch support-spark-preprocessing updated (475542c -> e2e92fc)
This is an automated email from the ASF dual-hosted git repository.
jlli pushed a change to branch support-spark-preprocessing
in repository https://gitbox.apache.org/repos/asf/pinot.git.
discard 475542c Support data preprocessing in Spark framework
add 94c695f Remove prefetch query option, since the whole prefetch-acquire-release is already withing an instance option (#7352)
add c907dca fixing permission check case issue in BasicAuthPrincipal (#7354)
add 3bbc2d0 Prepare for next development (0.9.0-SNAPSHOT) (#7356)
add f951cce Support ID_SET aggregation on MV columns (#7355)
add 767a149 Add endpoint to check minion task status for a single task. (#7353)
add c419c2c Showing partial status of segment and counting CONSUMING state as good segment status (#7327)
add 16e4e3f Allow logical identifier in filter (#7347)
add 2d1f7dc Add functions to return raw results for Percentile TDigest and Est (#7226)
add a5f8cf4 fix upload LLC segment endpoint truncated download URL (#7361)
add 7016d03 Cmd+enter shortcut to run query in query console (#7359)
add 70efd03 Fix task scheduling on table update (#7362)
add 93f2efb Fix metric method for ONLINE_MINION_INSTANCES metric (#7363)
add 004c8af Make JsonToPinotSchema consistent with AvroSchemaToPinotSchema (#7366)
add 0681e4b Bug fix: make currentOffset in consuming segment volatile (#7365)
add c5e8ddf Fix misleading error msg for missing URI (#7367)
add 3892812 Add "num rows in segments" and "num segments queried per host" to the output of Realtime Provisioning Rule (#7282)
add c62a1fb make getColumnIndices accurate (#7370)
add 21767c6 Increase code coverage for pinot-java-client - ConnectionFactory (#7350)
add 2302bd2 check for less than zero start and end time (#7372)
add d412ecf Fix SegmentZKMetadta time handling (#7375)
add f5f072c Optimize IndexedTable (#7373)
add 29f27f8 Add cleanup method for HadoopSegmentCreationJob (#7371)
add 71b6890 Check schema backward-compatibility when updating schema through addSchema with override (#7374)
add de4ee0f Remove obsolete indices during segment reloading for V3 format (wrap up) (#7286)
add 27b4da7 showing tooltip in SQL Editor (#7387)
add 8fab3be Make the SQL Editor box expandable (#7381)
add 8106b69 add more integ test cases for index removal and stronger assertion (#7385)
add 5976922 Optimize TableResizer (#7392)
add fe14d60 fixing pinot-presto docker image build script (#7401)
add 6294d1b Remove unused constant FIELD_VALUE (#7396)
add e7901ee Missing Prometheus metric name (#7406)
add 58fcd5b simplify association between Java Class and PinotDataType for faster mapping (#7402)
add c27ae81 Cache ~/.m2/repository to speed up CI speed (#7408)
add 91679c0 Adding presto-pinot-driver module (#7384)
add 3d28c0e skip .factorypath from rat (#7419)
add dc80d67 avoid creating stateless ParseContextImpl once per jsonpath evaluation, avoid varargs allocation (#7412)
add 514f469 Obfuscator base implementation (#7407)
add ffcf9b9 fixing compatibility check for presto-pinot-driver (#7423)
add c1cc61e Replace MINUS(strColum1, strColumn2) with STRCMP(strColumn1,strColumn2) (#7394)
add 1fa6eda By default skip maven-deploy-plugin, so mvn deploy command won't fail (#7426)
add fe03999 Fix retention for cleaning up segment lineage (#7424)
add f3068bc Introduce resultSize in IndexedTable (#7420)
add 0e66d68 remove unused exceptions (#7429)
add 0ce2546 modify segment generator to not return illegal filenames (#7085)
add 11f8550 adding in source/target version (#7431)
add ad2e063 Remove the flaky checks in MergeRollupMinionClusterIntegrationTest (#7432)
add a5f3dc5 Move segment download and segment reload into TableManager (#7319)
add 69a91ac Add aws sts dependency to enable auth using web identity token. (#7017)
add 0ee33e8 add a artifact upload step to compatibility verifier job (#7425)
add 6605bc0 upload missing LLC segment to segment store by controller periodic task (#6778)
add 2dbc982 Fix parsing error messages returned to FileUploadDownloadClient (#7428)
add 6a7cbf1 Add aws sts dependency to enable auth using web identity token (#7445)
add dc78c79 refactor RangeIndexReader to make it easier to evolve its implementation (#7435)
add b23a1f3 FetchContext to hold configs in acquire/release calls (#7447)
add 31ca84e ban prefix increment (#7442)
add 0069fc3 Fix checkstyle violation and limit the prefix inc/dec check in for look (#7451)
add 9c2a333 Mask credentials in debug endpoint /appconfigs (#7452)
add 53cfdf3 Offset based realtime consumption status checker (#7267)
add b990c57 Support REGEXP_EXTRACT (#7114)
add fabda2b create pinot-integration-test-base module test-jar (#7399)
add 2c74f20 exclude healthcheck endpoint from auth (#7455)
add 9224590 Fix classpath with plugins java 8 (#7400)
add 0f42295 fix manifest scan which drives /version endpoint (#7456)
add a3df342 Add additional override in SegmentWriter and SegmentUploader (#7458)
add 6976697 Update some usage of BatchConfig (#7459)
add 4972110 Unify CombineOperator multi-threading logic (#7450)
add 5b15f08 add SingleSpaceSeparator rule and fix recent violations (#7467)
add c22886a add causes to stack trace return (#7460)
add 72d05a7 fix auth token injection in SegmentGenerationAndPushTaskExecutor (#7464)
add 34d987d add a linter checker (#7466)
add fcae6e0 hotfix for #7459 (#7471)
add b2bac68 Fix missing rate limiter if brokerResourceEV becomes null due to ZK connection (#7470)
add c31d1a1 add license checker to linter and fail fast (#7475)
add 8aeca69 Replace emptiness checks with `isEmpty()` (#7468)
add 043c658 Fix testNoBrokerResource test (#7478)
add 48e5aaf Revert back to use BatchIngestionConfig parameter (#7473)
add 4000568 Fix race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks: (#7427)
add f67666a define and enforce static import code style (#7483)
add 8a05ad8 fix segemnt uploader property in auth quickstart (#7479)
add cd7cf77 do not allow qualification of member variables with this (#7484)
add d99eb3c bit-sliced range index for int, long, float, double, dictionarized SV columns (#7454)
add f381abe use MethodHandle to access vectorized unsigned comparison on JDK9+ (#7487)
add 240bcb8 Fix pql double quote checker exception (#7485)
add bd5bff1 create controller resource packages config key (#7488)
add debb132 Add option to limit thread usage per query (#7492)
add d638444 Move acquire/release to right before/after execution call (#7493)
add 1fdc2eb [issue-7495] Fix minion metrics exporter config (#7496)
add 70374e2 remove unused dependency (#7498)
add 9b42278 fix pinot-controller/broker recent fork timeout (#7490)
add cccc1e3 fix volatile stopped flag (#7508)
add 57614d1 Follow up to improve restlet API package config key (#7497)
add 76023ae remove unnecessary wait during broker test (#7505)
add 30f5490 1. Catch timeout exception to wait segments become online for endReplaceSegments(), so it can be retried. (#7509)
add c9e36dd [7264] Add Exception to Broker Response When Not All Segments Are Available (Partial Response) (#7397)
add 517a0dc test xmls are not updated with latest test classes (#7474)
add 02c4998 Add mergeRollupTask delay metrics (#7368)
add f306460 Do not allow query to override the execution threads over the instance-level limit (#7515)
add af90f79 Improved range queries (#7513)
add 9410aef adding favicon to the pinot controller ui (#7518)
add 63f7ad9 Enhance TableCache to support schema name different from table name (#7525)
add 7be21c8 Faster bitmap scans (#7530)
add 19f66fd fix segment generation commands (#7527)
add acb775d add validation for realtimeToOffline task (#7523)
add 1be1b48 Fix TableCacheTest by using unique schema and table name (#7538)
add 8f41621 Optimize EmptySegmentPruner to skip pruning when there is no empty segments (#7531)
add c8aa3e3 In TableRebalancer, allow increasing from 1 replica without downtime (#7532)
add 57a1f3f Support JSON queries with top-level array path expression. (#7511)
add 12b106b Fix SegmentGenerationWithTimeColumnTest (#7537)
add 6969b38 Introduce MinionConf, move END_REPLACE_SEGMENTS_TIMEOUT_MS to minion config instead of task config. (#7516)
add c36d2b9 Cleanup sys out (#7547)
add 9ef1f49 Add timezone as argument in toDateTime (#7552)
add 6c01ce2 Add IN function (#7542)
add 95b0eda map bitmaps through a bounded window to avoid excessive disk pressure… (#7535)
add 9a8ba67 [Refactor] Remove the index access within the PlanNode constructor (#7550)
add 7fe59d5 Return 0 in the response metadata when thread cpu time measurement is disabled (#7544)
add 493ed2c MergeRollupTaskGenerator enhancement: enable parallel buckets scheduling (#7481)
add 01cb4cd Make utility functions in `SegmentAssignmentUtils` public. (#7561)
add 1dda74a Return non zero from main with exception (#7482)
add 4ba8061 Adjust tuner api (#7553)
add bd39dd8 [Auto Reformat] Reformat TableRebalancer related classes (#7571)
add 8150322 Adding config for metrics library (#7551)
add ea31137 Refactor toDateTime with time zone (#7558)
add 5eee297 fix: Sorting tables by number of segments is broken (#7564)
add 0ecfc11 Add geo type conversion scalar functions (#7573)
add e681916 Enhance TableRebalancer to offload the segments from most loaded instances first (#7574)
add 9f0562e Adding generic util method for persisting task metadata in a ZNode (#7575)
add c9e2785 Improve get tenant API to differentiate offline and realtime tenants (#7548)
add ac49978 Need to make abstract methods public since its going to be overriden and called from different packages (#7579)
add 5106a92 Adding new generic method in ClusterInfoAccessor for getting ZNRecord (#7580)
add 0421197 Introduce Native Text Indices (Core Functionality) (#7405)
add 3ef5a37 Change FetchContext to have list of index per colum to fetch (#7587)
add 9d2a9f3 Fix mergeRollupTaskGenerator metrics and logs. (#7588)
add b8d3c2e allow rle compression of bitmaps for smaller file sizes (#7582)
add 85e0d9e add BOOLEAN_ARRAY and TIMESTAMP_ARRAY types (#7581)
add aed6ada Upgrade localstack utils version (#7598)
add 4246e0f Refactor query rewriter to interfaces and implementations to allow customization (#7576)
add d595346 Ignore shardIterator type when sequence number is available (#7592)
add a735bc9 Adding generic method in ClusterInfoAccessor for setting minion task metadata (#7602)
add 0648f77 In ServiceStartable, apply global cluster config in ZK to instance config (#7593)
add cb065e4 Refactor QueryOptions into QueryOptionsUtils to avoid redundant parsing (#7601)
add 471d16c Make dim tables creation bypass tenant validation (#7559)
add 0e49e74 use StandardCharsets.UTF_8 which is faster than looking up the charset and doesn't need to handle excetions (#7599)
add 5fd0231 Shade parquet class (#7570)
add 1bd899c Show error message on hover of error status of a segment (#7529)
add 6fef210 Stringutil cleanup (#7610)
add 92074f9 [Cleanup] Precompute and cache query properties in QueryContext (#7608)
add 1ee3c02 Add support for lowest partition id in case partition id 0 is not available (#7066)
add aed1307 Add MV raw forward index and MV `BYTES` data type (#7595)
add a114cd2 (fix) Query console doesn't load (#7618)
add 7bcbda1 Use maxEndTimeMs for merge/roll-up delay metrics. (#7617)
add 0327932 Fix star-tree index map when column name contains '.' (#7623)
add 72b8fc1 accelerate ByteArray.hashCode (#7622)
add c3d57c1 Parsing Support for FILTER Clauses (#7566)
add 3e4d70b make it possible to specify raw index properties for columns with JSON and RANGE indexes (#7615)
add ac7d176 clarify expectations of PartitionFunction implementations (#7628)
add d42e415 Enhance BloomFilter rule to include IN predicate(#7444) (#7624)
add 40cef4f avoid repeating column stats work when values are repeated (#7619)
add c15d2f0 add extra properties to the tuner, since tuner right now doesn't take TunerConfig properties (#7621)
add d4d6be4 implement FixedByteChunkMVForwardIndexReader, address post-review comments (#7629)
add 756f78a adding trino to pinot readme doc (#7635)
add ed335ef Push metricsRegistry init into synchronized get call (#7636)
add 8dc28e0 Allow Metadata and Dictionary Based Plans for No Op Filters (#7563)
add da3bce5 Add LASTWITHTIME aggregate function support #7315 (#7584)
add 011bc0f URLs need to be encoded in cluster manager as segment name can have % in it (#7639)
add 28894e4 Remove AggregationFunction.isIntermediateResultComparable() (#7640)
add 9c47f3b Adjust field config (#7631)
add 23f7002 Reject query with identifiers not in schema (#7590)
add 0166135 Round Robin IP addresses when retry uploading/downloading segments (#7585)
add 3734ee2 Support multi-value derived column in offline table reload (#7632)
add 611f3b1 Support segmentNamePostfix in segment name (#7646)
add 127f4c3 Add select segments API (#7651)
add 19eb97b introduce `LZ4_WITH_LENGTH` chunk compression type (#7655)
add 41cdc02 don't toString every multivalue (#7649)
add db11e0f Fix LIKE predicate (#7659)
add 24681e7 Feature. Provide ability to skip multiple value splitting in CSV (#7660)
add a910bee Enhance ColumnValueSegmentPruner and support bloom filter prefetch (#7654)
add a875c99 Apply the optimization on dictIds within the segment to DistinctCountHLL aggregation func (#7630)
add 372afc9 Do not log the actual exception when Arrays.compareUnsigned is unavailable (#7663)
add fbb9e8e Let the controller getTableInstance() call to return the list of live brokers of a table. (#7556)
add 5208caa fix fieldConfig nullable fields failed validation (#7648)
add 02338c0 A quickstart to demo how RealtimeToOfflineConv and MergeRollup minion tasks work with Realtime table (#7650)
add e814203 Move HelixManager initialization to init (#7666)
add 5becf5b Allow MV Field Support For Raw Columns in Text Indices (#7638)
add b2add14 During segment pruning, release the bloom filter after each segment is processed (#7668)
add 1b4241d Skip schema check for lookup function (#7670)
add 0ec77b5 Allow override distinctCount to segmentPartitionedDistinctCount (#7664)
add cc2d63c Add a quick start with both UPSERT and JSON index (#7669)
add da98e8b Add revertSegmentReplacement API (#7662)
add 990ae47 Add getIndexSegment() to the Operator interface (#7674)
add 6af5f4e Add table name with type to segment directory configs (#7667)
add 8948415 Allow text index on MV string in table config validation (#7677)
add 2107b2c Clear the reused record in PartitionUpsertMetadataManager (#7676)
add 5bf15cf Take acquire/release call to MinMaxValueBasedSelectionOrderByCombineOperator (#7681)
add 421645d When reloading all segments for a table, do not fail on one bad segment (#7675)
add 278baf8 Expose resource manager via ClusterInfoAccessor for task generator to use (#7680)
add 14c377d Fix JSONPath cache inefficient issue (#7409)
add d8961b0 Cleanup SegmentDirectoryLoader load (#7682)
add f20a364 allow headers for query runner (#7696)
add e80198d Fix common test (#7699)
add d7c0944 replace args4j with picocli for pinot-admin.sh (#7665)
add 7d61528 Fix verifyHostname issue in FileUploadDownloadClient (#7703)
add edb3af6 Add acquire/release for StreamingSelectionOnlyCombineOperator and GroupByCombineOperator (#7683)
add c594796 Fix TableCache schema to include the built-in virtual columns (#7706)
add 7cbd5ac to handle datetime column consistently (#7645)
add 91d2a95 Clean up controller-table related metrics in ControllerPeriodicTask (#7557)
add 3b0cce4 allow to carry headers with query requests (#7712)
add f6d7a51 optimise getUnpaddedString with SWAR padding search (#7708)
add 827c1ce lighter weight LiteralTransformFunction, avoid excessive array fills (#7707)
add 7a24845 inline binary comparison ops to prevent function call overhead (#7709)
add a17bdd7 Skip pinot.admin.system.exit check for quickstart (#7722)
add 2605122 Allow adding JSON data type for dimension column types (#7718)
add 52ec46e Remove the AutomatonBasedRegexpPredicateEvaluator and use regular RegexpLikePredicateEvaluator (#7715)
add 67b5b67 Fix the flaky SimpleMinionClusterIntegrationTest (#7723)
add ada94c2 follow up with picocli migration (#7705)
add 94a17f3 memoize literals in query context in order to deduplicate them (#7720)
add 39daba6 format query parser tests (#7727)
add 2075cbe correct way to send headers via url conn (#7728)
add ae6ee77 Fix SDF pattern in DataPreprocessingHelper (#7721)
add 8d83973 Remove the old deprecated SegmentConverter (#7725)
add 920093a Disable RealtimeKinesisIntegrationTest until the Localstack is fixed (#7735)
add c9ba48e Separate SegmentDirectoryLoader and tierBackend concepts (#7737)
add 4f22c86 Fixing DISTINCT with AS function (#7678)
add 13c9ee9 implement size balanced V4 raw chunk format (#7661)
add b66643a remove unused Number2ObjectPair class and auxiliaries (#7736)
add 928d4ef bump roaringbitmap version (#7734)
add af01aa5 Handle fields missing in the source in ParquetNativeRecordReader (#7742)
add 94d3c5f Add TIMESTAMP to datetime column Type (#7746)
add 540e70e update license and notice (#7750)
add 8442beb extend a bit on the FileUploadDownloadClient (#7751)
add 570e983 Handle null segment lineage ZNRecord for getSelectedSegments API (#7752)
add 4b2d31b 1. Change TLS to native implementation (https://netty.io/wiki/forked-tomcat-native.html) (#7653)
add e3d238a Add forceCleanup option for 'startReplaceSegments' API (#7744)
add 97ac119 fixing javadoc to compatible with jdk8 source (#7754)
add b313a67 Fix controller error response field names (#7757)
add 561b8a3 Use oldest offset on newly detected partitions (#7756)
add 068549c Support default null value in data preprocessing job (#7739)
add 781f5cd Support Native FST As An Index Subtype for FST Indices (#7729)
add 4a540ed Add link to the ZeroInWord algorithm. (#7764)
add 362f3e1 Add getName() to PartitionFunction interface (#7760)
add 05b74b4 set default Dockerfile CMD to -help (#7767)
add d547920 Add timeColumn, timeUnit and totalDocs to the json segment metadata (#7765)
add 5def5b9 Indicate QuickStart types for invalid/missing type (#7768)
add 6611ff8 Add config for enabling realtime offset based consumption status checker (#7753)
add d1606cd Check interrupt flag in acquire (#7769)
add e2e92fc Support data preprocessing in Spark framework
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (475542c)
\
N -- N -- N refs/heads/support-spark-preprocessing (e2e92fc)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/pinot_compatibility_tests.yml | 12 +
.github/workflows/pinot_tests.yml | 64 ++
.github/workflows/scripts/.pinot_linter.sh | 33 +
.github/workflows/scripts/.pinot_quickstart.sh | 3 +-
.github/workflows/scripts/.pinot_test.sh | 8 +-
LICENSE-binary | 37 +-
NOTICE-binary | 32 +-
README.md | 4 +-
compatibility-verifier/compCheck.sh | 15 +-
config/checkstyle.xml | 60 ++
config/suppressions.xml | 4 +
docker/images/pinot-presto/Dockerfile | 6 +-
docker/images/pinot/Dockerfile | 2 +-
.../etc/jmx_prometheus_javaagent/configs/pinot.yml | 8 +-
licenses-binary/LICENSE-args4j.txt | 19 -
licenses-binary/LICENSE-paranamer.txt | 29 +
licenses-binary/LICENSE-typelevel.txt | 49 +
pinot-broker/pom.xml | 2 +-
.../broker/broker/helix/BaseBrokerStarter.java | 70 +-
.../broker/broker/helix/ClusterChangeMediator.java | 2 +-
.../HelixExternalViewBasedQueryQuotaManager.java | 15 +-
.../pinot/broker/queryquota/MaxHitRateTracker.java | 2 +-
.../requesthandler/BaseBrokerRequestHandler.java | 322 +++++--
.../SingleConnectionBrokerRequestHandler.java | 13 +-
.../SegmentLineageBasedSegmentPreSelector.java | 2 +-
.../routing/segmentpruner/EmptySegmentPruner.java | 4 +
.../broker/HelixBrokerStarterHostnamePortTest.java | 3 +
.../broker/broker/HelixBrokerStarterTest.java | 35 +-
...elixExternalViewBasedQueryQuotaManagerTest.java | 4 +-
.../requesthandler/DistinctCountRewriteTest.java | 62 ++
.../broker/requesthandler/QueryValidationTest.java | 84 +-
.../routing/segmentpruner/SegmentPrunerTest.java | 11 +-
.../timeboundary/TimeBoundaryManagerTest.java | 11 +-
pinot-clients/pinot-java-client/pom.xml | 2 +-
.../org/apache/pinot/client/BrokerResponse.java | 2 +-
.../org/apache/pinot/client/BrokerSelector.java | 7 +
.../java/org/apache/pinot/client/Connection.java | 4 +-
.../org/apache/pinot/client/ConnectionFactory.java | 9 +-
.../apache/pinot/client/DynamicBrokerSelector.java | 23 +-
.../apache/pinot/client/SimpleBrokerSelector.java | 10 +-
.../apache/pinot/client/ConnectionFactoryTest.java | 48 +-
.../pinot/client/DynamicBrokerSelectorTest.java | 84 +-
pinot-clients/pinot-jdbc-client/pom.xml | 2 +-
.../org/apache/pinot/client/utils/Constants.java | 2 +-
pinot-clients/pom.xml | 2 +-
pinot-common/pom.xml | 30 +-
.../main/java/org/apache/pinot/common/Utils.java | 39 +-
.../pinot/common/exception/QueryException.java | 52 +-
.../SchemaAlreadyExistsException.java} | 12 +-
.../common/function/DateTimePatternHandler.java | 18 +-
.../pinot/common/function/JsonPathCache.java | 35 +-
.../common/function/TransformFunctionType.java | 5 +
.../common/function/scalar/DateTimeFunctions.java | 8 +
.../common/function/scalar/JsonFunctions.java | 47 +-
.../common/function/scalar/StringFunctions.java | 54 ++
.../pinot/common/lineage/LineageEntryState.java | 2 +-
.../pinot/common/lineage/SegmentLineageUtils.java | 2 +-
.../common/metadata/segment/SegmentZKMetadata.java | 32 +-
.../SegmentZKMetadataCustomMapModifier.java | 2 +-
.../pinot/common/metrics/AbstractMetrics.java | 12 +-
.../apache/pinot/common/metrics/BrokerMeter.java | 2 +
.../pinot/common/metrics/ControllerMeter.java | 4 +-
.../pinot/common/metrics/PinotMetricUtils.java | 61 +-
.../pinot/common/metrics/ValidationMetrics.java | 107 ++-
.../pinot/common/minion/BaseTaskMetadata.java | 47 +-
.../common/minion/MergeRollupTaskMetadata.java | 17 +-
.../common/minion/MinionTaskMetadataUtils.java | 30 +-
.../RealtimeToOfflineSegmentsTaskMetadata.java | 17 +-
.../request/context/RequestContextUtils.java | 97 +-
.../common/restlet/resources/SegmentErrorInfo.java | 24 +-
.../pinot/common/tier/PinotServerTierStorage.java | 34 +-
.../java/org/apache/pinot/common/tier/Tier.java | 5 +
.../org/apache/pinot/common/tier/TierFactory.java | 3 +-
.../common/tier/TimeBasedTierSegmentSelector.java | 5 +
.../common/utils/ClientSSLContextGenerator.java | 1 +
.../org/apache/pinot/common/utils/DataSchema.java | 201 ++--
.../apache/pinot/common/utils/DateTimeUtils.java | 35 +-
.../common/utils/FileUploadDownloadClient.java | 115 ++-
.../apache/pinot/common/utils/LLCSegmentName.java | 13 +-
.../pinot/common/utils/NamedThreadFactory.java | 2 +-
.../apache/pinot/common/utils/PinotAppConfigs.java | 3 +-
.../apache/pinot/common/utils/PinotDataType.java | 201 +++-
...Utils.java => RegexpPatternConverterUtils.java} | 43 +-
.../pinot/common/utils/RoundRobinURIProvider.java | 64 ++
.../pinot/common/utils/ServiceStartableUtils.java | 90 ++
.../apache/pinot/common/utils/ServiceStatus.java | 39 +-
.../org/apache/pinot/common/utils/StringUtil.java | 22 -
.../common/utils/config/TableConfigUtils.java | 17 +-
.../pinot/common/utils/config/TagNameUtils.java | 7 +
.../pinot/common/utils/config/TierConfigUtils.java | 12 +
.../common/utils/fetcher/HttpSegmentFetcher.java | 34 +-
.../pinot/common/utils/grpc/GrpcQueryClient.java | 40 +-
.../pinot/common/utils/helix/HelixHelper.java | 27 +-
.../common/utils/helix/LeadControllerUtils.java | 5 +-
.../pinot/common/utils/helix/TableCache.java | 101 +-
.../parsers/PinotQuery2BrokerRequestConverter.java | 4 +-
.../pinot/pql/parsers/pql2/ast/BaseAstNode.java | 2 +-
.../parsers/pql2/ast/BooleanOperatorAstNode.java | 2 +-
.../apache/pinot/sql/parsers/CalciteSqlParser.java | 476 +---------
.../pinot/sql/parsers/rewriter/AliasApplier.java | 116 +++
.../rewriter/CompileTimeFunctionsInvoker.java | 93 ++
...nAggregationGroupByToDistinctQueryRewriter.java | 75 ++
.../sql/parsers/rewriter/OrdinalsUpdater.java | 66 ++
.../rewriter/PredicateComparisonRewriter.java | 132 +++
.../parsers/rewriter/QueryRewriter.java} | 12 +-
.../sql/parsers/rewriter/QueryRewriterFactory.java | 73 ++
.../sql/parsers/rewriter/SelectionsRewriter.java | 91 ++
.../pinot/common/data/DateTimeFormatSpecTest.java | 4 +
.../pinot/common/exception/QueryExceptionTest.java | 53 ++
.../function/AggregationFunctionTypeTest.java | 10 +
.../pinot/common/http/MultiGetRequestTest.java | 12 +-
.../common/metadata/InstanceZKMetadataTest.java | 4 +-
.../pinot/common/metadata/MetadataUtils.java | 2 +-
.../pinot/common/metrics/PinotMetricUtilsTest.java | 13 +
.../common/minion/MinionTaskMetadataUtilsTest.java | 88 ++
.../pinot/common/tier/TierConfigUtilsTest.java | 39 +-
.../apache/pinot/common/utils/DataSchemaTest.java | 45 +-
.../LikeToRegexpLikePatternConverterUtilsTest.java | 77 --
.../pinot/common/utils/PinotDataTypeTest.java | 49 +-
.../utils/RegexpPatternConverterUtilsTest.java | 104 ++
.../common/utils/RoundRobinURIProviderTest.java | 179 ++++
.../pinot/common/utils/SegmentNameBuilderTest.java | 3 +
.../common/utils/config/TableConfigSerDeTest.java | 32 +-
.../pinot/common/utils/config/TableConfigTest.java | 61 ++
.../request/BrokerRequestSerializationTest.java | 38 -
.../pinot/request/BrokerResponseNativeTest.java | 2 -
.../pinot/sql/parsers/CalciteSqlCompilerTest.java | 218 +++--
...regationGroupByToDistinctQueryRewriterTest.java | 172 ++++
.../parsers/rewriter/QueryRewriterFactoryTest.java | 61 ++
.../resources/testConfigs/jsonIndexConfig.json | 35 +
.../resources/testConfigs/multipleIndexConfig.json | 33 +
.../test/resources/testConfigs/noIndexConfig.json | 28 +
.../resources/testConfigs/rangeIndexConfig.json | 28 +
.../resources/testConfigs/textIndexConfig.json | 35 +
pinot-connectors/pinot-spark-connector/pom.xml | 2 +-
pinot-connectors/pom.xml | 3 +-
pinot-connectors/presto-pinot-driver/pom.xml | 738 ++++++++++++++
.../presto/PinotScatterGatherQueryClient.java | 340 +++++++
.../presto/grpc/PinotStreamingQueryClient.java | 65 ++
.../apache/pinot/connector/presto/grpc/Utils.java | 54 ++
pinot-controller/pom.xml | 2 +-
.../pinot/controller/BaseControllerStarter.java | 17 +-
.../apache/pinot/controller/ControllerConf.java | 19 +-
.../api/ControllerAdminApiApplication.java | 9 +-
.../api/access/AuthenticationFilter.java | 2 +-
.../resources/PinotIngestionRestletResource.java | 4 +-
.../api/resources/PinotQueryResource.java | 2 +-
.../api/resources/PinotSchemaRestletResource.java | 11 +-
.../api/resources/PinotSegmentRestletResource.java | 41 +
.../PinotSegmentUploadDownloadRestletResource.java | 75 +-
.../api/resources/PinotTableInstances.java | 15 +
.../api/resources/PinotTableRestletResource.java | 3 +-
.../api/resources/PinotTaskRestletResource.java | 22 +-
.../api/resources/PinotTenantRestletResource.java | 33 +-
.../api/resources/TableConfigsRestletResource.java | 2 +-
.../api/resources/TableDebugResource.java | 2 +-
.../resources/WebApplicationExceptionMapper.java | 10 +-
.../pinot/controller/api/upload/ZKOperator.java | 24 +-
.../helix/ControllerRequestURLBuilder.java | 4 +
.../controller/helix/SegmentStatusChecker.java | 16 +
.../helix/core/PinotHelixResourceManager.java | 384 ++++++--
.../helix/core/PinotTableIdealStateBuilder.java | 2 +-
.../helix/core/SegmentDeletionManager.java | 2 +-
.../assignment/segment/SegmentAssignmentUtils.java | 14 +-
.../helix/core/minion/ClusterInfoAccessor.java | 96 +-
.../core/minion/PinotHelixTaskResourceManager.java | 148 +--
.../helix/core/minion/PinotTaskManager.java | 164 ++--
.../helix/core/minion/TaskMetricsEmitter.java | 2 +-
.../core/periodictask/ControllerPeriodicTask.java | 14 +
.../realtime/PinotLLCRealtimeSegmentManager.java | 165 +++-
.../core/realtime/PinotRealtimeSegmentManager.java | 2 +-
.../segment/DefaultFlushThresholdUpdater.java | 9 +-
.../realtime/segment/FlushThresholdUpdater.java | 7 +-
.../SegmentSizeBasedFlushThresholdUpdater.java | 15 +-
.../helix/core/rebalance/TableRebalancer.java | 256 +++--
.../helix/core/retention/RetentionManager.java | 50 +-
.../helix/core/util/ZKMetadataUtils.java | 2 +-
.../realtime/provisioning/MemoryEstimator.java | 14 +-
.../recommender/rules/impl/BloomFilterRule.java | 3 +-
.../rules/impl/RealtimeProvisioningRule.java | 31 +-
.../rules/impl/VariedLengthDictionaryRule.java | 2 +-
.../recommender/rules/io/configs/IndexConfig.java | 8 +-
.../params/InvertedSortedIndexJointRuleParams.java | 4 +-
.../utils/QueryInvertedSortedIndexRecommender.java | 2 +-
.../tuner/NoOpTableTableConfigTuner.java | 8 +-
.../controller/tuner/RealTimeAutoIndexTuner.java | 19 +-
.../pinot/controller/tuner/TableConfigTuner.java | 21 +-
.../controller/tuner/TableConfigTunerUtils.java | 37 +-
.../controller/util/AutoAddInvertedIndex.java | 2 +-
.../pinot/controller/util/FileIngestionHelper.java | 31 +-
.../util/TableIngestionStatusHelper.java | 2 +-
.../validation/OfflineSegmentIntervalChecker.java | 16 +
.../RealtimeSegmentValidationManager.java | 36 +-
.../Homepage/Operations/SchemaComponent.tsx | 12 +-
.../src/main/resources/app/components/Table.tsx | 62 +-
.../main/resources/app/components/TableToolbar.tsx | 16 +-
pinot-controller/src/main/resources/app/index.html | 4 +-
.../src/main/resources/app/pages/Query.tsx | 54 +-
.../main/resources/app/pages/SegmentDetails.tsx | 5 +-
.../src/main/resources/app/requests/index.ts | 5 +-
.../main/resources/app/utils/PinotMethodUtils.ts | 46 +-
.../src/main/resources/app/utils/Utils.tsx | 10 +-
pinot-controller/src/main/resources/favicon.ico | Bin 0 -> 15406 bytes
.../src/main/resources/package-lock.json | 153 +++
pinot-controller/src/main/resources/package.json | 2 +
.../src/main/resources/webpack.config.js | 8 +
...st.java => ControllerStarterStatelessTest.java} | 2 +-
.../pinot/controller/ControllerTestUtils.java | 38 +-
...> ConsumingSegmentInfoReaderStatelessTest.java} | 32 +-
.../api/PinotControllerAppConfigsTest.java | 12 +-
...inotIngestionRestletResourceStatelessTest.java} | 2 +-
.../api/PinotInstanceRestletResourceTest.java | 4 +-
.../api/PinotSchemaRestletResourceTest.java | 128 ++-
.../api/PinotSegmentRestletResourceTest.java | 4 +-
.../controller/api/PinotSegmentsMetadataTest.java | 24 +-
.../api/TableConfigsRestletResourceTest.java | 4 +-
.../pinot/controller/api/TableSizeReaderTest.java | 40 +-
.../PinotDummyExtraRestletResourceTest.java | 60 ++
.../PinotDummyExtraRestletResource.java | 38 +-
.../helix/ControllerInstanceToggleTest.java | 12 +-
.../controller/helix/ControllerSentinelTestV2.java | 2 +-
.../pinot/controller/helix/ControllerTest.java | 25 +-
.../controller/helix/PinotResourceManagerTest.java | 2 +-
.../pinot/controller/helix/TableCacheTest.java | 141 +--
.../PinotHelixResourceManagerStatelessTest.java | 23 +
.../helix/core/PinotHelixResourceManagerTest.java | 177 +++-
...NonReplicaGroupTieredSegmentAssignmentTest.java | 13 +-
...NonReplicaGroupTieredSegmentAssignmentTest.java | 13 +-
...> MinionInstancesCleanupTaskStatelessTest.java} | 16 +-
...est.java => PinotTaskManagerStatelessTest.java} | 110 ++-
.../PinotLLCRealtimeSegmentManagerTest.java | 158 +++
.../segment/FlushThresholdUpdaterTest.java | 97 +-
.../TableRebalancerClusterStatelessTest.java | 19 +-
.../helix/core/rebalance/TableRebalancerTest.java | 443 +++++----
.../helix/core/retention/RetentionManagerTest.java | 4 +-
.../core/retention/SegmentLineageCleanupTest.java | 4 +-
.../controller/recommender/TestConfigEngine.java | 17 +-
.../tuner/RealTimeAutoIndexTunerTest.java | 8 +-
.../pinot/controller/tuner/TunerRegistryTest.java | 8 +-
.../pinot/controller/utils/FakeHttpServer.java | 55 ++
.../validation/ValidationManagerTest.java | 3 +-
...ealtimeProvisioningInput_highIngestionRate.json | 75 ++
pinot-controller/testng-statefull.xml | 110 ++-
pinot-controller/testng-stateless.xml | 7 +-
pinot-core/pom.xml | 18 +-
.../apache/pinot/core/auth/BasicAuthPrincipal.java | 5 +-
.../pinot/core/common/BlockDocIdIterator.java | 6 +
.../apache/pinot/core/common/MinionConstants.java | 1 +
.../apache/pinot/core/common/ObjectSerDeUtils.java | 136 ++-
.../org/apache/pinot/core/common/Operator.java | 25 +-
.../pinot/core/common/datatable/BaseDataTable.java | 7 +-
.../core/common/datatable/DataTableImplV2.java | 7 +-
.../core/common/datatable/DataTableImplV3.java | 7 +-
.../core/common/datatable/DataTableUtils.java | 9 +-
.../core/data/manager/BaseTableDataManager.java | 234 ++++-
.../core/data/manager/InstanceDataManager.java | 26 +-
.../manager/offline/DimensionTableDataManager.java | 2 +-
.../realtime/LLRealtimeSegmentDataManager.java | 25 +-
.../manager/realtime/RealtimeTableDataManager.java | 32 +-
.../core/data/table/ConcurrentIndexedTable.java | 160 +---
.../apache/pinot/core/data/table/IndexedTable.java | 132 ++-
.../pinot/core/data/table/IntermediateRecord.java | 19 +-
.../java/org/apache/pinot/core/data/table/Key.java | 4 +
.../pinot/core/data/table/SimpleIndexedTable.java | 120 +--
.../apache/pinot/core/data/table/TableResizer.java | 270 ++++--
.../table/UnboundedConcurrentIndexedTable.java | 41 +-
.../function/ConstructFromTextFunction.java | 9 +-
.../function/ConstructFromWKBFunction.java | 9 +-
.../transform/function/ScalarFunctions.java | 52 +-
.../transform/function/StAsBinaryFunction.java | 13 +-
.../transform/function/StAsTextFunction.java | 16 +-
.../transform/function/StGeogFromTextFunction.java | 6 +-
.../transform/function/StGeogFromWKBFunction.java | 6 +-
.../transform/function/StGeomFromTextFunction.java | 10 +-
.../transform/function/StGeomFromWKBFunction.java | 10 +-
.../transform/function/StPolygonFunction.java | 6 +-
.../pinot/core/minion/RawIndexConverter.java | 4 +-
.../apache/pinot/core/minion/SegmentConverter.java | 260 -----
.../minion/rollup/MergeRollupSegmentConverter.java | 188 ----
.../core/minion/rollup/RollupRecordAggregator.java | 79 --
.../minion/segment/DefaultRecordPartitioner.java | 36 -
.../core/minion/segment/MapperRecordReader.java | 121 ---
.../core/minion/segment/RecordAggregator.java | 37 -
.../core/minion/segment/ReducerRecordReader.java | 135 ---
.../AcquireReleaseColumnsSegmentOperator.java | 66 +-
.../apache/pinot/core/operator/BaseOperator.java | 9 -
.../core/operator/InstanceResponseOperator.java | 78 +-
.../operator/blocks/IntermediateResultsBlock.java | 2 +-
.../combine/AggregationOnlyCombineOperator.java | 4 +-
.../core/operator/combine/BaseCombineOperator.java | 83 +-
.../operator/combine/CombineOperatorUtils.java | 11 +-
.../operator/combine/DistinctCombineOperator.java | 5 +-
.../operator/combine/GroupByCombineOperator.java | 123 ++-
.../combine/GroupByOrderByCombineOperator.java | 176 ++--
...xValueBasedSelectionOrderByCombineOperator.java | 90 +-
.../combine/SelectionOnlyCombineOperator.java | 12 +-
.../combine/SelectionOrderByCombineOperator.java | 16 +-
.../dociditerators/MVScanDocIdIterator.java | 29 +-
.../dociditerators/SVScanDocIdIterator.java | 27 +-
.../filter/RangeIndexBasedFilterOperator.java | 106 ++-
.../FSTBasedRegexpPredicateEvaluatorFactory.java | 67 +-
.../operator/query/AggregationGroupByOperator.java | 26 +-
.../query/AggregationGroupByOrderByOperator.java | 28 +-
.../query/DictionaryBasedAggregationOperator.java | 20 +-
.../core/operator/query/DistinctOperator.java | 5 +
.../core/operator/query/SelectionOnlyOperator.java | 5 +
.../operator/query/SelectionOrderByOperator.java | 9 +-
.../StreamingSelectionOnlyCombineOperator.java | 33 +-
.../core/operator/transform/TransformOperator.java | 18 +-
.../function/BinaryOperatorTransformFunction.java | 564 +++++++----
.../function/EqualsTransformFunction.java | 10 +-
.../GreaterThanOrEqualTransformFunction.java | 10 +-
.../function/GreaterThanTransformFunction.java | 10 +-
.../transform/function/InTransformFunction.java | 381 ++++++++
.../function/JsonExtractKeyTransformFunction.java | 64 +-
.../JsonExtractScalarTransformFunction.java | 49 +-
.../function/LessThanOrEqualTransformFunction.java | 10 +-
.../function/LessThanTransformFunction.java | 10 +-
.../function/LiteralTransformFunction.java | 72 +-
.../function/NotEqualsTransformFunction.java | 10 +-
.../function/RegexpExtractTransformFunction.java | 116 +++
.../function/TransformFunctionFactory.java | 28 +-
.../plan/AcquireReleaseColumnsSegmentPlanNode.java | 22 +-
.../plan/AggregationGroupByOrderByPlanNode.java | 94 +-
.../core/plan/AggregationGroupByPlanNode.java | 81 +-
.../pinot/core/plan/AggregationPlanNode.java | 117 ++-
.../apache/pinot/core/plan/CombinePlanNode.java | 66 +-
.../plan/DictionaryBasedAggregationPlanNode.java | 62 --
.../core/plan/DictionaryBasedDistinctPlanNode.java | 67 --
.../apache/pinot/core/plan/DistinctPlanNode.java | 46 +-
.../apache/pinot/core/plan/DocIdSetPlanNode.java | 21 +-
.../org/apache/pinot/core/plan/FilterPlanNode.java | 72 +-
.../pinot/core/plan/InstanceResponsePlanNode.java | 12 +-
.../plan/MetadataBasedAggregationPlanNode.java | 65 --
.../apache/pinot/core/plan/ProjectionPlanNode.java | 21 +-
.../apache/pinot/core/plan/SelectionPlanNode.java | 46 +-
.../core/plan/StreamingSelectionPlanNode.java | 15 +-
.../apache/pinot/core/plan/TransformPlanNode.java | 43 +-
.../core/plan/maker/InstancePlanMakerImplV2.java | 220 ++---
.../apache/pinot/core/plan/maker/PlanMaker.java | 5 +-
.../aggregation/function/AggregationFunction.java | 5 -
.../function/AggregationFunctionFactory.java | 63 ++
.../function/AggregationFunctionUtils.java | 10 -
.../function/AvgAggregationFunction.java | 5 -
.../function/CountAggregationFunction.java | 11 +-
.../function/DistinctAggregationFunction.java | 5 -
.../function/DistinctCountAggregationFunction.java | 5 -
.../DistinctCountBitmapAggregationFunction.java | 5 -
.../DistinctCountHLLAggregationFunction.java | 353 ++++---
.../DistinctCountHLLMVAggregationFunction.java | 64 +-
.../DistinctCountRawHLLAggregationFunction.java | 5 -
...istinctCountThetaSketchAggregationFunction.java | 5 -
.../function/FastHLLAggregationFunction.java | 5 -
.../function/IdSetAggregationFunction.java | 433 ++++++---
...LastDoubleValueWithTimeAggregationFunction.java | 126 +++
.../LastFloatValueWithTimeAggregationFunction.java | 127 +++
.../LastIntValueWithTimeAggregationFunction.java | 142 +++
.../LastLongValueWithTimeAggregationFunction.java | 126 +++
...LastStringValueWithTimeAggregationFunction.java | 124 +++
.../function/LastWithTimeAggregationFunction.java | 222 +++++
.../function/MaxAggregationFunction.java | 5 -
.../function/MinAggregationFunction.java | 5 -
.../function/MinMaxRangeAggregationFunction.java | 5 -
.../function/ModeAggregationFunction.java | 5 -
.../function/PercentileAggregationFunction.java | 5 -
.../function/PercentileEstAggregationFunction.java | 5 -
.../PercentileRawEstAggregationFunction.java | 136 +++
.../PercentileRawEstMVAggregationFunction.java | 43 +
.../PercentileRawTDigestAggregationFunction.java | 137 +++
.../PercentileRawTDigestMVAggregationFunction.java | 43 +
.../PercentileTDigestAggregationFunction.java | 5 -
...artitionedDistinctCountAggregationFunction.java | 5 -
.../function/StUnionAggregationFunction.java | 5 -
.../function/SumAggregationFunction.java | 5 -
.../function/SumPrecisionAggregationFunction.java | 5 -
.../groupby/AggregationGroupByTrimmingService.java | 3 +-
.../groupby/DefaultGroupByExecutor.java | 19 +-
.../groupby/DictionaryBasedGroupKeyGenerator.java | 20 +-
.../pinot/core/query/distinct/DistinctTable.java | 7 +
.../query/executor/ServerQueryExecutorV1Impl.java | 134 +--
.../pinot/core/query/optimizer/QueryOptimizer.java | 4 +-
.../optimizer/filter/NumericalFilterOptimizer.java | 4 +-
.../statement/JsonStatementOptimizer.java | 38 +-
.../statement/StringPredicateFilterOptimizer.java | 119 +++
.../query/pruner/ColumnValueSegmentPruner.java | 258 +++--
.../query/reduce/AggregationDataTableReducer.java | 12 +-
.../core/query/reduce/BrokerReduceService.java | 21 +-
.../query/reduce/DistinctDataTableReducer.java | 10 +-
.../core/query/reduce/GroupByDataTableReducer.java | 23 +-
.../query/reduce/SelectionDataTableReducer.java | 17 +-
.../core/query/request/context/QueryContext.java | 170 +++-
.../core/query/request/context/ThreadTimer.java | 4 +
.../BrokerRequestToQueryContextConverter.java | 47 +-
.../genericrow/GenericRowDeserializer.java | 11 +-
.../genericrow/GenericRowSerializer.java | 7 +-
.../apache/pinot/core/startree/StarTreeUtils.java | 21 +-
.../startree/executor/StarTreeGroupByExecutor.java | 10 +-
.../core/transport/InstanceRequestHandler.java | 18 +-
.../apache/pinot/core/transport/QueryServer.java | 18 +-
.../pinot/core/transport/ServerChannels.java | 4 +-
.../org/apache/pinot/core/transport/TlsConfig.java | 27 +
.../org/apache/pinot/core/util/GroupByUtils.java | 28 +-
.../pinot/core/util/MemoizedClassAssociation.java | 30 +-
.../pinot/core/util/PeerServerSegmentFinder.java | 2 +-
.../{QueryOptions.java => QueryOptionsUtils.java} | 68 +-
.../pinot/core/util/SortedRangeIntersection.java | 8 +-
.../java/org/apache/pinot/core/util/TlsUtils.java | 48 +-
.../apache/pinot/server}/access/AccessControl.java | 10 +-
.../pinot/server}/access/AccessControlFactory.java | 2 +-
.../server}/access/AllowAllAccessFactory.java | 8 +-
.../org/apache/pinot/core/auth/BasicAuthTest.java | 51 +
.../apache/pinot/core/common/DataFetcherTest.java | 9 +-
.../pinot/core/common/ObjectSerDeUtilsTest.java | 76 ++
.../core/common/datatable/DataTableSerDeTest.java | 5 +-
.../core/data/function/DateTimeFunctionsTest.java | 7 +
.../function/InbuiltFunctionEvaluatorTest.java | 36 +
...=> BaseTableDataManagerAcquireSegmentTest.java} | 39 +-
.../data/manager/BaseTableDataManagerTest.java | 664 ++++++-------
.../realtime/RealtimeTableDataManagerTest.java | 56 ++
.../pinot/core/data/table/IndexedTableTest.java | 28 +-
.../pinot/core/data/table/TableResizerTest.java | 136 +--
.../minion/MergeRollupSegmentConverterTest.java | 186 ----
.../pinot/core/minion/SegmentConverterTest.java | 212 -----
.../pinot/core/minion/SegmentPurgerTest.java | 11 +-
.../operator/combine/CombineSlowOperatorsTest.java | 31 +-
.../combine/SelectionCombineOperatorTest.java | 6 +-
.../function/InTransformFunctionTest.java | 194 ++++
.../function/RegexpTransformFunctionTest.java | 74 ++
.../pinot/core/plan/CombinePlanNodeTest.java | 22 +-
...adataAndDictionaryAggregationPlanMakerTest.java | 118 +--
.../DefaultAggregationExecutorTest.java | 2 +-
.../function/AggregationFunctionFactoryTest.java | 84 ++
.../query/aggregation/groupby/GroupByTrimTest.java | 27 +-
.../executor/QueryExecutorExceptionsTest.java | 188 ++++
.../statement/JsonStatementOptimizerTest.java | 123 +--
.../StringPredicateFilterOptimizerTest.java | 66 ++
.../core/query/optimizer/statement/TestHelper.java | 59 ++
.../query/pruner/ColumnValueSegmentPrunerTest.java | 6 +-
.../BrokerRequestToQueryContextConverterTest.java | 46 +-
.../pinot/core/startree/v2/BaseStarTreeV2Test.java | 11 +-
.../org/apache/pinot/core/util/CrcUtilsTest.java | 3 -
.../org/apache/pinot/queries/BaseQueriesTest.java | 8 +-
.../pinot/queries/DistinctCountQueriesTest.java | 81 +-
.../apache/pinot/queries/DistinctQueriesTest.java | 139 ++-
.../apache/pinot/queries/ExpectedQueryResult.java | 56 ++
.../queries/FSTBasedRegexpLikeQueriesTest.java | 39 +-
.../org/apache/pinot/queries/IdSetQueriesTest.java | 153 ++-
...nnerSegmentSelectionSingleValueQueriesTest.java | 2 +-
...terSegmentAggregationMultiValueQueriesTest.java | 113 +++
...erSegmentAggregationSingleValueQueriesTest.java | 118 +++
...erSegmentResultTableSingleValueQueriesTest.java | 21 +-
.../apache/pinot/queries/JsonPathQueriesTest.java | 68 +-
.../pinot/queries/LastWithTimeQueriesTest.java | 548 +++++++++++
.../NoDictionaryCompressionQueriesTest.java | 15 +-
.../org/apache/pinot/queries/QueriesTestUtils.java | 105 +-
...SegmentPartitionedDistinctCountQueriesTest.java | 101 +-
.../queries/SegmentWithNullValueVectorTest.java | 1 -
.../apache/pinot/queries/StUnionQueriesTest.java | 42 +-
.../pinot/queries/TextSearchQueriesTest.java | 204 ++--
pinot-distribution/pom.xml | 18 +-
.../pom.xml | 157 +--
.../tests/BaseClusterIntegrationTest.java | 0
.../integration/tests/BasicAuthTestUtils.java | 0
.../tests/ClusterIntegrationTestUtils.java | 6 +-
.../pinot/integration/tests/ClusterTest.java | 0
...ridClusterIntegrationTestCommandLineRunner.java | 0
.../pinot/integration/tests/QueryGenerator.java | 0
pinot-integration-tests/pom.xml | 11 +-
.../pinot/compat/tests/SqlResultComparator.java | 9 +-
.../org/apache/pinot/compat/tests/StreamOp.java | 11 +-
.../tests/BaseClusterIntegrationTestSet.java | 4 +-
.../tests/BasicAuthTlsRealtimeIntegrationTest.java | 44 +-
.../tests/JsonPathClusterIntegrationTest.java | 43 +-
.../MergeRollupMinionClusterIntegrationTest.java | 77 +-
.../MultiNodesOfflineClusterIntegrationTest.java | 14 -
.../tests/OfflineClusterIntegrationTest.java | 430 ++++++---
.../tests/RealtimeKinesisIntegrationTest.java | 9 +-
...fflineSegmentsMinionClusterIntegrationTest.java | 8 +-
.../tests/SimpleMinionClusterIntegrationTest.java | 13 +-
.../CertBasedTlsChannelAccessControlFactory.java | 74 ++
.../minion/tasks/TestTaskExecutorFactory.java | 5 +
...onulls_default_column_test_extra_columns.schema | 6 +
.../src/test/resources/tlstest.jks | Bin 4271 -> 2277 bytes
.../src/test/resources/tlstest.p12 | Bin 0 -> 10155 bytes
pinot-minion/pom.xml | 2 +-
.../org/apache/pinot/minion/BaseMinionStarter.java | 32 +-
.../java/org/apache/pinot/minion/MinionConf.java | 70 ++
.../executor/MinionTaskZkMetadataManager.java | 2 +-
.../minion/executor/PinotTaskExecutorFactory.java | 9 +
.../executor/TaskExecutorFactoryRegistry.java | 5 +-
pinot-perf/pom.xml | 37 +-
.../apache/pinot/perf/BenchmarkCombineGroupBy.java | 8 +-
.../java/org/apache/pinot/perf/BenchmarkFST.java | 100 ++
.../perf/BenchmarkFixedByteValueReaderWriter.java | 83 ++
.../pinot/perf/BenchmarkGeospatialSerde.java | 93 +-
.../apache/pinot/perf/BenchmarkIndexedTable.java | 9 +-
.../BenchmarkNoDictionaryStringCompression.java | 7 +-
...BenchmarkOffheapBitmapInvertedIndexCreator.java | 13 +-
.../pinot/perf/BenchmarkOfflineIndexReader.java | 10 +-
.../org/apache/pinot/perf/BenchmarkRangeIndex.java | 497 ++++++++++
.../pinot/perf/BenchmarkRoaringBitmapCreation.java | 217 +++++
.../pinot/perf/BenchmarkScanDocIdIterators.java | 246 +++++
.../pinot/perf/BenchmarkStringDictionary.java | 5 +-
.../java/org/apache/pinot/perf/BytesCounter.java | 40 +-
.../java/org/apache/pinot/perf/Distribution.java | 89 ++
.../org/apache/pinot/perf/RawIndexBenchmark.java | 33 +-
.../pinot-batch-ingestion-common/pom.xml | 2 +-
.../batch/common/SegmentGenerationTaskRunner.java | 3 +-
.../pinot-batch-ingestion-hadoop/pom.xml | 2 +-
.../batch/hadoop/HadoopSegmentCreationMapper.java | 3 +-
.../hadoop/HadoopSegmentGenerationJobRunner.java | 4 +-
.../pinot-batch-ingestion-spark/pom.xml | 2 +-
.../pinot-batch-ingestion-standalone/pom.xml | 2 +-
pinot-plugins/pinot-batch-ingestion/pom.xml | 2 +-
.../v0_deprecated/pinot-hadoop/pom.xml | 2 +-
.../pinot/hadoop/job/HadoopSegmentCreationJob.java | 19 +-
.../hadoop/job/HadoopSegmentPreprocessingJob.java | 3 +-
.../hadoop/job/mappers/SegmentCreationMapper.java | 3 +-
.../preprocess/HadoopDataPreprocessingHelper.java | 14 +-
.../HadoopDataPreprocessingHelperTest.java | 108 +++
.../v0_deprecated/pinot-ingestion-common/pom.xml | 2 +-
.../ingestion/jobs/SegmentPreprocessingJob.java | 7 +
.../preprocess/DataPreprocessingHelper.java | 12 +-
.../preprocess/OrcDataPreprocessingHelper.java | 5 +-
.../mappers/AvroDataPreprocessingMapper.java | 19 +-
.../mappers/OrcDataPreprocessingMapper.java | 11 +-
.../mappers/SegmentPreprocessingMapper.java | 3 +-
.../AvroDataPreprocessingPartitioner.java | 56 +-
.../partitioners/GenericPartitioner.java | 13 +-
.../OrcDataPreprocessingPartitioner.java | 68 +-
.../ingestion/utils/InternalConfigConstants.java | 2 +
.../ingestion/utils/preprocess/TextComparator.java | 5 +-
.../v0_deprecated/pinot-spark/pom.xml | 2 +-
.../spark/jobs/SparkSegmentCreationFunction.java | 11 +-
.../pinot/spark/jobs/SparkSegmentCreationJob.java | 5 +-
.../spark/jobs/SparkSegmentPreprocessingJob.java | 3 +-
.../preprocess/SparkDataPreprocessingHelper.java | 18 +-
.../SparkDataPreprocessingPartitioner.java | 16 +-
.../pinot-batch-ingestion/v0_deprecated/pom.xml | 2 +-
.../pinot-environment/pinot-azure/pom.xml | 2 +-
pinot-plugins/pinot-environment/pom.xml | 2 +-
pinot-plugins/pinot-file-system/pinot-adls/pom.xml | 2 +-
pinot-plugins/pinot-file-system/pinot-gcs/pom.xml | 2 +-
.../apache/pinot/plugin/filesystem/GcsPinotFS.java | 8 +-
.../org/apache/pinot/plugin/filesystem/GcsUri.java | 12 +-
pinot-plugins/pinot-file-system/pinot-hdfs/pom.xml | 2 +-
.../pinot/plugin/filesystem/HadoopPinotFS.java | 2 +-
pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 40 +-
.../apache/pinot/plugin/filesystem/S3PinotFS.java | 2 +-
pinot-plugins/pinot-file-system/pom.xml | 2 +-
.../pinot-input-format/pinot-avro-base/pom.xml | 9 +-
.../pinot/plugin/inputformat/avro/AvroUtils.java | 10 +-
.../plugin/inputformat/avro/AvroUtilsTest.java | 54 +-
.../pinot-input-format/pinot-avro/pom.xml | 12 +-
.../pinot-confluent-avro/pom.xml | 2 +-
pinot-plugins/pinot-input-format/pinot-csv/pom.xml | 12 +-
.../plugin/inputformat/csv/CSVRecordExtractor.java | 40 +-
.../inputformat/csv/CSVRecordExtractorConfig.java | 6 +-
.../plugin/inputformat/csv/CSVRecordReader.java | 6 +-
.../inputformat/csv/CSVRecordReaderConfig.java | 9 +
.../pinot-input-format/pinot-json/pom.xml | 12 +-
pinot-plugins/pinot-input-format/pinot-orc/pom.xml | 2 +-
.../plugin/inputformat/orc/ORCRecordReader.java | 9 +-
.../inputformat/orc/ORCRecordExtractorTest.java | 15 +-
.../inputformat/orc/ORCRecordReaderTest.java | 7 +-
.../pinot-input-format/pinot-parquet/pom.xml | 2 +-
.../parquet/ParquetNativeRecordExtractor.java | 2 +-
.../parquet/ParquetNativeRecordReaderTest.java | 66 ++
.../pinot-input-format/pinot-protobuf/pom.xml | 2 +-
.../protobuf/ProtoBufRecordExtractor.java | 2 +-
.../protobuf/ProtoBufRecordExtractorTest.java | 7 +-
.../pinot-input-format/pinot-thrift/pom.xml | 2 +-
pinot-plugins/pinot-input-format/pom.xml | 2 +-
.../pinot-metrics/pinot-dropwizard/pom.xml | 2 +-
pinot-plugins/pinot-metrics/pinot-yammer/pom.xml | 11 +-
pinot-plugins/pinot-metrics/pom.xml | 2 +-
.../pinot-minion-builtin-tasks/pom.xml | 2 +-
.../BaseMultipleSegmentsConversionExecutor.java | 11 +-
.../tasks/BaseSingleSegmentConversionExecutor.java | 4 +
.../minion/tasks/SegmentConversionUtils.java | 36 +-
.../ConvertToRawIndexTaskExecutorFactory.java | 5 +
.../ConvertToRawIndexTaskGenerator.java | 19 +-
.../tasks/mergerollup/MergeRollupTaskExecutor.java | 5 +
.../MergeRollupTaskExecutorFactory.java | 9 +-
.../mergerollup/MergeRollupTaskGenerator.java | 475 ++++++---
.../tasks/mergerollup/MergeRollupTaskUtils.java | 3 +-
.../tasks/purge/PurgeTaskExecutorFactory.java | 5 +
.../RealtimeToOfflineSegmentsTaskExecutor.java | 5 +-
...altimeToOfflineSegmentsTaskExecutorFactory.java | 10 +-
.../RealtimeToOfflineSegmentsTaskGenerator.java | 9 +-
.../SegmentGenerationAndPushTaskExecutor.java | 4 +-
...egmentGenerationAndPushTaskExecutorFactory.java | 5 +
.../mergerollup/MergeRollupTaskExecutorTest.java | 3 +-
.../mergerollup/MergeRollupTaskGeneratorTest.java | 214 ++++-
.../mergerollup/MergeRollupTaskUtilsTest.java | 4 +-
.../RealtimeToOfflineSegmentsTaskExecutorTest.java | 16 +-
...RealtimeToOfflineSegmentsTaskGeneratorTest.java | 38 +-
pinot-plugins/pinot-minion-tasks/pom.xml | 2 +-
.../pinot-segment-uploader-default/pom.xml | 2 +-
.../segmentuploader/SegmentUploaderDefault.java | 16 +-
pinot-plugins/pinot-segment-uploader/pom.xml | 2 +-
.../pinot-segment-writer-file-based/pom.xml | 2 +-
.../filebased/FileBasedSegmentWriter.java | 14 +-
pinot-plugins/pinot-segment-writer/pom.xml | 2 +-
.../pinot-stream-ingestion/pinot-kafka-0.9/pom.xml | 2 +-
.../stream/kafka09/KafkaStreamLevelConsumer.java | 2 +-
.../pinot-stream-ingestion/pinot-kafka-2.0/pom.xml | 2 +-
.../stream/kafka20/KafkaStreamLevelConsumer.java | 2 +-
.../pinot-kafka-base/pom.xml | 2 +-
.../pinot-stream-ingestion/pinot-kinesis/pom.xml | 39 +-
.../plugin/stream/kinesis/KinesisConsumer.java | 15 +-
.../pinot-stream-ingestion/pinot-pulsar/pom.xml | 3 +-
.../stream/pulsar/PulsarStreamLevelConsumer.java | 2 +-
pinot-plugins/pinot-stream-ingestion/pom.xml | 2 +-
pinot-plugins/pom.xml | 6 +-
pinot-segment-local/pom.xml | 11 +-
.../local/aggregator/ValueAggregatorFactory.java | 4 +
.../DoubleLongPair.java} | 30 +-
.../FloatLongPair.java} | 30 +-
.../IntLongPair.java} | 30 +-
.../LongLongPair.java} | 30 +-
.../segment/local/customobject/QuantileDigest.java | 25 +-
.../customobject/SerializedQuantileDigest.java | 50 +
.../local/customobject/SerializedTDigest.java | 49 +
.../StringLongPair.java} | 40 +-
.../segment/local/customobject/ValueLongPair.java | 31 +-
.../local/data/manager/TableDataManager.java | 27 +-
.../local/function/InbuiltFunctionEvaluator.java | 44 +-
.../immutable/ImmutableSegmentImpl.java | 13 +-
.../immutable/ImmutableSegmentLoader.java | 117 ++-
.../indexsegment/mutable/MutableSegmentImpl.java | 59 +-
.../io/compression/ChunkCompressorFactory.java | 35 +-
.../local/io/compression/LZ4Compressor.java | 24 +-
.../local/io/compression/LZ4Decompressor.java | 15 +-
...ompressor.java => LZ4WithLengthCompressor.java} | 34 +-
...pressor.java => LZ4WithLengthDecompressor.java} | 31 +-
.../io/compression/PassThroughCompressor.java | 18 +-
.../io/compression/PassThroughDecompressor.java | 13 +-
.../local/io/compression/SnappyCompressor.java | 18 +-
.../local/io/compression/SnappyDecompressor.java | 14 +-
.../local/io/compression/ZstandardCompressor.java | 20 +-
.../io/compression/ZstandardDecompressor.java | 14 +-
.../impl/FixedByteSingleValueMultiColReader.java | 5 +-
.../local/io/util/FixedByteValueReaderWriter.java | 47 +-
.../segment/local/io/util/PinotDataBitSet.java | 33 +-
.../segment/local/io/util/PinotDataBitSetV2.java | 15 +-
.../local/io/util/VarLengthValueReader.java | 5 +-
.../local/io/util/VarLengthValueWriter.java | 5 +-
.../writer/impl/BaseChunkSVForwardIndexWriter.java | 60 +-
.../impl/FixedByteChunkSVForwardIndexWriter.java | 4 +-
.../impl/FixedByteSingleValueMultiColWriter.java | 5 +-
.../impl/VarByteChunkSVForwardIndexWriter.java | 86 +-
.../impl/VarByteChunkSVForwardIndexWriterV4.java | 229 +++++
.../local/io/writer/impl/VarByteChunkWriter.java | 14 +-
...der.java => DefaultSegmentDirectoryLoader.java} | 20 +-
.../dictionary/StringOffHeapMutableDictionary.java | 11 +-
.../impl/forward/VarByteSVMutableForwardIndex.java | 7 +-
.../recordtransformer/DataTypeTransformer.java | 6 +-
.../creator/impl/SegmentColumnarIndexCreator.java | 252 +++--
.../creator/impl/SegmentDictionaryCreator.java | 5 +-
.../fwd/MultiValueFixedByteRawIndexCreator.java | 171 ++++
....java => MultiValueVarByteRawIndexCreator.java} | 63 +-
.../fwd/SingleValueVarByteRawIndexCreator.java | 10 +-
.../impl/inv/BitSlicedRangeIndexCreator.java | 143 +++
.../impl/inv/BitmapInvertedIndexWriter.java | 54 +-
.../creator/impl/inv/MmapFileWriter.java} | 41 +-
.../inv/OffHeapBitmapInvertedIndexCreator.java | 2 +-
.../impl/inv/OnHeapBitmapInvertedIndexCreator.java | 2 +-
.../creator/impl/inv/RangeIndexCreator.java | 15 +-
.../impl/inv/geospatial/OffHeapH3IndexCreator.java | 2 +-
.../impl/inv/json/BaseJsonIndexCreator.java | 2 +-
.../impl/inv/json/OffHeapJsonIndexCreator.java | 9 +-
.../impl/inv/json/OnHeapJsonIndexCreator.java | 5 +-
.../impl/inv/text/LuceneFSTIndexCreator.java | 5 +
.../stats/AbstractColumnStatisticsCollector.java | 7 +-
.../stats/BytesColumnPredIndexStatsCollector.java | 44 +-
.../stats/DoubleColumnPreIndexStatsCollector.java | 15 +-
.../stats/FloatColumnPreIndexStatsCollector.java | 15 +-
.../stats/IntColumnPreIndexStatsCollector.java | 15 +-
.../stats/LongColumnPreIndexStatsCollector.java | 15 +-
.../stats/SegmentPreIndexStatsCollectorImpl.java | 2 +-
.../stats/StringColumnPreIndexStatsCollector.java | 30 +-
.../creator/impl/text/LuceneTextIndexCreator.java | 20 +
.../index/column/IntermediateIndexContainer.java | 2 +-
.../index/column/PhysicalColumnIndexContainer.java | 52 +-
.../converter/SegmentV1V2ToV3FormatConverter.java | 14 +-
.../segment/index/datasource/BaseDataSource.java | 7 +-
.../index/datasource/MutableDataSource.java | 14 +-
...UpdateIndexException.java => IndexHandler.java} | 14 +-
.../segment/index/loader/IndexHandlerFactory.java | 63 ++
.../segment/index/loader/IndexLoadingConfig.java | 54 +-
.../local/segment/index/loader/LoaderUtils.java | 13 +-
.../segment/index/loader/SegmentPreProcessor.java | 52 +-
.../loader/bloomfilter/BloomFilterHandler.java | 56 +-
.../defaultcolumn/BaseDefaultColumnHandler.java | 103 +-
.../defaultcolumn/V1DefaultColumnHandler.java | 2 +-
.../defaultcolumn/V3DefaultColumnHandler.java | 50 +-
...neFSTIndexHandler.java => FSTIndexHandler.java} | 79 +-
.../index/loader/invertedindex/H3IndexHandler.java | 64 +-
.../loader/invertedindex/InvertedIndexHandler.java | 59 +-
.../loader/invertedindex/JsonIndexHandler.java | 61 +-
.../loader/invertedindex/RangeIndexHandler.java | 77 +-
.../loader/invertedindex/TextIndexHandler.java | 142 +--
.../index/readers/BaseImmutableDictionary.java | 7 +-
.../index/readers/BitSlicedRangeIndexReader.java | 137 +++
.../index/readers/BitmapInvertedIndexReader.java | 29 -
.../index/readers/NullValueVectorReaderImpl.java | 8 +-
...eIndexReader.java => RangeIndexReaderImpl.java} | 193 ++--
.../readers/bloom/GuavaBloomFilterReaderUtils.java | 5 +-
.../forward/BaseChunkSVForwardIndexReader.java | 2 +-
...ava => FixedByteChunkMVForwardIndexReader.java} | 112 +--
....java => VarByteChunkMVForwardIndexReader.java} | 99 +-
.../forward/VarByteChunkSVForwardIndexReader.java | 7 +-
.../VarByteChunkSVForwardIndexReaderV4.java | 296 ++++++
.../readers/MultiplePinotSegmentRecordReader.java | 2 +-
.../local/segment/store/FilePerIndexDirectory.java | 18 +-
.../segment/store/SegmentLocalFSDirectory.java | 8 +-
.../segment/store/SingleFileIndexDirectory.java | 9 +
.../virtualcolumn/VirtualColumnIndexContainer.java | 3 +-
.../segment/local/startree/OffHeapStarTree.java | 5 +-
.../local/startree/StarTreeBuilderUtils.java | 7 +-
.../startree/v2/store/StarTreeIndexContainer.java | 11 +-
.../startree/v2/store/StarTreeIndexMapUtils.java | 42 +-
.../upsert/PartitionUpsertMetadataManager.java | 1 +
.../segment/local/utils/CustomSerDeUtils.java | 9 +-
.../pinot/segment/local/utils/FPOrdering.java | 78 ++
.../pinot/segment/local/utils/GeometryUtils.java | 10 +
.../pinot/segment/local/utils/IngestionUtils.java | 18 +-
.../segment/local/utils/TableConfigUtils.java | 122 ++-
.../segment/local/utils/fst/RegexpMatcher.java | 2 +-
.../utils/nativefst/ByteSequenceIterator.java | 180 ++++
.../local/utils/nativefst/ConstantArcSizeFST.java | 157 +++
.../pinot/segment/local/utils/nativefst/FST.java | 438 +++++++++
.../segment/local/utils/nativefst/FSTFlags.java | 46 +-
.../segment/local/utils/nativefst/FSTHeader.java | 80 ++
.../local/utils/nativefst/FSTTraversal.java | 127 +++
.../local/utils/nativefst/ImmutableFST.java | 396 ++++++++
.../segment/local/utils/nativefst/MatchResult.java | 90 ++
.../nativefst/NativeFSTIndexCreator.java} | 52 +-
.../utils/nativefst/NativeFSTIndexReader.java | 75 ++
.../local/utils/nativefst/StateVisitor.java | 13 +-
.../local/utils/nativefst/automaton/Automaton.java | 658 +++++++++++++
.../nativefst/automaton/AutomatonMatcher.java | 264 +++++
.../nativefst/automaton/AutomatonProvider.java} | 22 +-
.../utils/nativefst/automaton/BasicAutomata.java | 250 +++++
.../utils/nativefst/automaton/BasicOperations.java | 647 +++++++++++++
.../nativefst/automaton/CharacterRunAutomaton.java | 66 ++
.../automaton/MinimizationOperations.java | 710 ++++++++++++++
.../local/utils/nativefst/automaton/README.md | 84 ++
.../local/utils/nativefst/automaton/RegExp.java | 771 +++++++++++++++
.../utils/nativefst/automaton/RunAutomaton.java | 311 ++++++
.../nativefst/automaton/SpecialOperations.java | 360 +++++++
.../local/utils/nativefst/automaton/State.java | 173 ++++
.../local/utils/nativefst/automaton/StatePair.java | 68 ++
.../nativefst/automaton/StringUnionOperations.java | 372 ++++++++
.../utils/nativefst/automaton/Transition.java | 149 +++
.../nativefst/automaton/TransitionComparator.java | 79 ++
.../local/utils/nativefst/builder/FSTBuilder.java | 566 +++++++++++
.../local/utils/nativefst/builder/FSTInfo.java | 143 +++
.../utils/nativefst/builder/FSTSerializer.java | 57 ++
.../utils/nativefst/builder/FSTSerializerImpl.java | 370 ++++++++
.../local/utils/nativefst/builder/FSTUtils.java | 180 ++++
.../local/utils/nativefst/utils/RegexpMatcher.java | 169 ++++
.../local/io/compression/TestCompression.java | 85 ++
.../io/util/VarLengthValueReaderWriterTest.java | 6 +-
.../segment/local/segment/creator/BlocksTest.java | 1 -
.../impl/SegmentColumnarIndexCreatorTest.java | 57 ++
.../segment/index/SegmentMetadataImplTest.java | 3 +
.../SegmentV1V2ToV3FormatConverterTest.java | 2 +-
.../index/creator/BitSlicedIndexCreatorTest.java | 521 ++++++++++
.../MultiValueFixedByteRawIndexCreatorTest.java | 178 ++++
.../MultiValueVarByteRawIndexCreatorTest.java | 141 +++
.../index/creator/NativeFSTIndexCreatorTest.java | 75 ++
.../index/creator/RangeIndexCreatorTest.java | 430 ++++++---
.../segment/index/creator/RawIndexCreatorTest.java | 145 ++-
.../SegmentGenerationWithBytesTypeTest.java | 10 +-
.../SegmentGenerationWithTimeColumnTest.java | 153 ++-
.../segment/index/creator/VarByteChunkV4Test.java | 167 ++++
.../creator/inv/BitmapInvertedIndexWriterTest.java | 144 +++
.../forward/VarByteChunkSVForwardIndexTest.java | 7 +-
.../mutable/VarByteSVMutableForwardIndexTest.java | 7 +-
.../local/segment/index/loader/LoaderTest.java | 24 +-
.../index/loader/SegmentPreProcessorTest.java | 413 ++++++--
.../FixedByteWidthRowColDataFileReaderTest.java | 2 -
.../FixedByteValueReaderWriterTest.java | 77 ++
...FixedByteWidthRowColForwardIndexWriterTest.java | 2 -
.../store/ColumnIndexDirectoryTestHelper.java | 2 +-
.../segment/store/FilePerIndexDirectoryTest.java | 42 +-
.../segment/store/SegmentLocalFSDirectoryTest.java | 2 +-
.../store/SingleFileIndexDirectoryTest.java | 59 +-
.../pinot/segment/local/utils/FPOrderingTest.java | 134 +++
.../segment/local/utils/TableConfigUtilsTest.java | 227 +++--
.../local/utils/nativefst/FSTBuilderTest.java | 99 ++
.../utils/nativefst/FSTRegexpWithWeirdTest.java | 85 ++
.../local/utils/nativefst/FSTSanityTest.java | 84 ++
.../local/utils/nativefst/FSTTestUtils.java | 139 +++
.../local/utils/nativefst/FSTTraversalTest.java | 324 +++++++
.../nativefst/ImmutableFSTDeserializedTest.java | 50 +
.../nativefst/ImmutableFSTSerializerTest.java | 13 +-
.../local/utils/nativefst/ImmutableFSTTest.java | 147 +++
.../segment/local/utils/nativefst/MinMax.java | 21 +-
.../local/utils/nativefst/SerializerTestBase.java | 195 ++++
.../src/test/resources/data/abc-numbers.native.fst | Bin 0 -> 29 bytes
.../src/test/resources/data/abc.native.fst | Bin 0 -> 24 bytes
.../src/test/resources/data/en_tst.dict | Bin 0 -> 1070678 bytes
.../src/test/resources/data/newColumnsSchema3.json | 4 -
...nsSchema4.json => newColumnsSchemaWithFST.json} | 0
...Schema.json => newColumnsSchemaWithH3Json.json} | 10 +-
...xtSchema.json => newColumnsSchemaWithText.json} | 10 +
.../src/test/resources/data/serfst.txt | Bin 0 -> 29951 bytes
.../src/test/resources/data/words.txt | 1004 ++++++++++++++++++++
pinot-segment-spi/pom.xml | 11 +-
.../pinot/segment/spi/AggregationFunctionType.java | 13 +
.../org/apache/pinot/segment/spi/FetchContext.java | 86 ++
.../org/apache/pinot/segment/spi/IndexSegment.java | 20 +-
.../org/apache/pinot/segment/spi/V1Constants.java | 1 +
.../spi/compression/ChunkCompressionType.java | 11 +-
.../segment/spi/compression/ChunkCompressor.java | 13 +
.../segment/spi/compression/ChunkDecompressor.java | 9 +
.../spi/creator/ColumnIndexCreationInfo.java | 4 +
.../segment/spi/creator/ColumnStatistics.java | 7 +
.../spi/creator/SegmentGeneratorConfig.java | 54 +-
.../creator/name/FixedSegmentNameGenerator.java | 2 +
.../name/NormalizedDateSegmentNameGenerator.java | 24 +-
.../spi/creator/name/SegmentNameGenerator.java | 12 +
.../creator/name/SegmentNameGeneratorFactory.java | 2 +-
.../creator/name/SimpleSegmentNameGenerator.java | 9 +
.../pinot/segment/spi/datasource/DataSource.java | 3 +-
.../spi/index/column/ColumnIndexContainer.java | 3 +-
.../creator/CombinedInvertedIndexCreator.java | 9 +-
.../spi/index/creator/ForwardIndexCreator.java | 9 +
.../spi/index/creator/TextIndexCreator.java | 5 +
.../spi/index/metadata/SegmentMetadataImpl.java | 3 +
.../spi/index/reader/ForwardIndexReader.java | 15 +
.../segment/spi/index/reader/RangeIndexReader.java | 120 +++
.../segment/spi/loader/SegmentDirectoryLoader.java | 8 +-
.../SegmentDirectoryLoaderContext.java} | 37 +-
.../spi/loader/SegmentDirectoryLoaderRegistry.java | 33 +-
.../pinot/segment/spi/memory/CleanerUtil.java | 10 +
.../spi/partition/ByteArrayPartitionFunction.java | 6 +
.../spi/partition/HashCodePartitionFunction.java | 6 +
.../spi/partition/ModuloPartitionFunction.java | 6 +
.../spi/partition/MurmurPartitionFunction.java | 11 +-
.../segment/spi/partition/PartitionFunction.java | 11 +
.../spi/partition/PartitionFunctionFactory.java | 3 +-
.../segment/spi/store/ColumnIndexDirectory.java | 13 +-
.../pinot/segment/spi/store/SegmentDirectory.java | 26 +-
.../segment/spi/store/SegmentDirectoryPaths.java | 6 +-
.../spi/creator/SegmentGeneratorConfigTest.java | 15 +-
.../NormalizedDateSegmentNameGeneratorTest.java | 62 +-
.../name/SimpleSegmentNameGeneratorTest.java | 27 +
.../misc/AggregationFunctionColumnPairTest.java | 18 +
.../spi/partition/PartitionFunctionTest.java | 55 +-
pinot-server/pom.xml | 2 +-
.../pinot/server/api/resources/TablesResource.java | 6 +-
.../org/apache/pinot/server/conf/ServerConf.java | 6 +-
.../pinot/server/starter/ServerInstance.java | 12 +-
.../server/starter/helix/AdminApiApplication.java | 2 +-
.../server/starter/helix/BaseServerStarter.java | 100 +-
.../starter/helix/HelixInstanceDataManager.java | 126 +--
.../helix/HelixInstanceDataManagerConfig.java | 27 +-
.../helix/OffsetBasedConsumptionStatusChecker.java | 114 +++
.../starter/helix/SegmentFetcherAndLoader.java | 264 -----
.../helix/SegmentMessageHandlerFactory.java | 20 +-
.../SegmentOnlineOfflineStateModelFactory.java | 9 +-
.../apache/pinot/server/api/AccessControlTest.java | 18 +-
.../apache/pinot/server/api/BaseResourceTest.java | 6 +-
.../server/api/PinotServerAppConfigsTest.java | 12 +-
.../OffsetBasedConsumptionStatusCheckerTest.java | 251 +++++
pinot-spi/pom.xml | 9 +-
.../org/apache/pinot/spi/config/ConfigUtils.java | 2 +-
.../config/instance/InstanceDataManagerConfig.java | 4 +-
.../org/apache/pinot/spi/config/table/FSTType.java | 8 +-
.../apache/pinot/spi/config/table/FieldConfig.java | 32 +-
.../pinot/spi/config/table/IndexingConfig.java | 21 +
.../apache/pinot/spi/config/table/TableConfig.java | 19 +-
.../apache/pinot/spi/config/table/TierConfig.java | 27 +-
.../pinot/spi/data/DateTimeFormatPatternSpec.java | 10 +-
.../apache/pinot/spi/env/PinotConfiguration.java | 19 +-
.../pinot/spi/ingestion/batch/spec/TlsSpec.java | 18 +
.../segment/uploader/SegmentUploader.java | 10 +-
.../ingestion/segment/writer/SegmentWriter.java | 10 +-
.../apache/pinot/spi/plugin/PluginClassLoader.java | 43 +-
.../apache/pinot/spi/stream/OffsetCriteria.java | 3 +
.../java/org/apache/pinot/spi/utils/ByteArray.java | 89 +-
.../apache/pinot/spi/utils/CommonConstants.java | 23 +-
.../java/org/apache/pinot/spi/utils/JsonUtils.java | 7 +-
.../org/apache/pinot/spi/utils/Obfuscator.java | 140 +++
.../java/org/apache/pinot/spi/utils/Pairs.java | 44 +-
.../org/apache/pinot/spi/utils/StringUtils.java | 53 --
.../spi/utils/builder/TableConfigBuilder.java | 8 +-
.../apache/pinot/spi/config/ConfigUtilsTest.java | 17 +
.../spi/data/readers/AbstractRecordReaderTest.java | 4 +-
.../org/apache/pinot/spi/utils/ByteArrayTest.java | 79 ++
.../org/apache/pinot/spi/utils/ObfuscatorTest.java | 119 +++
pinot-tools/pom.xml | 7 +-
.../apache/pinot/tools/AbstractBaseCommand.java | 13 -
.../org/apache/pinot/tools/AuthQuickstart.java | 8 +-
.../pinot/tools/AutoAddInvertedIndexTool.java | 43 +-
.../pinot/tools/BatchQuickstartWithMinion.java | 4 +
.../org/apache/pinot/tools/BootstrapTableTool.java | 16 +-
.../main/java/org/apache/pinot/tools/Command.java | 11 +-
.../org/apache/pinot/tools/HybridQuickstart.java | 6 +
.../org/apache/pinot/tools/JoinQuickStart.java | 6 +
.../apache/pinot/tools/JsonIndexQuickStart.java | 5 +
.../OfflineComplexTypeHandlingQuickStart.java | 4 +
.../org/apache/pinot/tools/PinotToolLauncher.java | 84 +-
.../org/apache/pinot/tools/QuickStartBase.java | 3 +
.../java/org/apache/pinot/tools/Quickstart.java | 5 +
.../RealtimeComplexTypeHandlingQuickStart.java | 5 +
.../pinot/tools/RealtimeJsonIndexQuickStart.java | 5 +
.../org/apache/pinot/tools/RealtimeQuickStart.java | 5 +
...tart.java => RealtimeQuickStartWithMinion.java} | 121 ++-
.../org/apache/pinot/tools/SegmentDumpTool.java | 20 +-
.../java/org/apache/pinot/tools/SpeedTest.java | 2 -
.../org/apache/pinot/tools/UpdateSegmentState.java | 21 +-
...exQuickStart.java => UpsertJsonQuickStart.java} | 19 +-
.../org/apache/pinot/tools/UpsertQuickStart.java | 6 +
.../apache/pinot/tools/ValidateTableRetention.java | 21 +-
.../pinot/tools/admin/PinotAdministrator.java | 188 ++--
.../tools/admin/command/AddSchemaCommand.java | 23 +-
.../pinot/tools/admin/command/AddTableCommand.java | 31 +-
.../tools/admin/command/AddTenantCommand.java | 34 +-
.../tools/admin/command/AnonymizeDataCommand.java | 48 +-
.../admin/command/AvroSchemaToPinotSchema.java | 59 +-
.../tools/admin/command/BootstrapTableCommand.java | 23 +-
.../admin/command/ChangeNumReplicasCommand.java | 17 +-
.../tools/admin/command/ChangeTableState.java | 23 +-
.../tools/admin/command/CreateSegmentCommand.java | 34 +-
.../tools/admin/command/DeleteClusterCommand.java | 11 +-
.../admin/command/EnumArrayOptionHandler.java | 67 --
.../tools/admin/command/GenerateDataCommand.java | 24 +-
.../command/GitHubEventsQuickStartCommand.java | 7 +-
.../tools/admin/command/ImportDataCommand.java | 33 +-
.../tools/admin/command/JsonToPinotSchema.java | 59 +-
.../command/LaunchDataIngestionJobCommand.java | 59 +-
.../tools/admin/command/MoveReplicaGroup.java | 55 +-
.../OfflineSegmentIntervalCheckerCommand.java | 13 +-
.../admin/command/OperateClusterConfigCommand.java | 25 +-
.../tools/admin/command/PostQueryCommand.java | 23 +-
.../tools/admin/command/QuickStartCommand.java | 120 +--
.../command/RealtimeProvisioningHelperCommand.java | 48 +-
.../tools/admin/command/RebalanceTableCommand.java | 47 +-
.../command/SegmentProcessorFrameworkCommand.java | 9 +-
.../admin/command/ShowClusterInfoCommand.java | 15 +-
.../tools/admin/command/StartBrokerCommand.java | 21 +-
.../admin/command/StartControllerCommand.java | 32 +-
.../tools/admin/command/StartKafkaCommand.java | 13 +-
.../tools/admin/command/StartMinionCommand.java | 20 +-
.../tools/admin/command/StartServerCommand.java | 37 +-
.../admin/command/StartServiceManagerCommand.java | 36 +-
.../tools/admin/command/StartZookeeperCommand.java | 16 +-
.../tools/admin/command/StopProcessCommand.java | 17 +-
.../admin/command/StreamAvroIntoKafkaCommand.java | 23 +-
.../admin/command/StreamGitHubEventsCommand.java | 21 +-
.../tools/admin/command/UploadSegmentCommand.java | 23 +-
.../tools/admin/command/ValidateConfigCommand.java | 27 +-
.../admin/command/VerifyClusterStateCommand.java | 17 +-
.../tools/admin/command/VerifySegmentState.java | 27 +-
.../anonymizer/PinotDataAndQueryAnonymizer.java | 2 +-
.../tools/filesystem/PinotFSBenchmarkRunner.java | 33 +-
.../pinot/tools/perf/PerfBenchmarkDriver.java | 15 +-
.../pinot/tools/perf/PerfBenchmarkRunner.java | 52 +-
.../org/apache/pinot/tools/perf/QueryRunner.java | 144 ++-
.../tools/query/comparison/ClusterStarter.java | 5 +-
.../tools/query/comparison/QueryComparison.java | 20 +-
.../tools/query/comparison/StatsGenerator.java | 4 +-
.../apache/pinot/tools/scan/query/AvgFunction.java | 2 +-
.../tools/scan/query/EqualsPredicateFilter.java | 2 +-
.../pinot/tools/scan/query/InPredicateFilter.java | 2 +-
.../tools/scan/query/NotInPredicateFilter.java | 2 +-
.../pinot/tools/scan/query/NotPredicateFilter.java | 2 +-
.../apache/pinot/tools/scan/query/Projection.java | 4 +-
.../pinot/tools/scan/query/QueryResponse.java | 8 +-
.../tools/scan/query/RangePredicateFilter.java | 4 +-
.../apache/pinot/tools/scan/query/ResultTable.java | 12 +-
.../tools/scan/query/ScanBasedQueryProcessor.java | 2 +-
.../tools/scan/query/SegmentQueryProcessor.java | 4 +-
.../converter/DictionaryToRawIndexConverter.java | 51 +-
.../converter/PinotSegmentConvertCommand.java | 29 +-
.../segment/converter/SegmentMergeCommand.java | 306 ------
.../pinot/tools/service/PinotServiceManager.java | 5 +-
.../service/PinotServiceManagerStatusCallback.java | 7 +-
.../PinotServiceManagerInstanceResource.java | 21 +-
.../pinot/tools/streams/MeetupRsvpJsonStream.java | 22 +-
.../pinot/tools/streams/MeetupRsvpStream.java | 9 +-
.../githubevents/PullRequestMergedEvent.java | 12 +-
.../githubEvents_offline_table_config.json | 42 +
.../githubEvents_realtime_table_config.json | 56 ++
.../stream/githubEvents/githubEvents_schema.json | 27 +
.../githubEvents/rawdata/2021-07-21-few-hours.json | 800 ++++++++++++++++
...sert_json_meetupRsvp_realtime_table_config.json | 101 ++
.../meetupRsvp/upsert_json_meetupRsvp_schema.json | 50 +
pinot-tools/src/main/resources/generator/README.md | 3 +-
.../tools/admin/command/TestQuickStartCommand.java | 114 +++
pom.xml | 100 +-
996 files changed, 41020 insertions(+), 12498 deletions(-)
create mode 100755 .github/workflows/scripts/.pinot_linter.sh
delete mode 100644 licenses-binary/LICENSE-args4j.txt
create mode 100644 licenses-binary/LICENSE-paranamer.txt
create mode 100644 licenses-binary/LICENSE-typelevel.txt
create mode 100644 pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/DistinctCountRewriteTest.java
copy pinot-common/src/main/java/org/apache/pinot/common/{lineage/LineageEntryState.java => exception/SchemaAlreadyExistsException.java} (81%)
copy pinot-tools/src/main/java/org/apache/pinot/tools/scan/query/NotPredicateFilter.java => pinot-common/src/main/java/org/apache/pinot/common/function/JsonPathCache.java (53%)
copy pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/segmentgenerationandpush/SegmentGenerationAndPushTaskExecutorFactory.java => pinot-common/src/main/java/org/apache/pinot/common/minion/BaseTaskMetadata.java (50%)
rename pinot-core/src/main/java/org/apache/pinot/core/minion/rollup/MergeType.java => pinot-common/src/main/java/org/apache/pinot/common/utils/DateTimeUtils.java (51%)
rename pinot-common/src/main/java/org/apache/pinot/common/utils/{LikeToRegexpLikePatternConverterUtils.java => RegexpPatternConverterUtils.java} (50%)
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/utils/RoundRobinURIProvider.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/common/utils/ServiceStartableUtils.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/AliasApplier.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/CompileTimeFunctionsInvoker.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/NonAggregationGroupByToDistinctQueryRewriter.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/OrdinalsUpdater.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/PredicateComparisonRewriter.java
copy pinot-common/src/main/java/org/apache/pinot/{common/lineage/LineageEntryState.java => sql/parsers/rewriter/QueryRewriter.java} (74%)
create mode 100644 pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/QueryRewriterFactory.java
create mode 100644 pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/SelectionsRewriter.java
create mode 100644 pinot-common/src/test/java/org/apache/pinot/common/exception/QueryExceptionTest.java
create mode 100644 pinot-common/src/test/java/org/apache/pinot/common/minion/MinionTaskMetadataUtilsTest.java
delete mode 100644 pinot-common/src/test/java/org/apache/pinot/common/utils/LikeToRegexpLikePatternConverterUtilsTest.java
create mode 100644 pinot-common/src/test/java/org/apache/pinot/common/utils/RegexpPatternConverterUtilsTest.java
create mode 100644 pinot-common/src/test/java/org/apache/pinot/common/utils/RoundRobinURIProviderTest.java
create mode 100644 pinot-common/src/test/java/org/apache/pinot/common/utils/config/TableConfigTest.java
create mode 100644 pinot-common/src/test/java/org/apache/pinot/sql/parsers/rewriter/NonAggregationGroupByToDistinctQueryRewriterTest.java
create mode 100644 pinot-common/src/test/java/org/apache/pinot/sql/parsers/rewriter/QueryRewriterFactoryTest.java
create mode 100644 pinot-common/src/test/resources/testConfigs/jsonIndexConfig.json
create mode 100644 pinot-common/src/test/resources/testConfigs/multipleIndexConfig.json
create mode 100644 pinot-common/src/test/resources/testConfigs/noIndexConfig.json
create mode 100644 pinot-common/src/test/resources/testConfigs/rangeIndexConfig.json
create mode 100644 pinot-common/src/test/resources/testConfigs/textIndexConfig.json
create mode 100644 pinot-connectors/presto-pinot-driver/pom.xml
create mode 100644 pinot-connectors/presto-pinot-driver/src/main/java/org/apache/pinot/connector/presto/PinotScatterGatherQueryClient.java
create mode 100644 pinot-connectors/presto-pinot-driver/src/main/java/org/apache/pinot/connector/presto/grpc/PinotStreamingQueryClient.java
create mode 100644 pinot-connectors/presto-pinot-driver/src/main/java/org/apache/pinot/connector/presto/grpc/Utils.java
create mode 100644 pinot-controller/src/main/resources/favicon.ico
rename pinot-controller/src/test/java/org/apache/pinot/controller/{ControllerStarterTest.java => ControllerStarterStatelessTest.java} (98%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{ConsumingSegmentInfoReaderTest.java => ConsumingSegmentInfoReaderStatelessTest.java} (94%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/api/{PinotIngestionRestletResourceTest.java => PinotIngestionRestletResourceStatelessTest.java} (98%)
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDummyExtraRestletResourceTest.java
copy pinot-common/src/main/java/org/apache/pinot/common/utils/grpc/GrpcQueryClient.java => pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/extrapackage/PinotDummyExtraRestletResource.java (55%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/minion/{MinionInstancesCleanupTaskTest.java => MinionInstancesCleanupTaskStatelessTest.java} (80%)
rename pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/minion/{PinotTaskManagerTest.java => PinotTaskManagerStatelessTest.java} (51%)
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/utils/FakeHttpServer.java
create mode 100644 pinot-controller/src/test/resources/recommenderInput/RealtimeProvisioningInput_highIngestionRate.json
delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/minion/SegmentConverter.java
delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/minion/rollup/MergeRollupSegmentConverter.java
delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/minion/rollup/RollupRecordAggregator.java
delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/minion/segment/DefaultRecordPartitioner.java
delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/minion/segment/MapperRecordReader.java
delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/minion/segment/RecordAggregator.java
delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/minion/segment/ReducerRecordReader.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/InTransformFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/RegexpExtractTransformFunction.java
delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/plan/DictionaryBasedAggregationPlanNode.java
delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/plan/DictionaryBasedDistinctPlanNode.java
delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/plan/MetadataBasedAggregationPlanNode.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/LastDoubleValueWithTimeAggregationFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/LastFloatValueWithTimeAggregationFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/LastIntValueWithTimeAggregationFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/LastLongValueWithTimeAggregationFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/LastStringValueWithTimeAggregationFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/LastWithTimeAggregationFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/PercentileRawEstAggregationFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/PercentileRawEstMVAggregationFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/PercentileRawTDigestAggregationFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/PercentileRawTDigestMVAggregationFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/statement/StringPredicateFilterOptimizer.java
copy pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/SimpleBrokerSelector.java => pinot-core/src/main/java/org/apache/pinot/core/util/MemoizedClassAssociation.java (57%)
rename pinot-core/src/main/java/org/apache/pinot/core/util/{QueryOptions.java => QueryOptionsUtils.java} (63%)
rename {pinot-server/src/main/java/org/apache/pinot/server/api => pinot-core/src/main/java/org/apache/pinot/server}/access/AccessControl.java (82%)
rename {pinot-server/src/main/java/org/apache/pinot/server/api => pinot-core/src/main/java/org/apache/pinot/server}/access/AccessControlFactory.java (95%)
rename {pinot-server/src/main/java/org/apache/pinot/server/api => pinot-core/src/main/java/org/apache/pinot/server}/access/AllowAllAccessFactory.java (84%)
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/auth/BasicAuthTest.java
copy pinot-core/src/test/java/org/apache/pinot/core/data/manager/{BaseTableDataManagerTest.java => BaseTableDataManagerAcquireSegmentTest.java} (94%)
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManagerTest.java
delete mode 100644 pinot-core/src/test/java/org/apache/pinot/core/minion/MergeRollupSegmentConverterTest.java
delete mode 100644 pinot-core/src/test/java/org/apache/pinot/core/minion/SegmentConverterTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/InTransformFunctionTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/RegexpTransformFunctionTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/query/executor/QueryExecutorExceptionsTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/StringPredicateFilterOptimizerTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/query/optimizer/statement/TestHelper.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/queries/ExpectedQueryResult.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/queries/LastWithTimeQueriesTest.java
copy {pinot-integration-tests => pinot-integration-test-base}/pom.xml (57%)
rename {pinot-integration-tests => pinot-integration-test-base}/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTest.java (100%)
rename {pinot-integration-tests => pinot-integration-test-base}/src/test/java/org/apache/pinot/integration/tests/BasicAuthTestUtils.java (100%)
rename {pinot-integration-tests => pinot-integration-test-base}/src/test/java/org/apache/pinot/integration/tests/ClusterIntegrationTestUtils.java (99%)
rename {pinot-integration-tests => pinot-integration-test-base}/src/test/java/org/apache/pinot/integration/tests/ClusterTest.java (100%)
rename {pinot-integration-tests => pinot-integration-test-base}/src/test/java/org/apache/pinot/integration/tests/HybridClusterIntegrationTestCommandLineRunner.java (100%)
rename {pinot-integration-tests => pinot-integration-test-base}/src/test/java/org/apache/pinot/integration/tests/QueryGenerator.java (100%)
create mode 100644 pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/access/CertBasedTlsChannelAccessControlFactory.java
create mode 100644 pinot-integration-tests/src/test/resources/tlstest.p12
create mode 100644 pinot-minion/src/main/java/org/apache/pinot/minion/MinionConf.java
create mode 100644 pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkFST.java
create mode 100644 pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkFixedByteValueReaderWriter.java
create mode 100644 pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkRangeIndex.java
create mode 100644 pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkRoaringBitmapCreation.java
create mode 100644 pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkScanDocIdIterators.java
copy pinot-common/src/main/java/org/apache/pinot/common/tier/PinotServerTierStorage.java => pinot-perf/src/main/java/org/apache/pinot/perf/BytesCounter.java (59%)
create mode 100644 pinot-perf/src/main/java/org/apache/pinot/perf/Distribution.java
create mode 100644 pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-hadoop/src/test/java/org/apache/pinot/hadoop/job/preprocess/HadoopDataPreprocessingHelperTest.java
create mode 100644 pinot-plugins/pinot-input-format/pinot-parquet/src/test/java/org/apache/pinot/plugin/inputformat/parquet/ParquetNativeRecordReaderTest.java
copy pinot-segment-local/src/main/java/org/apache/pinot/segment/local/{io/compression/SnappyCompressor.java => customobject/DoubleLongPair.java} (56%)
copy pinot-segment-local/src/main/java/org/apache/pinot/segment/local/{io/compression/SnappyCompressor.java => customobject/FloatLongPair.java} (57%)
copy pinot-segment-local/src/main/java/org/apache/pinot/segment/local/{io/compression/SnappyCompressor.java => customobject/IntLongPair.java} (57%)
copy pinot-segment-local/src/main/java/org/apache/pinot/segment/local/{io/compression/SnappyCompressor.java => customobject/LongLongPair.java} (57%)
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/SerializedQuantileDigest.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/SerializedTDigest.java
copy pinot-segment-local/src/main/java/org/apache/pinot/segment/local/{io/compression/PassThroughCompressor.java => customobject/StringLongPair.java} (51%)
copy pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/compression/ChunkCompressionType.java => pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/ValueLongPair.java (58%)
copy pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/compression/{LZ4Compressor.java => LZ4WithLengthCompressor.java} (58%)
copy pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/compression/{ZstandardDecompressor.java => LZ4WithLengthDecompressor.java} (60%)
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/writer/impl/VarByteChunkSVForwardIndexWriterV4.java
copy pinot-common/src/main/java/org/apache/pinot/common/lineage/LineageEntryState.java => pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/writer/impl/VarByteChunkWriter.java (80%)
rename pinot-segment-local/src/main/java/org/apache/pinot/segment/local/loader/{LocalSegmentDirectoryLoader.java => DefaultSegmentDirectoryLoader.java} (66%)
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/fwd/MultiValueFixedByteRawIndexCreator.java
copy pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/fwd/{SingleValueVarByteRawIndexCreator.java => MultiValueVarByteRawIndexCreator.java} (59%)
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/inv/BitSlicedRangeIndexCreator.java
copy pinot-segment-local/src/main/java/org/apache/pinot/segment/local/{io/compression/PassThroughCompressor.java => segment/creator/impl/inv/MmapFileWriter.java} (50%)
rename pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/{V3UpdateIndexException.java => IndexHandler.java} (74%)
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexHandlerFactory.java
rename pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/{LuceneFSTIndexHandler.java => FSTIndexHandler.java} (67%)
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/BitSlicedRangeIndexReader.java
rename pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/{RangeIndexReader.java => RangeIndexReaderImpl.java} (52%)
copy pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/{VarByteChunkSVForwardIndexReader.java => FixedByteChunkMVForwardIndexReader.java} (56%)
copy pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/{VarByteChunkSVForwardIndexReader.java => VarByteChunkMVForwardIndexReader.java} (66%)
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/VarByteChunkSVForwardIndexReaderV4.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/FPOrdering.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/ByteSequenceIterator.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/ConstantArcSizeFST.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/FST.java
rename pinot-core/src/main/java/org/apache/pinot/core/minion/segment/RecordPartitioner.java => pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/FSTFlags.java (50%)
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/FSTHeader.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/FSTTraversal.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/ImmutableFST.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/MatchResult.java
copy pinot-segment-local/src/main/java/org/apache/pinot/segment/local/{segment/creator/impl/inv/text/LuceneFSTIndexCreator.java => utils/nativefst/NativeFSTIndexCreator.java} (59%)
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/NativeFSTIndexReader.java
copy pinot-common/src/main/java/org/apache/pinot/common/lineage/LineageEntryState.java => pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/StateVisitor.java (78%)
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/automaton/Automaton.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/automaton/AutomatonMatcher.java
rename pinot-segment-local/src/main/java/org/apache/pinot/segment/local/{segment/index/loader/V3RemoveIndexException.java => utils/nativefst/automaton/AutomatonProvider.java} (67%)
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/automaton/BasicAutomata.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/automaton/BasicOperations.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/automaton/CharacterRunAutomaton.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/automaton/MinimizationOperations.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/automaton/README.md
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/automaton/RegExp.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/automaton/RunAutomaton.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/automaton/SpecialOperations.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/automaton/State.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/automaton/StatePair.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/automaton/StringUnionOperations.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/automaton/Transition.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/automaton/TransitionComparator.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/builder/FSTBuilder.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/builder/FSTInfo.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/builder/FSTSerializer.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/builder/FSTSerializerImpl.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/builder/FSTUtils.java
create mode 100644 pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/utils/RegexpMatcher.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/io/compression/TestCompression.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/creator/BitSlicedIndexCreatorTest.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/creator/MultiValueFixedByteRawIndexCreatorTest.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/creator/MultiValueVarByteRawIndexCreatorTest.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/creator/NativeFSTIndexCreatorTest.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/creator/VarByteChunkV4Test.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/creator/inv/BitmapInvertedIndexWriterTest.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/readerwriter/FixedByteValueReaderWriterTest.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/FPOrderingTest.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/nativefst/FSTBuilderTest.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/nativefst/FSTRegexpWithWeirdTest.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/nativefst/FSTSanityTest.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/nativefst/FSTTestUtils.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/nativefst/FSTTraversalTest.java
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/nativefst/ImmutableFSTDeserializedTest.java
copy pinot-common/src/main/java/org/apache/pinot/common/lineage/LineageEntryState.java => pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/nativefst/ImmutableFSTSerializerTest.java (71%)
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/nativefst/ImmutableFSTTest.java
copy pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/compression/ChunkCompressionType.java => pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/nativefst/MinMax.java (73%)
create mode 100644 pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/nativefst/SerializerTestBase.java
create mode 100644 pinot-segment-local/src/test/resources/data/abc-numbers.native.fst
create mode 100644 pinot-segment-local/src/test/resources/data/abc.native.fst
create mode 100644 pinot-segment-local/src/test/resources/data/en_tst.dict
rename pinot-segment-local/src/test/resources/data/{newColumnsSchema4.json => newColumnsSchemaWithFST.json} (100%)
copy pinot-segment-local/src/test/resources/data/{newColumnsWithTextSchema.json => newColumnsSchemaWithH3Json.json} (84%)
rename pinot-segment-local/src/test/resources/data/{newColumnsWithTextSchema.json => newColumnsSchemaWithText.json} (85%)
create mode 100644 pinot-segment-local/src/test/resources/data/serfst.txt
create mode 100644 pinot-segment-local/src/test/resources/data/words.txt
create mode 100644 pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/FetchContext.java
copy pinot-common/src/main/java/org/apache/pinot/common/lineage/LineageEntryState.java => pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/CombinedInvertedIndexCreator.java (81%)
create mode 100644 pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/RangeIndexReader.java
copy pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/{creator/name/FixedSegmentNameGenerator.java => loader/SegmentDirectoryLoaderContext.java} (50%)
create mode 100644 pinot-server/src/main/java/org/apache/pinot/server/starter/helix/OffsetBasedConsumptionStatusChecker.java
delete mode 100644 pinot-server/src/main/java/org/apache/pinot/server/starter/helix/SegmentFetcherAndLoader.java
create mode 100644 pinot-server/src/test/java/org/apache/pinot/server/starter/helix/OffsetBasedConsumptionStatusCheckerTest.java
copy pinot-common/src/main/java/org/apache/pinot/common/lineage/LineageEntryState.java => pinot-spi/src/main/java/org/apache/pinot/spi/config/table/FSTType.java (84%)
create mode 100644 pinot-spi/src/main/java/org/apache/pinot/spi/utils/Obfuscator.java
delete mode 100644 pinot-spi/src/main/java/org/apache/pinot/spi/utils/StringUtils.java
create mode 100644 pinot-spi/src/test/java/org/apache/pinot/spi/utils/ByteArrayTest.java
create mode 100644 pinot-spi/src/test/java/org/apache/pinot/spi/utils/ObfuscatorTest.java
copy pinot-tools/src/main/java/org/apache/pinot/tools/{RealtimeQuickStart.java => RealtimeQuickStartWithMinion.java} (50%)
copy pinot-tools/src/main/java/org/apache/pinot/tools/{RealtimeJsonIndexQuickStart.java => UpsertJsonQuickStart.java} (90%)
delete mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/EnumArrayOptionHandler.java
delete mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/segment/converter/SegmentMergeCommand.java
create mode 100644 pinot-tools/src/main/resources/examples/minions/stream/githubEvents/githubEvents_offline_table_config.json
create mode 100644 pinot-tools/src/main/resources/examples/minions/stream/githubEvents/githubEvents_realtime_table_config.json
create mode 100644 pinot-tools/src/main/resources/examples/minions/stream/githubEvents/githubEvents_schema.json
create mode 100644 pinot-tools/src/main/resources/examples/minions/stream/githubEvents/rawdata/2021-07-21-few-hours.json
create mode 100644 pinot-tools/src/main/resources/examples/stream/meetupRsvp/upsert_json_meetupRsvp_realtime_table_config.json
create mode 100644 pinot-tools/src/main/resources/examples/stream/meetupRsvp/upsert_json_meetupRsvp_schema.json
create mode 100644 pinot-tools/src/test/java/org/apache/pinot/tools/admin/command/TestQuickStartCommand.java
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org