You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by jl...@apache.org on 2021/01/22 22:24:07 UTC
[incubator-pinot] branch segment-creation-in-one-pass updated
(51aaa35 -> a96a8e0)
This is an automated email from the ASF dual-hosted git repository.
jlli pushed a change to branch segment-creation-in-one-pass
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
discard 51aaa35 Support data ingestion for offline segment in one pass
add 3e4c325 [INFRA] Cancel the previous builds (#6429)
add 3d24302 refactor AnomaliesResponses to avoid duplicate code (#6406)
add 19e3d51 [TE] endpoint - harleyjj/rca - add forecast, upper, lower, and yoXy to aggregate/chunk enpoint (#6432)
add 33830cb [TE] suppress the anomaly if current value is NaN (#6428)
add 68fbb9c [TE] frontend - harleyjj/rca - display metric funnel in metrics table when forecast selected (#6333)
add 99fe289 [TE] fix dimensional summary in emails (#6443)
add 72a7849 Support confluent schema registry (#6434)
add 832ece7 Bug Fix column metadata, read from the correct property config for hasFSTIndex (#6441)
add 8d3d4d4 Merge H3-index branch to master (#6409)
add e8c4636 Fixing the issue that raw table name extracts from segment metadata might contain _OFFLINE suffix (#6445)
add 950295a Fix gcs listFiles (#6426)
add d83e371 Adding ImportData sub command in pinot admin (#6396)
add 28882ba Adding pinot minion component into helm (#6430)
add 8085fb7 setting default Data Type while setting type in Add Schema dialog (#6452)
add ec29b82 additional scalar functions for array type (#6446)
add 5de1d4b Fixing pinot helm 0.2.3 package (#6459)
add b592c8c Adding CRON scheduler for Pinot tasks (#6451)
add 60c802c Fixing pinot hadoop fs isDirectory method (#6461)
add 174a77b [TE]frontend - Add support for feedback propagation for Entity Monitoring (#6447)
add 079f27a Cleanup dictionary and forward index loading in SegmentPreProcessor (#6455)
add a36d1c0 [TE] Feedback propagation (#6463)
add a56f330 Broker time segment pruner(#6189): (#6462)
add 0417e20 additional string scalar functions (#6458)
add 79c9460 Dimension table storage quota config and validation (#6465)
add 2e7cdcd Always store raw table name in segment metadata (#6457)
add 4a89534 Replace BrokerRequestOptimizer with QueryOptimizer to also optimize the PinotQuery (#6423)
add 0e8c0e2 [TE] clean up template as to not repeat code and reduce conditionals (#6448)
add f17be35 Adding cluster config to config number of concurrent tasks per instance for minion task: SegmentGenerationAndPushTaskGenerator (#6468)
add 8c8897e Simplify SegmentGenerationAndPushTask handling getting schema and table config (#6469)
add f7af798 Handle scheduler calls with proper response when it's disabled. (#6474)
add 7649f9f Remove the usage of deprecated range delimiter (#6475)
add 04e2bcc Update superset docker image repo (#6477)
add dde3c18 TLS-support for client-pinot and pinot-internode connections (#6418)
add a96a8e0 Support data ingestion for offline segment in one pass
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (51aaa35)
\
N -- N -- N refs/heads/segment-creation-in-one-pass (a96a8e0)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../workflows/pinot_tests-workflow-run.yml | 31 +-
docker/images/pinot-superset/Dockerfile | 2 +-
kubernetes/helm/index.yaml | 44 +-
kubernetes/helm/pinot-0.2.3.tgz | Bin 0 -> 24664 bytes
kubernetes/helm/pinot/Chart.yaml | 4 +-
kubernetes/helm/pinot/README.md | 21 +
kubernetes/helm/pinot/templates/_helpers.tpl | 23 +
.../helm/pinot/templates/minion/configmap.yaml | 26 +-
.../minion/service-headless.yaml} | 33 +-
.../{Chart.yaml => templates/minion/service.yaml} | 33 +-
.../helm/pinot/templates/minion/statefulset.yml | 108 +
kubernetes/helm/pinot/values.yaml | 50 +
.../broker/broker/BrokerAdminApiApplication.java | 23 +-
.../broker/broker/helix/HelixBrokerStarter.java | 36 +-
.../requesthandler/BaseBrokerRequestHandler.java | 122 +-
.../SingleConnectionBrokerRequestHandler.java | 5 +-
.../routing/segmentpruner/TimeSegmentPruner.java | 52 +-
.../request/PqlAndCalciteSqlCompatibilityTest.java | 161 +-
.../requesthandler/BrokerRequestOptionsTest.java | 44 +-
.../LiteralOnlyBrokerRequestTest.java | 4 +-
.../requesthandler/RangeMergeOptimizerTest.java | 20 +-
.../routing/segmentpruner/SegmentPrunerTest.java | 68 +-
pinot-common/pom.xml | 4 +
.../common/function/scalar/ArrayFunctions.java | 59 +
.../common/function/scalar/StringFunctions.java | 127 +-
.../common/metadata/segment/SegmentZKMetadata.java | 21 +-
.../apache/pinot/common/utils/CommonConstants.java | 25 +
.../common/utils/FileUploadDownloadClient.java | 40 +-
.../apache/pinot/parsers/utils/ParserUtils.java | 16 +-
.../parsers/pql2/ast/BetweenPredicateAstNode.java | 7 +-
.../pql2/ast/ComparisonPredicateAstNode.java | 17 +-
.../apache/pinot/pql/parsers/Pql2CompilerTest.java | 10 +-
.../spark/connector/PinotServerDataFetcher.scala | 2 +
pinot-controller/pom.xml | 4 +
.../apache/pinot/controller/ControllerConf.java | 47 +-
.../apache/pinot/controller/ControllerStarter.java | 29 +-
.../api/ControllerAdminApiApplication.java | 65 +-
.../api/resources/PinotQueryResource.java | 39 +-
.../api/resources/PinotTableRestletResource.java | 57 +
.../api/resources/PinotTaskRestletResource.java | 120 +
.../api/resources/ServerTableSizeReader.java | 2 +-
.../helix/ControllerRequestURLBuilder.java | 7 +
.../helix/core/PinotHelixResourceManager.java | 22 +-
.../helix/core/minion/ClusterInfoAccessor.java | 16 +
.../helix/core/minion/CronJobScheduleJob.java | 54 +
.../helix/core/minion/PinotTaskManager.java | 264 +-
.../core/minion/TableTaskSchedulerUpdater.java | 57 +
.../SegmentGenerationAndPushTaskGenerator.java | 76 +-
.../core/realtime/PinotRealtimeSegmentManager.java | 10 +-
.../controller/util/CompletionServiceHelper.java | 7 +-
.../util/ConsumingSegmentInfoReader.java | 2 +-
.../pinot/controller/util/ListenerConfigUtil.java | 93 -
.../util/ServerSegmentMetadataReader.java | 2 +-
.../Homepage/Operations/SchemaComponent.tsx | 42 +-
.../pinot/controller/ControllerTestUtils.java | 15 +-
.../api/ConsumingSegmentInfoReaderTest.java | 2 +-
.../controller/api/PinotSegmentsMetadataTest.java | 2 +-
.../api/PinotTableRestletResourceTest.java | 93 +
.../controller/api/ServerTableSizeReaderTest.java | 2 +-
.../pinot/controller/api/TableSizeReaderTest.java | 2 +-
.../helix/core/PinotHelixResourceManagerTest.java | 4 +-
.../helix/core/minion/PinotTaskManagerTest.java | 145 +
.../SegmentGenerationAndPushTaskGeneratorTest.java | 80 +
.../controller/util/ListenerConfigUtilTest.java | 44 +-
pinot-core/pom.xml | 17 +-
.../org/apache/pinot/core/common/DataSource.java | 7 +
.../apache/pinot/core/common/MinionConstants.java | 2 +
.../realtime/LLRealtimeSegmentDataManager.java | 3 +-
.../transform/function/ScalarFunctions.java | 27 +
.../transform/function/StDistanceFunction.java | 40 +-
.../generator/SegmentGeneratorConfig.java | 34 +-
.../indexsegment/mutable/MutableSegmentImpl.java | 36 +-
.../core/operator/filter/FilterOperatorUtils.java | 15 +-
.../operator/filter/H3IndexFilterOperator.java | 246 +
.../org/apache/pinot/core/plan/FilterPlanNode.java | 40 +
.../optimizer/filter/MergeEqInFilterOptimizer.java | 16 +-
.../filter/MergeRangeFilterOptimizer.java | 5 +-
.../request/context/predicate/RangePredicate.java | 18 +-
.../BrokerRequestToQueryContextConverter.java | 268 +-
.../core/realtime/impl/RealtimeSegmentConfig.java | 31 +-
.../realtime/impl/geospatial/MutableH3Index.java | 78 +
.../core/requesthandler/RangeMergeOptimizer.java | 3 +-
.../GeoSpatialIndexCreator.java} | 44 +-
.../creator/impl/SegmentColumnarIndexCreator.java | 47 +-
.../core/segment/creator/impl/V1Constants.java | 3 +-
.../impl/inv/geospatial/BaseH3IndexCreator.java | 168 +
.../creator/impl/inv/geospatial/H3IndexConfig.java | 50 +
.../impl/inv/geospatial/H3IndexResolution.java | 71 +
.../impl/inv/geospatial/OffHeapH3IndexCreator.java | 218 +
.../impl/inv/geospatial/OnHeapH3IndexCreator.java | 49 +
.../segment/index/column/ColumnIndexContainer.java | 6 +
.../index/column/IntermediateIndexContainer.java | 2 +-
.../index/column/PhysicalColumnIndexContainer.java | 20 +
.../segment/index/datasource/BaseDataSource.java | 13 +-
.../index/datasource/ImmutableDataSource.java | 2 +-
.../index/datasource/MutableDataSource.java | 6 +-
.../segment/index/loader/IndexLoadingConfig.java | 61 +-
.../core/segment/index/loader/LoaderUtils.java | 47 +
.../segment/index/loader/SegmentPreProcessor.java | 8 +
.../index/loader/invertedindex/H3IndexHandler.java | 161 +
.../loader/invertedindex/InvertedIndexHandler.java | 19 +-
.../loader/invertedindex/JsonIndexHandler.java | 24 +-
.../invertedindex/LuceneFSTIndexHandler.java | 23 +-
.../loader/invertedindex/RangeIndexHandler.java | 30 +-
.../loader/invertedindex/TextIndexHandler.java | 47 +-
.../segment/index/metadata/ColumnMetadata.java | 2 +-
.../segment/index/metadata/SegmentMetadata.java | 4 +
.../index/metadata/SegmentMetadataImpl.java | 49 +-
.../index/readers/BitmapInvertedIndexReader.java | 87 +-
.../readers/H3IndexReader.java} | 44 +-
.../readers/geospatial/ImmutableH3IndexReader.java | 84 +
.../pinot/core/segment/store/ColumnIndexType.java | 3 +-
.../virtualcolumn/VirtualColumnIndexContainer.java | 6 +
.../core/startree/v2/store/StarTreeDataSource.java | 5 +-
.../pinot/core/transport}/ListenerConfig.java | 36 +-
.../apache/pinot/core/transport/QueryRouter.java | 34 +-
.../apache/pinot/core/transport/QueryServer.java | 51 +
.../pinot/core/transport/ServerChannels.java | 44 +
.../pinot/core/transport/ServerInstance.java | 21 +
.../core/transport/ServerRoutingInstance.java | 30 +-
.../org/apache/pinot/core/transport/TlsConfig.java | 77 +
.../ColumnIndexType.java => util/H3Utils.java} | 34 +-
.../apache/pinot/core/util/ListenerConfigUtil.java | 207 +
.../apache/pinot/core/util/TableConfigUtils.java | 4 +-
.../java/org/apache/pinot/core/util/TlsUtils.java | 269 +
.../core/data/function/InbuiltFunctionsTest.java | 34 +
.../transform/StDistanceFunctionTest.java | 8 +-
.../core/indexsegment/IntermediateSegmentTest.java | 15 +-
.../NoDictionaryRangePredicateEvaluatorTest.java | 104 +-
.../ScalarTransformFunctionWrapperTest.java | 297 +-
.../core/query/optimizer/QueryOptimizerTest.java | 32 +-
.../inv/geospatial/H3IndexResolutionTest.java} | 37 +-
.../core/segment/index/ColumnMetadataTest.java | 1 +
.../pinot/core/segment/index/H3IndexTest.java | 110 +
.../org/apache/pinot/queries/BaseQueriesTest.java | 2 +-
.../apache/pinot/queries/DistinctQueriesTest.java | 2 +-
.../apache/pinot/queries/H3IndexQueriesTest.java | 233 +
.../tests/ClusterIntegrationTestUtils.java | 55 +-
.../org/apache/pinot/minion/MinionStarter.java | 14 +-
.../SegmentGenerationAndPushTaskExecutor.java | 24 +-
.../batch/common/SegmentGenerationTaskRunner.java | 14 +-
.../batch/hadoop/HadoopSegmentCreationMapper.java | 3 +-
.../spark/SparkSegmentGenerationJobRunner.java | 3 +-
.../standalone/SegmentGenerationJobRunner.java | 2 +-
.../apache/pinot/hadoop/io/PinotOutputFormat.java | 3 +-
.../apache/pinot/plugin/filesystem/GcsPinotFS.java | 23 +-
.../pinot/plugin/filesystem/HadoopPinotFS.java | 13 +-
...aConfluentSchemaRegistryAvroMessageDecoder.java | 2 +-
pinot-server/pom.xml | 4 -
.../org/apache/pinot/server/conf/ServerConf.java | 20 +-
.../pinot/server/starter/ServerInstance.java | 48 +-
.../server/starter/helix/AdminApiApplication.java | 25 +-
.../server/starter/helix/HelixServerStarter.java | 40 +-
.../apache/pinot/server/api/AccessControlTest.java | 11 +-
.../apache/pinot/server/api/BaseResourceTest.java | 12 +-
.../pinot/server/api/TablesResourceTest.java | 14 +-
.../apache/pinot/spi/config/table/FieldConfig.java | 2 +-
.../pinot/spi/config/table/IndexingConfig.java | 1 +
.../spi/ingestion/batch/BatchConfigProperties.java | 4 +-
.../batch/spec/SegmentGenerationJobSpec.java | 13 +
.../batch/spec/SegmentGenerationTaskSpec.java | 7 +-
.../pinot/spi/ingestion/batch/spec/TlsSpec.java | 45 +-
pinot-tools/pom.xml | 12 +-
.../org/apache/pinot/tools/BootstrapTableTool.java | 32 +-
.../org/apache/pinot/tools/GenericQuickstart.java | 149 +
.../pinot/tools/admin/PinotAdministrator.java | 2 +
.../tools/admin/command/AddSchemaCommand.java | 18 +-
.../pinot/tools/admin/command/AddTableCommand.java | 17 +-
.../tools/admin/command/AddTenantCommand.java | 13 +-
.../tools/admin/command/BootstrapTableCommand.java | 6 +-
.../tools/admin/command/ChangeTableState.java | 18 +-
.../tools/admin/command/ImportDataCommand.java | 390 ++
.../command/LaunchDataIngestionJobCommand.java | 9 +
.../admin/command/OperateClusterConfigCommand.java | 15 +-
.../tools/admin/command/PostQueryCommand.java | 15 +-
.../tools/admin/command/StartMinionCommand.java | 8 +-
.../tools/admin/command/UploadSegmentCommand.java | 15 +-
.../tools/scan/query/RangePredicateFilter.java | 36 +-
.../converter/DictionaryToRawIndexConverter.java | 4 +-
.../main/resources/conf/pinot-minion-log4j2.xml | 48 +
.../main/resources/conf/pinot-service-log4j2.xml | 8 +
.../batch/starbucksStores/ingestionJobSpec.yaml | 139 +
.../batch/starbucksStores/rawdata/data.csv | 6444 ++++++++++++++++++++
.../starbucksStores_offline_table_config.json | 34 +
.../starbucksStores/starbucksStores_schema.json | 29 +
.../stream/meetupRsvp/meetupRsvp_schema.json | 2 +-
.../upsert_meetupRsvp_realtime_table_config.json | 10 +
.../meetupRsvp/upsert_meetupRsvp_schema.json | 2 +-
pom.xml | 12 +
.../dashboard/resources/AnomalyResource.java | 44 +-
.../resources/v2/RootCauseMetricResource.java | 61 +-
.../dashboard/resources/v2/RootCauseResource.java | 11 +-
.../bao/TestMergedAnomalyResultManager.java | 19 +
.../app/pods/components/alert-details/template.hbs | 387 +-
.../pods/components/anomaly-summary/component.js | 14 +-
.../components/composite-anomalies/component.js | 14 +-
.../entity-metrics-anomalies/component.js | 3 +-
.../entity-metrics-anomalies/template.hbs | 2 +-
.../group-constituents-anomalies/component.js | 3 +-
.../group-constituents-anomalies/template.hbs | 2 +-
.../parent-anomalies/component.js | 3 +-
.../parent-anomalies/template.hbs | 2 +-
.../pods/components/rootcause-anomaly/component.js | 207 +-
.../pods/components/rootcause-metrics/component.js | 220 +-
.../resolution/component.js | 60 +-
.../resolution/template.hbs | 65 +-
.../pods/partials/rootcause/metrics/template.hbs | 1 +
.../app/pods/rootcause/controller.js | 687 ++-
.../app/pods/services/api/anomalies/service.js | 76 +-
.../services/rootcause-aggregates-cache/service.js | 74 +-
.../app/shared/metricFunnelColumns.js | 56 +
thirdeye/thirdeye-frontend/app/styles/app.scss | 2 +-
.../pods/custom/composite-anomalies-table.scss | 53 +
.../styles/pods/custom/parent-anomalies-table.scss | 24 -
.../app/styles/shared/_styles.scss | 2 -
.../app/utils/anomalies-tree-parser.js | 58 +-
thirdeye/thirdeye-frontend/app/utils/anomaly.js | 131 +-
.../app/utils/composite-anomalies.js | 2 +-
.../entity-metrics-anomalies/component-test.js | 4 +-
.../group-constituents-anomalies/component-test.js | 4 +-
.../parent-anomalies/component-test.js | 4 +-
.../tests/unit/utils/anomalies-tree-parser-test.js | 72 +-
.../resources/v2/BaselineParsingUtils.java | 15 +-
.../bao/jdbc/MergedAnomalyResultManagerImpl.java | 25 +-
.../detection/DetectionPipelineTaskRunner.java | 4 +-
.../rootcause/timeseries/BaselineAggregate.java | 21 +
.../datalayer/bao/MergedAnomalyResultManager.java | 2 +
227 files changed, 14512 insertions(+), 2424 deletions(-)
copy docker/images/pinot-superset/Dockerfile => .github/workflows/pinot_tests-workflow-run.yml (60%)
create mode 100644 kubernetes/helm/pinot-0.2.3.tgz
copy docker/images/pinot-superset/Dockerfile => kubernetes/helm/pinot/templates/minion/configmap.yaml (67%)
copy kubernetes/helm/pinot/{Chart.yaml => templates/minion/service-headless.yaml} (59%)
copy kubernetes/helm/pinot/{Chart.yaml => templates/minion/service.yaml} (59%)
create mode 100644 kubernetes/helm/pinot/templates/minion/statefulset.yml
create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/CronJobScheduleJob.java
create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/TableTaskSchedulerUpdater.java
delete mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/util/ListenerConfigUtil.java
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManagerTest.java
create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/minion/generator/SegmentGenerationAndPushTaskGeneratorTest.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/filter/H3IndexFilterOperator.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/geospatial/MutableH3Index.java
copy pinot-core/src/main/java/org/apache/pinot/core/segment/{store/ColumnIndexType.java => creator/GeoSpatialIndexCreator.java} (51%)
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/geospatial/BaseH3IndexCreator.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/geospatial/H3IndexConfig.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/geospatial/H3IndexResolution.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/geospatial/OffHeapH3IndexCreator.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/geospatial/OnHeapH3IndexCreator.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/invertedindex/H3IndexHandler.java
copy pinot-core/src/main/java/org/apache/pinot/core/segment/{store/ColumnIndexType.java => index/readers/H3IndexReader.java} (51%)
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/geospatial/ImmutableH3IndexReader.java
rename {pinot-controller/src/main/java/org/apache/pinot/controller/api/listeners => pinot-core/src/main/java/org/apache/pinot/core/transport}/ListenerConfig.java (67%)
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/transport/TlsConfig.java
copy pinot-core/src/main/java/org/apache/pinot/core/{segment/store/ColumnIndexType.java => util/H3Utils.java} (52%)
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/util/ListenerConfigUtil.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/util/TlsUtils.java
copy pinot-core/src/{main/java/org/apache/pinot/core/segment/store/ColumnIndexType.java => test/java/org/apache/pinot/core/segment/creator/impl/inv/geospatial/H3IndexResolutionTest.java} (51%)
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/segment/index/H3IndexTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/queries/H3IndexQueriesTest.java
rename pinot-controller/src/main/java/org/apache/pinot/controller/api/listeners/TlsConfiguration.java => pinot-spi/src/main/java/org/apache/pinot/spi/ingestion/batch/spec/TlsSpec.java (64%)
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/GenericQuickstart.java
create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/ImportDataCommand.java
create mode 100644 pinot-tools/src/main/resources/conf/pinot-minion-log4j2.xml
create mode 100644 pinot-tools/src/main/resources/examples/batch/starbucksStores/ingestionJobSpec.yaml
create mode 100644 pinot-tools/src/main/resources/examples/batch/starbucksStores/rawdata/data.csv
create mode 100644 pinot-tools/src/main/resources/examples/batch/starbucksStores/starbucksStores_offline_table_config.json
create mode 100644 pinot-tools/src/main/resources/examples/batch/starbucksStores/starbucksStores_schema.json
create mode 100644 thirdeye/thirdeye-frontend/app/shared/metricFunnelColumns.js
create mode 100644 thirdeye/thirdeye-frontend/app/styles/pods/custom/composite-anomalies-table.scss
delete mode 100644 thirdeye/thirdeye-frontend/app/styles/pods/custom/parent-anomalies-table.scss
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org