You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by je...@apache.org on 2019/11/06 17:42:37 UTC
[incubator-pinot] branch realtimeref updated (0a83fab -> c1d9cec)
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a change to branch realtimeref
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
from 0a83fab Adding test and addressing comments
add 6ee6a41 [TE] Add unit test for DetectionEmailAlerter - fix casting bug (#4717)
add 3579aae Remove the segment missing error message when acquiring segments (#4714)
add a78ac5c [TE][RCA] Fix the broken RCA algorithm tab (#4720)
add dc17239 Package parquet and orc record reader into shaded jar (#4716)
add f245b5e [TE] support row count metric for Pinot (#4718)
add e9e6a33 [TE] frontend - harleyjj/alert-details - handle concurrency errors in alert details component (#4721)
add a6f8f1e [Resource Assignment] Plug in resource assignment to LLC REALTIME table (#4695)
add 860cb92 Enable full stack trace when there is an exception during tests (#4727)
add 6bc6e55 Optimize MutableOffHeapByteArrayStore by directly calling the PinotDataBuffer API (#4362)
add c1a0944 [TE] Fix ignored timeout by checking Thread.interrupted (#4690)
add 15bde95 [TE] web - harleyjj/rca-session - save dimension-algorithm table settings with session and load them (#4712)
add 2f3cd43 [TE] put default settings for subscription YAML (#4723)
add f015c68 Add sql query command (#4726)
add 1b8cdd8 Fix comparison to pick the right minimum value for hash-set (#4588)
add 3a77194 Enhance the exception handling in TableRebalancer (#4734)
add ac5ada2 Catch exception when checking lead controller resource is enabled (#4736)
add b00b66c [TE] correct time format before send query to Vertica (#4548)
add 2f72432 Fix state change when consumption is stopped (#4739)
add 9c84de7 Changing error log to warn in SegmentPreprocessingMapper (#4735)
add 7d66875 Implement OffHeapMutableBytesStore for real-time var-length bytes store (#4385)
add 5af88ef [TE] frontend - harleyjj/alert-details - fix more preview comparison bugs (#4738)
add f3b6617 Make Kafka offset out of range as an transient exception. (#4618)
add 494ff8a Optimizations for IndexedTable resize (#4728)
add a8c6f8d Adding bootstrap mode for Pinot-hadoop job to output segments into relative directories. (#4742)
add 3c3c716 Fix a wrong option name for AddSchema in the information log (#4746)
add 1f5bf57 Presence vector (#4585)
add 74aea85 [SegmentGeneratorConfig Cleanup] Replace checkTimeColumnValidityDuringGeneration with skipTimeValueCheck (#4745)
add 04f2b7c AvroSchemaToPinotSchema should return immediately if outputDir doesn't exist (#4751)
add 716a3b2 Clean up duplicate/unused metrics in controller (#4750)
add 4f849ea [New Transformation function] DateTrunc: Identical semantics to presto's date_trunc (#4740)
add 4fc317b First commit, Fixed a typo in ReadMe (#4755)
add 5b97d06 fix issue #4115, change replaceAll() to replace (#4116)
add b30a692 [TE] frontend - harleyjj/alert-details - comparison revisions and facelift based on feedback (#4749)
add 8040b09 [TE][notification] Ability to reopen and reuse existing Jira tickets for alerts (#4748)
add 367f7e4 [TE] Merge anomaly properties (#4744)
add b48c412 [TE] frontend - harleyjj/alert-details - remove duplicate key from component (#4760)
add 07bd379 Bug fixing: Json decoder extracts wrong time field value if schema contains outgoingTimeFieldSpec (#4758)
add 5439e77 Helm chart templates for Pinot deployment and quickstart example (#4725)
add 6d3c7de Introduce a mode in the server to disable queries (#4767)
add 22aa781 [TE] deactivate metrics instead of deleting (#4768)
add 2ce7502 Fix the bug of releasing the segment when there are still threads working on it (#4764)
add 5d30495 [TE] endpoints - harleyjj/alert-activation - make endpoint for toggling alert activation (#4763)
add f0c95dd [TE] Clean up ADContentFormatterContext (#4765)
add 4afabca Show all items in the side menu (#4770)
add a3977b9 [TE] frontend - harleyjj/self-serve-alert-yaml-details - add toggle button for activating/deactivating alert (#4769)
add fa07579 Simplify the constant calculation in CombinePlanNode (#4772)
add 4f3d650 [TE] Configure default alerter before constructing properties (#4771)
add 19ccfe0 [TE] Extending Jira Merge capabilities for supporting dimensional alerter (#4766)
add faa068d [TE] add event trigger listener for scheduling (#4724)
add 1c66672 Adding superset demo example (#4779)
add 6026746 [TE][notification] Minor tweeking of jira alerter (#4776)
add b2b1852 Adding example for azure (#4781)
add 0614310 Prepare License and Notice files for 0.2.0 release (#4782)
add 23303ec [TE] minor naming change on ThirdEyeAnomalyConfiguration (#4785)
add 363d47f [TE] fix the bug that the moving window might span beyound the monitoring end time (#4787)
add 5193287 Enhance CombineOperator (#4773)
add 052b73b Enhance Selection Comparator 2 (#4788)
add 4445fdc Fix the bug of uneven task distribution for threads (#4793)
add c1d9cec Merge branch 'master' into realtimeref
No new revisions were added by this update.
Summary of changes:
.gitignore | 3 +
LICENSE-binary | 14 +
NOTICE-binary | 83 +-
README.md | 2 +-
.../pinot-superset/.dockerignore} | 13 +-
.../pinot-superset/.python-version} | 11 +-
docker/images/pinot-superset/Dockerfile | 112 +
docker/images/pinot-superset/Makefile | 64 +
docker/images/pinot-superset/README.md | 70 +
.../pinot-superset/requirements-db.txt} | 27 +-
docker/{ => images/pinot}/Dockerfile | 0
docker/{ => images/pinot}/README.md | 0
docker/{ => images/pinot}/docker-build-and-push.sh | 0
docker/{ => images/pinot}/docker-build.sh | 0
docker/{ => images/pinot}/docker-compose.yml | 0
docker/{ => images/pinot}/docker-push.sh | 0
docs/admin_guide.rst | 2 +
docs/dev_guide.rst | 2 +
docs/index.rst | 31 +-
docs/introduction.rst | 2 +
docs/pql_examples.rst | 17 +
docs/user_guide.rst | 2 +
.../Chart.yaml} | 25 +-
kubernetes/examples/helm/README.md | 333 +
.../helm-rbac.yaml} | 25 +-
.../open-superset-ui.sh} | 8 +-
.../examples/helm/pinot-realtime-quickstart.yml | 445 +
.../{gke/skaffold => helm}/query-pinot-data.sh | 0
.../examples/helm/requirements.lock | 17 +-
.../requirements.yaml} | 14 +-
.../{gke/skaffold/setup.sh => helm/setup_gke.sh} | 13 +-
kubernetes/examples/helm/superset.yaml | 2322 +++++
kubernetes/examples/helm/templates/_helpers.tpl | 143 +
.../templates/broker/configmap.yaml} | 16 +-
.../templates/broker/service-external.yaml} | 29 +-
.../templates/broker/service-headless.yaml} | 26 +-
.../templates/broker/service.yaml} | 26 +-
.../examples/helm/templates/broker/statefulset.yml | 91 +
.../templates/controller/configmap.yaml} | 20 +-
.../templates/controller/service-external.yaml} | 29 +-
.../templates/controller/service-headless.yaml} | 26 +-
.../templates/controller/service.yaml} | 26 +-
.../helm/templates/controller/statefulset.yaml | 100 +
.../templates/server/configmap.yaml} | 18 +-
.../templates/server/service-headless.yaml} | 26 +-
.../templates/server/service.yaml} | 26 +-
.../examples/helm/templates/server/statefulset.yml | 106 +
kubernetes/examples/helm/values.yaml | 212 +
.../{gke/skaffold => skaffold/gke}/README.md | 2 +-
.../{gke/skaffold => skaffold/gke}/cleanup.sh | 0
.../gke}/gke-storageclass-kafka-pd.yml | 0
.../gke}/gke-storageclass-pinot-controller-pd.yml | 0
.../gke}/gke-storageclass-pinot-server-pd.yml | 0
.../gke}/gke-storageclass-zk-pd.yml | 0
.../{gke/skaffold => skaffold/gke}/kafka.yml | 0
.../skaffold => skaffold/gke}/pinot-broker.yml | 0
.../skaffold => skaffold/gke}/pinot-controller.yml | 0
.../gke/pinot-realtime-quickstart.yml} | 6 +-
.../skaffold => skaffold/gke}/pinot-server.yml | 0
.../skaffold => skaffold/gke}/query-pinot-data.sh | 0
.../{gke/skaffold => skaffold/gke}/setup.sh | 0
.../{gke/skaffold => skaffold/gke}/skaffold.yaml | 0
.../{gke/skaffold => skaffold/gke}/zookeeper.yml | 0
.../routing/HelixExternalViewBasedRouting.java | 13 +-
.../PartitionAwareOfflineRoutingTableBuilder.java | 23 +-
.../builder/RoutingTableInstancePruner.java | 6 +
.../LargeClusterRoutingTableBuilderTest.java | 8 +-
.../LowLevelConsumerRoutingTableBuilderTest.java | 32 +-
...rtitionAwareOfflineRoutingTableBuilderTest.java | 44 +-
.../common/assignment/InstancePartitionsUtils.java | 36 +-
.../apache/pinot/common/config/IndexingConfig.java | 28 +-
.../SegmentsValidationAndRetentionConfig.java | 23 +-
.../apache/pinot/common/config/TableConfig.java | 10 -
.../apache/pinot/common/config/TagNameUtils.java | 15 -
.../pinot/common/metrics/ControllerMeter.java | 8 +-
.../pinot/common/metrics/ValidationMetrics.java | 37 -
.../common/partition/PartitionAssignment.java | 109 -
...roupBasedStreamPartitionAssignmentStrategy.java | 83 -
.../partition/ReplicaGroupPartitionAssignment.java | 124 -
.../ReplicaGroupPartitionAssignmentGenerator.java | 128 -
.../StreamPartitionAssignmentGenerator.java | 139 -
.../StreamPartitionAssignmentStrategy.java | 40 -
.../StreamPartitionAssignmentStrategyFactory.java | 45 -
.../UniformStreamPartitionAssignmentStrategy.java | 66 -
.../pinot/common/segment/SegmentMetadata.java | 2 +
.../apache/pinot/common/utils/CommonConstants.java | 1 +
.../org/apache/pinot/common/utils/HashUtil.java | 2 +-
.../apache/pinot/common/utils/time/TimeUtils.java | 35 +-
.../apache/pinot/pql/parsers/Pql2AstListener.java | 4 +-
.../pinot/common/config/TableConfigTest.java | 23 -
.../common/partition/IdealStateBuilderUtil.java | 179 -
...icaGroupBasedStreamPartitionAssignmentTest.java | 136 -
.../StreamPartitionAssignmentGeneratorTest.java | 364 -
.../UniformStreamPartitionAssignmentTest.java | 93 -
.../impl/kafka/KafkaConnectionHandler.java | 4 +-
.../pinot-connector-kafka-base/pom.xml | 9 +
.../impl/kafka/KafkaJSONMessageDecoder.java | 20 +-
.../impl/kafka/KafkaJSONMessageDecoderTest.java | 102 +
.../src/test/resources/data/test_sample_data.json | 10001 +++++++++++++++++++
.../test_sample_data_schema_no_time_field.json | 51 +
...sample_data_schema_with_outgoing_time_spec.json | 63 +
...ple_data_schema_without_outgoing_time_spec.json | 58 +
.../apache/pinot/controller/ControllerStarter.java | 2 +-
.../pinot/controller/LeadControllerManager.java | 23 +-
.../controller/api/upload/SegmentValidator.java | 49 +-
.../helix/core/PinotHelixResourceManager.java | 79 +-
.../helix/core/PinotTableIdealStateBuilder.java | 7 +-
.../instance/InstanceTagPoolSelector.java | 4 +-
.../assignment/segment/SegmentAssignmentUtils.java | 3 +-
.../realtime/PinotLLCRealtimeSegmentManager.java | 1404 +--
.../core/realtime/SegmentCompletionManager.java | 68 +-
.../helix/core/realtime/TableConfigCache.java | 12 +-
.../segment/DefaultFlushThresholdUpdater.java | 46 +-
.../segment/FlushThresholdUpdateManager.java | 32 +-
.../realtime/segment/FlushThresholdUpdater.java | 22 +-
.../SegmentSizeBasedFlushThresholdUpdater.java | 45 +-
.../helix/core/rebalance/TableRebalancer.java | 62 +-
.../controller/util/SegmentIntervalUtils.java | 12 +-
.../validation/OfflineSegmentIntervalChecker.java | 3 +-
.../RealtimeSegmentValidationManager.java | 6 +-
.../PinotLLCRealtimeSegmentManagerTest.java | 2114 ++--
.../helix/core/realtime/SegmentCompletionTest.java | 32 +-
.../segment/FlushThresholdUpdaterTest.java | 939 +-
.../org/apache/pinot/core/common/DataSource.java | 7 +
.../org/apache/pinot/core/data/GenericRow.java | 7 +
.../core/data/manager/BaseTableDataManager.java | 36 +-
.../pinot/core/data/manager/TableDataManager.java | 25 +-
.../manager/offline/OfflineTableDataManager.java | 3 +-
.../realtime/HLRealtimeSegmentDataManager.java | 5 +-
.../realtime/LLRealtimeSegmentDataManager.java | 11 +-
.../manager/realtime/RealtimeTableDataManager.java | 8 +-
.../apache/pinot/core/data/order/OrderByUtils.java | 328 -
.../core/data/table/ConcurrentIndexedTable.java | 157 +-
.../apache/pinot/core/data/table/IndexedTable.java | 73 +-
.../pinot/core/data/table/IndexedTableResizer.java | 303 +
.../pinot/core/data/table/SimpleIndexedTable.java | 149 +-
.../org/apache/pinot/core/data/table/Table.java | 18 +-
.../generator/SegmentGeneratorConfig.java | 24 +-
.../indexsegment/mutable/MutableSegmentImpl.java | 43 +-
.../writer/impl/MutableOffHeapByteArrayStore.java | 81 +-
.../core/io/writer/impl/OffHeapStringStore.java | 222 -
.../apache/pinot/core/minion/SegmentConverter.java | 29 +-
.../apache/pinot/core/operator/BaseOperator.java | 3 +
.../core/operator/CombineGroupByOperator.java | 33 +-
.../operator/CombineGroupByOrderByOperator.java | 45 +-
.../pinot/core/operator/CombineOperator.java | 155 +-
.../operator/blocks/IntermediateResultsBlock.java | 7 +-
.../query/AggregationGroupByOrderByOperator.java | 4 +
.../operator/query/EmptySelectionOperator.java | 2 +-
.../core/operator/query/SelectionOnlyOperator.java | 2 +-
.../operator/query/SelectionOrderByOperator.java | 79 +-
.../function/DateTruncTransformFunction.java | 256 +
.../operator/transform/function/TimeZoneKey.java | 266 +
.../function/TransformFunctionFactory.java | 1 +
.../apache/pinot/core/plan/CombinePlanNode.java | 75 +-
.../apache/pinot/core/plan/SelectionPlanNode.java | 2 +-
.../apache/pinot/core/plan/TransformPlanNode.java | 33 +-
.../groupby/AggregationGroupByTrimmingService.java | 3 +-
.../query/executor/ServerQueryExecutorV1Impl.java | 31 +-
.../core/query/reduce/BrokerReduceService.java | 12 +-
.../query/selection/SelectionOperatorService.java | 40 +-
.../query/selection/SelectionOperatorUtils.java | 108 +-
.../converter/RealtimeSegmentConverter.java | 9 +-
.../core/realtime/impl/RealtimeSegmentConfig.java | 16 +-
.../impl/dictionary/OffHeapMutableBytesStore.java | 259 +
.../RealtimeNullValueVectorReaderWriter.java} | 33 +-
.../ConsumingSegmentAssignmentStrategy.java | 61 -
.../segment/RealtimeSegmentAssignmentStrategy.java | 41 -
.../stream/PartitionLevelStreamConfig.java | 66 +-
.../pinot/core/realtime/stream/StreamConfig.java | 155 +-
.../creator/impl/SegmentColumnarIndexCreator.java | 186 +-
.../core/segment/creator/impl/V1Constants.java | 1 +
.../impl/nullvalue/NullValueVectorCreator.java | 65 +
.../core/segment/index/SegmentMetadataImpl.java | 9 +-
.../segment/index/column/ColumnIndexContainer.java | 11 +
.../index/column/PhysicalColumnIndexContainer.java | 15 +
.../converter/SegmentV1V2ToV3FormatConverter.java | 10 +-
.../index/data/source/ColumnDataSource.java | 19 +-
.../loader/bloomfilter/BloomFilterHandler.java | 2 +-
.../index/readers/NullValueVectorReader.java | 28 +-
.../readers/NullValueVectorReaderImpl.java} | 32 +-
.../pinot/core/segment/memory/PinotDataBuffer.java | 2 +-
.../core/segment/store/ColumnIndexDirectory.java | 20 +
.../pinot/core/segment/store/ColumnIndexType.java | 3 +-
.../core/segment/store/FilePerIndexDirectory.java | 17 +
.../segment/store/SegmentLocalFSDirectory.java | 5 +
.../segment/store/SingleFileIndexDirectory.java | 12 +
.../virtualcolumn/VirtualColumnIndexContainer.java | 6 +
.../v2/store/StarTreeDimensionDataSource.java | 6 +
.../v2/store/StarTreeMetricDataSource.java | 6 +
.../org/apache/pinot/core/util/GroupByUtils.java | 21 +-
.../server/realtime/ControllerLeaderLocator.java | 16 +-
.../transform/function/zone-index.properties | 2215 ++++
.../core/common/RealtimeNoDictionaryTest.java | 8 +-
.../pinot/core/data/order/OrderByUtilsTest.java | 265 -
.../core/data/table/IndexedTableResizerTest.java | 434 +
.../pinot/core/data/table/IndexedTableTest.java | 221 +-
.../MutableSegmentImplNullValueVectorTest.java | 94 +
.../mutable/MutableSegmentImplTestUtils.java | 10 +-
.../io/writer/impl/OffHeapStringStoreTest.java | 119 -
.../pinot/core/minion/SegmentConverterTest.java | 78 +-
.../core/operator/CombineSlowOperatorsTest.java | 150 +
.../function/DateTruncTransformFunctionTest.java | 172 +
.../pinot/core/plan/CombinePlanNodeTest.java | 11 +-
...adataAndDictionaryAggregationPlanMakerTest.java | 9 +-
.../dictionary/OffHeapMutableBytesStoreTest.java | 142 +
.../impl/fakestream/FakeStreamConfigUtils.java | 10 -
.../RealtimeNullValueVectorReaderWriterTest.java | 51 +
.../ConsumingSegmentAssignmentStrategyTest.java | 285 -
.../core/realtime/stream/StreamConfigTest.java | 242 +-
.../impl/nullvalue/NullValueVectorCreatorTest.java | 68 +
.../core/segment/index/ColumnMetadataTest.java | 5 +-
.../segment/index/SegmentMetadataImplTest.java | 8 +-
.../SegmentV1V2ToV3FormatConverterTest.java | 3 +-
.../SegmentGenerationWithNullValueVectorTest.java | 178 +
.../SegmentGenerationWithTimeColumnTest.java | 21 +-
.../index/loader/SegmentPreProcessorTest.java | 9 +-
.../readers/NullValueVectorReaderImplTest.java | 68 +
.../store/ColumnIndexDirectoryTestHelper.java | 13 +
.../startree/StarTreeIndexTestSegmentHelper.java | 44 +-
.../pinot/core/startree/TestStarTreeMetadata.java | 33 +-
.../hll/SegmentWithHllIndexCreateHelper.java | 6 +-
.../org/apache/pinot/core/util/CrcUtilsTest.java | 3 +-
.../pinot/queries/BaseMultiValueQueriesTest.java | 2 +-
.../pinot/queries/BaseSingleValueQueriesTest.java | 2 +-
.../apache/pinot/queries/DistinctQueriesTest.java | 1 -
.../apache/pinot/queries/FastHllQueriesTest.java | 2 +-
.../apache/pinot/queries/TransformQueriesTest.java | 55 +
.../selection/SelectionOperatorServiceTest.java | 4 +-
.../segments/v1/creator/DictionariesTest.java | 2 +-
.../pinot/segments/v1/creator/IntArraysTest.java | 7 +-
.../data/test_null_value_vector_data.json | 12 +
.../data/test_null_value_vector_pinot_schema.json | 26 +
pinot-hadoop/pom.xml | 26 +-
.../apache/pinot/hadoop/job/BaseSegmentJob.java | 4 +
.../pinot/hadoop/job/JobConfigConstants.java | 3 +
.../hadoop/job/mappers/SegmentCreationMapper.java | 32 +-
.../job/mappers/SegmentPreprocessingMapper.java | 6 +-
.../hadoop/job/mappers/SegmentCreationTest.java | 75 +
.../tests/PinotURIUploadIntegrationTest.java | 32 -
.../apache/pinot/perf/BenchmarkCombineGroupBy.java | 12 +-
.../apache/pinot/perf/BenchmarkIndexedTable.java | 7 +-
.../perf/BenchmarkOffHeapMutableBytesStore.java | 147 +
.../pinot/perf/BenchmarkPinotDataBuffer.java | 139 +
.../tools/admin/command/AddSchemaCommand.java | 2 +-
.../admin/command/AvroSchemaToPinotSchema.java | 3 +-
.../OfflineSegmentIntervalCheckerCommand.java | 3 +-
.../tools/admin/command/PostQueryCommand.java | 19 +-
pom.xml | 3 +
.../app/pods/anomalies/controller.js | 15 +-
.../app/pods/components/alert-details/component.js | 454 +-
.../app/pods/components/alert-details/template.hbs | 22 +-
.../pods/components/detection-yaml/component.js | 41 +-
.../pods/components/detection-yaml/template.hbs | 8 -
.../rootcause-dimensions-algorithm/component.js | 23 +-
.../rootcause-dimensions-settings/template.hbs | 2 +
.../self-serve-alert-yaml-details/component.js | 19 +-
.../self-serve-alert-yaml-details/template.hbs | 27 +-
.../app/pods/components/stats-cards/component.js | 15 +-
.../app/pods/components/stats-cards/template.hbs | 60 +-
.../pods/components/subscription-yaml/component.js | 24 +-
.../pods/components/subscription-yaml/template.hbs | 8 -
.../app/pods/components/yaml-editor/component.js | 565 --
.../app/pods/components/yaml-editor/template.hbs | 174 -
.../anomalies-table/current-wow/template.hbs | 2 +-
.../anomalies-table/dimensions-only/template.hbs | 4 +-
.../app/pods/manage/alerts/index/route.js | 3 +-
.../app/pods/manage/explore/controller.js | 25 +
.../app/pods/manage/explore/route.js | 13 +-
.../app/pods/manage/explore/template.hbs | 5 +-
.../app/pods/manage/yaml/controller.js | 18 +
.../app/pods/manage/yaml/route.js | 33 +-
.../app/pods/manage/yaml/template.hbs | 3 +
.../partials/rootcause/dimensions/template.hbs | 3 +
.../app/pods/rootcause/controller.js | 21 +-
.../thirdeye-frontend/app/pods/rootcause/route.js | 9 +-
.../app/pods/self-serve/create-alert/route.js | 6 +-
.../app/pods/services/api/anomalies/service.js | 1 +
thirdeye/thirdeye-frontend/app/styles/app.scss | 3 +-
.../app/styles/components/button.scss | 18 +
.../{yaml-editor.scss => detection-yaml.scss} | 4 +-
.../{yaml-editor.scss => subscription-yaml.scss} | 4 +-
.../app/styles/components/te-anomaly-table.scss | 16 +-
.../app/styles/components/timeseries-chart.scss | 4 +-
.../app/styles/shared/_styles.scss | 52 +-
.../thirdeye-frontend/app/templates/dropdown.hbs | 28 +-
thirdeye/thirdeye-frontend/app/utils/anomaly.js | 19 +-
.../thirdeye-frontend/app/utils/api/self-serve.js | 10 +-
thirdeye/thirdeye-frontend/app/utils/constants.js | 74 -
thirdeye/thirdeye-frontend/app/utils/rca-utils.js | 4 +-
thirdeye/thirdeye-frontend/app/utils/utils.js | 69 +-
thirdeye/thirdeye-frontend/app/utils/yaml-tools.js | 347 +
.../acceptance/self-serve-alert-tuning-test.js | 2 +-
.../component-test.js | 31 +-
.../component-test.js | 4 +-
.../components/subscription-yaml/component-test.js | 57 +
thirdeye/thirdeye-frontend/yarn.lock | 31 +-
.../push/DeleteOverlappingSegmentsInPinot.java | 2 +-
thirdeye/thirdeye-pinot/pom.xml | 36 +-
.../anomaly/ThirdEyeAnomalyApplication.java | 9 +
.../anomaly/ThirdEyeAnomalyConfiguration.java | 23 +-
.../detection/trigger/DataAvailabilityEvent.java | 49 +-
.../trigger/DataAvailabilityEventListener.java | 102 +
.../DataAvailabilityEventListenerDriver.java | 101 +
.../trigger/DataAvailabilityKafkaConsumer.java | 61 +
.../trigger/filter/ActiveDatasetFilter.java | 24 +-
.../filter/DataAvailabilityEventFilter.java} | 9 +-
.../detection/trigger/filter/OnTimeFilter.java | 24 +-
.../DataAvailabilityListenerConfiguration.java | 128 +
.../trigger/utils/DatasetTriggerInfoRepo.java | 147 +
.../anomaly/utils/ThirdeyeMetricsUtil.java | 6 +
.../onboard/AutoOnboardPinotMetadataSource.java | 58 +-
.../thirdeye/cube/summary/SummaryResponse.java | 18 +
.../dashboard/resources/SummaryResource.java | 60 +-
.../resources/v2/RootCauseSessionResource.java | 6 +
.../datalayer/bao/DatasetConfigManager.java | 4 +
.../datalayer/bao/DetectionConfigManager.java | 3 +-
.../bao/jdbc/DatasetConfigManagerImpl.java | 7 +
.../bao/jdbc/DetectionConfigManagerImpl.java | 7 +
.../thirdeye/datalayer/dao/GenericPojoDao.java | 4 +-
.../datalayer/dto/MergedAnomalyResultDTO.java | 12 +
.../datalayer/entity/DatasetConfigIndex.java | 12 +
.../thirdeye/datalayer/pojo/DatasetConfigBean.java | 10 +
.../datalayer/pojo/RootcauseSessionBean.java | 17 +-
.../pinot/resultset/ThirdEyeResultSetUtils.java | 4 +-
.../pinot/thirdeye/datasource/sql/SqlUtils.java | 10 +-
.../thirdeye/detection/DetectionPipelineJob.java | 4 +-
.../thirdeye/detection/DetectionResource.java | 36 +
.../alert/DetectionAlertFilterNotification.java | 26 +-
.../detection/alert/DetectionAlertTaskFactory.java | 7 +-
.../filter/DimensionsRecipientAlertFilter.java | 17 +-
.../alert/scheme/DetectionAlertScheme.java | 7 +-
.../alert/scheme/DetectionEmailAlerter.java | 81 +-
.../alert/scheme/DetectionJiraAlerter.java | 122 +-
.../detection/algorithm/DimensionWrapper.java | 10 +
.../thirdeye/detection/algorithm/MergeWrapper.java | 6 +-
.../detection/wrapper/AnomalyDetectorWrapper.java | 7 +-
.../wrapper/ChildKeepingMergeWrapper.java | 4 +
.../thirdeye/detection/yaml/YamlResource.java | 1 +
.../translator/SubscriptionConfigTranslator.java | 4 +
.../formatter/DetectionConfigFormatter.java | 36 +-
.../thirdeye/notification/commons/JiraEntity.java | 117 +
.../notification/commons/ThirdEyeJiraClient.java | 155 +
.../content/BaseNotificationContent.java | 8 +-
.../notification/content/NotificationContent.java | 5 +-
.../content/templates/EntityGroupKeyContent.java | 9 +-
.../templates/HierarchicalAnomaliesContent.java | 10 +-
.../content/templates/MetricAnomaliesContent.java | 9 +-
.../formatter/ADContentFormatterContext.java | 68 -
.../formatter/channels/AlertContentFormatter.java | 10 +-
.../formatter/channels/EmailContentFormatter.java | 10 +-
.../formatter/channels/JiraContentFormatter.java | 52 +-
.../apache/pinot/thirdeye/util/ThirdEyeUtils.java | 85 +-
.../src/main/resources/schema/create-schema.sql | 2 +
.../trigger/DataAvailabilityEventListenerTest.java | 134 +
.../trigger/DatasetTriggerInfoRepoTest.java | 138 +
.../trigger/MockConsumerDataAvailability.java | 53 +
.../trigger/MockDataAvailabilityEvent.java | 74 +
.../AutoOnboardPinotMetricsServiceTest.java | 9 +-
.../pinot/thirdeye/datalayer/DaoTestUtils.java | 1 +
.../pinot/PinotThirdEyeDataSourceTest.java | 2 +-
.../detection/alert/filter/AlertFilterUtils.java | 23 +-
.../filter/DimensionsDetectionAlertFilterTest.java | 31 +-
.../ToAllRecipientsDetectionAlertFilterTest.java | 2 -
.../alert/scheme/AnotherRandomAlerter.java | 6 +-
.../alert/scheme/DetectionEmailAlerterTest.java | 154 +
.../alert/scheme/DetectionJiraAlerterTest.java | 144 +
.../detection/alert/scheme/RandomAlerter.java | 6 +-
.../thirdeye/detection/yaml/YamlResourceTest.java | 1 +
.../templates/TestEntityGroupKeyContent.java | 12 +-
.../TestHierarchicalAnomaliesContent.java | 10 +-
.../templates/TestMetricAnomaliesContent.java | 10 +-
.../channels/TestJiraContentFormatter.java | 197 +-
.../rootcause/MockDatasetConfigManager.java | 5 +
.../pinot/thirdeye/util/ThirdEyeUtilsTest.java | 20 +
.../src/test/resources/sample-alert-config.yml | 10 -
.../resources/test-jira-anomalies-template.ftl | 4 +-
377 files changed, 26884 insertions(+), 9535 deletions(-)
copy docker/{docker-push.sh => images/pinot-superset/.dockerignore} (82%)
mode change 100755 => 100644
copy docker/{docker-push.sh => images/pinot-superset/.python-version} (82%)
mode change 100755 => 100644
create mode 100644 docker/images/pinot-superset/Dockerfile
create mode 100644 docker/images/pinot-superset/Makefile
create mode 100644 docker/images/pinot-superset/README.md
copy docker/{docker-build-and-push.sh => images/pinot-superset/requirements-db.txt} (74%)
mode change 100755 => 100644
rename docker/{ => images/pinot}/Dockerfile (100%)
rename docker/{ => images/pinot}/README.md (100%)
copy docker/{ => images/pinot}/docker-build-and-push.sh (100%)
rename docker/{ => images/pinot}/docker-build.sh (100%)
rename docker/{ => images/pinot}/docker-compose.yml (100%)
rename docker/{ => images/pinot}/docker-push.sh (100%)
copy kubernetes/examples/{gke/skaffold/gke-storageclass-pinot-controller-pd.yml => helm/Chart.yaml} (58%)
create mode 100644 kubernetes/examples/helm/README.md
copy kubernetes/examples/{gke/skaffold/gke-storageclass-pinot-server-pd.yml => helm/helm-rbac.yaml} (70%)
copy kubernetes/examples/{gke/skaffold/query-pinot-data.sh => helm/open-superset-ui.sh} (76%)
create mode 100644 kubernetes/examples/helm/pinot-realtime-quickstart.yml
copy kubernetes/examples/{gke/skaffold => helm}/query-pinot-data.sh (100%)
rename docker/docker-build-and-push.sh => kubernetes/examples/helm/requirements.lock (76%)
mode change 100755 => 100644
copy kubernetes/examples/{gke/skaffold/gke-storageclass-kafka-pd.yml => helm/requirements.yaml} (80%)
copy kubernetes/examples/{gke/skaffold/setup.sh => helm/setup_gke.sh} (84%)
create mode 100644 kubernetes/examples/helm/superset.yaml
create mode 100644 kubernetes/examples/helm/templates/_helpers.tpl
copy kubernetes/examples/{gke/skaffold/gke-storageclass-pinot-server-pd.yml => helm/templates/broker/configmap.yaml} (74%)
copy kubernetes/examples/{gke/skaffold/gke-storageclass-pinot-controller-pd.yml => helm/templates/broker/service-external.yaml} (57%)
copy kubernetes/examples/{gke/skaffold/gke-storageclass-pinot-controller-pd.yml => helm/templates/broker/service-headless.yaml} (59%)
copy kubernetes/examples/{gke/skaffold/gke-storageclass-pinot-controller-pd.yml => helm/templates/broker/service.yaml} (59%)
create mode 100644 kubernetes/examples/helm/templates/broker/statefulset.yml
copy kubernetes/examples/{gke/skaffold/gke-storageclass-pinot-server-pd.yml => helm/templates/controller/configmap.yaml} (62%)
copy kubernetes/examples/{gke/skaffold/gke-storageclass-pinot-controller-pd.yml => helm/templates/controller/service-external.yaml} (56%)
copy kubernetes/examples/{gke/skaffold/gke-storageclass-pinot-controller-pd.yml => helm/templates/controller/service-headless.yaml} (58%)
copy kubernetes/examples/{gke/skaffold/gke-storageclass-pinot-controller-pd.yml => helm/templates/controller/service.yaml} (58%)
create mode 100644 kubernetes/examples/helm/templates/controller/statefulset.yaml
copy kubernetes/examples/{gke/skaffold/gke-storageclass-pinot-server-pd.yml => helm/templates/server/configmap.yaml} (67%)
copy kubernetes/examples/{gke/skaffold/gke-storageclass-pinot-controller-pd.yml => helm/templates/server/service-headless.yaml} (59%)
copy kubernetes/examples/{gke/skaffold/gke-storageclass-pinot-controller-pd.yml => helm/templates/server/service.yaml} (59%)
create mode 100644 kubernetes/examples/helm/templates/server/statefulset.yml
create mode 100644 kubernetes/examples/helm/values.yaml
rename kubernetes/examples/{gke/skaffold => skaffold/gke}/README.md (98%)
rename kubernetes/examples/{gke/skaffold => skaffold/gke}/cleanup.sh (100%)
rename kubernetes/examples/{gke/skaffold => skaffold/gke}/gke-storageclass-kafka-pd.yml (100%)
rename kubernetes/examples/{gke/skaffold => skaffold/gke}/gke-storageclass-pinot-controller-pd.yml (100%)
rename kubernetes/examples/{gke/skaffold => skaffold/gke}/gke-storageclass-pinot-server-pd.yml (100%)
rename kubernetes/examples/{gke/skaffold => skaffold/gke}/gke-storageclass-zk-pd.yml (100%)
rename kubernetes/examples/{gke/skaffold => skaffold/gke}/kafka.yml (100%)
rename kubernetes/examples/{gke/skaffold => skaffold/gke}/pinot-broker.yml (100%)
rename kubernetes/examples/{gke/skaffold => skaffold/gke}/pinot-controller.yml (100%)
rename kubernetes/examples/{gke/skaffold/pinot-example-loader.yml => skaffold/gke/pinot-realtime-quickstart.yml} (92%)
rename kubernetes/examples/{gke/skaffold => skaffold/gke}/pinot-server.yml (100%)
rename kubernetes/examples/{gke/skaffold => skaffold/gke}/query-pinot-data.sh (100%)
rename kubernetes/examples/{gke/skaffold => skaffold/gke}/setup.sh (100%)
rename kubernetes/examples/{gke/skaffold => skaffold/gke}/skaffold.yaml (100%)
rename kubernetes/examples/{gke/skaffold => skaffold/gke}/zookeeper.yml (100%)
delete mode 100644 pinot-common/src/main/java/org/apache/pinot/common/partition/PartitionAssignment.java
delete mode 100644 pinot-common/src/main/java/org/apache/pinot/common/partition/ReplicaGroupBasedStreamPartitionAssignmentStrategy.java
delete mode 100644 pinot-common/src/main/java/org/apache/pinot/common/partition/ReplicaGroupPartitionAssignment.java
delete mode 100644 pinot-common/src/main/java/org/apache/pinot/common/partition/ReplicaGroupPartitionAssignmentGenerator.java
delete mode 100644 pinot-common/src/main/java/org/apache/pinot/common/partition/StreamPartitionAssignmentGenerator.java
delete mode 100644 pinot-common/src/main/java/org/apache/pinot/common/partition/StreamPartitionAssignmentStrategy.java
delete mode 100644 pinot-common/src/main/java/org/apache/pinot/common/partition/StreamPartitionAssignmentStrategyFactory.java
delete mode 100644 pinot-common/src/main/java/org/apache/pinot/common/partition/UniformStreamPartitionAssignmentStrategy.java
delete mode 100644 pinot-common/src/test/java/org/apache/pinot/common/partition/IdealStateBuilderUtil.java
delete mode 100644 pinot-common/src/test/java/org/apache/pinot/common/partition/ReplicaGroupBasedStreamPartitionAssignmentTest.java
delete mode 100644 pinot-common/src/test/java/org/apache/pinot/common/partition/StreamPartitionAssignmentGeneratorTest.java
delete mode 100644 pinot-common/src/test/java/org/apache/pinot/common/partition/UniformStreamPartitionAssignmentTest.java
create mode 100644 pinot-connectors/pinot-connector-kafka-base/src/test/java/org/apache/pinot/core/realtime/impl/kafka/KafkaJSONMessageDecoderTest.java
create mode 100644 pinot-connectors/pinot-connector-kafka-base/src/test/resources/data/test_sample_data.json
create mode 100644 pinot-connectors/pinot-connector-kafka-base/src/test/resources/data/test_sample_data_schema_no_time_field.json
create mode 100644 pinot-connectors/pinot-connector-kafka-base/src/test/resources/data/test_sample_data_schema_with_outgoing_time_spec.json
create mode 100644 pinot-connectors/pinot-connector-kafka-base/src/test/resources/data/test_sample_data_schema_without_outgoing_time_spec.json
delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/data/order/OrderByUtils.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/data/table/IndexedTableResizer.java
delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/OffHeapStringStore.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/DateTruncTransformFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TimeZoneKey.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/dictionary/OffHeapMutableBytesStore.java
copy pinot-core/src/main/java/org/apache/pinot/core/{segment/store/ColumnIndexType.java => realtime/impl/nullvalue/RealtimeNullValueVectorReaderWriter.java} (56%)
delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/realtime/segment/ConsumingSegmentAssignmentStrategy.java
delete mode 100644 pinot-core/src/main/java/org/apache/pinot/core/realtime/segment/RealtimeSegmentAssignmentStrategy.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/nullvalue/NullValueVectorCreator.java
copy pinot-common/src/main/java/org/apache/pinot/common/config/StreamConsumptionConfig.java => pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/NullValueVectorReader.java (60%)
copy pinot-core/src/main/java/org/apache/pinot/core/segment/{store/ColumnIndexType.java => index/readers/NullValueVectorReaderImpl.java} (56%)
create mode 100644 pinot-core/src/main/resources/org/apache/pinot/core/operator/transform/function/zone-index.properties
delete mode 100644 pinot-core/src/test/java/org/apache/pinot/core/data/order/OrderByUtilsTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/data/table/IndexedTableResizerTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImplNullValueVectorTest.java
delete mode 100644 pinot-core/src/test/java/org/apache/pinot/core/io/writer/impl/OffHeapStringStoreTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/operator/CombineSlowOperatorsTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/DateTruncTransformFunctionTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/realtime/impl/dictionary/OffHeapMutableBytesStoreTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/realtime/impl/nullvalue/RealtimeNullValueVectorReaderWriterTest.java
delete mode 100644 pinot-core/src/test/java/org/apache/pinot/core/realtime/segment/ConsumingSegmentAssignmentStrategyTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/segment/creator/impl/nullvalue/NullValueVectorCreatorTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/segment/index/creator/SegmentGenerationWithNullValueVectorTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/segment/index/readers/NullValueVectorReaderImplTest.java
create mode 100644 pinot-core/src/test/resources/data/test_null_value_vector_data.json
create mode 100644 pinot-core/src/test/resources/data/test_null_value_vector_pinot_schema.json
create mode 100644 pinot-hadoop/src/test/java/org/apache/pinot/hadoop/job/mappers/SegmentCreationTest.java
create mode 100644 pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkOffHeapMutableBytesStore.java
create mode 100644 pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkPinotDataBuffer.java
delete mode 100644 thirdeye/thirdeye-frontend/app/pods/components/yaml-editor/component.js
delete mode 100644 thirdeye/thirdeye-frontend/app/pods/components/yaml-editor/template.hbs
copy thirdeye/thirdeye-frontend/app/styles/components/{yaml-editor.scss => detection-yaml.scss} (87%)
rename thirdeye/thirdeye-frontend/app/styles/components/{yaml-editor.scss => subscription-yaml.scss} (85%)
create mode 100644 thirdeye/thirdeye-frontend/app/utils/yaml-tools.js
rename thirdeye/thirdeye-frontend/tests/integration/pods/components/{yaml-editor => detection-yaml}/component-test.js (65%)
create mode 100644 thirdeye/thirdeye-frontend/tests/integration/pods/components/subscription-yaml/component-test.js
copy pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/ColumnIndexContainer.java => thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/detection/trigger/DataAvailabilityEvent.java (50%)
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/detection/trigger/DataAvailabilityEventListener.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/detection/trigger/DataAvailabilityEventListenerDriver.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/detection/trigger/DataAvailabilityKafkaConsumer.java
copy pinot-common/src/main/java/org/apache/pinot/common/config/StreamConsumptionConfig.java => thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/detection/trigger/filter/ActiveDatasetFilter.java (57%)
copy thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/{datalayer/bao/DetectionConfigManager.java => anomaly/detection/trigger/filter/DataAvailabilityEventFilter.java} (77%)
rename pinot-common/src/main/java/org/apache/pinot/common/config/StreamConsumptionConfig.java => thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/detection/trigger/filter/OnTimeFilter.java (57%)
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/detection/trigger/utils/DataAvailabilityListenerConfiguration.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/detection/trigger/utils/DatasetTriggerInfoRepo.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/notification/commons/JiraEntity.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/notification/commons/ThirdEyeJiraClient.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/notification/formatter/ADContentFormatterContext.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/anomaly/detection/trigger/DataAvailabilityEventListenerTest.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/anomaly/detection/trigger/DatasetTriggerInfoRepoTest.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/anomaly/detection/trigger/MockConsumerDataAvailability.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/anomaly/detection/trigger/MockDataAvailabilityEvent.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/alert/scheme/DetectionEmailAlerterTest.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/alert/scheme/DetectionJiraAlerterTest.java
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org