You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by ja...@apache.org on 2019/06/20 00:22:10 UTC
[incubator-pinot] branch llc_routing_hotfix updated (77954ec ->
ff86e60)
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a change to branch llc_routing_hotfix
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
discard 77954ec Within a partition, only allow querying the first CONSUMING segment for the real-time table routing
omit dcb261e Release branch only:Upgrade helix to 0.8.4 for release
omit f026f45 Fix the potentail negative sleep time in HelixServerStarter (#4277)
omit 4c9ac58 [TE] pass alertId to preview in Edit Alert (#4275)
omit 4e3c7d9 [TE] Change default auto onboard frequency to 6 hours (#4258)
omit 3ecdc5c Avoid negative sleep time in startupServiceStatusCheck (#4276)
omit ccab9a1 Fix controller metric (#4274)
omit 69f29c3 [TE] Cube algorithm for ratio metrics (#4246)
omit 55a20b2 [TE] detection config semantic validation fix (#4267)
omit 00c2204 pinot-connector-kafka-0.9 module (#4252)
omit 22b56ed Update the documentation for FASTHLL (#4251)
omit 0c7a20f [TE] frontend - harleyjj/preview - ensure zoom buttons are over shaded region (#4254)
omit f5c19f6 [TE] frontend - remove unnecessary flag for noExistingSubscriptionGroup (#4256)
omit d540d95 Change ControllerTest default prot to 18998 (#4259)
omit 27b61a1 [TE] add padding to graphs in email and Anomalies route (#4263)
omit e768195 [TE] add current time series if it's unavailable in predicted baseline endpoint (#4262)
omit 3b9ebea [TE] add padding to baseline endpoints (#4260)
omit 62ce57f Add RealtimeConsumptionCatchupServiceCallback (#4218)
omit 7abfb12 SoC - Separate out Tuning from Translator (#4250)
omit 951a781 Unify controller base url in integration test (#4257)
omit 3ce335a [TE] frontend - harleyjj/application - redirect http to https (#4238)
omit be62932 [TE] Change process timeout to 2 minutes (#4255)
omit 8c8c0ae Add config to disable HLC realtime segment completion (#4235)
new d238c6b Within a partition, only allow querying the first CONSUMING segment for the real-time table routing
new ff86e60 Bump up Helix
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (77954ec)
\
N -- N -- N refs/heads/llc_routing_hotfix (ff86e60)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.gitignore | 6 +
docs/pql_examples.rst | 6 +-
.../broker/broker/helix/HelixBrokerStarter.java | 34 +--
pinot-common/pom.xml | 18 ++
.../pinot/common/config/TableNameBuilder.java | 7 -
.../apache/pinot/common/utils/CommonConstants.java | 3 -
.../pinot/common/utils}/KafkaStarterUtils.java | 3 +-
.../apache/pinot/common/utils/ServiceStatus.java | 107 ++++----
pinot-connectors/README.md | 23 --
.../pinot-connector-kafka-0.9/README.md | 24 --
pinot-connectors/pinot-connector-kafka-0.9/pom.xml | 67 -----
pinot-connectors/pom.xml | 63 -----
.../apache/pinot/controller/ControllerConf.java | 15 +-
.../apache/pinot/controller/ControllerStarter.java | 18 +-
.../helix/core/PinotHelixResourceManager.java | 25 +-
.../pinot/controller/helix/ControllerTest.java | 2 +-
pinot-core/pom.xml | 16 ++
.../kafka}/AvroRecordToPinotRowGenerator.java | 0
.../realtime/impl/kafka/ConsumerAndIterator.java | 0
.../impl/kafka/KafkaAvroMessageDecoder.java | 0
.../realtime/impl/kafka/KafkaBrokerWrapper.java | 0
.../impl/kafka/KafkaConnectionHandler.java | 0
.../realtime/impl/kafka/KafkaConsumerFactory.java | 0
.../realtime/impl/kafka/KafkaConsumerManager.java | 0
.../impl/kafka/KafkaHighLevelStreamConfig.java | 0
.../impl/kafka/KafkaJSONMessageDecoder.java | 0
.../impl/kafka/KafkaLowLevelStreamConfig.java | 0
.../impl/kafka/KafkaPartitionLevelConsumer.java | 0
.../impl/kafka/KafkaSimpleConsumerFactory.java | 0
.../impl/kafka/KafkaSimpleConsumerFactoryImpl.java | 0
.../impl/kafka/KafkaStreamConfigProperties.java | 0
.../impl/kafka/KafkaStreamLevelConsumer.java | 0
.../impl/kafka/KafkaStreamMetadataProvider.java | 0
.../kafka}/SimpleAvroMessageDecoder.java | 0
.../realtime/impl/kafka/SimpleConsumerFactory.java | 0
.../impl/kafka/SimpleConsumerMessageBatch.java | 0
.../pinot/core/realtime/stream/StreamConfig.java | 4 +-
.../kafka}/AvroRecordToPinotRowGeneratorTest.java | 0
.../impl/kafka/KafkaLowLevelStreamConfigTest.java | 0
.../kafka/KafkaPartitionLevelConsumerTest.java | 0
pinot-integration-tests/pom.xml | 6 -
.../tests/BaseClusterIntegrationTest.java | 2 +-
.../ControllerPeriodicTasksIntegrationTests.java | 2 +-
.../tests/HybridClusterIntegrationTest.java | 3 +-
...ridClusterIntegrationTestCommandLineRunner.java | 2 +-
.../tests/LLCRealtimeClusterIntegrationTest.java | 40 ---
.../tests/NewConfigApplyIntegrationTest.java | 12 +-
.../tests/OfflineClusterIntegrationTest.java | 38 +--
.../tests/PinotURIUploadIntegrationTest.java | 2 +-
.../tests/RealtimeClusterIntegrationTest.java | 3 +-
pinot-perf/pom.xml | 6 -
.../perf/BenchmarkRealtimeConsumptionSpeed.java | 2 +-
.../org/apache/pinot/perf/RealtimeStressTest.java | 2 +-
.../server/starter/helix/HelixServerStarter.java | 104 ++------
.../request/ScheduledRequestHandlerTest.java | 10 +-
pinot-tools/pom.xml | 6 -
.../org/apache/pinot/tools/HybridQuickstart.java | 2 +-
.../org/apache/pinot/tools/RealtimeQuickStart.java | 2 +-
.../tools/admin/command/StartKafkaCommand.java | 2 +-
.../admin/command/StreamAvroIntoKafkaCommand.java | 2 +-
.../pinot/tools/streams/AirlineDataStream.java | 2 +-
.../pinot/tools/streams/MeetupRsvpStream.java | 2 +-
pom.xml | 16 +-
.../app/pods/application/route.js | 8 +-
.../app/pods/components/alert-details/component.js | 59 +++--
.../app/pods/components/alert-details/template.hbs | 5 +-
.../pods/components/anomaly-summary/component.js | 28 +-
.../pods/components/timeseries-chart/component.js | 57 +----
.../app/pods/manage/yaml/controller.js | 31 ++-
.../app/pods/manage/yaml/template.hbs | 1 -
.../app/pods/screenshot/controller.js | 21 +-
.../thirdeye-frontend/app/pods/screenshot/route.js | 7 +-
.../anomaly/alert/util/EmailScreenshotHelper.java | 2 +-
.../api/application/ApplicationResource.java | 2 +-
.../auto/onboard/AutoOnboardConfiguration.java | 2 +-
.../thirdeye/cube/additive/AdditiveCubeNode.java | 53 ++--
.../thirdeye/cube/additive/AdditiveDBClient.java | 12 +-
.../pinot/thirdeye/cube/additive/AdditiveRow.java | 7 +-
.../MultiDimensionalSummary.java | 38 +--
.../MultiDimensionalSummaryCLITool.java | 6 +-
.../pinot/thirdeye/cube/cost/CostFunction.java | 1 -
.../thirdeye/cube/cost/RatioCostFunction.java | 136 ----------
.../apache/pinot/thirdeye/cube/data/cube/Cube.java | 3 +-
.../pinot/thirdeye/cube/data/cube/CubeUtils.java | 25 --
.../cube/data/cube/DimNameValueCostEntry.java | 54 ++--
.../cube/data/dbclient/BaseCubePinotClient.java | 12 -
.../pinot/thirdeye/cube/data/dbrow/BaseRow.java | 25 --
.../thirdeye/cube/data/node/BaseCubeNode.java | 49 +---
.../pinot/thirdeye/cube/data/node/CubeNode.java | 12 +-
.../thirdeye/cube/data/node/CubeNodeUtils.java | 7 +-
.../cube/entry/MultiDimensionalRatioSummary.java | 94 -------
.../pinot/thirdeye/cube/entry/SummaryUtils.java | 43 ----
.../pinot/thirdeye/cube/ratio/RatioCubeNode.java | 285 ---------------------
.../pinot/thirdeye/cube/ratio/RatioDBClient.java | 113 --------
.../apache/pinot/thirdeye/cube/ratio/RatioRow.java | 190 --------------
.../thirdeye/cube/summary/BaseResponseRow.java | 7 +-
.../pinot/thirdeye/cube/summary/Summary.java | 43 ++--
.../thirdeye/cube/summary/SummaryResponse.java | 33 ++-
.../dashboard/resources/SummaryResource.java | 78 +-----
.../thirdeye/detection/DetectionPipeline.java | 17 +-
.../thirdeye/detection/DetectionResource.java | 41 +--
.../pinot/thirdeye/detection/DetectionUtils.java | 18 +-
.../annotation/registry/DetectionRegistry.java | 4 +-
.../components/AbsoluteChangeRuleDetector.java | 5 +-
.../components/PercentageChangeRuleDetector.java | 5 +-
.../detection/components/RuleBaselineProvider.java | 13 +-
.../components/ThresholdRuleDetector.java | 1 +
.../onboard/YamlOnboardingTaskRunner.java | 18 +-
.../detection/validators/ConfigValidator.java | 2 +-
.../validators/SubscriptionConfigValidator.java | 2 +-
.../detection/wrapper/AnomalyDetectorWrapper.java | 2 +-
.../detection/wrapper/AnomalyFilterWrapper.java | 2 +-
.../wrapper/BaselineFillingMergeWrapper.java | 12 +-
.../thirdeye/detection/wrapper/GrouperWrapper.java | 2 +-
.../CompositePipelineConfigTranslator.java | 113 +++++---
.../detection/yaml/DetectionConfigTuner.java | 180 -------------
.../YamlDetectionAlertConfigTranslator.java | 48 ++--
.../YamlDetectionConfigTranslator.java | 57 ++++-
.../YamlDetectionTranslatorLoader.java | 2 +-
.../thirdeye/detection/yaml/YamlResource.java | 36 +--
.../detection/yaml/YamlTranslationResult.java | 87 +++++++
.../yaml/translator/ConfigTranslator.java | 52 ----
.../thirdeye/rootcause/impl/MetricEntity.java | 13 -
.../MultiDimensionalSummaryCLIToolTest.java | 1 -
.../pinot/thirdeye/cube/data/cube/CubeTest.java | 14 +-
.../cube/data/cube/DimNameValueCostEntryTest.java | 7 +-
.../cube/data/dbrow/DimensionValuesTest.java | 1 +
.../thirdeye/cube/data/dbrow/DimensionsTest.java | 1 +
.../cube/data/node/AdditiveCubeNodeTest.java | 61 -----
.../thirdeye/cube/data/node/CubeNodeTest.java | 49 ++--
.../thirdeye/cube/data/node/RatioCubeNodeTest.java | 112 --------
.../thirdeye/detection/DetectionUtilsTest.java | 51 ----
.../components/RuleBaselineProviderTest.java | 6 +-
.../CompositePipelineConfigTranslatorTest.java | 38 +--
.../yaml/MockYamlDetectionConfigTranslator.java | 21 ++
.../YamlDetectionAlertConfigTranslatorTest.java | 13 +-
.../yaml/YamlDetectionConfigTranslatorTest.java | 60 +++++
.../yaml/translator/YamlTranslationResult.java | 54 ----
.../compositePipelineTranslatorTestResult-1.json | 6 +-
.../compositePipelineTranslatorTestResult-2.json | 0
.../yaml/{translator => }/pipeline-config-1.yaml | 2 +-
.../yaml/{translator => }/pipeline-config-2.yaml | 0
.../yaml/{translator => }/pipeline-config-3.yaml | 0
143 files changed, 892 insertions(+), 2595 deletions(-)
rename {pinot-connectors/pinot-connector-kafka-0.9/src/main/java/org/apache/pinot/core/realtime/impl/kafka => pinot-common/src/main/java/org/apache/pinot/common/utils}/KafkaStarterUtils.java (98%)
delete mode 100644 pinot-connectors/README.md
delete mode 100644 pinot-connectors/pinot-connector-kafka-0.9/README.md
delete mode 100644 pinot-connectors/pinot-connector-kafka-0.9/pom.xml
delete mode 100644 pinot-connectors/pom.xml
rename pinot-core/src/main/java/org/apache/pinot/core/realtime/{stream => impl/kafka}/AvroRecordToPinotRowGenerator.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9 => pinot-core}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/ConsumerAndIterator.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9 => pinot-core}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaAvroMessageDecoder.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9 => pinot-core}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaBrokerWrapper.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9 => pinot-core}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaConnectionHandler.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9 => pinot-core}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaConsumerFactory.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9 => pinot-core}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaConsumerManager.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9 => pinot-core}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaHighLevelStreamConfig.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9 => pinot-core}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaJSONMessageDecoder.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9 => pinot-core}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaLowLevelStreamConfig.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9 => pinot-core}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaPartitionLevelConsumer.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9 => pinot-core}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaSimpleConsumerFactory.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9 => pinot-core}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaSimpleConsumerFactoryImpl.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9 => pinot-core}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaStreamConfigProperties.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9 => pinot-core}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaStreamLevelConsumer.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9 => pinot-core}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaStreamMetadataProvider.java (100%)
rename pinot-core/src/main/java/org/apache/pinot/core/realtime/{stream => impl/kafka}/SimpleAvroMessageDecoder.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9 => pinot-core}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/SimpleConsumerFactory.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9 => pinot-core}/src/main/java/org/apache/pinot/core/realtime/impl/kafka/SimpleConsumerMessageBatch.java (100%)
rename pinot-core/src/test/java/org/apache/pinot/core/realtime/{stream => impl/kafka}/AvroRecordToPinotRowGeneratorTest.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9 => pinot-core}/src/test/java/org/apache/pinot/core/realtime/impl/kafka/KafkaLowLevelStreamConfigTest.java (100%)
rename {pinot-connectors/pinot-connector-kafka-0.9/src/test/java/org/apache/pinot/core/realtime/impl => pinot-core/src/test/java/org/apache/pinot/core/realtime}/kafka/KafkaPartitionLevelConsumerTest.java (100%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/cube/{entry => additive}/MultiDimensionalSummary.java (80%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/cube/{entry => additive}/MultiDimensionalSummaryCLITool.java (97%)
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/cube/cost/RatioCostFunction.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/cube/entry/MultiDimensionalRatioSummary.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/cube/entry/SummaryUtils.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/cube/ratio/RatioCubeNode.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/cube/ratio/RatioDBClient.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/cube/ratio/RatioRow.java
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/{translator => }/CompositePipelineConfigTranslator.java (79%)
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/DetectionConfigTuner.java
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/{translator => }/YamlDetectionAlertConfigTranslator.java (83%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/{translator => }/YamlDetectionConfigTranslator.java (54%)
rename thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/{translator => }/YamlDetectionTranslatorLoader.java (96%)
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/YamlTranslationResult.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/translator/ConfigTranslator.java
delete mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/cube/data/node/AdditiveCubeNodeTest.java
delete mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/cube/data/node/RatioCubeNodeTest.java
delete mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/DetectionUtilsTest.java
rename thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/yaml/{translator => }/CompositePipelineConfigTranslatorTest.java (78%)
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/yaml/MockYamlDetectionConfigTranslator.java
rename thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/yaml/{translator => }/YamlDetectionAlertConfigTranslatorTest.java (88%)
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/yaml/YamlDetectionConfigTranslatorTest.java
delete mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/yaml/translator/YamlTranslationResult.java
rename thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/yaml/{translator => }/compositePipelineTranslatorTestResult-1.json (96%)
rename thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/yaml/{translator => }/compositePipelineTranslatorTestResult-2.json (100%)
rename thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/yaml/{translator => }/pipeline-config-1.yaml (97%)
rename thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/yaml/{translator => }/pipeline-config-2.yaml (100%)
rename thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/yaml/{translator => }/pipeline-config-3.yaml (100%)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[incubator-pinot] 02/02: Bump up Helix
Posted by ja...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a commit to branch llc_routing_hotfix
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit ff86e608e5d2f4041673b56eea074751574a6ac3
Author: Jackie (Xiaotian) Jiang <xa...@linkedin.com>
AuthorDate: Wed Jun 19 17:21:47 2019 -0700
Bump up Helix
---
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index 83ed620..cadc6c4 100644
--- a/pom.xml
+++ b/pom.xml
@@ -117,7 +117,7 @@
<!-- Configuration for unit/integration tests section 1 of 3 (properties) ENDS HERE.-->
<avro.version>1.7.6</avro.version>
<parquet.version>1.8.0</parquet.version>
- <helix.version>0.8.2</helix.version>
+ <helix.version>0.8.4</helix.version>
<!-- jfim: for Kafka 0.9.0.0, use zkclient 0.7 -->
<kafka.version>0.9.0.1</kafka.version>
<zkclient.version>0.7</zkclient.version>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[incubator-pinot] 01/02: Within a partition,
only allow querying the first CONSUMING segment for the real-time
table routing
Posted by ja...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a commit to branch llc_routing_hotfix
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit d238c6bb549f9af27eac9f3ea511fd45cb3b2434
Author: Jackie (Xiaotian) Jiang <xa...@linkedin.com>
AuthorDate: Tue Jun 18 18:34:55 2019 -0700
Within a partition, only allow querying the first CONSUMING segment for the real-time table routing
Within a partition, we only allow querying the first CONSUMING segment (segment with instances in CONSUMING
state) for the following reasons:
- If within a partition, there are multiple CONSUMING segments (typically caused by the delay of CONSUMING to
ONLINE state transition), we can only query the first CONSUMING segment because it might contain records that
overlapped with the records in the next segment (over-consumed).
- If the instance states for a segment is partial ONLINE and partial CONSUMING (some instances finished the
CONSUMING to ONLINE state transition, others didn't), we count the segment as CONSUMING segment (most likely
the first CONSUMING segment because it is already committed and is performing the CONSUMING to ONLINE state
transition). If we don't count the segment as CONSUMING segment, then this segment is not allowed to be in the
CONSUMING state for routing purpose, and we will route all queries to the ONLINE instances which can
potentially overwhelm instance.
- It is possible that the latest CONSUMING segment is not allowed for routing purpose and we won't query it, but
it should only last for a short period of time. Once the older CONSUMING segment becomes ONLINE (all instances
finished the CONSUMING to ONLINE state transition), the latest CONSUMING segment will become the first
CONSUMING segment and will be allowed for routing purpose.
---
.../LowLevelConsumerRoutingTableBuilder.java | 6 +-
.../builder/LowLevelRoutingTableBuilderUtil.java | 76 ++++++++++------------
2 files changed, 36 insertions(+), 46 deletions(-)
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/builder/LowLevelConsumerRoutingTableBuilder.java b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/builder/LowLevelConsumerRoutingTableBuilder.java
index f6c40b2..4a39da2 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/builder/LowLevelConsumerRoutingTableBuilder.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/builder/LowLevelConsumerRoutingTableBuilder.java
@@ -122,9 +122,9 @@ public class LowLevelConsumerRoutingTableBuilder extends GeneratorBasedRoutingTa
continue;
}
- // Replicas in CONSUMING state are only allowed on the last segment
- if (state.equalsIgnoreCase(CommonConstants.Helix.StateModel.RealtimeSegmentOnlineOfflineStateModel.CONSUMING)
- && segmentName.equals(validConsumingSegment)) {
+ // If the server is in CONSUMING status, the segment has to be match with the valid consuming segment
+ if (state.equals(CommonConstants.Helix.StateModel.RealtimeSegmentOnlineOfflineStateModel.CONSUMING)
+ && validConsumingSegment != null && segmentNameStr.equals(validConsumingSegment.getSegmentName())) {
validServers.add(instance);
}
}
diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/builder/LowLevelRoutingTableBuilderUtil.java b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/builder/LowLevelRoutingTableBuilderUtil.java
index 8490397..3ccf841 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/builder/LowLevelRoutingTableBuilderUtil.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/builder/LowLevelRoutingTableBuilderUtil.java
@@ -22,7 +22,7 @@ import java.util.HashMap;
import java.util.Map;
import java.util.SortedSet;
import org.apache.helix.model.ExternalView;
-import org.apache.pinot.common.utils.CommonConstants;
+import org.apache.pinot.common.utils.CommonConstants.Helix.StateModel.RealtimeSegmentOnlineOfflineStateModel;
import org.apache.pinot.common.utils.SegmentName;
@@ -32,7 +32,30 @@ import org.apache.pinot.common.utils.SegmentName;
public class LowLevelRoutingTableBuilderUtil {
/**
- * Compute the map of allowed 'consuming' segments for each partition.
+ * Compute the map of allowed CONSUMING segments for each partition for routing purpose.
+ * <p>Within a partition, we only allow querying the first CONSUMING segment (segment with instances in CONSUMING
+ * state) for the following reasons:
+ * <ul>
+ * <li>
+ * If within a partition, there are multiple CONSUMING segments (typically caused by the delay of CONSUMING to
+ * ONLINE state transition), we can only query the first CONSUMING segment because it might contain records that
+ * overlapped with the records in the next segment (over-consumed).
+ * </li>
+ * <li>
+ * If the instance states for a segment is partial ONLINE and partial CONSUMING (some instances finished the
+ * CONSUMING to ONLINE state transition, others didn't), we count the segment as CONSUMING segment (most likely
+ * the first CONSUMING segment because it is already committed and is performing the CONSUMING to ONLINE state
+ * transition). If we don't count the segment as CONSUMING segment, then this segment is not allowed to be in the
+ * CONSUMING state for routing purpose, and we will route all queries to the ONLINE instances which can
+ * potentially overwhelm instance.
+ * </li>
+ * <li>
+ * It is possible that the latest CONSUMING segment is not allowed for routing purpose and we won't query it, but
+ * it should only last for a short period of time. Once the older CONSUMING segment becomes ONLINE (all instances
+ * finished the CONSUMING to ONLINE state transition), the latest CONSUMING segment will become the first
+ * CONSUMING segment and will be allowed for routing purpose.
+ * </li>
+ * </ul>
*
* @param externalView helix external view
* @param sortedSegmentsByPartition map of partition to sorted set of segment names.
@@ -40,51 +63,18 @@ public class LowLevelRoutingTableBuilderUtil {
*/
public static Map<String, SegmentName> getAllowedConsumingStateSegments(ExternalView externalView,
Map<String, SortedSet<SegmentName>> sortedSegmentsByPartition) {
- Map<String, SegmentName> allowedSegmentInConsumingStateByPartition = new HashMap<>();
- for (String partition : sortedSegmentsByPartition.keySet()) {
- SortedSet<SegmentName> sortedSegmentsForPartition = sortedSegmentsByPartition.get(partition);
- SegmentName lastAllowedSegmentInConsumingState = null;
-
+ Map<String, SegmentName> allowedConsumingSegments = new HashMap<>();
+ for (Map.Entry<String, SortedSet<SegmentName>> entry : sortedSegmentsByPartition.entrySet()) {
+ String partitionId = entry.getKey();
+ SortedSet<SegmentName> sortedSegmentsForPartition = entry.getValue();
for (SegmentName segmentName : sortedSegmentsForPartition) {
- Map<String, String> helixPartitionState = externalView.getStateMap(segmentName.getSegmentName());
- boolean allInConsumingState = true;
- int replicasInConsumingState = 0;
-
- // Only keep the segment if all replicas have it in CONSUMING state
- for (String externalViewState : helixPartitionState.values()) {
- // Ignore ERROR state
- if (externalViewState
- .equalsIgnoreCase(CommonConstants.Helix.StateModel.RealtimeSegmentOnlineOfflineStateModel.ERROR)) {
- continue;
- }
-
- // Not all segments are in CONSUMING state, therefore don't consider the last segment assignable to CONSUMING
- // replicas
- if (externalViewState
- .equalsIgnoreCase(CommonConstants.Helix.StateModel.RealtimeSegmentOnlineOfflineStateModel.ONLINE)) {
- allInConsumingState = false;
- break;
- }
-
- // Otherwise count the replica as being in CONSUMING state
- if (externalViewState
- .equalsIgnoreCase(CommonConstants.Helix.StateModel.RealtimeSegmentOnlineOfflineStateModel.CONSUMING)) {
- replicasInConsumingState++;
- }
- }
-
- // If all replicas have this segment in consuming state (and not all of them are in ERROR state), then pick this
- // segment to be the last allowed segment to be in CONSUMING state
- if (allInConsumingState && 0 < replicasInConsumingState) {
- lastAllowedSegmentInConsumingState = segmentName;
+ if (externalView.getStateMap(segmentName.getSegmentName())
+ .containsValue(RealtimeSegmentOnlineOfflineStateModel.CONSUMING)) {
+ allowedConsumingSegments.put(partitionId, segmentName);
break;
}
}
-
- if (lastAllowedSegmentInConsumingState != null) {
- allowedSegmentInConsumingStateByPartition.put(partition, lastAllowedSegmentInConsumingState);
- }
}
- return allowedSegmentInConsumingStateByPartition;
+ return allowedConsumingSegments;
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org