You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/06/10 00:28:16 UTC
[GitHub] [pinot] sajjad-moradi opened a new pull request, #8877: Emit metrics if there's no consuming segment for a partition
sajjad-moradi opened a new pull request, #8877:
URL: https://github.com/apache/pinot/pull/8877
## Description
Currently if there's no consuming segment for a partition, it remains undetected.
With the changes in this PR, the following two metrics get emitted if for a partition group has no consuming segment.
- Number of missing consuming segments
- Maximum duration (in minutes) that a partition hasn't had a consuming segment
The check happens at the beginning of the realtime segment validation periodic task.
## Testing Done
- Unit tests
- Ran LLC cluster integration test. Then deleted a consuming segment in ideal state and after that ran the periodic task. Here are the MBeans for the emitted metrics:

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
sajjad-moradi commented on code in PR #8877:
URL: https://github.com/apache/pinot/pull/8877#discussion_r896234956
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/MissingConsumingSegmentFinder.java:
##########
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.helix.AccessOption;
+import org.apache.helix.ZNRecord;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+import org.apache.pinot.common.metadata.ZKMetadataProvider;
+import org.apache.pinot.common.metrics.ControllerGauge;
+import org.apache.pinot.common.metrics.ControllerMeter;
+import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.apache.zookeeper.data.Stat;
+
+
+/**
+ * For a given table, this class finds out if there is any partition group for which there's no consuming segment in
+ * ideal state. If so, it emits two metrics:
+ * - Number of missing consuming segments
+ * - Maximum duration (in minutes) that a partition hasn't had a consuming segment
+ */
+public class MissingConsumingSegmentFinder {
+
+ private String _realtimeTableName;
+ private ControllerMetrics _controllerMetrics;
+ private SegmentMetadataFetcher _segmentMetadataFetcher;
+
+ public MissingConsumingSegmentFinder(String realtimeTableName, ZkHelixPropertyStore<ZNRecord> propertyStore,
+ ControllerMetrics controllerMetrics) {
+ _realtimeTableName = realtimeTableName;
+ _controllerMetrics = controllerMetrics;
+ _segmentMetadataFetcher = new SegmentMetadataFetcher(propertyStore, controllerMetrics);
+ }
+
+ @VisibleForTesting
+ MissingConsumingSegmentFinder(String realtimeTableName, SegmentMetadataFetcher segmentMetadataFetcher) {
+ _realtimeTableName = realtimeTableName;
+ _segmentMetadataFetcher = segmentMetadataFetcher;
+ }
+
+ public void findAndEmitMetrics(IdealState idealState) {
+ MissingSegmentInfo info = findMissingSegments(idealState.getRecord().getMapFields(), Instant.now());
+ _controllerMetrics.setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_COUNT,
+ info._count);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_MAX_DURATION_MINUTES,
+ info._maxDurationInMinutes);
+ }
+
+ @VisibleForTesting
+ MissingSegmentInfo findMissingSegments(Map<String, Map<String, String>> idealStateMap, Instant now) {
+ // create the maps
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestConsumingSegmentMap = new HashMap<>();
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestCompletedSegmentMap = new HashMap<>();
+ idealStateMap.forEach((segmentName, instanceToStatusMap) -> {
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName != null) { // Skip the uploaded realtime segments that don't conform to llc naming
+ if (instanceToStatusMap.containsValue(SegmentStateModel.CONSUMING)) {
+ updateMap(partitionGroupIdToLatestConsumingSegmentMap, llcSegmentName);
Review Comment:
1) My understanding is that if a consuming segment encounters a permanent error, it calls controller to mark it as OFFLINE in ideal state. So for these cases, the last consumed time is when the previous completed segment went to ONLINE state.
2) We can ignore the new partition cases as Segment Validation Manager is going to create a new consuming segment for them. If we wanted to detect them, then we'd need to fetch the metadata for the stream which may be problematic (please refer to next comment).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] sajjad-moradi commented on pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
sajjad-moradi commented on PR #8877:
URL: https://github.com/apache/pinot/pull/8877#issuecomment-1170437363
@Jackie-Jiang @npawar could you please take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] KKcorps commented on pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
KKcorps commented on PR #8877:
URL: https://github.com/apache/pinot/pull/8877#issuecomment-1171563261
I have taken a quick look and it looks fine to me!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on a diff in pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on code in PR #8877:
URL: https://github.com/apache/pinot/pull/8877#discussion_r906897792
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/SegmentStatusChecker.java:
##########
@@ -131,8 +137,7 @@ protected void postprocess(Context context) {
* Updates metrics related to the table config.
* If table config not found, resets the metrics
*/
- private void updateTableConfigMetrics(String tableNameWithType) {
- TableConfig tableConfig = _pinotHelixResourceManager.getTableConfig(tableNameWithType);
+ private void updateTableConfigMetrics(String tableNameWithType, TableConfig tableConfig) {
Review Comment:
I was going to ask to remove the `tableNameWithType` argument (given that table config is passed in), but then I saw we are checking for table config null and resetting a metric. @npawar this seems to be a part of pr #8761. Can you share the case in which table config will be null? It would be useful to add that as comments here.
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/MissingConsumingSegmentFinder.java:
##########
@@ -0,0 +1,221 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.helix.AccessOption;
+import org.apache.helix.ZNRecord;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+import org.apache.pinot.common.metadata.ZKMetadataProvider;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ControllerGauge;
+import org.apache.pinot.common.metrics.ControllerMeter;
+import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder;
+import org.apache.pinot.spi.stream.OffsetCriteria;
+import org.apache.pinot.spi.stream.PartitionLevelStreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * For a given table, this class finds out if there is any partition group for which there's no consuming segment in
+ * ideal state. If so, it emits three metrics:
+ * - Total number of partitions with missing consuming segments including
+ * - Number of newly added partitions for which there's no consuming segment (there's no completed segment either)
+ * - Maximum duration (in minutes) that a partition hasn't had a consuming segment
+ */
+public class MissingConsumingSegmentFinder {
+ private static final Logger LOGGER = LoggerFactory.getLogger(MissingConsumingSegmentFinder.class);
+
+ private final String _realtimeTableName;
+ private final SegmentMetadataFetcher _segmentMetadataFetcher;
+ private final Map<Integer, StreamPartitionMsgOffset> _partitionGroupIdToLargestStreamOffsetMap;
+ private final StreamPartitionMsgOffsetFactory _streamPartitionMsgOffsetFactory;
+
+ private ControllerMetrics _controllerMetrics;
+
+ public MissingConsumingSegmentFinder(String realtimeTableName, ZkHelixPropertyStore<ZNRecord> propertyStore,
+ ControllerMetrics controllerMetrics, PartitionLevelStreamConfig streamConfig) {
+ _realtimeTableName = realtimeTableName;
+ _controllerMetrics = controllerMetrics;
+ _segmentMetadataFetcher = new SegmentMetadataFetcher(propertyStore, controllerMetrics);
+ _streamPartitionMsgOffsetFactory =
+ StreamConsumerFactoryProvider.create(streamConfig).createStreamMsgOffsetFactory();
+
+ // create partition group id to largest stream offset map
+ _partitionGroupIdToLargestStreamOffsetMap = new HashMap<>();
+ streamConfig.setOffsetCriteria(OffsetCriteria.LARGEST_OFFSET_CRITERIA);
+ try {
+ PinotTableIdealStateBuilder.getPartitionGroupMetadataList(streamConfig, Collections.emptyList())
+ .forEach(metadata -> {
+ _partitionGroupIdToLargestStreamOffsetMap.put(metadata.getPartitionGroupId(), metadata.getStartOffset());
+ });
+ } catch (Exception e) {
+ LOGGER.warn("Problem encountered in fetching stream metadata for topic: {} of table: {}. "
+ + "Continue finding missing consuming segment only with ideal state information.",
+ streamConfig.getTopicName(), streamConfig.getTableNameWithType());
+ }
+ }
+
+ @VisibleForTesting
+ MissingConsumingSegmentFinder(String realtimeTableName, SegmentMetadataFetcher segmentMetadataFetcher,
+ Map<Integer, StreamPartitionMsgOffset> partitionGroupIdToLargestStreamOffsetMap,
+ StreamPartitionMsgOffsetFactory streamPartitionMsgOffsetFactory) {
+ _realtimeTableName = realtimeTableName;
+ _segmentMetadataFetcher = segmentMetadataFetcher;
+ _partitionGroupIdToLargestStreamOffsetMap = partitionGroupIdToLargestStreamOffsetMap;
+ _streamPartitionMsgOffsetFactory = streamPartitionMsgOffsetFactory;
+ }
+
+ public void findAndEmitMetrics(IdealState idealState) {
+ MissingSegmentInfo info = findMissingSegments(idealState.getRecord().getMapFields(), Instant.now());
+ _controllerMetrics.setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_TOTAL_COUNT,
+ info._totalCount);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_NEW_PARTITION_COUNT,
+ info._newPartitionGroupCount);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_MAX_DURATION_MINUTES,
+ info._maxDurationInMinutes);
+ }
+
+ @VisibleForTesting
+ MissingSegmentInfo findMissingSegments(Map<String, Map<String, String>> idealStateMap, Instant now) {
+ // create the maps
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestConsumingSegmentMap = new HashMap<>();
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestCompletedSegmentMap = new HashMap<>();
+ idealStateMap.forEach((segmentName, instanceToStatusMap) -> {
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName != null) { // Skip the uploaded realtime segments that don't conform to llc naming
+ if (instanceToStatusMap.containsValue(SegmentStateModel.CONSUMING)) {
+ updateMap(partitionGroupIdToLatestConsumingSegmentMap, llcSegmentName);
+ } else if (instanceToStatusMap.containsValue(SegmentStateModel.ONLINE)) {
+ updateMap(partitionGroupIdToLatestCompletedSegmentMap, llcSegmentName);
+ }
+ }
+ });
+
+ MissingSegmentInfo missingSegmentInfo = new MissingSegmentInfo();
+ if (!_partitionGroupIdToLargestStreamOffsetMap.isEmpty()) {
+ _partitionGroupIdToLargestStreamOffsetMap.forEach((partitionGroupId, largestStreamOffset) -> {
+ if (!partitionGroupIdToLatestConsumingSegmentMap.containsKey(partitionGroupId)) {
Review Comment:
would be good to add a comment below line 130 that this is the case when consuming segment is missing for the partitionGroupId. But it is not a problem if the partitionGroupId is closed.
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/MissingConsumingSegmentFinder.java:
##########
@@ -0,0 +1,221 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.helix.AccessOption;
+import org.apache.helix.ZNRecord;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+import org.apache.pinot.common.metadata.ZKMetadataProvider;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ControllerGauge;
+import org.apache.pinot.common.metrics.ControllerMeter;
+import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder;
+import org.apache.pinot.spi.stream.OffsetCriteria;
+import org.apache.pinot.spi.stream.PartitionLevelStreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * For a given table, this class finds out if there is any partition group for which there's no consuming segment in
+ * ideal state. If so, it emits three metrics:
+ * - Total number of partitions with missing consuming segments including
+ * - Number of newly added partitions for which there's no consuming segment (there's no completed segment either)
+ * - Maximum duration (in minutes) that a partition hasn't had a consuming segment
+ */
+public class MissingConsumingSegmentFinder {
+ private static final Logger LOGGER = LoggerFactory.getLogger(MissingConsumingSegmentFinder.class);
+
+ private final String _realtimeTableName;
+ private final SegmentMetadataFetcher _segmentMetadataFetcher;
+ private final Map<Integer, StreamPartitionMsgOffset> _partitionGroupIdToLargestStreamOffsetMap;
+ private final StreamPartitionMsgOffsetFactory _streamPartitionMsgOffsetFactory;
+
+ private ControllerMetrics _controllerMetrics;
+
+ public MissingConsumingSegmentFinder(String realtimeTableName, ZkHelixPropertyStore<ZNRecord> propertyStore,
+ ControllerMetrics controllerMetrics, PartitionLevelStreamConfig streamConfig) {
+ _realtimeTableName = realtimeTableName;
+ _controllerMetrics = controllerMetrics;
+ _segmentMetadataFetcher = new SegmentMetadataFetcher(propertyStore, controllerMetrics);
+ _streamPartitionMsgOffsetFactory =
+ StreamConsumerFactoryProvider.create(streamConfig).createStreamMsgOffsetFactory();
+
+ // create partition group id to largest stream offset map
+ _partitionGroupIdToLargestStreamOffsetMap = new HashMap<>();
+ streamConfig.setOffsetCriteria(OffsetCriteria.LARGEST_OFFSET_CRITERIA);
+ try {
+ PinotTableIdealStateBuilder.getPartitionGroupMetadataList(streamConfig, Collections.emptyList())
+ .forEach(metadata -> {
+ _partitionGroupIdToLargestStreamOffsetMap.put(metadata.getPartitionGroupId(), metadata.getStartOffset());
+ });
+ } catch (Exception e) {
+ LOGGER.warn("Problem encountered in fetching stream metadata for topic: {} of table: {}. "
+ + "Continue finding missing consuming segment only with ideal state information.",
+ streamConfig.getTopicName(), streamConfig.getTableNameWithType());
+ }
+ }
+
+ @VisibleForTesting
+ MissingConsumingSegmentFinder(String realtimeTableName, SegmentMetadataFetcher segmentMetadataFetcher,
+ Map<Integer, StreamPartitionMsgOffset> partitionGroupIdToLargestStreamOffsetMap,
+ StreamPartitionMsgOffsetFactory streamPartitionMsgOffsetFactory) {
+ _realtimeTableName = realtimeTableName;
+ _segmentMetadataFetcher = segmentMetadataFetcher;
+ _partitionGroupIdToLargestStreamOffsetMap = partitionGroupIdToLargestStreamOffsetMap;
+ _streamPartitionMsgOffsetFactory = streamPartitionMsgOffsetFactory;
+ }
+
+ public void findAndEmitMetrics(IdealState idealState) {
+ MissingSegmentInfo info = findMissingSegments(idealState.getRecord().getMapFields(), Instant.now());
+ _controllerMetrics.setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_TOTAL_COUNT,
+ info._totalCount);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_NEW_PARTITION_COUNT,
+ info._newPartitionGroupCount);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_MAX_DURATION_MINUTES,
+ info._maxDurationInMinutes);
+ }
+
+ @VisibleForTesting
+ MissingSegmentInfo findMissingSegments(Map<String, Map<String, String>> idealStateMap, Instant now) {
+ // create the maps
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestConsumingSegmentMap = new HashMap<>();
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestCompletedSegmentMap = new HashMap<>();
+ idealStateMap.forEach((segmentName, instanceToStatusMap) -> {
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName != null) { // Skip the uploaded realtime segments that don't conform to llc naming
+ if (instanceToStatusMap.containsValue(SegmentStateModel.CONSUMING)) {
+ updateMap(partitionGroupIdToLatestConsumingSegmentMap, llcSegmentName);
+ } else if (instanceToStatusMap.containsValue(SegmentStateModel.ONLINE)) {
+ updateMap(partitionGroupIdToLatestCompletedSegmentMap, llcSegmentName);
+ }
+ }
+ });
+
+ MissingSegmentInfo missingSegmentInfo = new MissingSegmentInfo();
+ if (!_partitionGroupIdToLargestStreamOffsetMap.isEmpty()) {
+ _partitionGroupIdToLargestStreamOffsetMap.forEach((partitionGroupId, largestStreamOffset) -> {
+ if (!partitionGroupIdToLatestConsumingSegmentMap.containsKey(partitionGroupId)) {
+ LLCSegmentName latestCompletedSegment = partitionGroupIdToLatestCompletedSegmentMap.get(partitionGroupId);
+ if (latestCompletedSegment == null) {
+ // There's no consuming or completed segment for this partition group. Possibilities:
+ // 1) it's a new partition group that has not yet been detected
+ // 2) the first consuming segment has been deleted from ideal state manually
+ missingSegmentInfo._newPartitionGroupCount++;
+ missingSegmentInfo._totalCount++;
+ } else {
+ // completed segment is available, but there's no consuming segment
+ SegmentZKMetadata segmentZKMetadata = _segmentMetadataFetcher
+ .fetchSegmentZkMetadata(_realtimeTableName, latestCompletedSegment.getSegmentName());
+ StreamPartitionMsgOffset completedSegmentEndOffset =
+ _streamPartitionMsgOffsetFactory.create(segmentZKMetadata.getEndOffset());
+ if (completedSegmentEndOffset.compareTo(largestStreamOffset) < 0) {
+ // there are unconsumed messages available on the stream
+ missingSegmentInfo._totalCount++;
+ updateMaxDurationInfo(missingSegmentInfo, partitionGroupId, segmentZKMetadata.getCreationTime(), now);
Review Comment:
You should compare the mod time of the znode, not the creation time
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/MissingConsumingSegmentFinder.java:
##########
@@ -0,0 +1,221 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.helix.AccessOption;
+import org.apache.helix.ZNRecord;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+import org.apache.pinot.common.metadata.ZKMetadataProvider;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ControllerGauge;
+import org.apache.pinot.common.metrics.ControllerMeter;
+import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder;
+import org.apache.pinot.spi.stream.OffsetCriteria;
+import org.apache.pinot.spi.stream.PartitionLevelStreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * For a given table, this class finds out if there is any partition group for which there's no consuming segment in
+ * ideal state. If so, it emits three metrics:
+ * - Total number of partitions with missing consuming segments including
+ * - Number of newly added partitions for which there's no consuming segment (there's no completed segment either)
+ * - Maximum duration (in minutes) that a partition hasn't had a consuming segment
+ */
+public class MissingConsumingSegmentFinder {
+ private static final Logger LOGGER = LoggerFactory.getLogger(MissingConsumingSegmentFinder.class);
+
+ private final String _realtimeTableName;
+ private final SegmentMetadataFetcher _segmentMetadataFetcher;
+ private final Map<Integer, StreamPartitionMsgOffset> _partitionGroupIdToLargestStreamOffsetMap;
+ private final StreamPartitionMsgOffsetFactory _streamPartitionMsgOffsetFactory;
+
+ private ControllerMetrics _controllerMetrics;
+
+ public MissingConsumingSegmentFinder(String realtimeTableName, ZkHelixPropertyStore<ZNRecord> propertyStore,
+ ControllerMetrics controllerMetrics, PartitionLevelStreamConfig streamConfig) {
+ _realtimeTableName = realtimeTableName;
+ _controllerMetrics = controllerMetrics;
+ _segmentMetadataFetcher = new SegmentMetadataFetcher(propertyStore, controllerMetrics);
+ _streamPartitionMsgOffsetFactory =
+ StreamConsumerFactoryProvider.create(streamConfig).createStreamMsgOffsetFactory();
+
+ // create partition group id to largest stream offset map
+ _partitionGroupIdToLargestStreamOffsetMap = new HashMap<>();
+ streamConfig.setOffsetCriteria(OffsetCriteria.LARGEST_OFFSET_CRITERIA);
+ try {
+ PinotTableIdealStateBuilder.getPartitionGroupMetadataList(streamConfig, Collections.emptyList())
+ .forEach(metadata -> {
+ _partitionGroupIdToLargestStreamOffsetMap.put(metadata.getPartitionGroupId(), metadata.getStartOffset());
+ });
+ } catch (Exception e) {
+ LOGGER.warn("Problem encountered in fetching stream metadata for topic: {} of table: {}. "
+ + "Continue finding missing consuming segment only with ideal state information.",
+ streamConfig.getTopicName(), streamConfig.getTableNameWithType());
+ }
+ }
+
+ @VisibleForTesting
+ MissingConsumingSegmentFinder(String realtimeTableName, SegmentMetadataFetcher segmentMetadataFetcher,
+ Map<Integer, StreamPartitionMsgOffset> partitionGroupIdToLargestStreamOffsetMap,
+ StreamPartitionMsgOffsetFactory streamPartitionMsgOffsetFactory) {
+ _realtimeTableName = realtimeTableName;
+ _segmentMetadataFetcher = segmentMetadataFetcher;
+ _partitionGroupIdToLargestStreamOffsetMap = partitionGroupIdToLargestStreamOffsetMap;
+ _streamPartitionMsgOffsetFactory = streamPartitionMsgOffsetFactory;
+ }
+
+ public void findAndEmitMetrics(IdealState idealState) {
+ MissingSegmentInfo info = findMissingSegments(idealState.getRecord().getMapFields(), Instant.now());
+ _controllerMetrics.setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_TOTAL_COUNT,
+ info._totalCount);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_NEW_PARTITION_COUNT,
+ info._newPartitionGroupCount);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_MAX_DURATION_MINUTES,
+ info._maxDurationInMinutes);
+ }
+
+ @VisibleForTesting
+ MissingSegmentInfo findMissingSegments(Map<String, Map<String, String>> idealStateMap, Instant now) {
+ // create the maps
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestConsumingSegmentMap = new HashMap<>();
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestCompletedSegmentMap = new HashMap<>();
+ idealStateMap.forEach((segmentName, instanceToStatusMap) -> {
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName != null) { // Skip the uploaded realtime segments that don't conform to llc naming
+ if (instanceToStatusMap.containsValue(SegmentStateModel.CONSUMING)) {
+ updateMap(partitionGroupIdToLatestConsumingSegmentMap, llcSegmentName);
+ } else if (instanceToStatusMap.containsValue(SegmentStateModel.ONLINE)) {
+ updateMap(partitionGroupIdToLatestCompletedSegmentMap, llcSegmentName);
+ }
+ }
+ });
+
+ MissingSegmentInfo missingSegmentInfo = new MissingSegmentInfo();
+ if (!_partitionGroupIdToLargestStreamOffsetMap.isEmpty()) {
+ _partitionGroupIdToLargestStreamOffsetMap.forEach((partitionGroupId, largestStreamOffset) -> {
+ if (!partitionGroupIdToLatestConsumingSegmentMap.containsKey(partitionGroupId)) {
+ LLCSegmentName latestCompletedSegment = partitionGroupIdToLatestCompletedSegmentMap.get(partitionGroupId);
+ if (latestCompletedSegment == null) {
+ // There's no consuming or completed segment for this partition group. Possibilities:
+ // 1) it's a new partition group that has not yet been detected
+ // 2) the first consuming segment has been deleted from ideal state manually
+ missingSegmentInfo._newPartitionGroupCount++;
+ missingSegmentInfo._totalCount++;
+ } else {
+ // completed segment is available, but there's no consuming segment
+ SegmentZKMetadata segmentZKMetadata = _segmentMetadataFetcher
+ .fetchSegmentZkMetadata(_realtimeTableName, latestCompletedSegment.getSegmentName());
+ StreamPartitionMsgOffset completedSegmentEndOffset =
+ _streamPartitionMsgOffsetFactory.create(segmentZKMetadata.getEndOffset());
+ if (completedSegmentEndOffset.compareTo(largestStreamOffset) < 0) {
+ // there are unconsumed messages available on the stream
+ missingSegmentInfo._totalCount++;
+ updateMaxDurationInfo(missingSegmentInfo, partitionGroupId, segmentZKMetadata.getCreationTime(), now);
+ }
+ }
+ }
+ });
+ } else {
+ partitionGroupIdToLatestCompletedSegmentMap.forEach((partitionGroupId, latestCompletedSegment) -> {
+ if (!partitionGroupIdToLatestConsumingSegmentMap.containsKey(partitionGroupId)) {
+ missingSegmentInfo._totalCount++;
+ long segmentCompletionTimeMillis = _segmentMetadataFetcher
+ .fetchSegmentCreationTime(_realtimeTableName, latestCompletedSegment.getSegmentName());
+ updateMaxDurationInfo(missingSegmentInfo, partitionGroupId, segmentCompletionTimeMillis, now);
+ }
+ });
+ }
+ return missingSegmentInfo;
+ }
+
+ private void updateMaxDurationInfo(MissingSegmentInfo missingSegmentInfo, Integer partitionGroupId,
+ long segmentCompletionTimeMillis, Instant now) {
+ long duration = Duration.between(Instant.ofEpochMilli(segmentCompletionTimeMillis), now).toMinutes();
+ if (duration > missingSegmentInfo._maxDurationInMinutes) {
+ missingSegmentInfo._maxDurationInMinutes = duration;
+ }
+ LOGGER.warn("PartitionGroupId {} hasn't had a consuming segment for {} minutes!", partitionGroupId, duration);
+ }
+
+ private void updateMap(Map<Integer, LLCSegmentName> partitionGroupIdToLatestSegmentMap,
+ LLCSegmentName llcSegmentName) {
+ int partitionGroupId = llcSegmentName.getPartitionGroupId();
+ partitionGroupIdToLatestSegmentMap.compute(partitionGroupId, (pid, existingSegment) -> {
+ if (existingSegment == null) {
+ return llcSegmentName;
+ } else {
+ return existingSegment.getSequenceNumber() > llcSegmentName.getSequenceNumber() ? existingSegment
+ : llcSegmentName;
+ }
+ });
+ }
+
+ @VisibleForTesting
+ static class MissingSegmentInfo {
+ long _totalCount;
+ long _newPartitionGroupCount;
+ long _maxDurationInMinutes;
+ }
+
+ static class SegmentMetadataFetcher {
+ private ZkHelixPropertyStore<ZNRecord> _propertyStore;
+ private ControllerMetrics _controllerMetrics;
+
+ public SegmentMetadataFetcher(ZkHelixPropertyStore<ZNRecord> propertyStore, ControllerMetrics controllerMetrics) {
+ _propertyStore = propertyStore;
+ _controllerMetrics = controllerMetrics;
+ }
+
+ public SegmentZKMetadata fetchSegmentZkMetadata(String tableName, String segmentName) {
+ try {
+ ZNRecord znRecord = _propertyStore
+ .get(ZKMetadataProvider.constructPropertyStorePathForSegment(tableName, segmentName), null,
+ AccessOption.PERSISTENT);
+ Preconditions.checkState(znRecord != null, "Failed to find segment ZK metadata for segment: %s of table: %s",
+ segmentName, tableName);
+ return new SegmentZKMetadata(znRecord);
+ } catch (Exception e) {
+ _controllerMetrics.addMeteredTableValue(tableName, ControllerMeter.LLC_ZOOKEEPER_FETCH_FAILURES, 1L);
+ throw e;
+ }
+ }
+
+ public long fetchSegmentCreationTime(String tableName, String segmentName) {
+ return fetchSegmentZkMetadata(tableName, segmentName).getCreationTime();
Review Comment:
need mod time of znode
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/MissingConsumingSegmentFinder.java:
##########
@@ -0,0 +1,221 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.helix.AccessOption;
+import org.apache.helix.ZNRecord;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+import org.apache.pinot.common.metadata.ZKMetadataProvider;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ControllerGauge;
+import org.apache.pinot.common.metrics.ControllerMeter;
+import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder;
+import org.apache.pinot.spi.stream.OffsetCriteria;
+import org.apache.pinot.spi.stream.PartitionLevelStreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * For a given table, this class finds out if there is any partition group for which there's no consuming segment in
+ * ideal state. If so, it emits three metrics:
+ * - Total number of partitions with missing consuming segments including
+ * - Number of newly added partitions for which there's no consuming segment (there's no completed segment either)
+ * - Maximum duration (in minutes) that a partition hasn't had a consuming segment
+ */
+public class MissingConsumingSegmentFinder {
+ private static final Logger LOGGER = LoggerFactory.getLogger(MissingConsumingSegmentFinder.class);
+
+ private final String _realtimeTableName;
+ private final SegmentMetadataFetcher _segmentMetadataFetcher;
+ private final Map<Integer, StreamPartitionMsgOffset> _partitionGroupIdToLargestStreamOffsetMap;
+ private final StreamPartitionMsgOffsetFactory _streamPartitionMsgOffsetFactory;
+
+ private ControllerMetrics _controllerMetrics;
+
+ public MissingConsumingSegmentFinder(String realtimeTableName, ZkHelixPropertyStore<ZNRecord> propertyStore,
+ ControllerMetrics controllerMetrics, PartitionLevelStreamConfig streamConfig) {
+ _realtimeTableName = realtimeTableName;
+ _controllerMetrics = controllerMetrics;
+ _segmentMetadataFetcher = new SegmentMetadataFetcher(propertyStore, controllerMetrics);
+ _streamPartitionMsgOffsetFactory =
+ StreamConsumerFactoryProvider.create(streamConfig).createStreamMsgOffsetFactory();
+
+ // create partition group id to largest stream offset map
+ _partitionGroupIdToLargestStreamOffsetMap = new HashMap<>();
+ streamConfig.setOffsetCriteria(OffsetCriteria.LARGEST_OFFSET_CRITERIA);
+ try {
+ PinotTableIdealStateBuilder.getPartitionGroupMetadataList(streamConfig, Collections.emptyList())
+ .forEach(metadata -> {
+ _partitionGroupIdToLargestStreamOffsetMap.put(metadata.getPartitionGroupId(), metadata.getStartOffset());
+ });
+ } catch (Exception e) {
+ LOGGER.warn("Problem encountered in fetching stream metadata for topic: {} of table: {}. "
+ + "Continue finding missing consuming segment only with ideal state information.",
+ streamConfig.getTopicName(), streamConfig.getTableNameWithType());
+ }
+ }
+
+ @VisibleForTesting
+ MissingConsumingSegmentFinder(String realtimeTableName, SegmentMetadataFetcher segmentMetadataFetcher,
+ Map<Integer, StreamPartitionMsgOffset> partitionGroupIdToLargestStreamOffsetMap,
+ StreamPartitionMsgOffsetFactory streamPartitionMsgOffsetFactory) {
+ _realtimeTableName = realtimeTableName;
+ _segmentMetadataFetcher = segmentMetadataFetcher;
+ _partitionGroupIdToLargestStreamOffsetMap = partitionGroupIdToLargestStreamOffsetMap;
+ _streamPartitionMsgOffsetFactory = streamPartitionMsgOffsetFactory;
+ }
+
+ public void findAndEmitMetrics(IdealState idealState) {
+ MissingSegmentInfo info = findMissingSegments(idealState.getRecord().getMapFields(), Instant.now());
+ _controllerMetrics.setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_TOTAL_COUNT,
+ info._totalCount);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_NEW_PARTITION_COUNT,
+ info._newPartitionGroupCount);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_MAX_DURATION_MINUTES,
+ info._maxDurationInMinutes);
+ }
+
+ @VisibleForTesting
+ MissingSegmentInfo findMissingSegments(Map<String, Map<String, String>> idealStateMap, Instant now) {
+ // create the maps
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestConsumingSegmentMap = new HashMap<>();
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestCompletedSegmentMap = new HashMap<>();
+ idealStateMap.forEach((segmentName, instanceToStatusMap) -> {
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName != null) { // Skip the uploaded realtime segments that don't conform to llc naming
+ if (instanceToStatusMap.containsValue(SegmentStateModel.CONSUMING)) {
+ updateMap(partitionGroupIdToLatestConsumingSegmentMap, llcSegmentName);
+ } else if (instanceToStatusMap.containsValue(SegmentStateModel.ONLINE)) {
+ updateMap(partitionGroupIdToLatestCompletedSegmentMap, llcSegmentName);
+ }
+ }
+ });
+
+ MissingSegmentInfo missingSegmentInfo = new MissingSegmentInfo();
+ if (!_partitionGroupIdToLargestStreamOffsetMap.isEmpty()) {
+ _partitionGroupIdToLargestStreamOffsetMap.forEach((partitionGroupId, largestStreamOffset) -> {
+ if (!partitionGroupIdToLatestConsumingSegmentMap.containsKey(partitionGroupId)) {
+ LLCSegmentName latestCompletedSegment = partitionGroupIdToLatestCompletedSegmentMap.get(partitionGroupId);
+ if (latestCompletedSegment == null) {
+ // There's no consuming or completed segment for this partition group. Possibilities:
+ // 1) it's a new partition group that has not yet been detected
+ // 2) the first consuming segment has been deleted from ideal state manually
+ missingSegmentInfo._newPartitionGroupCount++;
+ missingSegmentInfo._totalCount++;
+ } else {
+ // completed segment is available, but there's no consuming segment
+ SegmentZKMetadata segmentZKMetadata = _segmentMetadataFetcher
+ .fetchSegmentZkMetadata(_realtimeTableName, latestCompletedSegment.getSegmentName());
+ StreamPartitionMsgOffset completedSegmentEndOffset =
+ _streamPartitionMsgOffsetFactory.create(segmentZKMetadata.getEndOffset());
+ if (completedSegmentEndOffset.compareTo(largestStreamOffset) < 0) {
Review Comment:
Is this the only way we know that a partition group is closed? Is there a corner case that if the partitionGroup does not produce any new events after we completed the last segment, but we did not create a consuming segment (which we should have) due to some reason, then we will never flag it? Since you are taking the trouble of calling the stream, is there an API that returns ONLY active partitionGroups ?(maybe with their offsets)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on PR #8877:
URL: https://github.com/apache/pinot/pull/8877#issuecomment-1154380690
We should move this to SegmentStatusChecker, that is run more frequently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on a diff in pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
npawar commented on code in PR #8877:
URL: https://github.com/apache/pinot/pull/8877#discussion_r894908251
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/MissingConsumingSegmentFinder.java:
##########
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.helix.AccessOption;
+import org.apache.helix.ZNRecord;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+import org.apache.pinot.common.metadata.ZKMetadataProvider;
+import org.apache.pinot.common.metrics.ControllerGauge;
+import org.apache.pinot.common.metrics.ControllerMeter;
+import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.apache.zookeeper.data.Stat;
+
+
+/**
+ * For a given table, this class finds out if there is any partition group for which there's no consuming segment in
+ * ideal state. If so, it emits two metrics:
+ * - Number of missing consuming segments
+ * - Maximum duration (in minutes) that a partition hasn't had a consuming segment
+ */
+public class MissingConsumingSegmentFinder {
+
+ private String _realtimeTableName;
+ private ControllerMetrics _controllerMetrics;
+ private SegmentMetadataFetcher _segmentMetadataFetcher;
+
+ public MissingConsumingSegmentFinder(String realtimeTableName, ZkHelixPropertyStore<ZNRecord> propertyStore,
+ ControllerMetrics controllerMetrics) {
+ _realtimeTableName = realtimeTableName;
+ _controllerMetrics = controllerMetrics;
+ _segmentMetadataFetcher = new SegmentMetadataFetcher(propertyStore, controllerMetrics);
+ }
+
+ @VisibleForTesting
+ MissingConsumingSegmentFinder(String realtimeTableName, SegmentMetadataFetcher segmentMetadataFetcher) {
+ _realtimeTableName = realtimeTableName;
+ _segmentMetadataFetcher = segmentMetadataFetcher;
+ }
+
+ public void findAndEmitMetrics(IdealState idealState) {
+ MissingSegmentInfo info = findMissingSegments(idealState.getRecord().getMapFields(), Instant.now());
+ _controllerMetrics.setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_COUNT,
+ info._count);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_MAX_DURATION_MINUTES,
+ info._maxDurationInMinutes);
+ }
+
+ @VisibleForTesting
+ MissingSegmentInfo findMissingSegments(Map<String, Map<String, String>> idealStateMap, Instant now) {
+ // create the maps
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestConsumingSegmentMap = new HashMap<>();
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestCompletedSegmentMap = new HashMap<>();
+ idealStateMap.forEach((segmentName, instanceToStatusMap) -> {
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName != null) { // Skip the uploaded realtime segments that don't conform to llc naming
+ if (instanceToStatusMap.containsValue(SegmentStateModel.CONSUMING)) {
+ updateMap(partitionGroupIdToLatestConsumingSegmentMap, llcSegmentName);
Review Comment:
have you on purpose omitted the cases where there's no CONSUMING segment, but the ValidationManager will be fixing it?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/MissingConsumingSegmentFinder.java:
##########
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.helix.AccessOption;
+import org.apache.helix.ZNRecord;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+import org.apache.pinot.common.metadata.ZKMetadataProvider;
+import org.apache.pinot.common.metrics.ControllerGauge;
+import org.apache.pinot.common.metrics.ControllerMeter;
+import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.apache.zookeeper.data.Stat;
+
+
+/**
+ * For a given table, this class finds out if there is any partition group for which there's no consuming segment in
+ * ideal state. If so, it emits two metrics:
+ * - Number of missing consuming segments
+ * - Maximum duration (in minutes) that a partition hasn't had a consuming segment
+ */
+public class MissingConsumingSegmentFinder {
+
+ private String _realtimeTableName;
+ private ControllerMetrics _controllerMetrics;
+ private SegmentMetadataFetcher _segmentMetadataFetcher;
+
+ public MissingConsumingSegmentFinder(String realtimeTableName, ZkHelixPropertyStore<ZNRecord> propertyStore,
+ ControllerMetrics controllerMetrics) {
+ _realtimeTableName = realtimeTableName;
+ _controllerMetrics = controllerMetrics;
+ _segmentMetadataFetcher = new SegmentMetadataFetcher(propertyStore, controllerMetrics);
+ }
+
+ @VisibleForTesting
+ MissingConsumingSegmentFinder(String realtimeTableName, SegmentMetadataFetcher segmentMetadataFetcher) {
+ _realtimeTableName = realtimeTableName;
+ _segmentMetadataFetcher = segmentMetadataFetcher;
+ }
+
+ public void findAndEmitMetrics(IdealState idealState) {
+ MissingSegmentInfo info = findMissingSegments(idealState.getRecord().getMapFields(), Instant.now());
+ _controllerMetrics.setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_COUNT,
+ info._count);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_MAX_DURATION_MINUTES,
+ info._maxDurationInMinutes);
+ }
+
+ @VisibleForTesting
+ MissingSegmentInfo findMissingSegments(Map<String, Map<String, String>> idealStateMap, Instant now) {
+ // create the maps
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestConsumingSegmentMap = new HashMap<>();
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestCompletedSegmentMap = new HashMap<>();
+ idealStateMap.forEach((segmentName, instanceToStatusMap) -> {
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName != null) { // Skip the uploaded realtime segments that don't conform to llc naming
+ if (instanceToStatusMap.containsValue(SegmentStateModel.CONSUMING)) {
+ updateMap(partitionGroupIdToLatestConsumingSegmentMap, llcSegmentName);
Review Comment:
Like 1) OFFLINE state segment, or 2) when it was the very first segment that was deleted, hence no completed segment recorded 3) new partition
I feel this should be a generic metric which says how many are not CONSUMING, and be oblivious to what is correctable ad what's not
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -874,6 +871,8 @@ public void ensureAllPartitionsConsuming(TableConfig tableConfig, PartitionLevel
HelixHelper.updateIdealState(_helixManager, realtimeTableName, idealState -> {
assert idealState != null;
if (idealState.isEnabled()) {
+ new MissingConsumingSegmentFinder(realtimeTableName, _propertyStore, _controllerMetrics)
Review Comment:
should this be run after the validation manager is done with the current round? Bit confusing that we first try to emit for what's missing and then immediately proceed to fixing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on PR #8877:
URL: https://github.com/apache/pinot/pull/8877#issuecomment-1156815198
IIRC, it is normal for certain stream type (e.g. Kinesis) to have no consuming segment for some partitions. cc @npawar @KKcorps to confirm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
sajjad-moradi commented on code in PR #8877:
URL: https://github.com/apache/pinot/pull/8877#discussion_r907629889
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/MissingConsumingSegmentFinder.java:
##########
@@ -0,0 +1,221 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.helix.AccessOption;
+import org.apache.helix.ZNRecord;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+import org.apache.pinot.common.metadata.ZKMetadataProvider;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ControllerGauge;
+import org.apache.pinot.common.metrics.ControllerMeter;
+import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder;
+import org.apache.pinot.spi.stream.OffsetCriteria;
+import org.apache.pinot.spi.stream.PartitionLevelStreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * For a given table, this class finds out if there is any partition group for which there's no consuming segment in
+ * ideal state. If so, it emits three metrics:
+ * - Total number of partitions with missing consuming segments including
+ * - Number of newly added partitions for which there's no consuming segment (there's no completed segment either)
+ * - Maximum duration (in minutes) that a partition hasn't had a consuming segment
+ */
+public class MissingConsumingSegmentFinder {
+ private static final Logger LOGGER = LoggerFactory.getLogger(MissingConsumingSegmentFinder.class);
+
+ private final String _realtimeTableName;
+ private final SegmentMetadataFetcher _segmentMetadataFetcher;
+ private final Map<Integer, StreamPartitionMsgOffset> _partitionGroupIdToLargestStreamOffsetMap;
+ private final StreamPartitionMsgOffsetFactory _streamPartitionMsgOffsetFactory;
+
+ private ControllerMetrics _controllerMetrics;
+
+ public MissingConsumingSegmentFinder(String realtimeTableName, ZkHelixPropertyStore<ZNRecord> propertyStore,
+ ControllerMetrics controllerMetrics, PartitionLevelStreamConfig streamConfig) {
+ _realtimeTableName = realtimeTableName;
+ _controllerMetrics = controllerMetrics;
+ _segmentMetadataFetcher = new SegmentMetadataFetcher(propertyStore, controllerMetrics);
+ _streamPartitionMsgOffsetFactory =
+ StreamConsumerFactoryProvider.create(streamConfig).createStreamMsgOffsetFactory();
+
+ // create partition group id to largest stream offset map
+ _partitionGroupIdToLargestStreamOffsetMap = new HashMap<>();
+ streamConfig.setOffsetCriteria(OffsetCriteria.LARGEST_OFFSET_CRITERIA);
+ try {
+ PinotTableIdealStateBuilder.getPartitionGroupMetadataList(streamConfig, Collections.emptyList())
+ .forEach(metadata -> {
+ _partitionGroupIdToLargestStreamOffsetMap.put(metadata.getPartitionGroupId(), metadata.getStartOffset());
+ });
+ } catch (Exception e) {
+ LOGGER.warn("Problem encountered in fetching stream metadata for topic: {} of table: {}. "
+ + "Continue finding missing consuming segment only with ideal state information.",
+ streamConfig.getTopicName(), streamConfig.getTableNameWithType());
+ }
+ }
+
+ @VisibleForTesting
+ MissingConsumingSegmentFinder(String realtimeTableName, SegmentMetadataFetcher segmentMetadataFetcher,
+ Map<Integer, StreamPartitionMsgOffset> partitionGroupIdToLargestStreamOffsetMap,
+ StreamPartitionMsgOffsetFactory streamPartitionMsgOffsetFactory) {
+ _realtimeTableName = realtimeTableName;
+ _segmentMetadataFetcher = segmentMetadataFetcher;
+ _partitionGroupIdToLargestStreamOffsetMap = partitionGroupIdToLargestStreamOffsetMap;
+ _streamPartitionMsgOffsetFactory = streamPartitionMsgOffsetFactory;
+ }
+
+ public void findAndEmitMetrics(IdealState idealState) {
+ MissingSegmentInfo info = findMissingSegments(idealState.getRecord().getMapFields(), Instant.now());
+ _controllerMetrics.setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_TOTAL_COUNT,
+ info._totalCount);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_NEW_PARTITION_COUNT,
+ info._newPartitionGroupCount);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_MAX_DURATION_MINUTES,
+ info._maxDurationInMinutes);
+ }
+
+ @VisibleForTesting
+ MissingSegmentInfo findMissingSegments(Map<String, Map<String, String>> idealStateMap, Instant now) {
+ // create the maps
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestConsumingSegmentMap = new HashMap<>();
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestCompletedSegmentMap = new HashMap<>();
+ idealStateMap.forEach((segmentName, instanceToStatusMap) -> {
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName != null) { // Skip the uploaded realtime segments that don't conform to llc naming
+ if (instanceToStatusMap.containsValue(SegmentStateModel.CONSUMING)) {
+ updateMap(partitionGroupIdToLatestConsumingSegmentMap, llcSegmentName);
+ } else if (instanceToStatusMap.containsValue(SegmentStateModel.ONLINE)) {
+ updateMap(partitionGroupIdToLatestCompletedSegmentMap, llcSegmentName);
+ }
+ }
+ });
+
+ MissingSegmentInfo missingSegmentInfo = new MissingSegmentInfo();
+ if (!_partitionGroupIdToLargestStreamOffsetMap.isEmpty()) {
+ _partitionGroupIdToLargestStreamOffsetMap.forEach((partitionGroupId, largestStreamOffset) -> {
+ if (!partitionGroupIdToLatestConsumingSegmentMap.containsKey(partitionGroupId)) {
+ LLCSegmentName latestCompletedSegment = partitionGroupIdToLatestCompletedSegmentMap.get(partitionGroupId);
+ if (latestCompletedSegment == null) {
+ // There's no consuming or completed segment for this partition group. Possibilities:
+ // 1) it's a new partition group that has not yet been detected
+ // 2) the first consuming segment has been deleted from ideal state manually
+ missingSegmentInfo._newPartitionGroupCount++;
+ missingSegmentInfo._totalCount++;
+ } else {
+ // completed segment is available, but there's no consuming segment
+ SegmentZKMetadata segmentZKMetadata = _segmentMetadataFetcher
+ .fetchSegmentZkMetadata(_realtimeTableName, latestCompletedSegment.getSegmentName());
+ StreamPartitionMsgOffset completedSegmentEndOffset =
+ _streamPartitionMsgOffsetFactory.create(segmentZKMetadata.getEndOffset());
+ if (completedSegmentEndOffset.compareTo(largestStreamOffset) < 0) {
Review Comment:
I think it's ok if we don't flag in this case because there's no message available in the stream to consume. As soon as the messages become available, the next run of the periodic task will find out the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on PR #8877:
URL: https://github.com/apache/pinot/pull/8877#issuecomment-1158163964
> IIRC, it is normal for certain stream type (e.g. Kinesis) to have no consuming segment for some partitions. cc @npawar @KKcorps to confirm
True. Partitions have a concept of EOL in Kinesis. We need to consider that before declaring that there is no consuming segment for the partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
sajjad-moradi commented on code in PR #8877:
URL: https://github.com/apache/pinot/pull/8877#discussion_r896232278
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -874,6 +871,8 @@ public void ensureAllPartitionsConsuming(TableConfig tableConfig, PartitionLevel
HelixHelper.updateIdealState(_helixManager, realtimeTableName, idealState -> {
assert idealState != null;
if (idealState.isEnabled()) {
+ new MissingConsumingSegmentFinder(realtimeTableName, _propertyStore, _controllerMetrics)
Review Comment:
The reason it was at the beginning of the validation manager was to avoid stream connection issues. One root cause for not having consuming segments in ideal state is that stream client can throw exception during stream metadata for creating new consuming segment either in 1) segment commit or in 2) validation manager for detecting new partition groups. By just looking at ideal state, we can detect for how long we haven't had a consuming segment for a partition.
Anyway I moved this SegmentStatusChecker job per Subbu's suggestion, so it runs more frequently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on a diff in pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on code in PR #8877:
URL: https://github.com/apache/pinot/pull/8877#discussion_r907638311
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/MissingConsumingSegmentFinder.java:
##########
@@ -0,0 +1,221 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.helix.AccessOption;
+import org.apache.helix.ZNRecord;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+import org.apache.pinot.common.metadata.ZKMetadataProvider;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ControllerGauge;
+import org.apache.pinot.common.metrics.ControllerMeter;
+import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder;
+import org.apache.pinot.spi.stream.OffsetCriteria;
+import org.apache.pinot.spi.stream.PartitionLevelStreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * For a given table, this class finds out if there is any partition group for which there's no consuming segment in
+ * ideal state. If so, it emits three metrics:
+ * - Total number of partitions with missing consuming segments including
+ * - Number of newly added partitions for which there's no consuming segment (there's no completed segment either)
+ * - Maximum duration (in minutes) that a partition hasn't had a consuming segment
+ */
+public class MissingConsumingSegmentFinder {
+ private static final Logger LOGGER = LoggerFactory.getLogger(MissingConsumingSegmentFinder.class);
+
+ private final String _realtimeTableName;
+ private final SegmentMetadataFetcher _segmentMetadataFetcher;
+ private final Map<Integer, StreamPartitionMsgOffset> _partitionGroupIdToLargestStreamOffsetMap;
+ private final StreamPartitionMsgOffsetFactory _streamPartitionMsgOffsetFactory;
+
+ private ControllerMetrics _controllerMetrics;
+
+ public MissingConsumingSegmentFinder(String realtimeTableName, ZkHelixPropertyStore<ZNRecord> propertyStore,
+ ControllerMetrics controllerMetrics, PartitionLevelStreamConfig streamConfig) {
+ _realtimeTableName = realtimeTableName;
+ _controllerMetrics = controllerMetrics;
+ _segmentMetadataFetcher = new SegmentMetadataFetcher(propertyStore, controllerMetrics);
+ _streamPartitionMsgOffsetFactory =
+ StreamConsumerFactoryProvider.create(streamConfig).createStreamMsgOffsetFactory();
+
+ // create partition group id to largest stream offset map
+ _partitionGroupIdToLargestStreamOffsetMap = new HashMap<>();
+ streamConfig.setOffsetCriteria(OffsetCriteria.LARGEST_OFFSET_CRITERIA);
+ try {
+ PinotTableIdealStateBuilder.getPartitionGroupMetadataList(streamConfig, Collections.emptyList())
+ .forEach(metadata -> {
+ _partitionGroupIdToLargestStreamOffsetMap.put(metadata.getPartitionGroupId(), metadata.getStartOffset());
+ });
+ } catch (Exception e) {
+ LOGGER.warn("Problem encountered in fetching stream metadata for topic: {} of table: {}. "
+ + "Continue finding missing consuming segment only with ideal state information.",
+ streamConfig.getTopicName(), streamConfig.getTableNameWithType());
+ }
+ }
+
+ @VisibleForTesting
+ MissingConsumingSegmentFinder(String realtimeTableName, SegmentMetadataFetcher segmentMetadataFetcher,
+ Map<Integer, StreamPartitionMsgOffset> partitionGroupIdToLargestStreamOffsetMap,
+ StreamPartitionMsgOffsetFactory streamPartitionMsgOffsetFactory) {
+ _realtimeTableName = realtimeTableName;
+ _segmentMetadataFetcher = segmentMetadataFetcher;
+ _partitionGroupIdToLargestStreamOffsetMap = partitionGroupIdToLargestStreamOffsetMap;
+ _streamPartitionMsgOffsetFactory = streamPartitionMsgOffsetFactory;
+ }
+
+ public void findAndEmitMetrics(IdealState idealState) {
+ MissingSegmentInfo info = findMissingSegments(idealState.getRecord().getMapFields(), Instant.now());
+ _controllerMetrics.setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_TOTAL_COUNT,
+ info._totalCount);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_NEW_PARTITION_COUNT,
+ info._newPartitionGroupCount);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_MAX_DURATION_MINUTES,
+ info._maxDurationInMinutes);
+ }
+
+ @VisibleForTesting
+ MissingSegmentInfo findMissingSegments(Map<String, Map<String, String>> idealStateMap, Instant now) {
+ // create the maps
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestConsumingSegmentMap = new HashMap<>();
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestCompletedSegmentMap = new HashMap<>();
+ idealStateMap.forEach((segmentName, instanceToStatusMap) -> {
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName != null) { // Skip the uploaded realtime segments that don't conform to llc naming
+ if (instanceToStatusMap.containsValue(SegmentStateModel.CONSUMING)) {
+ updateMap(partitionGroupIdToLatestConsumingSegmentMap, llcSegmentName);
+ } else if (instanceToStatusMap.containsValue(SegmentStateModel.ONLINE)) {
+ updateMap(partitionGroupIdToLatestCompletedSegmentMap, llcSegmentName);
+ }
+ }
+ });
+
+ MissingSegmentInfo missingSegmentInfo = new MissingSegmentInfo();
+ if (!_partitionGroupIdToLargestStreamOffsetMap.isEmpty()) {
+ _partitionGroupIdToLargestStreamOffsetMap.forEach((partitionGroupId, largestStreamOffset) -> {
+ if (!partitionGroupIdToLatestConsumingSegmentMap.containsKey(partitionGroupId)) {
+ LLCSegmentName latestCompletedSegment = partitionGroupIdToLatestCompletedSegmentMap.get(partitionGroupId);
+ if (latestCompletedSegment == null) {
+ // There's no consuming or completed segment for this partition group. Possibilities:
+ // 1) it's a new partition group that has not yet been detected
+ // 2) the first consuming segment has been deleted from ideal state manually
+ missingSegmentInfo._newPartitionGroupCount++;
+ missingSegmentInfo._totalCount++;
+ } else {
+ // completed segment is available, but there's no consuming segment
+ SegmentZKMetadata segmentZKMetadata = _segmentMetadataFetcher
+ .fetchSegmentZkMetadata(_realtimeTableName, latestCompletedSegment.getSegmentName());
+ StreamPartitionMsgOffset completedSegmentEndOffset =
+ _streamPartitionMsgOffsetFactory.create(segmentZKMetadata.getEndOffset());
+ if (completedSegmentEndOffset.compareTo(largestStreamOffset) < 0) {
Review Comment:
I understand. I was more intrigued that there is no way to return only live partitions from the underlying stream. @npawar is this the case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on a diff in pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
npawar commented on code in PR #8877:
URL: https://github.com/apache/pinot/pull/8877#discussion_r896260018
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/MissingConsumingSegmentFinder.java:
##########
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.helix.AccessOption;
+import org.apache.helix.ZNRecord;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+import org.apache.pinot.common.metadata.ZKMetadataProvider;
+import org.apache.pinot.common.metrics.ControllerGauge;
+import org.apache.pinot.common.metrics.ControllerMeter;
+import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.apache.zookeeper.data.Stat;
+
+
+/**
+ * For a given table, this class finds out if there is any partition group for which there's no consuming segment in
+ * ideal state. If so, it emits two metrics:
+ * - Number of missing consuming segments
+ * - Maximum duration (in minutes) that a partition hasn't had a consuming segment
+ */
+public class MissingConsumingSegmentFinder {
+
+ private String _realtimeTableName;
+ private ControllerMetrics _controllerMetrics;
+ private SegmentMetadataFetcher _segmentMetadataFetcher;
+
+ public MissingConsumingSegmentFinder(String realtimeTableName, ZkHelixPropertyStore<ZNRecord> propertyStore,
+ ControllerMetrics controllerMetrics) {
+ _realtimeTableName = realtimeTableName;
+ _controllerMetrics = controllerMetrics;
+ _segmentMetadataFetcher = new SegmentMetadataFetcher(propertyStore, controllerMetrics);
+ }
+
+ @VisibleForTesting
+ MissingConsumingSegmentFinder(String realtimeTableName, SegmentMetadataFetcher segmentMetadataFetcher) {
+ _realtimeTableName = realtimeTableName;
+ _segmentMetadataFetcher = segmentMetadataFetcher;
+ }
+
+ public void findAndEmitMetrics(IdealState idealState) {
+ MissingSegmentInfo info = findMissingSegments(idealState.getRecord().getMapFields(), Instant.now());
+ _controllerMetrics.setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_COUNT,
+ info._count);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_MAX_DURATION_MINUTES,
+ info._maxDurationInMinutes);
+ }
+
+ @VisibleForTesting
+ MissingSegmentInfo findMissingSegments(Map<String, Map<String, String>> idealStateMap, Instant now) {
+ // create the maps
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestConsumingSegmentMap = new HashMap<>();
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestCompletedSegmentMap = new HashMap<>();
+ idealStateMap.forEach((segmentName, instanceToStatusMap) -> {
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName != null) { // Skip the uploaded realtime segments that don't conform to llc naming
+ if (instanceToStatusMap.containsValue(SegmentStateModel.CONSUMING)) {
+ updateMap(partitionGroupIdToLatestConsumingSegmentMap, llcSegmentName);
Review Comment:
`So for these cases, the last consumed time is when the previous completed segment went to ONLINE state.` - I think this misses the case when the very first segment itself is OFFLINE.
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/MissingConsumingSegmentFinder.java:
##########
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.helix.AccessOption;
+import org.apache.helix.ZNRecord;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+import org.apache.pinot.common.metadata.ZKMetadataProvider;
+import org.apache.pinot.common.metrics.ControllerGauge;
+import org.apache.pinot.common.metrics.ControllerMeter;
+import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.apache.zookeeper.data.Stat;
+
+
+/**
+ * For a given table, this class finds out if there is any partition group for which there's no consuming segment in
+ * ideal state. If so, it emits two metrics:
+ * - Number of missing consuming segments
+ * - Maximum duration (in minutes) that a partition hasn't had a consuming segment
+ */
+public class MissingConsumingSegmentFinder {
+
+ private String _realtimeTableName;
+ private ControllerMetrics _controllerMetrics;
+ private SegmentMetadataFetcher _segmentMetadataFetcher;
+
+ public MissingConsumingSegmentFinder(String realtimeTableName, ZkHelixPropertyStore<ZNRecord> propertyStore,
+ ControllerMetrics controllerMetrics) {
+ _realtimeTableName = realtimeTableName;
+ _controllerMetrics = controllerMetrics;
+ _segmentMetadataFetcher = new SegmentMetadataFetcher(propertyStore, controllerMetrics);
+ }
+
+ @VisibleForTesting
+ MissingConsumingSegmentFinder(String realtimeTableName, SegmentMetadataFetcher segmentMetadataFetcher) {
+ _realtimeTableName = realtimeTableName;
+ _segmentMetadataFetcher = segmentMetadataFetcher;
+ }
+
+ public void findAndEmitMetrics(IdealState idealState) {
+ MissingSegmentInfo info = findMissingSegments(idealState.getRecord().getMapFields(), Instant.now());
+ _controllerMetrics.setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_COUNT,
+ info._count);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_MAX_DURATION_MINUTES,
+ info._maxDurationInMinutes);
+ }
+
+ @VisibleForTesting
+ MissingSegmentInfo findMissingSegments(Map<String, Map<String, String>> idealStateMap, Instant now) {
+ // create the maps
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestConsumingSegmentMap = new HashMap<>();
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestCompletedSegmentMap = new HashMap<>();
+ idealStateMap.forEach((segmentName, instanceToStatusMap) -> {
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName != null) { // Skip the uploaded realtime segments that don't conform to llc naming
+ if (instanceToStatusMap.containsValue(SegmentStateModel.CONSUMING)) {
+ updateMap(partitionGroupIdToLatestConsumingSegmentMap, llcSegmentName);
Review Comment:
And for the new partition case, my strong preference would still be to include them here so this becomes a reliable metric covering all cases. It would be hard to reason about and explain to users about the cases this includes and excludes (like if they delete a CONSUMING segment that had seqId 3, we would detect it, but if they deleted CONSUMING segment with segId 0, we wouldnt detect it, and then proceed to explain how that is okay to not detect because validation manager would anyway fix it.)
But if keeping this lightweight and free of stream connections and stream error handling is a string requirement, I guess it's fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] sajjad-moradi merged pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
sajjad-moradi merged PR #8877:
URL: https://github.com/apache/pinot/pull/8877
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on PR #8877:
URL: https://github.com/apache/pinot/pull/8877#issuecomment-1171560444
We have this problem in production, so we need to commit soon. I suggest we commit it now, and the re-visit if @KKcorps finds any issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on PR #8877:
URL: https://github.com/apache/pinot/pull/8877#issuecomment-1171190684
cc @KKcorps to also take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #8877:
URL: https://github.com/apache/pinot/pull/8877#issuecomment-1152682304
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8877?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8877](https://codecov.io/gh/apache/pinot/pull/8877?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a6e9de4) into [master](https://codecov.io/gh/apache/pinot/commit/90fa0f910449ef194b253a081ddd4cbe02710aa8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (90fa0f9) will **decrease** coverage by `3.37%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #8877 +/- ##
============================================
- Coverage 69.74% 66.37% -3.38%
- Complexity 4683 4812 +129
============================================
Files 1805 1353 -452
Lines 94006 68385 -25621
Branches 13992 10675 -3317
============================================
- Hits 65563 45389 -20174
+ Misses 23904 19761 -4143
+ Partials 4539 3235 -1304
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `66.37% <0.00%> (-0.04%)` | :arrow_down: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8877?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...g/apache/pinot/common/metrics/ControllerGauge.java](https://codecov.io/gh/apache/pinot/pull/8877/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-97.68%)` | :arrow_down: |
| [...va/org/apache/pinot/core/routing/RoutingTable.java](https://codecov.io/gh/apache/pinot/pull/8877/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yb3V0aW5nL1JvdXRpbmdUYWJsZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/pinot/common/config/NettyConfig.java](https://codecov.io/gh/apache/pinot/pull/8877/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL05ldHR5Q29uZmlnLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/8877/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/8877/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/8877/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/8877/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...apache/pinot/common/helix/ExtraInstanceConfig.java](https://codecov.io/gh/apache/pinot/pull/8877/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vaGVsaXgvRXh0cmFJbnN0YW5jZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ache/pinot/server/access/AccessControlFactory.java](https://codecov.io/gh/apache/pinot/pull/8877/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL2FjY2Vzcy9BY2Nlc3NDb250cm9sRmFjdG9yeS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/pinot/common/messages/SegmentReloadMessage.java](https://codecov.io/gh/apache/pinot/pull/8877/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvU2VnbWVudFJlbG9hZE1lc3NhZ2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [691 more](https://codecov.io/gh/apache/pinot/pull/8877/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8877?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8877?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [90fa0f9...a6e9de4](https://codecov.io/gh/apache/pinot/pull/8877?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on a diff in pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on code in PR #8877:
URL: https://github.com/apache/pinot/pull/8877#discussion_r910361302
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/MissingConsumingSegmentFinder.java:
##########
@@ -0,0 +1,221 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.helix.AccessOption;
+import org.apache.helix.ZNRecord;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+import org.apache.pinot.common.metadata.ZKMetadataProvider;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ControllerGauge;
+import org.apache.pinot.common.metrics.ControllerMeter;
+import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder;
+import org.apache.pinot.spi.stream.OffsetCriteria;
+import org.apache.pinot.spi.stream.PartitionLevelStreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * For a given table, this class finds out if there is any partition group for which there's no consuming segment in
+ * ideal state. If so, it emits three metrics:
+ * - Total number of partitions with missing consuming segments including
+ * - Number of newly added partitions for which there's no consuming segment (there's no completed segment either)
+ * - Maximum duration (in minutes) that a partition hasn't had a consuming segment
+ */
+public class MissingConsumingSegmentFinder {
+ private static final Logger LOGGER = LoggerFactory.getLogger(MissingConsumingSegmentFinder.class);
+
+ private final String _realtimeTableName;
+ private final SegmentMetadataFetcher _segmentMetadataFetcher;
+ private final Map<Integer, StreamPartitionMsgOffset> _partitionGroupIdToLargestStreamOffsetMap;
+ private final StreamPartitionMsgOffsetFactory _streamPartitionMsgOffsetFactory;
+
+ private ControllerMetrics _controllerMetrics;
+
+ public MissingConsumingSegmentFinder(String realtimeTableName, ZkHelixPropertyStore<ZNRecord> propertyStore,
+ ControllerMetrics controllerMetrics, PartitionLevelStreamConfig streamConfig) {
+ _realtimeTableName = realtimeTableName;
+ _controllerMetrics = controllerMetrics;
+ _segmentMetadataFetcher = new SegmentMetadataFetcher(propertyStore, controllerMetrics);
+ _streamPartitionMsgOffsetFactory =
+ StreamConsumerFactoryProvider.create(streamConfig).createStreamMsgOffsetFactory();
+
+ // create partition group id to largest stream offset map
+ _partitionGroupIdToLargestStreamOffsetMap = new HashMap<>();
+ streamConfig.setOffsetCriteria(OffsetCriteria.LARGEST_OFFSET_CRITERIA);
+ try {
+ PinotTableIdealStateBuilder.getPartitionGroupMetadataList(streamConfig, Collections.emptyList())
+ .forEach(metadata -> {
+ _partitionGroupIdToLargestStreamOffsetMap.put(metadata.getPartitionGroupId(), metadata.getStartOffset());
+ });
+ } catch (Exception e) {
+ LOGGER.warn("Problem encountered in fetching stream metadata for topic: {} of table: {}. "
+ + "Continue finding missing consuming segment only with ideal state information.",
+ streamConfig.getTopicName(), streamConfig.getTableNameWithType());
+ }
+ }
+
+ @VisibleForTesting
+ MissingConsumingSegmentFinder(String realtimeTableName, SegmentMetadataFetcher segmentMetadataFetcher,
+ Map<Integer, StreamPartitionMsgOffset> partitionGroupIdToLargestStreamOffsetMap,
+ StreamPartitionMsgOffsetFactory streamPartitionMsgOffsetFactory) {
+ _realtimeTableName = realtimeTableName;
+ _segmentMetadataFetcher = segmentMetadataFetcher;
+ _partitionGroupIdToLargestStreamOffsetMap = partitionGroupIdToLargestStreamOffsetMap;
+ _streamPartitionMsgOffsetFactory = streamPartitionMsgOffsetFactory;
+ }
+
+ public void findAndEmitMetrics(IdealState idealState) {
+ MissingSegmentInfo info = findMissingSegments(idealState.getRecord().getMapFields(), Instant.now());
+ _controllerMetrics.setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_TOTAL_COUNT,
+ info._totalCount);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_NEW_PARTITION_COUNT,
+ info._newPartitionGroupCount);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_MAX_DURATION_MINUTES,
+ info._maxDurationInMinutes);
+ }
+
+ @VisibleForTesting
+ MissingSegmentInfo findMissingSegments(Map<String, Map<String, String>> idealStateMap, Instant now) {
+ // create the maps
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestConsumingSegmentMap = new HashMap<>();
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestCompletedSegmentMap = new HashMap<>();
+ idealStateMap.forEach((segmentName, instanceToStatusMap) -> {
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName != null) { // Skip the uploaded realtime segments that don't conform to llc naming
+ if (instanceToStatusMap.containsValue(SegmentStateModel.CONSUMING)) {
+ updateMap(partitionGroupIdToLatestConsumingSegmentMap, llcSegmentName);
+ } else if (instanceToStatusMap.containsValue(SegmentStateModel.ONLINE)) {
+ updateMap(partitionGroupIdToLatestCompletedSegmentMap, llcSegmentName);
+ }
+ }
+ });
+
+ MissingSegmentInfo missingSegmentInfo = new MissingSegmentInfo();
+ if (!_partitionGroupIdToLargestStreamOffsetMap.isEmpty()) {
+ _partitionGroupIdToLargestStreamOffsetMap.forEach((partitionGroupId, largestStreamOffset) -> {
+ if (!partitionGroupIdToLatestConsumingSegmentMap.containsKey(partitionGroupId)) {
+ LLCSegmentName latestCompletedSegment = partitionGroupIdToLatestCompletedSegmentMap.get(partitionGroupId);
+ if (latestCompletedSegment == null) {
+ // There's no consuming or completed segment for this partition group. Possibilities:
+ // 1) it's a new partition group that has not yet been detected
+ // 2) the first consuming segment has been deleted from ideal state manually
+ missingSegmentInfo._newPartitionGroupCount++;
+ missingSegmentInfo._totalCount++;
+ } else {
+ // completed segment is available, but there's no consuming segment
+ SegmentZKMetadata segmentZKMetadata = _segmentMetadataFetcher
+ .fetchSegmentZkMetadata(_realtimeTableName, latestCompletedSegment.getSegmentName());
+ StreamPartitionMsgOffset completedSegmentEndOffset =
+ _streamPartitionMsgOffsetFactory.create(segmentZKMetadata.getEndOffset());
+ if (completedSegmentEndOffset.compareTo(largestStreamOffset) < 0) {
Review Comment:
[Capturing our offline discussion]
You brought up a good point that we may be catching up from a closed partition. The logic you have here seems fine to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] KKcorps commented on a diff in pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
KKcorps commented on code in PR #8877:
URL: https://github.com/apache/pinot/pull/8877#discussion_r911358944
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/MissingConsumingSegmentFinder.java:
##########
@@ -0,0 +1,221 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.helix.AccessOption;
+import org.apache.helix.ZNRecord;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.store.zk.ZkHelixPropertyStore;
+import org.apache.pinot.common.metadata.ZKMetadataProvider;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.metrics.ControllerGauge;
+import org.apache.pinot.common.metrics.ControllerMeter;
+import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder;
+import org.apache.pinot.spi.stream.OffsetCriteria;
+import org.apache.pinot.spi.stream.PartitionLevelStreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffset;
+import org.apache.pinot.spi.stream.StreamPartitionMsgOffsetFactory;
+import org.apache.pinot.spi.utils.CommonConstants.Helix.StateModel.SegmentStateModel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * For a given table, this class finds out if there is any partition group for which there's no consuming segment in
+ * ideal state. If so, it emits three metrics:
+ * - Total number of partitions with missing consuming segments including
+ * - Number of newly added partitions for which there's no consuming segment (there's no completed segment either)
+ * - Maximum duration (in minutes) that a partition hasn't had a consuming segment
+ */
+public class MissingConsumingSegmentFinder {
+ private static final Logger LOGGER = LoggerFactory.getLogger(MissingConsumingSegmentFinder.class);
+
+ private final String _realtimeTableName;
+ private final SegmentMetadataFetcher _segmentMetadataFetcher;
+ private final Map<Integer, StreamPartitionMsgOffset> _partitionGroupIdToLargestStreamOffsetMap;
+ private final StreamPartitionMsgOffsetFactory _streamPartitionMsgOffsetFactory;
+
+ private ControllerMetrics _controllerMetrics;
+
+ public MissingConsumingSegmentFinder(String realtimeTableName, ZkHelixPropertyStore<ZNRecord> propertyStore,
+ ControllerMetrics controllerMetrics, PartitionLevelStreamConfig streamConfig) {
+ _realtimeTableName = realtimeTableName;
+ _controllerMetrics = controllerMetrics;
+ _segmentMetadataFetcher = new SegmentMetadataFetcher(propertyStore, controllerMetrics);
+ _streamPartitionMsgOffsetFactory =
+ StreamConsumerFactoryProvider.create(streamConfig).createStreamMsgOffsetFactory();
+
+ // create partition group id to largest stream offset map
+ _partitionGroupIdToLargestStreamOffsetMap = new HashMap<>();
+ streamConfig.setOffsetCriteria(OffsetCriteria.LARGEST_OFFSET_CRITERIA);
+ try {
+ PinotTableIdealStateBuilder.getPartitionGroupMetadataList(streamConfig, Collections.emptyList())
+ .forEach(metadata -> {
+ _partitionGroupIdToLargestStreamOffsetMap.put(metadata.getPartitionGroupId(), metadata.getStartOffset());
+ });
+ } catch (Exception e) {
+ LOGGER.warn("Problem encountered in fetching stream metadata for topic: {} of table: {}. "
+ + "Continue finding missing consuming segment only with ideal state information.",
+ streamConfig.getTopicName(), streamConfig.getTableNameWithType());
+ }
+ }
+
+ @VisibleForTesting
+ MissingConsumingSegmentFinder(String realtimeTableName, SegmentMetadataFetcher segmentMetadataFetcher,
+ Map<Integer, StreamPartitionMsgOffset> partitionGroupIdToLargestStreamOffsetMap,
+ StreamPartitionMsgOffsetFactory streamPartitionMsgOffsetFactory) {
+ _realtimeTableName = realtimeTableName;
+ _segmentMetadataFetcher = segmentMetadataFetcher;
+ _partitionGroupIdToLargestStreamOffsetMap = partitionGroupIdToLargestStreamOffsetMap;
+ _streamPartitionMsgOffsetFactory = streamPartitionMsgOffsetFactory;
+ }
+
+ public void findAndEmitMetrics(IdealState idealState) {
+ MissingSegmentInfo info = findMissingSegments(idealState.getRecord().getMapFields(), Instant.now());
+ _controllerMetrics.setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_TOTAL_COUNT,
+ info._totalCount);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_NEW_PARTITION_COUNT,
+ info._newPartitionGroupCount);
+ _controllerMetrics
+ .setValueOfTableGauge(_realtimeTableName, ControllerGauge.MISSING_CONSUMING_SEGMENT_MAX_DURATION_MINUTES,
+ info._maxDurationInMinutes);
+ }
+
+ @VisibleForTesting
+ MissingSegmentInfo findMissingSegments(Map<String, Map<String, String>> idealStateMap, Instant now) {
+ // create the maps
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestConsumingSegmentMap = new HashMap<>();
+ Map<Integer, LLCSegmentName> partitionGroupIdToLatestCompletedSegmentMap = new HashMap<>();
+ idealStateMap.forEach((segmentName, instanceToStatusMap) -> {
+ LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+ if (llcSegmentName != null) { // Skip the uploaded realtime segments that don't conform to llc naming
+ if (instanceToStatusMap.containsValue(SegmentStateModel.CONSUMING)) {
+ updateMap(partitionGroupIdToLatestConsumingSegmentMap, llcSegmentName);
+ } else if (instanceToStatusMap.containsValue(SegmentStateModel.ONLINE)) {
+ updateMap(partitionGroupIdToLatestCompletedSegmentMap, llcSegmentName);
+ }
+ }
+ });
+
+ MissingSegmentInfo missingSegmentInfo = new MissingSegmentInfo();
+ if (!_partitionGroupIdToLargestStreamOffsetMap.isEmpty()) {
+ _partitionGroupIdToLargestStreamOffsetMap.forEach((partitionGroupId, largestStreamOffset) -> {
+ if (!partitionGroupIdToLatestConsumingSegmentMap.containsKey(partitionGroupId)) {
+ LLCSegmentName latestCompletedSegment = partitionGroupIdToLatestCompletedSegmentMap.get(partitionGroupId);
+ if (latestCompletedSegment == null) {
+ // There's no consuming or completed segment for this partition group. Possibilities:
+ // 1) it's a new partition group that has not yet been detected
+ // 2) the first consuming segment has been deleted from ideal state manually
+ missingSegmentInfo._newPartitionGroupCount++;
+ missingSegmentInfo._totalCount++;
+ } else {
+ // completed segment is available, but there's no consuming segment
+ SegmentZKMetadata segmentZKMetadata = _segmentMetadataFetcher
+ .fetchSegmentZkMetadata(_realtimeTableName, latestCompletedSegment.getSegmentName());
+ StreamPartitionMsgOffset completedSegmentEndOffset =
+ _streamPartitionMsgOffsetFactory.create(segmentZKMetadata.getEndOffset());
+ if (completedSegmentEndOffset.compareTo(largestStreamOffset) < 0) {
Review Comment:
I was only concerned about `.compareTo` working for Kinesis or not but after looking at the code it should.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] sajjad-moradi commented on pull request #8877: Emit metrics if there's no consuming segment for a partition
Posted by GitBox <gi...@apache.org>.
sajjad-moradi commented on PR #8877:
URL: https://github.com/apache/pinot/pull/8877#issuecomment-1163240646
> IIRC, it is normal for certain stream type (e.g. Kinesis) to have no consuming segment for some partitions. cc @npawar @KKcorps to confirm
Good catch. This scenario didn't occur to me as I was thinking mainly about Kafka.
Anyway, I was hoping to avoid fetching stream metadata in case of issue connecting to stream, but I guess I need to do so to cover this case as well as the ones neha was pointing out. I'll refactor today or tomorrow (I just got back from vacation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org