You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "vrajat (via GitHub)" <gi...@apache.org> on 2024/03/08 18:05:39 UTC
[PR] Detect expired messages in Kafka. Log and set a gauge. [pinot]
vrajat opened a new pull request, #12608:
URL: https://github.com/apache/pinot/pull/12608
Pinot may take multiple hours between polling a partition in a Kafka topic. One specific example is that Pinot took a long time to flush a segment to disk. In the meantime, messages in Kafka can expire if message retention time is small.
If `auto.offset.reset` is set to smallest, then Kafka will silently move the offset to the first available message leading to data loss.
RealtimeSegmentValidationManager is a cron that runs every hour and detects where the offset of a segment in zookeeper is in the past when compared to the smallest offset in Kafka. However since it runs every hour, it may miss the data loss if it happens between runs.
This commit compares the startOffset to the batchFirstOffset.
* startOffset: Offset requested by the database for the next batch.
* batchFirstOffset: First offset of the batch of messages received from the stream.
If startOffset < batchFirstOffset, then log the condition as well as set a meter to 1.
This test is implemented only for Kafka Streams.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Detect expired messages in Kafka. Log and set a gauge. [pinot]
Posted by "vrajat (via GitHub)" <gi...@apache.org>.
vrajat commented on code in PR #12608:
URL: https://github.com/apache/pinot/pull/12608#discussion_r1518177154
##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/KafkaPartitionLevelConsumer.java:
##########
@@ -69,8 +69,12 @@ public synchronized MessageBatch<StreamMessage<byte[]>> fetchMessages(long start
ConsumerRecords<String, Bytes> consumerRecords = _consumer.poll(Duration.ofMillis(timeoutMillis));
List<ConsumerRecord<String, Bytes>> messageAndOffsets = consumerRecords.records(_topicPartition);
List<StreamMessage<byte[]>> filtered = new ArrayList<>(messageAndOffsets.size());
+ long firstOffset = startOffset;
long lastOffset = startOffset;
StreamMessageMetadata rowMetadata = null;
+ if (!consumerRecords.isEmpty()) {
+ firstOffset = consumerRecords.iterator().next().offset();
+ }
Review Comment:
regarding the suggestion, it is better to check the offset of the first message. Lets say in a batch only the last message is from a topic. Then this will give the wrong signal that the offset jumped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Detect expired messages in Kafka. Log and set a gauge. [pinot]
Posted by "vrajat (via GitHub)" <gi...@apache.org>.
vrajat commented on PR #12608:
URL: https://github.com/apache/pinot/pull/12608#issuecomment-1986390389
I tested this commit manually. Gist of quickstart: https://gist.github.com/vrajat/5db2715535db325dde9af8e4a029da47
The workflow is as follows:
* Insert a batch of 400 messages.
* Pinot ingests these messages
* Pause consumption
* Insert another batch of 400 messages
* Delete records up to offset 500. So 100 records are missing.
* Resume consumption.
O/p
```
***** Cluster is running *****
***** Publish first batch of data to kafka streams *****
***** Starting githubEvents data stream and publishing to Kafka *****
***** Pausing consumption *****
***** Publish second batch of data to kafka streams *****
***** Starting githubEvents data stream and publishing to Kafka *****
Executing records delete operation
Records delete operation completed:
partition: githubEvents-1 low_watermark: 500
partition: githubEvents-0 low_watermark: 500
Offset is 500
Offset is 500
***** Publish third batch of data to kafka streams *****
***** Starting githubEvents data stream and publishing to Kafka *****
***** Resume consumption *****
You can always go to http://localhost:9000 to play around in the query console
2024/03/08 12:38:01.004 ERROR [PinotLLCRealtimeSegmentManager] [pool-23-thread-2] Data lost from offset: 400 to: 500 for partition: 0 of table: githubEvents_REALTIME
2024/03/08 12:38:01.013 ERROR [PinotLLCRealtimeSegmentManager] [pool-23-thread-2] Data lost from offset: 400 to: 500 for partition: 1 of table: githubEvents_REALTIME
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Detect expired messages in Kafka. Log and set a gauge. [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12608:
URL: https://github.com/apache/pinot/pull/12608#discussion_r1518393375
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##########
@@ -899,6 +904,26 @@ public Map<String, PartitionLagState> getPartitionToLagState(
return _partitionMetadataProvider.getCurrentPartitionLagState(consumerPartitionStateMap);
}
+ /**
+ * Checks if the begin offset of the stream partition has been fast-forwarded.
+ * batchFirstOffset should be less than or equal to startOffset.
+ * If batchFirstOffset is greater, then some messages were not received.
+ *
+ * @param startOffset The offset of the first message desired, inclusive.
+ * @param batchFirstOffset The offset of the first message in the batch.
+ */
+ private void validateStartOffset(StreamPartitionMsgOffset startOffset, StreamPartitionMsgOffset batchFirstOffset) {
+ if (batchFirstOffset.compareTo(startOffset) > 0) {
+ _serverMetrics.addMeteredTableValue(_tableStreamName, ServerMeter.STREAM_DATA_LOSS, 1L);
+ String message = "startOffset(" + startOffset
+ + ") is older than topic's beginning offset(" + batchFirstOffset + ")";
+ _segmentLogger.error(message);
+ _realtimeTableDataManager.addSegmentError(_segmentNameStr,
Review Comment:
(format) Let's reformat the change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Detect expired messages in Kafka. Log and set a gauge. [pinot]
Posted by "vrajat (via GitHub)" <gi...@apache.org>.
vrajat commented on code in PR #12608:
URL: https://github.com/apache/pinot/pull/12608#discussion_r1518174041
##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/KafkaPartitionLevelConsumer.java:
##########
@@ -69,8 +69,12 @@ public synchronized MessageBatch<StreamMessage<byte[]>> fetchMessages(long start
ConsumerRecords<String, Bytes> consumerRecords = _consumer.poll(Duration.ofMillis(timeoutMillis));
List<ConsumerRecord<String, Bytes>> messageAndOffsets = consumerRecords.records(_topicPartition);
List<StreamMessage<byte[]>> filtered = new ArrayList<>(messageAndOffsets.size());
+ long firstOffset = startOffset;
long lastOffset = startOffset;
StreamMessageMetadata rowMetadata = null;
+ if (!consumerRecords.isEmpty()) {
+ firstOffset = consumerRecords.iterator().next().offset();
+ }
Review Comment:
Yes. that condition is checking if the offset of a message is between start, end offset while it is looping through the messages in the batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Detect expired messages in Kafka. Log and set a gauge. [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #12608:
URL: https://github.com/apache/pinot/pull/12608
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Detect expired messages in Kafka. Log and set a gauge. [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #12608:
URL: https://github.com/apache/pinot/pull/12608#issuecomment-1986342309
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12608?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `45.45455%` with `12 lines` in your changes are missing coverage. Please review.
> Project coverage is 61.81%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`fe011a5`)](https://app.codecov.io/gh/apache/pinot/pull/12608?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 100 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/pinot/pull/12608?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...a/manager/realtime/RealtimeSegmentDataManager.java](https://app.codecov.io/gh/apache/pinot/pull/12608?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVTZWdtZW50RGF0YU1hbmFnZXIuamF2YQ==) | 8.33% | [10 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12608?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...pinot/plugin/stream/kafka20/KafkaMessageBatch.java](https://app.codecov.io/gh/apache/pinot/pull/12608?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcGx1Z2lucy9waW5vdC1zdHJlYW0taW5nZXN0aW9uL3Bpbm90LWthZmthLTIuMC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcGx1Z2luL3N0cmVhbS9rYWZrYTIwL0thZmthTWVzc2FnZUJhdGNoLmphdmE=) | 66.66% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12608?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #12608 +/- ##
============================================
+ Coverage 61.75% 61.81% +0.06%
+ Complexity 207 198 -9
============================================
Files 2436 2450 +14
Lines 133233 133767 +534
Branches 20636 20745 +109
============================================
+ Hits 82274 82687 +413
- Misses 44911 44973 +62
- Partials 6048 6107 +59
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/12608/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/12608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/12608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/12608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/12608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.73% <45.45%> (+0.02%)` | :arrow_up: |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/12608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.70% <45.45%> (+0.07%)` | :arrow_up: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.78% <45.45%> (+0.04%)` | :arrow_up: |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.66% <45.45%> (+33.93%)` | :arrow_up: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/12608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.81% <45.45%> (+0.06%)` | :arrow_up: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/12608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.80% <45.45%> (+0.06%)` | :arrow_up: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.92% <26.66%> (+0.03%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12608/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.71% <27.27%> (-0.03%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/12608?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Detect expired messages in Kafka. Log and set a gauge. [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12608:
URL: https://github.com/apache/pinot/pull/12608#discussion_r1518138970
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##########
@@ -456,6 +456,8 @@ protected boolean consumeLoop()
throw t;
}
+ validateStartOffset(_currentOffset, messageBatch.getFirstMessageOffset());
Review Comment:
Consider moving the `null` check here:
```suggestion
StreamPartitionMsgOffset batchFirstOffset = messageBatch.getFirstMessageOffset();
if (batchFirstOffset != null) {
validateStartOffset(_currentOffset, batchFirstOffset);
}
```
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##########
@@ -899,6 +901,29 @@ public Map<String, PartitionLagState> getPartitionToLagState(
return _partitionMetadataProvider.getCurrentPartitionLagState(consumerPartitionStateMap);
}
+ /**
+ * Checks if the begin offset of the stream partition has been fast-forwarded.
+ * batchFirstOffset should be less than or equal to startOffset.
+ * If batchFirstOffset is greater, then some messages were not received.
+ *
+ * @param startOffset The offset of the first message desired, inclusive.
+ * @param batchFirstOffset The offset of the first message in the batch.
+ */
+ private void validateStartOffset(StreamPartitionMsgOffset startOffset, StreamPartitionMsgOffset batchFirstOffset) {
+ if (batchFirstOffset != null && batchFirstOffset.compareTo(startOffset) > 0) {
+ _serverMetrics.addMeteredTableValue(_tableStreamName, ServerMeter.STREAM_DATA_LOSS, 1L);
+ String message = "startOffset(" + startOffset
+ + ") is older than topic's beginning offset(" + batchFirstOffset + ")";
+ _segmentLogger.error(message);
+ _realtimeTableDataManager.addSegmentError(_segmentNameStr,
+ new SegmentErrorInfo(now(), message, null)
+ );
+ } else {
+ // Record that this batch has no data loss.
+ _serverMetrics.addMeteredTableValue(_tableStreamName, ServerMeter.STREAM_DATA_LOSS, 0L);
Review Comment:
We don't need to emit meter when there is no loss
##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/KafkaPartitionLevelConsumer.java:
##########
@@ -69,8 +69,12 @@ public synchronized MessageBatch<StreamMessage<byte[]>> fetchMessages(long start
ConsumerRecords<String, Bytes> consumerRecords = _consumer.poll(Duration.ofMillis(timeoutMillis));
List<ConsumerRecord<String, Bytes>> messageAndOffsets = consumerRecords.records(_topicPartition);
List<StreamMessage<byte[]>> filtered = new ArrayList<>(messageAndOffsets.size());
+ long firstOffset = startOffset;
long lastOffset = startOffset;
StreamMessageMetadata rowMetadata = null;
+ if (!consumerRecords.isEmpty()) {
+ firstOffset = consumerRecords.iterator().next().offset();
+ }
Review Comment:
There is also a check for `offset >= startOffset` underlying. Is it even possible?
##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/KafkaPartitionLevelConsumer.java:
##########
@@ -69,8 +69,12 @@ public synchronized MessageBatch<StreamMessage<byte[]>> fetchMessages(long start
ConsumerRecords<String, Bytes> consumerRecords = _consumer.poll(Duration.ofMillis(timeoutMillis));
List<ConsumerRecord<String, Bytes>> messageAndOffsets = consumerRecords.records(_topicPartition);
List<StreamMessage<byte[]>> filtered = new ArrayList<>(messageAndOffsets.size());
+ long firstOffset = startOffset;
long lastOffset = startOffset;
StreamMessageMetadata rowMetadata = null;
+ if (!consumerRecords.isEmpty()) {
+ firstOffset = consumerRecords.iterator().next().offset();
+ }
Review Comment:
Should we get offset from records under the topic partition?
```suggestion
if (!messageAndOffsets.isEmpty()) {
firstOffset = messageAndOffsets.get(0).offset();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Detect expired messages in Kafka. Log and set a gauge. [pinot]
Posted by "vrajat (via GitHub)" <gi...@apache.org>.
vrajat commented on code in PR #12608:
URL: https://github.com/apache/pinot/pull/12608#discussion_r1518165799
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##########
@@ -899,6 +901,29 @@ public Map<String, PartitionLagState> getPartitionToLagState(
return _partitionMetadataProvider.getCurrentPartitionLagState(consumerPartitionStateMap);
}
+ /**
+ * Checks if the begin offset of the stream partition has been fast-forwarded.
+ * batchFirstOffset should be less than or equal to startOffset.
+ * If batchFirstOffset is greater, then some messages were not received.
+ *
+ * @param startOffset The offset of the first message desired, inclusive.
+ * @param batchFirstOffset The offset of the first message in the batch.
+ */
+ private void validateStartOffset(StreamPartitionMsgOffset startOffset, StreamPartitionMsgOffset batchFirstOffset) {
+ if (batchFirstOffset != null && batchFirstOffset.compareTo(startOffset) > 0) {
+ _serverMetrics.addMeteredTableValue(_tableStreamName, ServerMeter.STREAM_DATA_LOSS, 1L);
+ String message = "startOffset(" + startOffset
+ + ") is older than topic's beginning offset(" + batchFirstOffset + ")";
+ _segmentLogger.error(message);
+ _realtimeTableDataManager.addSegmentError(_segmentNameStr,
+ new SegmentErrorInfo(now(), message, null)
+ );
+ } else {
+ // Record that this batch has no data loss.
+ _serverMetrics.addMeteredTableValue(_tableStreamName, ServerMeter.STREAM_DATA_LOSS, 0L);
Review Comment:
I am thinking of an alert that is triggered when this metric goes to 1. A spike is sufficient to trigger an alert. Without the else block, the metric will not be reset to 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org