You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "lkuchars (via GitHub)" <gi...@apache.org> on 2026/02/10 20:41:32 UTC
[PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
lkuchars opened a new pull request, #10881:
URL: https://github.com/apache/nifi/pull/10881
<!-- Licensed to the Apache Software Foundation (ASF) under one or more -->
<!-- contributor license agreements. See the NOTICE file distributed with -->
<!-- this work for additional information regarding copyright ownership. -->
<!-- The ASF licenses this file to You under the Apache License, Version 2.0 -->
<!-- (the "License"); you may not use this file except in compliance with -->
<!-- the License. You may obtain a copy of the License at -->
<!-- http://www.apache.org/licenses/LICENSE-2.0 -->
<!-- Unless required by applicable law or agreed to in writing, software -->
<!-- distributed under the License is distributed on an "AS IS" BASIS, -->
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
<!-- See the License for the specific language governing permissions and -->
<!-- limitations under the License. -->
# Summary
[NIFI-15576](https://issues.apache.org/jira/browse/NIFI-15576)
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
- [x] Pull request contains [commits signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) with a registered key indicating `Verified` status
### Pull Request Formatting
- [x] Pull Request based on current revision of the `main` branch
- [x] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [x] Build completed using `./mvnw clean install -P contrib-check`
- [ ] JDK 21
- [ ] JDK 25
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [ ] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
Posted by "awelless (via GitHub)" <gi...@apache.org>.
awelless commented on code in PR #10881:
URL: https://github.com/apache/nifi/pull/10881#discussion_r2817792182
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesis.java:
##########
@@ -696,20 +705,28 @@ private void checkInitializationResult(final InitializationResult initialization
}
}
+
+
private void processRecordsFromBuffer(final ProcessSession session, final Lease lease) {
try {
- final List<KinesisClientRecord> records = recordBuffer.consumeRecords(lease);
+ final ConsumeRecordsResult result = recordBuffer.consumeRecords(lease);
+ final List<KinesisClientRecord> records = result.records();
+
+ final String shardId = lease.shardId();
+ final Long millisBehindLatest = result.millisBehindLatest();
+ if (millisBehindLatest != null) {
+ session.recordGauge(makeMillisBehindLatestGaugeName(streamName, shardId), millisBehindLatest, CommitTiming.SESSION_COMMITTED);
+ }
Review Comment:
Extreme nitpicking. Can we put it near `session.adjustCounter` below? So that all counter and gauges are updated in a single place.
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesis.java:
##########
@@ -761,7 +770,8 @@ public void processRecords(final ProcessRecordsInput processRecordsInput) {
if (bufferId == null) {
throw new IllegalStateException("Buffer ID not found: Record Processor not initialized");
}
- recordBuffer.addRecords(bufferId, processRecordsInput.records(), processRecordsInput.checkpointer());
+ recordBuffer.addRecords(bufferId, processRecordsInput.records(),
+ processRecordsInput.checkpointer(), processRecordsInput.millisBehindLatest());
Review Comment:
Could you mention this gauge and a meaning behind it in [this file](https://github.com/apache/nifi/blob/main/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/resources/docs/org.apache.nifi.processors.aws.kinesis.ConsumeKinesis/additionalDetails.md)? Then it will be visible in processor documentation in NiFi
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesis.java:
##########
@@ -145,6 +146,8 @@ Uses DynamoDB for check pointing and coordination, and (optional) CloudWatch for
description = "A SubSequence Number of the last Kinesis Record in the FlowFile. Generated by KPL when aggregating records into a single Kinesis Record"),
@WritesAttribute(attribute = APPROXIMATE_ARRIVAL_TIMESTAMP,
description = "Approximate arrival timestamp of the last Kinesis Record in the FlowFile"),
+ @WritesAttribute(attribute = MILLIS_BEHIND_LATEST,
+ description = "Milliseconds behind the latest record in the shard at the time records were consumed"),
Review Comment:
If this attribute will be needed downstream, no problems.
But I wouldn't put it in FlowFile attributes if it's only caused by the gauge name format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
Posted by "sfc-gh-lkucharski (via GitHub)" <gi...@apache.org>.
sfc-gh-lkucharski commented on code in PR #10881:
URL: https://github.com/apache/nifi/pull/10881#discussion_r2813854694
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesis.java:
##########
@@ -145,6 +146,8 @@ Uses DynamoDB for check pointing and coordination, and (optional) CloudWatch for
description = "A SubSequence Number of the last Kinesis Record in the FlowFile. Generated by KPL when aggregating records into a single Kinesis Record"),
@WritesAttribute(attribute = APPROXIMATE_ARRIVAL_TIMESTAMP,
description = "Approximate arrival timestamp of the last Kinesis Record in the FlowFile"),
+ @WritesAttribute(attribute = MILLIS_BEHIND_LATEST,
+ description = "Milliseconds behind the latest record in the shard at the time records were consumed"),
Review Comment:
I decided to add it as an FF attribute to be able to report this metric even if the `makeMillisBehindLatestGaugeName` format will be deemed too OpenFlow specific by the maintainers.
Our current internal OF format for metric with attributes is `some.metric.name[arg.1="123",arg.2="456"]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
Posted by "pvillard31 (via GitHub)" <gi...@apache.org>.
pvillard31 commented on code in PR #10881:
URL: https://github.com/apache/nifi/pull/10881#discussion_r2822763838
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/test/java/org/apache/nifi/processors/aws/kinesis/ReaderRecordProcessorTest.java:
##########
@@ -363,6 +365,45 @@ void testInvalidRecordsWithSchemaEvolution() {
secondFailureFlowFile.assertContentEquals(KinesisRecordPayload.extract(records.get(4)), UTF_8);
}
+ @Test
+ void testMillisBehindLatestAttributeOnSuccessAndFailure() {
+ final List<KinesisClientRecord> records = List.of(
+ createKinesisRecord(USER_JSON_1, "1"),
+ createKinesisRecord(INVALID_JSON, "2")
+ );
+ final long millis = 5000L;
+ final ConsumeRecordsResult consumeResult = new ConsumeRecordsResult(records, millis);
+
+ final ProcessingResult result = processor.processRecords(session, TEST_STREAM_NAME, TEST_SHARD_ID, consumeResult);
+
+ assertEquals(1, result.successFlowFiles().size());
+ assertEquals(1, result.parseFailureFlowFiles().size());
+
+ final FlowFile successFlowFile = result.successFlowFiles().getFirst();
+ assertEquals(String.valueOf(millis), successFlowFile.getAttribute(MILLIS_BEHIND_LATEST));
+
+ final FlowFile failureFlowFile = result.parseFailureFlowFiles().getFirst();
+ assertEquals(String.valueOf(millis), failureFlowFile.getAttribute(MILLIS_BEHIND_LATEST));
+ }
+
+ @Test
+ void testNullMillisBehindLatestAttributeNotSet() {
+ final List<KinesisClientRecord> records = List.of(
+ createKinesisRecord(USER_JSON_1, "1")
+ );
+
+ final ProcessingResult result = processor.processRecords(session, TEST_STREAM_NAME, TEST_SHARD_ID, new ConsumeRecordsResult(records, null));
+
+ assertEquals(1, result.successFlowFiles().size());
+
+ final FlowFile successFlowFile = result.successFlowFiles().getFirst();
+ assertEquals(null, successFlowFile.getAttribute(MILLIS_BEHIND_LATEST));
Review Comment:
assertNull
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/test/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesisIT.java:
##########
@@ -233,6 +241,12 @@ void testConsumeSingleMessageFromSingleShard_withoutEnhancedFanOut() {
assertTrue(
streamClient.getEnhancedFanOutConsumerNames().isEmpty(),
"No enhanced fan-out consumers should be created for Shared Throughput consumer type");
+ final String shardId = flowFile.getAttribute("aws.kinesis.shard.id");
+ final String gaugeName = makeMillisBehindLatestGaugeName(streamName, shardId);
+ final List<Double> gaugeValues = runner.getGaugeValues(gaugeName);
+ assertFalse(gaugeValues.isEmpty(), "Expected millisBehindLatest gauge to be recorded");
+
+
Review Comment:
extra lines not needed
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/test/java/org/apache/nifi/processors/aws/kinesis/MemoryBoundRecordBufferTest.java:
##########
@@ -559,26 +561,67 @@ void testCommitRecordsWhileNewRecordsArrive() {
// Add records to pending queue.
final List<KinesisClientRecord> batch1 = createTestRecords(2);
- recordBuffer.addRecords(bufferId, batch1, checkpointer1);
+ recordBuffer.addRecords(bufferId, batch1, checkpointer1, 100L);
// Consume batch1 records (moves from pending to in-progress).
final Lease lease = recordBuffer.acquireBufferLease().orElseThrow();
- final List<KinesisClientRecord> consumedRecords = recordBuffer.consumeRecords(lease);
+ final List<KinesisClientRecord> consumedRecords = recordBuffer.consumeRecords(lease).records();
assertEquals(batch1, consumedRecords);
// Add more records while others are in-progress.
final List<KinesisClientRecord> batch2 = createTestRecords(1);
- recordBuffer.addRecords(bufferId, batch2, checkpointer2);
+ recordBuffer.addRecords(bufferId, batch2, checkpointer2, 100L);
// Commit in-progress records.
recordBuffer.commitConsumedRecords(lease);
// Consume batch2 records.
- final List<KinesisClientRecord> remainingRecords = recordBuffer.consumeRecords(lease);
+ final List<KinesisClientRecord> remainingRecords = recordBuffer.consumeRecords(lease).records();
assertEquals(batch2, remainingRecords);
}
+ @Test
+ void testConsumeRecordsReturnsLastMillisBehindLatest() {
+ final ShardBufferId bufferId = recordBuffer.createBuffer(SHARD_ID_1);
+
+ recordBuffer.addRecords(bufferId, createTestRecords(1), checkpointer1, 200L);
+ recordBuffer.addRecords(bufferId, createTestRecords(1), checkpointer2, 500L);
+
+ final Lease lease = recordBuffer.acquireBufferLease().orElseThrow();
+ final ConsumeRecordsResult result = recordBuffer.consumeRecords(lease);
+
+ assertEquals(2, result.records().size());
+ assertEquals(500L, result.millisBehindLatest());
+ }
+
+ @Test
+ void testConsumeRecordsWithNullMillisBehindLatest() {
+ final ShardBufferId bufferId = recordBuffer.createBuffer(SHARD_ID_1);
+
+ recordBuffer.addRecords(bufferId, createTestRecords(1), checkpointer1, null);
+
+ final Lease lease = recordBuffer.acquireBufferLease().orElseThrow();
+ final ConsumeRecordsResult result = recordBuffer.consumeRecords(lease);
+
+ assertEquals(1, result.records().size());
+ assertNull(result.millisBehindLatest());
+ }
+
+ @Test
+ void testConsumeRecordsWithMixedMillisBehindLatest() {
+ final ShardBufferId bufferId = recordBuffer.createBuffer(SHARD_ID_1);
+
+ recordBuffer.addRecords(bufferId, createTestRecords(1), checkpointer1, 300L);
+ recordBuffer.addRecords(bufferId, createTestRecords(1), checkpointer2, null);
+
+ final Lease lease = recordBuffer.acquireBufferLease().orElseThrow();
+ final ConsumeRecordsResult result = recordBuffer.consumeRecords(lease);
+
+ assertEquals(2, result.records().size());
+ assertEquals(300, result.millisBehindLatest());
Review Comment:
```suggestion
assertEquals(300L, result.millisBehindLatest());
```
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesis.java:
##########
@@ -696,22 +705,29 @@ private void checkInitializationResult(final InitializationResult initialization
}
}
+
+
Review Comment:
extra lines not needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
Posted by "sfc-gh-lkucharski (via GitHub)" <gi...@apache.org>.
sfc-gh-lkucharski commented on code in PR #10881:
URL: https://github.com/apache/nifi/pull/10881#discussion_r2797629857
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/MemoryBoundRecordBuffer.java:
##########
@@ -504,7 +508,13 @@ List<KinesisClientRecord> consumeRecords() {
recordsToConsume.addAll(batch.records());
}
- return recordsToConsume;
+ final Long maxMillisBehindLatest = inProgressBatches.stream()
+ .map(RecordBatch::millisBehindLatest)
+ .filter(Objects::nonNull)
+ .min(Long::compareTo)
Review Comment:
@awelless I think it's better to take the last value as you suggest. Thanks. As for the nullability, since the value returned is of type Long and I did not see anything about non null in the docs I'd feel safer to check this for null:
<img width="2242" height="201" alt="image" src="https://github.com/user-attachments/assets/379e7548-e302-45b6-8c57-12d528caa71c" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
Posted by "sfc-gh-lkucharski (via GitHub)" <gi...@apache.org>.
sfc-gh-lkucharski commented on code in PR #10881:
URL: https://github.com/apache/nifi/pull/10881#discussion_r2798383333
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesis.java:
##########
@@ -761,7 +770,8 @@ public void processRecords(final ProcessRecordsInput processRecordsInput) {
if (bufferId == null) {
throw new IllegalStateException("Buffer ID not found: Record Processor not initialized");
}
- recordBuffer.addRecords(bufferId, processRecordsInput.records(), processRecordsInput.checkpointer());
+ recordBuffer.addRecords(bufferId, processRecordsInput.records(),
+ processRecordsInput.checkpointer(), processRecordsInput.millisBehindLatest());
Review Comment:
I think millisBehindLatetest is what this metric should be about. The fact that records spent some time in caches will be factored into e2e metrics.
I'll add the information what this metric means in the documentation once it's ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
Posted by "sfc-gh-lkucharski (via GitHub)" <gi...@apache.org>.
sfc-gh-lkucharski commented on code in PR #10881:
URL: https://github.com/apache/nifi/pull/10881#discussion_r2797782554
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/test/java/org/apache/nifi/processors/aws/kinesis/MemoryBoundRecordBufferTest.java:
##########
@@ -559,26 +561,67 @@ void testCommitRecordsWhileNewRecordsArrive() {
// Add records to pending queue.
final List<KinesisClientRecord> batch1 = createTestRecords(2);
- recordBuffer.addRecords(bufferId, batch1, checkpointer1);
+ recordBuffer.addRecords(bufferId, batch1, checkpointer1, 100L);
// Consume batch1 records (moves from pending to in-progress).
final Lease lease = recordBuffer.acquireBufferLease().orElseThrow();
- final List<KinesisClientRecord> consumedRecords = recordBuffer.consumeRecords(lease);
+ final List<KinesisClientRecord> consumedRecords = recordBuffer.consumeRecords(lease).records();
assertEquals(batch1, consumedRecords);
// Add more records while others are in-progress.
final List<KinesisClientRecord> batch2 = createTestRecords(1);
- recordBuffer.addRecords(bufferId, batch2, checkpointer2);
+ recordBuffer.addRecords(bufferId, batch2, checkpointer2, 100L);
// Commit in-progress records.
recordBuffer.commitConsumedRecords(lease);
// Consume batch2 records.
- final List<KinesisClientRecord> remainingRecords = recordBuffer.consumeRecords(lease);
+ final List<KinesisClientRecord> remainingRecords = recordBuffer.consumeRecords(lease).records();
assertEquals(batch2, remainingRecords);
}
+ @Test
+ void testConsumeRecordsReturnsMinMillisBehindLatest() {
+ final ShardBufferId bufferId = recordBuffer.createBuffer(SHARD_ID_1);
+
+ recordBuffer.addRecords(bufferId, createTestRecords(1), checkpointer1, 200L);
+ recordBuffer.addRecords(bufferId, createTestRecords(1), checkpointer2, 500L);
+
+ final Lease lease = recordBuffer.acquireBufferLease().orElseThrow();
+ final ConsumeRecordsResult result = recordBuffer.consumeRecords(lease);
+
+ assertEquals(2, result.records().size());
+ assertEquals(200L, result.maxMillisBehindLatest());
Review Comment:
Changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
Posted by "sfc-gh-lkucharski (via GitHub)" <gi...@apache.org>.
sfc-gh-lkucharski commented on code in PR #10881:
URL: https://github.com/apache/nifi/pull/10881#discussion_r2823701552
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesis.java:
##########
@@ -696,22 +705,29 @@ private void checkInitializationResult(final InitializationResult initialization
}
}
+
+
Review Comment:
Sorry, I didn't see that. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
Posted by "sfc-gh-lkucharski (via GitHub)" <gi...@apache.org>.
sfc-gh-lkucharski commented on PR #10881:
URL: https://github.com/apache/nifi/pull/10881#issuecomment-3929941596
> Thanks for working on this addition @lkuchars.
>
> The general approach of recording lag is useful, but the naming of `millisBehindLatest` is not very intuitive. The PR title indicates `lag`, so something that includes `lag` seems better, and removing `millis`, although perhaps less clear, would make the name less verbose. The main issue is the object of `latest` being unclear. This seems analogous to the `currentLag` feature of Kafka in some ways, so what do you think about calling it `current.lag` instead?
Thanks for the review @exceptionfactory! Yep, `current.lag` sounds much better. Changes commited. PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
Posted by "sfc-gh-lkucharski (via GitHub)" <gi...@apache.org>.
sfc-gh-lkucharski commented on code in PR #10881:
URL: https://github.com/apache/nifi/pull/10881#discussion_r2813862352
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/MemoryBoundRecordBuffer.java:
##########
@@ -504,17 +503,13 @@ ConsumeRecordsResult consumeRecords() {
}
final List<KinesisClientRecord> recordsToConsume = new ArrayList<>();
+ Long lastMillisBehindLatest = null;
for (final RecordBatch batch : inProgressBatches) {
recordsToConsume.addAll(batch.records());
+ lastMillisBehindLatest = batch.millisBehindLatest;
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
Posted by "sfc-gh-lkucharski (via GitHub)" <gi...@apache.org>.
sfc-gh-lkucharski commented on code in PR #10881:
URL: https://github.com/apache/nifi/pull/10881#discussion_r2813854694
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesis.java:
##########
@@ -145,6 +146,8 @@ Uses DynamoDB for check pointing and coordination, and (optional) CloudWatch for
description = "A SubSequence Number of the last Kinesis Record in the FlowFile. Generated by KPL when aggregating records into a single Kinesis Record"),
@WritesAttribute(attribute = APPROXIMATE_ARRIVAL_TIMESTAMP,
description = "Approximate arrival timestamp of the last Kinesis Record in the FlowFile"),
+ @WritesAttribute(attribute = MILLIS_BEHIND_LATEST,
+ description = "Milliseconds behind the latest record in the shard at the time records were consumed"),
Review Comment:
I decided to add it as an FF attribute to be able to report this metric even if the `makeMillisBehindLatestGaugeName` format will be deemed too OpenFlow specific by the maintainers.
Our current internal OF format for metric with attributes is `some.metric.name[arg.1="123",arg.2="456"]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
Posted by "sfc-gh-lkucharski (via GitHub)" <gi...@apache.org>.
sfc-gh-lkucharski commented on code in PR #10881:
URL: https://github.com/apache/nifi/pull/10881#discussion_r2797772731
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/test/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesisIT.java:
##########
@@ -205,6 +206,12 @@ void testConsumeSingleMessageFromSingleShard() {
List.of(applicationName),
streamClient.getEnhancedFanOutConsumerNames(),
"Expected a single enhanced fan-out consumer with an application name");
+
+ // Verify millisBehindLatest gauge is recorded.
+ final String shardId = flowFile.getAttribute("aws.kinesis.shard.id");
+ final String gaugeName = "kinesis.stream." + streamName + ".shard." + shardId + ".millisBehindLatest";
+ final List<Double> gaugeValues = runner.getGaugeValues(gaugeName);
+ assertFalse(gaugeValues.isEmpty(), "Expected millisBehindLatest gauge to be recorded");
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
Posted by "sfc-gh-lkucharski (via GitHub)" <gi...@apache.org>.
sfc-gh-lkucharski commented on code in PR #10881:
URL: https://github.com/apache/nifi/pull/10881#discussion_r2797744621
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/RecordBuffer.java:
##########
@@ -37,7 +37,7 @@ interface ForKinesisClientLibrary {
ShardBufferId createBuffer(String shardId);
- void addRecords(ShardBufferId bufferId, List<KinesisClientRecord> records, RecordProcessorCheckpointer checkpointer);
+ void addRecords(ShardBufferId bufferId, List<KinesisClientRecord> records, RecordProcessorCheckpointer checkpointer, Long millisBehindLatest);
Review Comment:
Since we talked about it before the implementation and agreed to just add another parameter I'm going to leave the refactor for the next soul
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
Posted by "lkuchars (via GitHub)" <gi...@apache.org>.
lkuchars commented on code in PR #10881:
URL: https://github.com/apache/nifi/pull/10881#discussion_r2822668004
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesis.java:
##########
@@ -761,7 +770,8 @@ public void processRecords(final ProcessRecordsInput processRecordsInput) {
if (bufferId == null) {
throw new IllegalStateException("Buffer ID not found: Record Processor not initialized");
}
- recordBuffer.addRecords(bufferId, processRecordsInput.records(), processRecordsInput.checkpointer());
+ recordBuffer.addRecords(bufferId, processRecordsInput.records(),
+ processRecordsInput.checkpointer(), processRecordsInput.millisBehindLatest());
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
Posted by "awelless (via GitHub)" <gi...@apache.org>.
awelless commented on code in PR #10881:
URL: https://github.com/apache/nifi/pull/10881#discussion_r2813078196
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/MemoryBoundRecordBuffer.java:
##########
@@ -504,17 +503,13 @@ ConsumeRecordsResult consumeRecords() {
}
final List<KinesisClientRecord> recordsToConsume = new ArrayList<>();
+ Long lastMillisBehindLatest = null;
for (final RecordBatch batch : inProgressBatches) {
recordsToConsume.addAll(batch.records());
+ lastMillisBehindLatest = batch.millisBehindLatest;
Review Comment:
Shall we update it only if `batch.millisBehindLatest` is not null? Is it better to have some value for an older batch, or a `null` value for a newer batch?
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesis.java:
##########
@@ -145,6 +146,8 @@ Uses DynamoDB for check pointing and coordination, and (optional) CloudWatch for
description = "A SubSequence Number of the last Kinesis Record in the FlowFile. Generated by KPL when aggregating records into a single Kinesis Record"),
@WritesAttribute(attribute = APPROXIMATE_ARRIVAL_TIMESTAMP,
description = "Approximate arrival timestamp of the last Kinesis Record in the FlowFile"),
+ @WritesAttribute(attribute = MILLIS_BEHIND_LATEST,
+ description = "Milliseconds behind the latest record in the shard at the time records were consumed"),
Review Comment:
Originally the issue was about writing millis behind latest in a gauge.
Do we need to write it to FlowFile attributes too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
Posted by "lkuchars (via GitHub)" <gi...@apache.org>.
lkuchars commented on code in PR #10881:
URL: https://github.com/apache/nifi/pull/10881#discussion_r2822673105
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesis.java:
##########
@@ -696,20 +705,28 @@ private void checkInitializationResult(final InitializationResult initialization
}
}
+
+
private void processRecordsFromBuffer(final ProcessSession session, final Lease lease) {
try {
- final List<KinesisClientRecord> records = recordBuffer.consumeRecords(lease);
+ final ConsumeRecordsResult result = recordBuffer.consumeRecords(lease);
+ final List<KinesisClientRecord> records = result.records();
+
+ final String shardId = lease.shardId();
+ final Long millisBehindLatest = result.millisBehindLatest();
+ if (millisBehindLatest != null) {
+ session.recordGauge(makeMillisBehindLatestGaugeName(streamName, shardId), millisBehindLatest, CommitTiming.SESSION_COMMITTED);
+ }
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
Posted by "awelless (via GitHub)" <gi...@apache.org>.
awelless commented on code in PR #10881:
URL: https://github.com/apache/nifi/pull/10881#discussion_r2793047072
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesis.java:
##########
@@ -761,7 +770,8 @@ public void processRecords(final ProcessRecordsInput processRecordsInput) {
if (bufferId == null) {
throw new IllegalStateException("Buffer ID not found: Record Processor not initialized");
}
- recordBuffer.addRecords(bufferId, processRecordsInput.records(), processRecordsInput.checkpointer());
+ recordBuffer.addRecords(bufferId, processRecordsInput.records(),
+ processRecordsInput.checkpointer(), processRecordsInput.millisBehindLatest());
Review Comment:
I'm not sure if `millisBehindLatest` alone is going to accurately represent the lag. The value is computed by Kinesis when returning records from a stream ([GetRecords docs](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html#API_GetRecords_ResponseSyntax), [SubscribeToShardEvent docs](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_SubscribeToShardEvent.html#Streams-Type-SubscribeToShardEvent-MillisBehindLatest)).
But it **does not** take into account the time records spend in KCL caches (`processRecordsInput.timeSpendInCache()`) nor the time spent in `MemoryBoundRecordBuffer`.
Because of that it might hide lag increases in scenarios where `ConsumeKinesis` backpressure kicks in.
Some of the records are still delivered from Kinesis, so `millisBehindLatest` stays relatively low while the actual lag grows.
After some time `millisBehindLatest` will increase as well. Yet it will be visible only when these records are consumed from the buffer.
I'm fine with exposing `millisBehindLatest` provided we mention what this gauge really represents altogether with the caveates related to it in the documentation.
At the moment `additionalDetails.md` seems to be the best place for that.
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/MemoryBoundRecordBuffer.java:
##########
@@ -504,7 +508,13 @@ List<KinesisClientRecord> consumeRecords() {
recordsToConsume.addAll(batch.records());
}
- return recordsToConsume;
+ final Long maxMillisBehindLatest = inProgressBatches.stream()
+ .map(RecordBatch::millisBehindLatest)
+ .filter(Objects::nonNull)
+ .min(Long::compareTo)
Review Comment:
I think we should just use the last `millisBehindLatest` here. Since the batches are ordered, the last batch will store the latest value.
Consider the following scenario:
1. Batch 1 arrives with 200 ms value.
2. Batch 2 arrives with 10 ms value.
3. Batch 3 arrives with 1500 ms value.
Having a value of 10 ms would be misleading, as the delay was 1500 ms when the records were received for the last time.
For some reason this fields is made as nullable, although Amazon docs claim it must be present.
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/test/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesisIT.java:
##########
@@ -205,6 +206,12 @@ void testConsumeSingleMessageFromSingleShard() {
List.of(applicationName),
streamClient.getEnhancedFanOutConsumerNames(),
"Expected a single enhanced fan-out consumer with an application name");
+
+ // Verify millisBehindLatest gauge is recorded.
+ final String shardId = flowFile.getAttribute("aws.kinesis.shard.id");
+ final String gaugeName = "kinesis.stream." + streamName + ".shard." + shardId + ".millisBehindLatest";
+ final List<Double> gaugeValues = runner.getGaugeValues(gaugeName);
+ assertFalse(gaugeValues.isEmpty(), "Expected millisBehindLatest gauge to be recorded");
Review Comment:
Let's add this in `testConsumeSingleMessageFromSingleShard_withoutEnhancedFanOut` to make sure the gauge is recorder for `SHARED_THROUGHPUT` consumers as well.
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/MemoryBoundRecordBuffer.java:
##########
@@ -504,7 +508,13 @@ List<KinesisClientRecord> consumeRecords() {
recordsToConsume.addAll(batch.records());
}
- return recordsToConsume;
+ final Long maxMillisBehindLatest = inProgressBatches.stream()
+ .map(RecordBatch::millisBehindLatest)
+ .filter(Objects::nonNull)
+ .min(Long::compareTo)
Review Comment:
Also nit: since the buffer class methods are executed very often, it was decided to use plain for loops to avoid any unnecessary overhead.
But this shouldn't be the case, since iteration won't be necessary when taking the latest value.
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesis.java:
##########
@@ -761,7 +770,8 @@ public void processRecords(final ProcessRecordsInput processRecordsInput) {
if (bufferId == null) {
throw new IllegalStateException("Buffer ID not found: Record Processor not initialized");
}
- recordBuffer.addRecords(bufferId, processRecordsInput.records(), processRecordsInput.checkpointer());
+ recordBuffer.addRecords(bufferId, processRecordsInput.records(),
+ processRecordsInput.checkpointer(), processRecordsInput.millisBehindLatest());
Review Comment:
@exceptionfactory in a long run, what do you think about introducing an annotation, indicating a processor exposes a particular gauge/counter?
Then the gauges/counters can be placed in the processor documentation automatically. Similar to how `@WritesAttribute` works.
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/test/java/org/apache/nifi/processors/aws/kinesis/MemoryBoundRecordBufferTest.java:
##########
@@ -559,26 +561,67 @@ void testCommitRecordsWhileNewRecordsArrive() {
// Add records to pending queue.
final List<KinesisClientRecord> batch1 = createTestRecords(2);
- recordBuffer.addRecords(bufferId, batch1, checkpointer1);
+ recordBuffer.addRecords(bufferId, batch1, checkpointer1, 100L);
// Consume batch1 records (moves from pending to in-progress).
final Lease lease = recordBuffer.acquireBufferLease().orElseThrow();
- final List<KinesisClientRecord> consumedRecords = recordBuffer.consumeRecords(lease);
+ final List<KinesisClientRecord> consumedRecords = recordBuffer.consumeRecords(lease).records();
assertEquals(batch1, consumedRecords);
// Add more records while others are in-progress.
final List<KinesisClientRecord> batch2 = createTestRecords(1);
- recordBuffer.addRecords(bufferId, batch2, checkpointer2);
+ recordBuffer.addRecords(bufferId, batch2, checkpointer2, 100L);
// Commit in-progress records.
recordBuffer.commitConsumedRecords(lease);
// Consume batch2 records.
- final List<KinesisClientRecord> remainingRecords = recordBuffer.consumeRecords(lease);
+ final List<KinesisClientRecord> remainingRecords = recordBuffer.consumeRecords(lease).records();
assertEquals(batch2, remainingRecords);
}
+ @Test
+ void testConsumeRecordsReturnsMinMillisBehindLatest() {
+ final ShardBufferId bufferId = recordBuffer.createBuffer(SHARD_ID_1);
+
+ recordBuffer.addRecords(bufferId, createTestRecords(1), checkpointer1, 200L);
+ recordBuffer.addRecords(bufferId, createTestRecords(1), checkpointer2, 500L);
+
+ final Lease lease = recordBuffer.acquireBufferLease().orElseThrow();
+ final ConsumeRecordsResult result = recordBuffer.consumeRecords(lease);
+
+ assertEquals(2, result.records().size());
+ assertEquals(200L, result.maxMillisBehindLatest());
Review Comment:
Mentioned in the `MemoryBoundRecordBuffer` already.
I believe we should see 500 ms as the gauge value in this scenario.
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/RecordBuffer.java:
##########
@@ -37,7 +37,7 @@ interface ForKinesisClientLibrary {
ShardBufferId createBuffer(String shardId);
- void addRecords(ShardBufferId bufferId, List<KinesisClientRecord> records, RecordProcessorCheckpointer checkpointer);
+ void addRecords(ShardBufferId bufferId, List<KinesisClientRecord> records, RecordProcessorCheckpointer checkpointer, Long millisBehindLatest);
Review Comment:
Nit: the idea to pass the whole `ProcessRecordInput` to the buffer might be neater, especially if we decide to get more fields from that class in the future.
I don't have any strong preference in regard to whether we should pass `records`, `checkpointer`, `millis...` separately or in the `ProcessRecordInput`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
Posted by "dariuszseweryn (via GitHub)" <gi...@apache.org>.
dariuszseweryn commented on code in PR #10881:
URL: https://github.com/apache/nifi/pull/10881#discussion_r2872039015
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/resources/docs/org.apache.nifi.processors.aws.kinesis.ConsumeKinesis/additionalDetails.md:
##########
@@ -283,3 +283,12 @@ Here is an example of FlowFile content that is emitted by JsonRecordSetWriter wh
}
]
```
+
+## Metrics
+
+_ConsumeKinesis_ reports the following metrics:
+
+| Name | Type | Description |
+|----------------------------------------------------------------------|---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `aws.kinesis.current.lag[stream.name="<stream>",shard.id="<shard>"]` | Gauge | The number of milliseconds the consumer is behind the tip of the shard, as reported by the Kinesis Client Library. There is one gauge per stream/shard combination. The gauge is updated each time a batch of records is successfully processed and the session is committed. A value of `0` means the consumer is caught up. |
Review Comment:
I've added a mention
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds [nifi]
Posted by "lkuchars (via GitHub)" <gi...@apache.org>.
lkuchars closed pull request #10881: NIFI-15576 ConsumeKinesis processor should log lag metric in milliseconds
URL: https://github.com/apache/nifi/pull/10881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org