You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "awelless (via GitHub)" <gi...@apache.org> on 2025/11/28 15:19:50 UTC
[PR] NIFI-15270 Introduce Shared Throughput consumer in ConsumeKinesis [nifi]
awelless opened a new pull request, #10578:
URL: https://github.com/apache/nifi/pull/10578
<!-- Licensed to the Apache Software Foundation (ASF) under one or more -->
<!-- contributor license agreements. See the NOTICE file distributed with -->
<!-- this work for additional information regarding copyright ownership. -->
<!-- The ASF licenses this file to You under the Apache License, Version 2.0 -->
<!-- (the "License"); you may not use this file except in compliance with -->
<!-- the License. You may obtain a copy of the License at -->
<!-- http://www.apache.org/licenses/LICENSE-2.0 -->
<!-- Unless required by applicable law or agreed to in writing, software -->
<!-- distributed under the License is distributed on an "AS IS" BASIS, -->
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
<!-- See the License for the specific language governing permissions and -->
<!-- limitations under the License. -->
# Summary
[NIFI-15270](https://issues.apache.org/jira/browse/NIFI-15270)
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
### Pull Request Formatting
- [x] Pull Request based on current revision of the `main` branch
- [x] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [ ] Build completed using `./mvnw clean install -P contrib-check`
- [x] JDK 21
- [ ] JDK 25
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [ ] Documentation formatting appears as expected in rendered files - in progress
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15270 Introduce Shared Throughput consumer in ConsumeKinesis [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #10578:
URL: https://github.com/apache/nifi/pull/10578#discussion_r2572744118
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesis.java:
##########
@@ -199,6 +202,17 @@ Ensure that the credentials provided have access to Kinesis, DynamoDB and (optio
.identifiesControllerService(AwsCredentialsProviderService.class)
.build();
+ static final PropertyDescriptor CONSUMER_TYPE = new PropertyDescriptor.Builder()
+ .name("Consumer Type")
+ .description("""
+ The Consumer Type that is used to consume from the Kinesis records.
+ (See processor's additional details for more information.)
Review Comment:
Recommend avoiding the mention of additional details, since it is generally implied that additional details contains more information.
```suggestion
Strategy for reading records from Amazon Kinesis streams.
```
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/resources/docs/org.apache.nifi.processors.aws.kinesis.ConsumeKinesis/additionalDetails.md:
##########
@@ -116,6 +116,16 @@ The following is an example policy document for `ConsumeKinesis`.
**Note:** Replace "{Region}", "{Account}", "{Stream Name}", and "{Application Name}" in the ARNs with your own AWS region,
AWS account number, Kinesis data stream name, and `ConsumeKinesis` _Application Name_ property respectively.
+## Consumer Type
+
+Comparison of different Consumer Types from [Amazon Kinesis Streams documentation](https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html):
+
+| Characteristics | Shared throughput consumers without enhanced fan-out | Enhanced fan-out consumers |
+|---------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Read throughput | Fixed at a total of 2 MB/sec per shard. If there are multiple consumers reading from the same shard, they all share this throughput. The sum of the throughputs they receive from the shard doesn't exceed 2 MB/sec. | Scales as consumers register to use enhanced fan-out. Each consumer registered to use enhanced fan-out receives its own read throughput per shard, up to 2 MB/sec, independently of other consumers. |
+| Message propagation delay | An average of around 200 ms if you have one consumer reading from the stream. This average goes up to around 1000 ms if you have five consumers. | Typically an average of 70 ms whether you have one consumer or five consumers. |
+| Cost | Not applicable | There is a data retrieval cost and a consumer-shard hour cost. For more information, see [Amazon Kinesis Data Streams Pricing](https://aws.amazon.com/kinesis/data-streams/pricing/?nc=sn&loc=3). |
Review Comment:
The table layout does not seem like the best way to present the information, both in text-based views and in HTML rendering. What do you think about a bullet list style?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15270 Introduce Shared Throughput consumer in ConsumeKinesis [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #10578:
URL: https://github.com/apache/nifi/pull/10578#discussion_r2577743446
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesis.java:
##########
@@ -199,6 +202,17 @@ Ensure that the credentials provided have access to Kinesis, DynamoDB and (optio
.identifiesControllerService(AwsCredentialsProviderService.class)
.build();
+ static final PropertyDescriptor CONSUMER_TYPE = new PropertyDescriptor.Builder()
+ .name("Consumer Type")
+ .description("""
+ The Consumer Type that is used to consume from the Kinesis records.
+ (See processor's additional details for more information.)
Review Comment:
Thanks for making the adjustment. Yes, it would be useful to remove the comment from other descriptions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15270 Introduce Shared Throughput consumer in ConsumeKinesis [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory merged PR #10578:
URL: https://github.com/apache/nifi/pull/10578
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15270 Introduce Shared Throughput consumer in ConsumeKinesis [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #10578:
URL: https://github.com/apache/nifi/pull/10578#discussion_r2577749805
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/resources/docs/org.apache.nifi.processors.aws.kinesis.ConsumeKinesis/additionalDetails.md:
##########
@@ -116,6 +116,16 @@ The following is an example policy document for `ConsumeKinesis`.
**Note:** Replace "{Region}", "{Account}", "{Stream Name}", and "{Application Name}" in the ARNs with your own AWS region,
AWS account number, Kinesis data stream name, and `ConsumeKinesis` _Application Name_ property respectively.
+## Consumer Type
+
+Comparison of different Consumer Types from [Amazon Kinesis Streams documentation](https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html):
+
+| Characteristics | Shared throughput consumers without enhanced fan-out | Enhanced fan-out consumers |
+|---------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Read throughput | Fixed at a total of 2 MB/sec per shard. If there are multiple consumers reading from the same shard, they all share this throughput. The sum of the throughputs they receive from the shard doesn't exceed 2 MB/sec. | Scales as consumers register to use enhanced fan-out. Each consumer registered to use enhanced fan-out receives its own read throughput per shard, up to 2 MB/sec, independently of other consumers. |
+| Message propagation delay | An average of around 200 ms if you have one consumer reading from the stream. This average goes up to around 1000 ms if you have five consumers. | Typically an average of 70 ms whether you have one consumer or five consumers. |
+| Cost | Not applicable | There is a data retrieval cost and a consumer-shard hour cost. For more information, see [Amazon Kinesis Data Streams Pricing](https://aws.amazon.com/kinesis/data-streams/pricing/?nc=sn&loc=3). |
Review Comment:
Thanks, I see this follows the general format from the AWS documentation, so under the circumstances, this seems reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15270 Introduce Shared Throughput consumer in ConsumeKinesis [nifi]
Posted by "awelless (via GitHub)" <gi...@apache.org>.
awelless commented on code in PR #10578:
URL: https://github.com/apache/nifi/pull/10578#discussion_r2576207346
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesis.java:
##########
@@ -199,6 +202,17 @@ Ensure that the credentials provided have access to Kinesis, DynamoDB and (optio
.identifiesControllerService(AwsCredentialsProviderService.class)
.build();
+ static final PropertyDescriptor CONSUMER_TYPE = new PropertyDescriptor.Builder()
+ .name("Consumer Type")
+ .description("""
+ The Consumer Type that is used to consume from the Kinesis records.
+ (See processor's additional details for more information.)
Review Comment:
Done. Additional details are mentioned in a few other properties and described values. Should it be removed from there as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15270 Introduce Shared Throughput consumer in ConsumeKinesis [nifi]
Posted by "awelless (via GitHub)" <gi...@apache.org>.
awelless commented on code in PR #10578:
URL: https://github.com/apache/nifi/pull/10578#discussion_r2576234070
##########
nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/resources/docs/org.apache.nifi.processors.aws.kinesis.ConsumeKinesis/additionalDetails.md:
##########
@@ -116,6 +116,16 @@ The following is an example policy document for `ConsumeKinesis`.
**Note:** Replace "{Region}", "{Account}", "{Stream Name}", and "{Application Name}" in the ARNs with your own AWS region,
AWS account number, Kinesis data stream name, and `ConsumeKinesis` _Application Name_ property respectively.
+## Consumer Type
+
+Comparison of different Consumer Types from [Amazon Kinesis Streams documentation](https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html):
+
+| Characteristics | Shared throughput consumers without enhanced fan-out | Enhanced fan-out consumers |
+|---------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Read throughput | Fixed at a total of 2 MB/sec per shard. If there are multiple consumers reading from the same shard, they all share this throughput. The sum of the throughputs they receive from the shard doesn't exceed 2 MB/sec. | Scales as consumers register to use enhanced fan-out. Each consumer registered to use enhanced fan-out receives its own read throughput per shard, up to 2 MB/sec, independently of other consumers. |
+| Message propagation delay | An average of around 200 ms if you have one consumer reading from the stream. This average goes up to around 1000 ms if you have five consumers. | Typically an average of 70 ms whether you have one consumer or five consumers. |
+| Cost | Not applicable | There is a data retrieval cost and a consumer-shard hour cost. For more information, see [Amazon Kinesis Data Streams Pricing](https://aws.amazon.com/kinesis/data-streams/pricing/?nc=sn&loc=3). |
Review Comment:
Considering the amount of provided information, table format provides better readability for the side-by-side comparisons.
We can put the characteristics into the bulletins, and then compare each of the options there. But when providing all the details it doesn't seem readable. And I reckon we don't want to omit the specifics in the additional details document.
Alternatively, it's possible to list the characteristics for each consumer type separately. The downside of this approach is that during comparison of both consumers a reader must jump back and forth to evaluate the differences between the options.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org