You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2020/06/17 23:29:43 UTC
[GitHub] [nifi] gitfirenz opened a new pull request #4346: NIFI-7522 : Add Kafka 2.5 Processors set
gitfirenz opened a new pull request #4346:
URL: https://github.com/apache/nifi/pull/4346
#### Description of PR
Add Kafka 2.5 Processors set (NIFI-7522), in order to :
* Have the last fixes from Kafka
* Use the static Membership Consumers (to reduce rebalancing)
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder?
-> The full test suite is stuck with RAT licence issues.
- [x] Have you written or updated unit tests to verify your changes?
- [x] Have you verified that the full build is successful on JDK 8?
- [NOT YET] Have you verified that the full build is successful on JDK 11?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
### Testing
**Infra :**
The static kafka membership has been tested with a kafka 2.5 single node, topic "test" with 5 partitions (Windows 10 1909, jdk1.8.0_212)
**Code :** A NIFI template with 3 consumers is attached (NIFI template).
The 3 consumers have a group.instance.id 01 02 03 and a session.timeout.ms of 10minutes.
**Results :**
* The consumer consumes (like in 2.0)
* The kafka server log show the join of the static consumer : "Static member Some(01) of group CG with unknown member id rejoins".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] MikeThomsen commented on pull request #4346: NIFI-7522 : Add Kafka 2.5 Processors set
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on pull request #4346:
URL: https://github.com/apache/nifi/pull/4346#issuecomment-648351851
Read over your Jira ticket, and updating the Kafka 2.X processors might make more sense. We're running out of space in our convenience binary so it might be a hard fit to have Kafka 2.0 and 2.5 processors at the same time. Could also make it a lot easier to review if we can merge this into the 2.X processors. Do you know of any breaking changes between 2.0 and 2.5?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] gitfirenz edited a comment on pull request #4346: NIFI-7522 : Add Kafka 2.5 Processors set
Posted by GitBox <gi...@apache.org>.
gitfirenz edited a comment on pull request #4346:
URL: https://github.com/apache/nifi/pull/4346#issuecomment-648657221
Hello, i've just finished compiling with JDK8 and JDK11 on Centos8. So i'm technically ready
My POV is :
- putting 2.5 client in the 2.0 processor is a bit confusing for the user, despite doc
- the 2.5 client is 2.0-protocol-compatible, but i haven't tested.
- The 2.6 client will implement "incremental rebalancing", so i'd be good to anticipate it.
- There will be a Kafka 3.x [Roadmap](https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals)
- Maybe can we create a "2.5+" Processor with the 2.5 and further 2.6, 2.7 etc
Edit : Kafka 2.6 will be released today, so i will propose a PR for 2.6 instead of 2.5
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] gitfirenz closed pull request #4346: NIFI-7522 : Add Kafka 2.5 Processors set
Posted by GitBox <gi...@apache.org>.
gitfirenz closed pull request #4346:
URL: https://github.com/apache/nifi/pull/4346
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] joewitt commented on pull request #4346: NIFI-7522 : Add Kafka 2.5 Processors set
Posted by GitBox <gi...@apache.org>.
joewitt commented on pull request #4346:
URL: https://github.com/apache/nifi/pull/4346#issuecomment-648437572
I'd punt 0.9 and 0.10 for sure. I still run into 0.11 in the wild so prob good to hang onto that a bit longer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] joewitt commented on pull request #4346: NIFI-7522 : Add Kafka 2.5 Processors set
Posted by GitBox <gi...@apache.org>.
joewitt commented on pull request #4346:
URL: https://github.com/apache/nifi/pull/4346#issuecomment-648355056
Kafka is a different animal. We should just dump an older nar from the assembly but we should add a new Kafka 2.5 specific nar.
I'll review this as it gets closer to ready for review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] MikeThomsen commented on pull request #4346: NIFI-7522 : Add Kafka 2.5 Processors set
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on pull request #4346:
URL: https://github.com/apache/nifi/pull/4346#issuecomment-648408135
We have 0.9, 0.10 and 0.11 in nifi-assembly. Are any of them worth keeping in the convenience binary at this point vs moving them to profiles?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] gitfirenz commented on pull request #4346: NIFI-7522 : Add Kafka 2.5 Processors set
Posted by GitBox <gi...@apache.org>.
gitfirenz commented on pull request #4346:
URL: https://github.com/apache/nifi/pull/4346#issuecomment-648657221
Hello, i've just finished compiling with JDK8 and JDK11 on Centos8. So i'm technically ready
My POV is :
- putting 2.5 client in the 2.0 processor is a bit confusing for the user, despite doc
- the 2.5 client is 2.0-protocol-compatible, but i haven't tested.
- The 2.6 client will implement "incremental rebalancing", so i'd be good to anticipate it.
- There will be a Kafka 3.x [Roadmap](https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals)
- Maybe can we create a "2.5+" Processor with the 2.5 and further 2.6, 2.7 etc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org