Acks from Kafka not received (PutKafka, PublishKafka)

[James McMahon <js...@gmail.com>]

We are attempting to send flowfiles from a NiFi clustered configuration to
Kafka services on remote hosts. We are getting errors that indicate no acks
are received from Kafka.

When Kafka acks a PutKafka or a PublishKafka, what port does it use to
reach back to my nifi cluster node hosts, and what configuration param
tells it that?

I have proven that I can ping my remote Kafka host from each of my four
nifi cluster nodes. I have also been able to telnet from each of my cluster
nodes to the Kafka host on port 9092, which is the port named in the Known
Brokers configuration in my PutKafka attempt and in my PublishKafka
attempt. I suspect the comms failure is in the other direction, when Kafka
attempts to ack back to my nifi cluster nodes.

My Known Brokers list appears like so (the name of my Kafka hosts changed
to a generic):
Known Brokers          a-b-c123:9092,a-b-c124:9092

I am running nifi version 1.11.

Can anyone recommend other stepps to debug this and get it working? Thank
you.

Re: Acks from Kafka not received (PutKafka, PublishKafka)

[Vijay Chhipa <vc...@apple.com>]

James, 

Port 9092 is the only port you need. 
See this post: https://stackoverflow.com/questions/38531054/kafka-and-firewall-rules <https://stackoverflow.com/questions/38531054/kafka-and-firewall-rules>

Check that the firewall rules allow bidirectional traffic on that port. 

This is my setup: 

Kafka Brokers : a-b-c123:9092,a-b-c124:9092
Security protocol : SSL
SASL Mechanism : GSSAPI
SSLConextService : StandardSSLContextService

I am on NiFi 1.10  and Kafka 2.x

HTH
Vijay



> On Oct 30, 2020, at 5:57 AM, James McMahon <js...@gmail.com> wrote:
> 
> We are attempting to send flowfiles from a NiFi clustered configuration to Kafka services on remote hosts. We are getting errors that indicate no acks are received from Kafka.
> 
> When Kafka acks a PutKafka or a PublishKafka, what port does it use to reach back to my nifi cluster node hosts, and what configuration param tells it that?
> 
> I have proven that I can ping my remote Kafka host from each of my four nifi cluster nodes. I have also been able to telnet from each of my cluster nodes to the Kafka host on port 9092, which is the port named in the Known Brokers configuration in my PutKafka attempt and in my PublishKafka attempt. I suspect the comms failure is in the other direction, when Kafka attempts to ack back to my nifi cluster nodes.
> 
> My Known Brokers list appears like so (the name of my Kafka hosts changed to a generic):
> Known Brokers          a-b-c123:9092,a-b-c124:9092
> 
> I am running nifi version 1.11.
> 
> Can anyone recommend other stepps to debug this and get it working? Thank you.