Console producer can connect locally but not locally

[Zac Harvey <za...@welltok.com>]

I have a single Kafka node at, say, IP address 1.2.3.4.  If I SSH into that node from 2 different terminal windows, and run the console consumer from 1 terminal, and the console producer from another terminal, everything works great:


# Run the consumer from terminal 1

kafka-console-consumer.sh --zookeeper zkA:2181 --topic simpletest


# Run the producer from terminal 2

kafka-console-producer.sh --broker-list localhost:9092 --topic simpletest

# Now I can enter messages into terminal 2 and see them show up in terminal 1 (the consumer)


If I kill the console producer, but leave the consumer running, and then SSH into a different server (say with IP address of 5.6.7.8), then run the producer, and then try to send a message to Kafka (so that the running consumer picks it up), I get the following warnings:


# Run the producer from terminal 2

kafka-console-producer.sh --broker-list 1.2.3.4:9092 --topic simpletest

# Now enter a simple string message to try and send to consumer running on Kafka
hello
[2017-01-25 22:27:21,439] WARN Bootstrap broker 1.2.3.4:9092 disconnected (org.apache.kafka.clients.NetworkClient)

[2017-01-25 22:27:21,439] WARN Bootstrap broker 1.2.3.4:9092 disconnected (org.apache.kafka.clients.NetworkClient)

[2017-01-25 22:27:21,439] WARN Bootstrap broker 1.2.3.4:9092 disconnected (org.apache.kafka.clients.NetworkClient)

...


These warnings keep being generated until I kill the producer. Most importantly, the message never arrives and the consumer (again, running on the Kafka node, terminal 1) never spits the "hello" message to the console/STDOUT.


I have confirmed that the server at 5.6.7.8 has network access to 1.2.3.4:9092:


telnet 1.2.3.4 9092

Trying 1.2.3.4...
Connected to 1.2.3.4.
Escape character is '^]'.

etc. So in summary, I can run the console consumer locally on the Kafka node, and I can run the console producer locally on the Kafka node, and they work fine. But if I have the console consumer running locally on Kafka, and then try to run + send messages from a remote console producer, I get producer warnings and the messages never arrive. I have confirmed that the remote producer has network access to the Kafka node at port 9092.


Any ideas as to what's going wrong and/or how I could troubleshoot?

Re: Console producer can connect locally but not locally

[Peter Kopias <ko...@gmail.com>]

Hi.

This looks like some network / config issue.

1. Check that zookeeper (zkA)  is available and reachable from both hosts
(and it can connect). (check etc/hosts on both machines, there might be
some 127.0.0.1 issues there).

2. Verify, that the process bound to the public network interface:9092 is
kafka (it is possible that something else listens there theoretically. Use
lsof ¦ grep LISTEN

3. You might wanna check all firewall settings you have, that is the
primary source of network disconnects :), so my bet is on this.

4. If everything else fails, run a netcat (nc -l 9093)  on 9093 (or 9092
and set kafka to 9093),and try to connect there. This way you can make sure
that the network is fine.

So that's what I'd check in the first 5 minutes,before starting heavy tools
like wireshark or strace. :)

Best wishes,

Peter



On Jan 25, 2017 11:41 PM, "Zac Harvey" <za...@welltok.com> wrote:

> I have a single Kafka node at, say, IP address 1.2.3.4.  If I SSH into
> that node from 2 different terminal windows, and run the console consumer
> from 1 terminal, and the console producer from another terminal, everything
> works great:
>
>
> # Run the consumer from terminal 1
>
> kafka-console-consumer.sh --zookeeper zkA:2181 --topic simpletest
>
>
> # Run the producer from terminal 2
>
> kafka-console-producer.sh --broker-list localhost:9092 --topic simpletest
>
> # Now I can enter messages into terminal 2 and see them show up in
> terminal 1 (the consumer)
>
>
> If I kill the console producer, but leave the consumer running, and then
> SSH into a different server (say with IP address of 5.6.7.8), then run the
> producer, and then try to send a message to Kafka (so that the running
> consumer picks it up), I get the following warnings:
>
>
> # Run the producer from terminal 2
>
> kafka-console-producer.sh --broker-list 1.2.3.4:9092 --topic simpletest
>
> # Now enter a simple string message to try and send to consumer running on
> Kafka
> hello
> [2017-01-25 22:27:21,439] WARN Bootstrap broker 1.2.3.4:9092 disconnected
> (org.apache.kafka.clients.NetworkClient)
>
> [2017-01-25 22:27:21,439] WARN Bootstrap broker 1.2.3.4:9092 disconnected
> (org.apache.kafka.clients.NetworkClient)
>
> [2017-01-25 22:27:21,439] WARN Bootstrap broker 1.2.3.4:9092 disconnected
> (org.apache.kafka.clients.NetworkClient)
>
> ...
>
>
> These warnings keep being generated until I kill the producer. Most
> importantly, the message never arrives and the consumer (again, running on
> the Kafka node, terminal 1) never spits the "hello" message to the
> console/STDOUT.
>
>
> I have confirmed that the server at 5.6.7.8 has network access to
> 1.2.3.4:9092:
>
>
> telnet 1.2.3.4 9092
>
> Trying 1.2.3.4...
> Connected to 1.2.3.4.
> Escape character is '^]'.
>
> etc. So in summary, I can run the console consumer locally on the Kafka
> node, and I can run the console producer locally on the Kafka node, and
> they work fine. But if I have the console consumer running locally on
> Kafka, and then try to run + send messages from a remote console producer,
> I get producer warnings and the messages never arrive. I have confirmed
> that the remote producer has network access to the Kafka node at port 9092.
>
>
> Any ideas as to what's going wrong and/or how I could troubleshoot?
>