You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Valentin Kulichenko <va...@gmail.com> on 2020/04/28 03:43:49 UTC
Kafka Connect ignores the bootstrap.servers parameter
Greetings to the Kafka Community!
I'm a newbie in Kafka and only recently went beyond a local installation
described in the Quickstart. I have faced a weird issue that I can't
explain.
I want to deploy on two machines:
- Machine #1 runs ZooKeeper and a single Kafka broker. I use default
configuration files.
- Machine #2 runs a single instance of Kafka Connect. I use the source
connector for Ignite/GridGain (
https://www.confluent.io/hub/gridgain/ignite-connector). I doubt it
matters, though, as the issue seems to be related to connectivity between
the connector and the broker.
After starting the ZooKeeper and the broker on machine #1, I did the
following on machine #2:
1. Updated plugin.path and bootstrap.servers parameters in the
connect-standalone.properties; bootstrap.servers points to the IP of the #1.
2. Created a properties file for the GridGain connector.
3. Start the connector using this command:
bin/connect-standalone.sh config/connect-standalone.properties
config/gridgain-kafka-connect-source.properties
The issue is that it doesn't connect to the remote broker, and still tries
localhost ignoring the bootstrap.servers settings. The output looks like
this:
[2020-04-27 19:37:08,334] INFO StandaloneConfig values:
...
bootstrap.servers = [98.1.96.147:9092]
...
[2020-04-27 19:37:08,334] INFO Creating Kafka admin client
(org.apache.kafka.connect.util.ConnectUtils:43)
[2020-04-27 19:37:08,337] INFO AdminClientConfig values:
bootstrap.servers = [98.1.96.147:9092]
...
[2020-04-27 19:37:08,950] WARN [AdminClient clientId=adminclient-1]
Connection to node 0 (localhost/127.0.0.1:9092) could not be established.
Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
[2020-04-27 19:37:09,052] WARN [AdminClient clientId=adminclient-1]
Connection to node 0 (localhost/127.0.0.1:9092) could not be established.
Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
[2020-04-27 19:37:09,253] WARN [AdminClient clientId=adminclient-1]
Connection to node 0 (localhost/127.0.0.1:9092) could not be established.
Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
The warning goes on for a while until the process stops after a timeout.
Does anyone have any idea why Kafka Connect can try to establish a
connection with the localhost, although it has picked up the correct IP
from the configuration file?
The issue doesn't seem to be reproducible on any environment, so I
would appreciate any pointers on how to investigate this further.
Kafka's version is 2.4.1. Config files and the log are attached. If
anything else is needed, let me know.
Regards,
Val
Re: Kafka Connect ignores the bootstrap.servers parameter
Posted by Valentin Kulichenko <va...@gmail.com>.
Hi Goran,
Thanks for your reply. We've tried this and got interesting results.
First, we found out that there is a port forwarding between the client and
the server. So we put the local IP (192.168.x.x) into the 'listeners'
parameter, and the external IP (98.1.96.147) into the
'advertised.listeners'. I believe this is a correct way of configuring in
this case, but correct me if I'm wrong.
Second, after we changed the configuration, we got the same warning, but
now with the external IP address:
[2020-04-28 10:02:07,695] WARN [Controller id=0, targetBrokerId=0]
Connection to node 0 (/98.1.96.147:9092) could not be established. Broker
may not be available. (org.apache.kafka.clients.NetworkClient)
So it looks like 98.1.96.147:9092 is actually accessible from the client
because otherwise, the behavior would not change after the server
configuration change (I guess). But if that's the case, why it doesn't work
afterward then? I'm sure I might be easily missing something, but this
seems a little confusing to me.
Unfortunately, I will most likely not be able to test this going forward,
as we're moving to a different set of servers with more straightforward
network setup. But thanks everyone for the help and for such quick
responses!
-Val
On Tue, Apr 28, 2020 at 10:20 AM Goran Sliskovic <gs...@yahoo.com.invalid>
wrote:
> Client connects to IP address defined by bootstrap.servers and gets
> metada that contain IP address where kafka is available. That is configured
> in server.properties:
> # Hostname and port the broker will advertise to producers and consumers.
> If not set, # it uses the value for "listeners" if configured. Otherwise,
> it will use the value# returned from
> java.net.InetAddress.getCanonicalHostName().#advertised.listeners=PLAINTEXT://
> your.host.name:9092
> you have to set that to 98.1.96.147:9092
> advertised.listeners=PLAINTEXT://98,1,96.147:9092
>
> Also you may need to change:
> # The address the socket server listens on. It will get the value returned
> from # java.net.InetAddress.getCanonicalHostName() if not configured.#
> FORMAT:# listeners = listener_name://host_name:port# EXAMPLE:#
> listeners = PLAINTEXT://your.host.name:9092#listeners=PLAINTEXT://:9092
> listeners=PLAINTEXT://98,1,96.147:9092
> <http://your.host.name:9092#listeners=PLAINTEXT://:9092listeners=PLAINTEXT://98,1,96.147:9092>
>
> Link to article:
> Kafka Listeners - Explained
>
> |
> |
> |
> | | |
>
> |
>
> |
> |
> | |
> Kafka Listeners - Explained
>
> How to connect clients to Kafka hosted in separate networks, such as
> Docker, AWS EC2, GCP, Azure, etc
> |
>
> |
>
> |
>
>
>
>
>
>
>
> On Tuesday, April 28, 2020, 07:04:42 PM GMT+2, Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
> Hi Suresh,
> Thanks for your quick response. All the configuration files, except for
> the connect-standalone.properties are unmodified and use all the default
> parameters. Attaching them anyway just in case.
> This setup works perfectly on my laptop with VirtualBox (ZooKeeper and the
> broker on VM, the connector on the host). So the issue seems to be
> dependable on the environment.
> Are there any particular scenarios that can cause Kafka Connect to try
> localhost even if bootstrap.servers parameter provides a different IP? NAT?
> Port forwarding? Something else?
> I'm trying to get more details about the environment but would appreciate
> any pointers in the meantime.
> -Val
> On Tue, Apr 28, 2020 at 12:07 AM Suresh Chidambaram <
> chida.suresh@gmail.com> wrote:
>
> Hi Val,
>
> Could you share the server.properties and zookeeper.properties?
>
> Thanks
> C Suresh
>
>
> On Tuesday, April 28, 2020, Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
> > Greetings to the Kafka Community!
> >
> > I'm a newbie in Kafka and only recently went beyond a local installation
> > described in the Quickstart. I have faced a weird issue that I can't
> > explain.
> >
> > I want to deploy on two machines:
> > - Machine #1 runs ZooKeeper and a single Kafka broker. I use default
> > configuration files.
> > - Machine #2 runs a single instance of Kafka Connect. I use the source
> > connector for Ignite/GridGain (https://www.confluent.io/hub/
> > gridgain/ignite-connector). I doubt it matters, though, as the issue
> > seems to be related to connectivity between the connector and the broker.
> >
> > After starting the ZooKeeper and the broker on machine #1, I did the
> > following on machine #2:
> >
> > 1. Updated plugin.path and bootstrap.servers parameters in the
> > connect-standalone.properties; bootstrap.servers points to the IP of the
> #1.
> > 2. Created a properties file for the GridGain connector.
> > 3. Start the connector using this command:
> >
> > bin/connect-standalone.sh config/connect-standalone.properties
> > config/gridgain-kafka-connect-source.properties
> >
> > The issue is that it doesn't connect to the remote broker, and still
> tries
> > localhost ignoring the bootstrap.servers settings. The output looks like
> > this:
> >
> > [2020-04-27 19:37:08,334] INFO StandaloneConfig values:
> > ...
> > bootstrap.servers = [98.1.96.147:9092]
> > ...
> > [2020-04-27 19:37:08,334] INFO Creating Kafka admin client
> > (org.apache.kafka.connect.util.ConnectUtils:43)
> > [2020-04-27 19:37:08,337] INFO AdminClientConfig values:
> > bootstrap.servers = [98.1.96.147:9092]
> > ...
> > [2020-04-27 19:37:08,950] WARN [AdminClient clientId=adminclient-1]
> > Connection to node 0 (localhost/127.0.0.1:9092) could not be
> established.
> > Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
> > [2020-04-27 19:37:09,052] WARN [AdminClient clientId=adminclient-1]
> > Connection to node 0 (localhost/127.0.0.1:9092) could not be
> established.
> > Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
> > [2020-04-27 19:37:09,253] WARN [AdminClient clientId=adminclient-1]
> > Connection to node 0 (localhost/127.0.0.1:9092) could not be
> established.
> > Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
> >
> > The warning goes on for a while until the process stops after a timeout.
> >
> > Does anyone have any idea why Kafka Connect can try to establish a
> > connection with the localhost, although it has picked up the correct IP
> > from the configuration file?
> >
> > The issue doesn't seem to be reproducible on any environment, so I
> > would appreciate any pointers on how to investigate this further.
> >
> > Kafka's version is 2.4.1. Config files and the log are attached. If
> > anything else is needed, let me know.
> >
> > Regards,
> > Val
> >
>
>
Re: Kafka Connect ignores the bootstrap.servers parameter
Posted by Goran Sliskovic <gs...@yahoo.com.INVALID>.
Client connects to IP address defined by bootstrap.servers and gets metada that contain IP address where kafka is available. That is configured in server.properties:
# Hostname and port the broker will advertise to producers and consumers. If not set, # it uses the value for "listeners" if configured. Otherwise, it will use the value# returned from java.net.InetAddress.getCanonicalHostName().#advertised.listeners=PLAINTEXT://your.host.name:9092
you have to set that to 98.1.96.147:9092
advertised.listeners=PLAINTEXT://98,1,96.147:9092
Also you may need to change:
# The address the socket server listens on. It will get the value returned from # java.net.InetAddress.getCanonicalHostName() if not configured.# FORMAT:# listeners = listener_name://host_name:port# EXAMPLE:# listeners = PLAINTEXT://your.host.name:9092#listeners=PLAINTEXT://:9092
listeners=PLAINTEXT://98,1,96.147:9092
Link to article:
Kafka Listeners - Explained
|
|
|
| | |
|
|
|
| |
Kafka Listeners - Explained
How to connect clients to Kafka hosted in separate networks, such as Docker, AWS EC2, GCP, Azure, etc
|
|
|
On Tuesday, April 28, 2020, 07:04:42 PM GMT+2, Valentin Kulichenko <va...@gmail.com> wrote:
Hi Suresh,
Thanks for your quick response. All the configuration files, except for the connect-standalone.properties are unmodified and use all the default parameters. Attaching them anyway just in case.
This setup works perfectly on my laptop with VirtualBox (ZooKeeper and the broker on VM, the connector on the host). So the issue seems to be dependable on the environment.
Are there any particular scenarios that can cause Kafka Connect to try localhost even if bootstrap.servers parameter provides a different IP? NAT? Port forwarding? Something else?
I'm trying to get more details about the environment but would appreciate any pointers in the meantime.
-Val
On Tue, Apr 28, 2020 at 12:07 AM Suresh Chidambaram <ch...@gmail.com> wrote:
Hi Val,
Could you share the server.properties and zookeeper.properties?
Thanks
C Suresh
On Tuesday, April 28, 2020, Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Greetings to the Kafka Community!
>
> I'm a newbie in Kafka and only recently went beyond a local installation
> described in the Quickstart. I have faced a weird issue that I can't
> explain.
>
> I want to deploy on two machines:
> - Machine #1 runs ZooKeeper and a single Kafka broker. I use default
> configuration files.
> - Machine #2 runs a single instance of Kafka Connect. I use the source
> connector for Ignite/GridGain (https://www.confluent.io/hub/
> gridgain/ignite-connector). I doubt it matters, though, as the issue
> seems to be related to connectivity between the connector and the broker.
>
> After starting the ZooKeeper and the broker on machine #1, I did the
> following on machine #2:
>
> 1. Updated plugin.path and bootstrap.servers parameters in the
> connect-standalone.properties; bootstrap.servers points to the IP of the #1.
> 2. Created a properties file for the GridGain connector.
> 3. Start the connector using this command:
>
> bin/connect-standalone.sh config/connect-standalone.properties
> config/gridgain-kafka-connect-source.properties
>
> The issue is that it doesn't connect to the remote broker, and still tries
> localhost ignoring the bootstrap.servers settings. The output looks like
> this:
>
> [2020-04-27 19:37:08,334] INFO StandaloneConfig values:
> ...
> bootstrap.servers = [98.1.96.147:9092]
> ...
> [2020-04-27 19:37:08,334] INFO Creating Kafka admin client
> (org.apache.kafka.connect.util.ConnectUtils:43)
> [2020-04-27 19:37:08,337] INFO AdminClientConfig values:
> bootstrap.servers = [98.1.96.147:9092]
> ...
> [2020-04-27 19:37:08,950] WARN [AdminClient clientId=adminclient-1]
> Connection to node 0 (localhost/127.0.0.1:9092) could not be established.
> Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
> [2020-04-27 19:37:09,052] WARN [AdminClient clientId=adminclient-1]
> Connection to node 0 (localhost/127.0.0.1:9092) could not be established.
> Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
> [2020-04-27 19:37:09,253] WARN [AdminClient clientId=adminclient-1]
> Connection to node 0 (localhost/127.0.0.1:9092) could not be established.
> Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
>
> The warning goes on for a while until the process stops after a timeout.
>
> Does anyone have any idea why Kafka Connect can try to establish a
> connection with the localhost, although it has picked up the correct IP
> from the configuration file?
>
> The issue doesn't seem to be reproducible on any environment, so I
> would appreciate any pointers on how to investigate this further.
>
> Kafka's version is 2.4.1. Config files and the log are attached. If
> anything else is needed, let me know.
>
> Regards,
> Val
>
Re: Kafka Connect ignores the bootstrap.servers parameter
Posted by Valentin Kulichenko <va...@gmail.com>.
Hi Suresh,
Thanks for your quick response. All the configuration files, except for
the connect-standalone.properties are unmodified and use all the default
parameters. Attaching them anyway just in case.
This setup works perfectly on my laptop with VirtualBox (ZooKeeper and the
broker on VM, the connector on the host). So the issue seems to be
dependable on the environment.
Are there any particular scenarios that can cause Kafka Connect to try
localhost even if bootstrap.servers parameter provides a different IP? NAT?
Port forwarding? Something else?
I'm trying to get more details about the environment but would appreciate
any pointers in the meantime.
-Val
On Tue, Apr 28, 2020 at 12:07 AM Suresh Chidambaram <ch...@gmail.com>
wrote:
> Hi Val,
>
> Could you share the server.properties and zookeeper.properties?
>
> Thanks
> C Suresh
>
>
> On Tuesday, April 28, 2020, Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
> > Greetings to the Kafka Community!
> >
> > I'm a newbie in Kafka and only recently went beyond a local installation
> > described in the Quickstart. I have faced a weird issue that I can't
> > explain.
> >
> > I want to deploy on two machines:
> > - Machine #1 runs ZooKeeper and a single Kafka broker. I use default
> > configuration files.
> > - Machine #2 runs a single instance of Kafka Connect. I use the source
> > connector for Ignite/GridGain (https://www.confluent.io/hub/
> > gridgain/ignite-connector). I doubt it matters, though, as the issue
> > seems to be related to connectivity between the connector and the broker.
> >
> > After starting the ZooKeeper and the broker on machine #1, I did the
> > following on machine #2:
> >
> > 1. Updated plugin.path and bootstrap.servers parameters in the
> > connect-standalone.properties; bootstrap.servers points to the IP of the
> #1.
> > 2. Created a properties file for the GridGain connector.
> > 3. Start the connector using this command:
> >
> > bin/connect-standalone.sh config/connect-standalone.properties
> > config/gridgain-kafka-connect-source.properties
> >
> > The issue is that it doesn't connect to the remote broker, and still
> tries
> > localhost ignoring the bootstrap.servers settings. The output looks like
> > this:
> >
> > [2020-04-27 19:37:08,334] INFO StandaloneConfig values:
> > ...
> > bootstrap.servers = [98.1.96.147:9092]
> > ...
> > [2020-04-27 19:37:08,334] INFO Creating Kafka admin client
> > (org.apache.kafka.connect.util.ConnectUtils:43)
> > [2020-04-27 19:37:08,337] INFO AdminClientConfig values:
> > bootstrap.servers = [98.1.96.147:9092]
> > ...
> > [2020-04-27 19:37:08,950] WARN [AdminClient clientId=adminclient-1]
> > Connection to node 0 (localhost/127.0.0.1:9092) could not be
> established.
> > Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
> > [2020-04-27 19:37:09,052] WARN [AdminClient clientId=adminclient-1]
> > Connection to node 0 (localhost/127.0.0.1:9092) could not be
> established.
> > Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
> > [2020-04-27 19:37:09,253] WARN [AdminClient clientId=adminclient-1]
> > Connection to node 0 (localhost/127.0.0.1:9092) could not be
> established.
> > Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
> >
> > The warning goes on for a while until the process stops after a timeout.
> >
> > Does anyone have any idea why Kafka Connect can try to establish a
> > connection with the localhost, although it has picked up the correct IP
> > from the configuration file?
> >
> > The issue doesn't seem to be reproducible on any environment, so I
> > would appreciate any pointers on how to investigate this further.
> >
> > Kafka's version is 2.4.1. Config files and the log are attached. If
> > anything else is needed, let me know.
> >
> > Regards,
> > Val
> >
>
Kafka Connect ignores the bootstrap.servers parameter
Posted by Suresh Chidambaram <ch...@gmail.com>.
Hi Val,
Could you share the server.properties and zookeeper.properties?
Thanks
C Suresh
On Tuesday, April 28, 2020, Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Greetings to the Kafka Community!
>
> I'm a newbie in Kafka and only recently went beyond a local installation
> described in the Quickstart. I have faced a weird issue that I can't
> explain.
>
> I want to deploy on two machines:
> - Machine #1 runs ZooKeeper and a single Kafka broker. I use default
> configuration files.
> - Machine #2 runs a single instance of Kafka Connect. I use the source
> connector for Ignite/GridGain (https://www.confluent.io/hub/
> gridgain/ignite-connector). I doubt it matters, though, as the issue
> seems to be related to connectivity between the connector and the broker.
>
> After starting the ZooKeeper and the broker on machine #1, I did the
> following on machine #2:
>
> 1. Updated plugin.path and bootstrap.servers parameters in the
> connect-standalone.properties; bootstrap.servers points to the IP of the #1.
> 2. Created a properties file for the GridGain connector.
> 3. Start the connector using this command:
>
> bin/connect-standalone.sh config/connect-standalone.properties
> config/gridgain-kafka-connect-source.properties
>
> The issue is that it doesn't connect to the remote broker, and still tries
> localhost ignoring the bootstrap.servers settings. The output looks like
> this:
>
> [2020-04-27 19:37:08,334] INFO StandaloneConfig values:
> ...
> bootstrap.servers = [98.1.96.147:9092]
> ...
> [2020-04-27 19:37:08,334] INFO Creating Kafka admin client
> (org.apache.kafka.connect.util.ConnectUtils:43)
> [2020-04-27 19:37:08,337] INFO AdminClientConfig values:
> bootstrap.servers = [98.1.96.147:9092]
> ...
> [2020-04-27 19:37:08,950] WARN [AdminClient clientId=adminclient-1]
> Connection to node 0 (localhost/127.0.0.1:9092) could not be established.
> Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
> [2020-04-27 19:37:09,052] WARN [AdminClient clientId=adminclient-1]
> Connection to node 0 (localhost/127.0.0.1:9092) could not be established.
> Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
> [2020-04-27 19:37:09,253] WARN [AdminClient clientId=adminclient-1]
> Connection to node 0 (localhost/127.0.0.1:9092) could not be established.
> Broker may not be available. (org.apache.kafka.clients.NetworkClient:756)
>
> The warning goes on for a while until the process stops after a timeout.
>
> Does anyone have any idea why Kafka Connect can try to establish a
> connection with the localhost, although it has picked up the correct IP
> from the configuration file?
>
> The issue doesn't seem to be reproducible on any environment, so I
> would appreciate any pointers on how to investigate this further.
>
> Kafka's version is 2.4.1. Config files and the log are attached. If
> anything else is needed, let me know.
>
> Regards,
> Val
>