You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mich Talebzadeh <mi...@gmail.com> on 2018/07/14 18:42:41 UTC
Accessing Zookepper and Kafka through designated ports in a container
Hi,
I have created an Ubuntu image docker on a physical host running RHES75.
I installed both Zookeeper and Kafka binaries in this container and they
both started OK, I ran my Zookeeper on port 3400 and Kafka running on port
12092
hduser@bbe7e47d5609: /home/hduser> jps
7026 QuorumPeerMain
7079 Kafka
I can connect from contain to any host as long /etc/hosts has the relevant
host information.
I mapped port 3400 and 12092 on the physical host running the container as
port forwarding.
[root@rhes75 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND
CREATED STATUS PORTS NAMES
bbe7e47d5609 ubuntu "bash" 28 hours
ago Up 25 hours testdocker
I did the mapping of ports as follows:
iptables -t nat -A DOCKER -p tcp --dport 4300 -j DNAT --to-destination
172.17.0.2:4300
iptables -t nat -A DOCKER -p tcp --dport 12092 -j DNAT --to-destination
172.17.0.2:12092
So if I go to another host and try to telnet to port 4300 on the docker it
works
hduser@rhes564: /home/hduser/dba/log> telnet rhes75 4300
Trying 50.140.197.220...
Connected to rhes75 (50.140.197.220).
Escape character is '^]'.
hduser@rhes564: /home/hduser/dba/log> telnet rhes75 12092
Trying 50.140.197.220...
Connected to rhes75 (50.140.197.220).
Escape character is '^]'.
If I stop Zookeeper (running on 4300), the connection will be refused
I created a topic called "test" on a remote host as follows
${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:4300
--replication-factor 1 --partitions 1 --topic test
And if I go into docker I can see that topic on localhost:4300
${KAFKA_HOME}/bin/kafka-topics.sh --describe --zookeeper *localhost:4300*
--topic test
Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 1 Replicas: 1
Isr: 1
The problem I see is that I cannot publish messages to the broker on this
container
From remote host I start publishing some test prices
cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh --broker-list
rhes75:12092 --topic test --property "parse.key=true" --property
"key.separator=:"
I get this error that indicates there is an issue talking to broker on the
container!
[2018-07-14 19:49:22,838] WARN [Producer clientId=console-producer]
Connection to node 1 could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
[2018-07-14 19:49:22,925] WARN [Producer clientId=console-producer]
Connection to node 1 could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
[2018-07-14 19:49:23,088] WARN [Producer clientId=console-producer]
Connection to node 1 could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
[2018-07-14 19:49:23,491] WARN [Producer clientId=console-producer]
Connection to node 1 could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
[2018-07-14 19:49:24,188] WARN [Producer clientId=console-producer]
Connection to node 1 could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
[2018-07-14 19:49:25,124] WARN [Producer clientId=console-producer]
Connection to node 1 could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
[2018-07-14 19:49:25,326] ERROR Error when sending message to topic test
with key: 54 bytes, value: 12 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s) for
test-0: 1542 ms has passed since batch creation plus linger time
[2018-07-14 19:49:25,329] ERROR Error when sending message to topic test
with key: 54 bytes, value: 12 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s) for
test-0: 1542 ms has passed since batch creation plus linger time
[2018-07-14 19:49:25,329] ERROR Error when sending message to topic test
with key: 55 bytes, value: 11 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s) for
test-0: 1542 ms has passed since batch creation plus linger time
I have been working on this for a day or two and have not found any
resolution yet!
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
Re: Accessing Zookepper and Kafka through designated ports in a container
Posted by "M. Manna" <ma...@gmail.com>.
So was it your sasl or something else?
Regards,
On Sun, 15 Jul 2018 at 08:40, Mich Talebzadeh <mi...@gmail.com>
wrote:
> resolved this now
>
> Thanks
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn *
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sat, 14 Jul 2018 at 22:40, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
> > Some additional info from the client side
> >
> > [zk: rhes75:4300(CONNECTED) 3] *ls /brokers/topics * #Gives the list of
> > topics
> > [test, md]
> > [zk: rhes75:4300(CONNECTED) 5] *ls /brokers/ids*
> > [1]
> > [zk: rhes75:4300(CONNECTED) 4] *get /brokers/ids/1 * #detailed info about
> > broker ID '1'
> >
> >
> {"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://localhost:12092"],"jmx_port":9999,"host":"localhost","timestamp":"1531512017557","port":12092,"version":4}
> > cZxid = 0x16f
> > ctime = Fri Jul 13 21:00:17 BST 2018
> > mZxid = 0x16f
> > mtime = Fri Jul 13 21:00:17 BST 2018
> > pZxid = 0x16f
> > cversion = 0
> > dataVersion = 0
> > aclVersion = 0
> > ephemeralOwner = 0x164953a10100000
> > dataLength = 192
> > numChildren = 0
> >
> > [zk: rhes75:4300(CONNECTED) 6] *get /controller*
> > {"version":1,"brokerid":1,"timestamp":"1531512017622"}
> > cZxid = 0x170
> > ctime = Fri Jul 13 21:00:17 BST 2018
> > mZxid = 0x170
> > mtime = Fri Jul 13 21:00:17 BST 2018
> > pZxid = 0x170
> > cversion = 0
> > dataVersion = 0
> > aclVersion = 0
> > ephemeralOwner = 0x164953a10100000
> > dataLength = 54
> > numChildren = 0
> >
> > Dr Mich Talebzadeh
> >
> >
> >
> > LinkedIn *
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > <
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >*
> >
> >
> >
> > http://talebzadehmich.wordpress.com
> >
> >
> > *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> > loss, damage or destruction of data or any other property which may arise
> > from relying on this email's technical content is explicitly disclaimed.
> > The author will in no case be liable for any monetary damages arising
> from
> > such loss, damage or destruction.
> >
> >
> >
> >
> > On Sat, 14 Jul 2018 at 21:20, Mich Talebzadeh <mich.talebzadeh@gmail.com
> >
> > wrote:
> >
> >> Apologies correction. Got the port number incorrect. It should be 4300
> >>
> >> on the docker itself confirming Zookeeper port
> >>
> >> *netstat -plten|grep 4300*
> >> (Not all processes could be identified, non-owned process info
> >> will not be shown, you would have to be root to see it all.)
> >> tcp 0 0 0.0.0.0:4300 0.0.0.0:*
> >> LISTEN 1000 4638251 7026/java
> >>
> >> *jps|grep 7026*7026 QuorumPeerMain
> >>
> >> On the remote host
> >>
> >> hduser@rhes564: /home/hduser/zookeeper-3.4.6/bin> ./zkCli.sh -server
> >> rhes75:4300
> >> *Connecting to rhes75:4300*
> >> 2018-07-14 21:27:35,560 [myid:] - INFO [main:Environment@100] - Client
> >> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
> GMT
> >> 2018-07-14 21:27:35,563 [myid:] - INFO [main:Environment@100] - Client
> >> environment:host.name=rhes564
> >> 2018-07-14 21:27:35,563 [myid:] - INFO [main:Environment@100] - Client
> >> environment:java.version=1.8.0_77
> >> 2018-07-14 21:27:35,565 [myid:] - INFO [main:Environment@100] - Client
> >> environment:java.vendor=Oracle Corporation
> >> 2018-07-14 21:27:35,565 [myid:] - INFO [main:Environment@100] - Client
> >> environment:java.home=/usr/java/jdk1.8.0_77/jre
> >> 2018-07-14 21:27:35,565 [myid:] - INFO [main:Environment@100] - Client
> >>
> environment:java.class.path=/home/hduser/zookeeper-3.4.6/bin/../build/classes:/home/hduser/zookeeper-3.4.6/bin/../build/lib/*.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/netty-3.7.0.Final.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/log4j-1.2.16.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper-3.4.6/bin/../zookeeper-3.4.6.jar:/home/hduser/zookeeper-3.4.6/bin/../src/java/lib/*.jar:/home/hduser/zookeeper-3.4.6/bin/../conf:
> >> 2018-07-14 21:27:35,565 [myid:] - INFO [main:Environment@100] - Client
> >>
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> >> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
> >> environment:java.io.tmpdir=/tmp
> >> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
> >> environment:java.compiler=<NA>
> >> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
> >> environment:os.name=Linux
> >> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
> >> environment:os.arch=amd64
> >> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
> >> environment:os.version=2.6.18-92.el5
> >> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
> >> environment:user.name=hduser
> >> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
> >> environment:user.home=/home/hduser
> >> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
> >> environment:user.dir=/home/hduser/zookeeper-3.4.6/bin
> >> 2018-07-14 21:27:35,567 [myid:] - INFO [main:ZooKeeper@438] -
> >> Initiating client connection, connectString=rhes75:4300
> >> sessionTimeout=30000
> >> watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589e
> >> Welcome to ZooKeeper!
> >> 2018-07-14 21:27:35,588 [myid:] - INFO
> >> [main-SendThread(rhes75:4300):ClientCnxn$SendThread@975] - Opening
> >> socket connection to server rhes75/50.140.197.220:4300. Will not
> attempt
> >> to authenticate using SASL (unknown error)
> >> JLine support is enabled
> >> 2018-07-14 21:27:35,644 [myid:] - INFO
> >> [main-SendThread(rhes75:4300):ClientCnxn$SendThread@852] - Socket
> >> connection established to rhes75/50.140.197.220:4300, initiating
> session
> >> [zk: rhes75:4300(CONNECTING) 0] 2018-07-14 21:27:35,666 [myid:] - INFO
> >> [main-SendThread(rhes75:4300):ClientCnxn$SendThread@1235] - Session
> >> establishment complete on server rhes75/50.140.197.220:4300, sessionid
> =
> >> 0x164953a1010000a, negotiated timeout = 30000
> >> WATCHER::
> >> WatchedEvent state:SyncConnected type:None path:null
> >>
> >>
> >> Dr Mich Talebzadeh
> >>
> >>
> >>
> >> LinkedIn *
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >> <
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >*
> >>
> >>
> >>
> >> http://talebzadehmich.wordpress.com
> >>
> >>
> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for
> >> any loss, damage or destruction of data or any other property which may
> >> arise from relying on this email's technical content is explicitly
> >> disclaimed. The author will in no case be liable for any monetary
> damages
> >> arising from such loss, damage or destruction.
> >>
> >>
> >>
> >>
> >> On Sat, 14 Jul 2018 at 21:01, Mich Talebzadeh <
> mich.talebzadeh@gmail.com>
> >> wrote:
> >>
> >>> Hi
> >>>
> >>> This is from remote host
> >>>
> >>> ./zkCli.sh -server rhes75:3400
> >>> *Connecting to rhes75:3400*
> >>> 2018-07-14 21:08:41,104 [myid:] - INFO [main:Environment@100] -
> Client
> >>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
> GMT
> >>> 2018-07-14 21:08:41,107 [myid:] - INFO [main:Environment@100] -
> Client
> >>> environment:host.name=rhes564
> >>> 2018-07-14 21:08:41,107 [myid:] - INFO [main:Environment@100] -
> Client
> >>> environment:java.version=1.8.0_77
> >>> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] -
> Client
> >>> environment:java.vendor=Oracle Corporation
> >>> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] -
> Client
> >>> environment:java.home=/usr/java/jdk1.8.0_77/jre
> >>> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] -
> Client
> >>>
> environment:java.class.path=/home/hduser/zookeeper-3.4.6/bin/../build/classes:/home/hduser/zookeeper-3.4.6/bin/../build/lib/*.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/netty-3.7.0.Final.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/log4j-1.2.16.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper-3.4.6/bin/../zookeeper-3.4.6.jar:/home/hduser/zookeeper-3.4.6/bin/../src/java/lib/*.jar:/home/hduser/zookeeper-3.4.6/bin/../conf:
> >>> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] -
> Client
> >>>
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> >>> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] -
> Client
> >>> environment:java.io.tmpdir=/tmp
> >>> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] -
> Client
> >>> environment:java.compiler=<NA>
> >>> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] -
> Client
> >>> environment:os.name=Linux
> >>> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] -
> Client
> >>> environment:os.arch=amd64
> >>> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] -
> Client
> >>> environment:os.version=2.6.18-92.el5
> >>> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] -
> Client
> >>> environment:user.name=hduser
> >>> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] -
> Client
> >>> environment:user.home=/home/hduser
> >>> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] -
> Client
> >>> environment:user.dir=/home/hduser/zookeeper-3.4.6/bin
> >>> 2018-07-14 21:08:41,111 [myid:] - INFO [main:ZooKeeper@438] -
> >>> Initiating client connection, connectString=rhes75:3400
> >>> sessionTimeout=30000
> >>> watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589e
> >>> Welcome to ZooKeeper!
> >>> 2018-07-14 21:08:41,131 [myid:] - INFO
> >>> [main-SendThread(rhes75:3400):ClientCnxn$SendThread@975] - Opening
> >>> socket connection to server rhes75/50.140.197.220:3400. Will not
> >>> attempt to authenticate using SASL (unknown error)
> >>> JLine support is enabled
> >>> 2018-07-14 21:08:41,191 [myid:] - WARN
> >>> [main-SendThread(rhes75:3400):ClientCnxn$SendThread@1102] - Session
> 0x0
> >>> for server null, unexpected error, closing socket connection and
> attempting
> >>> reconnect
> >>> java.net.ConnectException: Connection refused
> >>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>> at
> >>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> >>> at
> >>>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> >>> at
> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> >>> [zk: rhes75:3400(CONNECTING) 0] 2018-07-14 21:08:42,299 [myid:] - INFO
> >>> [main-SendThread(rhes75:3400):ClientCnxn$SendThread@975] - Opening
> >>> socket connection to server rhes75/50.140.197.220:3400. Will not
> >>> attempt to authenticate using SASL (unknown error)
> >>> 2018-07-14 21:08:42,300 [myid:] - WARN
> >>> [main-SendThread(rhes75:3400):ClientCnxn$SendThread@1102] - Session
> 0x0
> >>> for server null, unexpected error, closing socket connection and
> attempting
> >>> reconnect
> >>>
> >>> Thanks
> >>>
> >>> Dr Mich Talebzadeh
> >>>
> >>>
> >>>
> >>> LinkedIn *
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >>> <
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >*
> >>>
> >>>
> >>>
> >>> http://talebzadehmich.wordpress.com
> >>>
> >>>
> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
> >>> any loss, damage or destruction of data or any other property which may
> >>> arise from relying on this email's technical content is explicitly
> >>> disclaimed. The author will in no case be liable for any monetary
> damages
> >>> arising from such loss, damage or destruction.
> >>>
> >>>
> >>>
> >>>
> >>> On Sat, 14 Jul 2018 at 20:34, M. Manna <ma...@gmail.com> wrote:
> >>>
> >>>> Can you print out zookeeper admin:
> >>>>
> >>>> ./zookeeper-shell.sh rhes75:3400
> >>>>
> >>>> And show us here?
> >>>>
> >>>> On Sat, 14 Jul 2018, 19:43 Mich Talebzadeh, <
> mich.talebzadeh@gmail.com>
> >>>> wrote:
> >>>>
> >>>> > Hi,
> >>>> >
> >>>> > I have created an Ubuntu image docker on a physical host running
> >>>> RHES75.
> >>>> >
> >>>> > I installed both Zookeeper and Kafka binaries in this container and
> >>>> they
> >>>> > both started OK, I ran my Zookeeper on port 3400 and Kafka running
> on
> >>>> port
> >>>> > 12092
> >>>> >
> >>>> > hduser@bbe7e47d5609: /home/hduser> jps
> >>>> > 7026 QuorumPeerMain
> >>>> > 7079 Kafka
> >>>> >
> >>>> > I can connect from contain to any host as long /etc/hosts has the
> >>>> relevant
> >>>> > host information.
> >>>> >
> >>>> > I mapped port 3400 and 12092 on the physical host running the
> >>>> container as
> >>>> > port forwarding.
> >>>> >
> >>>> > [root@rhes75 ~]# docker ps -a
> >>>> > CONTAINER ID IMAGE COMMAND
> >>>> > CREATED STATUS PORTS NAMES
> >>>> > bbe7e47d5609 ubuntu "bash" 28 hours
> >>>> > ago Up 25 hours testdocker
> >>>> >
> >>>> > I did the mapping of ports as follows:
> >>>> >
> >>>> > iptables -t nat -A DOCKER -p tcp --dport 4300 -j DNAT
> --to-destination
> >>>> > 172.17.0.2:4300
> >>>> > iptables -t nat -A DOCKER -p tcp --dport 12092 -j DNAT
> >>>> --to-destination
> >>>> > 172.17.0.2:12092
> >>>> >
> >>>> > So if I go to another host and try to telnet to port 4300 on the
> >>>> docker it
> >>>> > works
> >>>> >
> >>>> >
> >>>> > hduser@rhes564: /home/hduser/dba/log> telnet rhes75 4300
> >>>> > Trying 50.140.197.220...
> >>>> > Connected to rhes75 (50.140.197.220).
> >>>> > Escape character is '^]'.
> >>>> >
> >>>> >
> >>>> > hduser@rhes564: /home/hduser/dba/log> telnet rhes75 12092
> >>>> > Trying 50.140.197.220...
> >>>> > Connected to rhes75 (50.140.197.220).
> >>>> > Escape character is '^]'.
> >>>> >
> >>>> >
> >>>> > If I stop Zookeeper (running on 4300), the connection will be
> refused
> >>>> >
> >>>> > I created a topic called "test" on a remote host as follows
> >>>> >
> >>>> > ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:4300
> >>>> > --replication-factor 1 --partitions 1 --topic test
> >>>> >
> >>>> > And if I go into docker I can see that topic on localhost:4300
> >>>> >
> >>>> > ${KAFKA_HOME}/bin/kafka-topics.sh --describe --zookeeper
> >>>> *localhost:4300*
> >>>> > --topic test
> >>>> > Topic:test PartitionCount:1 ReplicationFactor:1
> >>>> Configs:
> >>>> > Topic: test Partition: 0 Leader: 1 Replicas: 1
> >>>> > Isr: 1
> >>>> >
> >>>> > The problem I see is that I cannot publish messages to the broker on
> >>>> this
> >>>> > container
> >>>> >
> >>>> > From remote host I start publishing some test prices
> >>>> > cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh
> >>>> --broker-list
> >>>> > rhes75:12092 --topic test --property "parse.key=true" --property
> >>>> > "key.separator=:"
> >>>> >
> >>>> > I get this error that indicates there is an issue talking to broker
> >>>> on the
> >>>> > container!
> >>>> >
> >>>> > [2018-07-14 19:49:22,838] WARN [Producer clientId=console-producer]
> >>>> > Connection to node 1 could not be established. Broker may not be
> >>>> available.
> >>>> > (org.apache.kafka.clients.NetworkClient)
> >>>> > [2018-07-14 19:49:22,925] WARN [Producer clientId=console-producer]
> >>>> > Connection to node 1 could not be established. Broker may not be
> >>>> available.
> >>>> > (org.apache.kafka.clients.NetworkClient)
> >>>> > [2018-07-14 19:49:23,088] WARN [Producer clientId=console-producer]
> >>>> > Connection to node 1 could not be established. Broker may not be
> >>>> available.
> >>>> > (org.apache.kafka.clients.NetworkClient)
> >>>> > [2018-07-14 19:49:23,491] WARN [Producer clientId=console-producer]
> >>>> > Connection to node 1 could not be established. Broker may not be
> >>>> available.
> >>>> > (org.apache.kafka.clients.NetworkClient)
> >>>> > [2018-07-14 19:49:24,188] WARN [Producer clientId=console-producer]
> >>>> > Connection to node 1 could not be established. Broker may not be
> >>>> available.
> >>>> > (org.apache.kafka.clients.NetworkClient)
> >>>> > [2018-07-14 19:49:25,124] WARN [Producer clientId=console-producer]
> >>>> > Connection to node 1 could not be established. Broker may not be
> >>>> available.
> >>>> > (org.apache.kafka.clients.NetworkClient)
> >>>> > [2018-07-14 19:49:25,326] ERROR Error when sending message to topic
> >>>> test
> >>>> > with key: 54 bytes, value: 12 bytes with error:
> >>>> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> >>>> > org.apache.kafka.common.errors.TimeoutException: Expiring 10
> >>>> record(s) for
> >>>> > test-0: 1542 ms has passed since batch creation plus linger time
> >>>> > [2018-07-14 19:49:25,329] ERROR Error when sending message to topic
> >>>> test
> >>>> > with key: 54 bytes, value: 12 bytes with error:
> >>>> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> >>>> > org.apache.kafka.common.errors.TimeoutException: Expiring 10
> >>>> record(s) for
> >>>> > test-0: 1542 ms has passed since batch creation plus linger time
> >>>> > [2018-07-14 19:49:25,329] ERROR Error when sending message to topic
> >>>> test
> >>>> > with key: 55 bytes, value: 11 bytes with error:
> >>>> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> >>>> > org.apache.kafka.common.errors.TimeoutException: Expiring 10
> >>>> record(s) for
> >>>> > test-0: 1542 ms has passed since batch creation plus linger time
> >>>> >
> >>>> > I have been working on this for a day or two and have not found any
> >>>> > resolution yet!
> >>>> >
> >>>> > Thanks
> >>>> >
> >>>> > Dr Mich Talebzadeh
> >>>> >
> >>>> >
> >>>> >
> >>>> > LinkedIn *
> >>>> >
> >>>>
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >>>> > <
> >>>> >
> >>>>
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >>>> > >*
> >>>> >
> >>>> >
> >>>> >
> >>>> > http://talebzadehmich.wordpress.com
> >>>> >
> >>>> >
> >>>> > *Disclaimer:* Use it at your own risk. Any and all responsibility
> for
> >>>> any
> >>>> > loss, damage or destruction of data or any other property which may
> >>>> arise
> >>>> > from relying on this email's technical content is explicitly
> >>>> disclaimed.
> >>>> > The author will in no case be liable for any monetary damages
> arising
> >>>> from
> >>>> > such loss, damage or destruction.
> >>>> >
> >>>>
> >>>
>
Re: Accessing Zookepper and Kafka through designated ports in a container
Posted by Mich Talebzadeh <mi...@gmail.com>.
resolved this now
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Sat, 14 Jul 2018 at 22:40, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Some additional info from the client side
>
> [zk: rhes75:4300(CONNECTED) 3] *ls /brokers/topics * #Gives the list of
> topics
> [test, md]
> [zk: rhes75:4300(CONNECTED) 5] *ls /brokers/ids*
> [1]
> [zk: rhes75:4300(CONNECTED) 4] *get /brokers/ids/1 * #detailed info about
> broker ID '1'
>
> {"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://localhost:12092"],"jmx_port":9999,"host":"localhost","timestamp":"1531512017557","port":12092,"version":4}
> cZxid = 0x16f
> ctime = Fri Jul 13 21:00:17 BST 2018
> mZxid = 0x16f
> mtime = Fri Jul 13 21:00:17 BST 2018
> pZxid = 0x16f
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x164953a10100000
> dataLength = 192
> numChildren = 0
>
> [zk: rhes75:4300(CONNECTED) 6] *get /controller*
> {"version":1,"brokerid":1,"timestamp":"1531512017622"}
> cZxid = 0x170
> ctime = Fri Jul 13 21:00:17 BST 2018
> mZxid = 0x170
> mtime = Fri Jul 13 21:00:17 BST 2018
> pZxid = 0x170
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x164953a10100000
> dataLength = 54
> numChildren = 0
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sat, 14 Jul 2018 at 21:20, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Apologies correction. Got the port number incorrect. It should be 4300
>>
>> on the docker itself confirming Zookeeper port
>>
>> *netstat -plten|grep 4300*
>> (Not all processes could be identified, non-owned process info
>> will not be shown, you would have to be root to see it all.)
>> tcp 0 0 0.0.0.0:4300 0.0.0.0:*
>> LISTEN 1000 4638251 7026/java
>>
>> *jps|grep 7026*7026 QuorumPeerMain
>>
>> On the remote host
>>
>> hduser@rhes564: /home/hduser/zookeeper-3.4.6/bin> ./zkCli.sh -server
>> rhes75:4300
>> *Connecting to rhes75:4300*
>> 2018-07-14 21:27:35,560 [myid:] - INFO [main:Environment@100] - Client
>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
>> 2018-07-14 21:27:35,563 [myid:] - INFO [main:Environment@100] - Client
>> environment:host.name=rhes564
>> 2018-07-14 21:27:35,563 [myid:] - INFO [main:Environment@100] - Client
>> environment:java.version=1.8.0_77
>> 2018-07-14 21:27:35,565 [myid:] - INFO [main:Environment@100] - Client
>> environment:java.vendor=Oracle Corporation
>> 2018-07-14 21:27:35,565 [myid:] - INFO [main:Environment@100] - Client
>> environment:java.home=/usr/java/jdk1.8.0_77/jre
>> 2018-07-14 21:27:35,565 [myid:] - INFO [main:Environment@100] - Client
>> environment:java.class.path=/home/hduser/zookeeper-3.4.6/bin/../build/classes:/home/hduser/zookeeper-3.4.6/bin/../build/lib/*.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/netty-3.7.0.Final.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/log4j-1.2.16.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper-3.4.6/bin/../zookeeper-3.4.6.jar:/home/hduser/zookeeper-3.4.6/bin/../src/java/lib/*.jar:/home/hduser/zookeeper-3.4.6/bin/../conf:
>> 2018-07-14 21:27:35,565 [myid:] - INFO [main:Environment@100] - Client
>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
>> environment:java.io.tmpdir=/tmp
>> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
>> environment:java.compiler=<NA>
>> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
>> environment:os.name=Linux
>> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
>> environment:os.arch=amd64
>> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
>> environment:os.version=2.6.18-92.el5
>> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
>> environment:user.name=hduser
>> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
>> environment:user.home=/home/hduser
>> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
>> environment:user.dir=/home/hduser/zookeeper-3.4.6/bin
>> 2018-07-14 21:27:35,567 [myid:] - INFO [main:ZooKeeper@438] -
>> Initiating client connection, connectString=rhes75:4300
>> sessionTimeout=30000
>> watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589e
>> Welcome to ZooKeeper!
>> 2018-07-14 21:27:35,588 [myid:] - INFO
>> [main-SendThread(rhes75:4300):ClientCnxn$SendThread@975] - Opening
>> socket connection to server rhes75/50.140.197.220:4300. Will not attempt
>> to authenticate using SASL (unknown error)
>> JLine support is enabled
>> 2018-07-14 21:27:35,644 [myid:] - INFO
>> [main-SendThread(rhes75:4300):ClientCnxn$SendThread@852] - Socket
>> connection established to rhes75/50.140.197.220:4300, initiating session
>> [zk: rhes75:4300(CONNECTING) 0] 2018-07-14 21:27:35,666 [myid:] - INFO
>> [main-SendThread(rhes75:4300):ClientCnxn$SendThread@1235] - Session
>> establishment complete on server rhes75/50.140.197.220:4300, sessionid =
>> 0x164953a1010000a, negotiated timeout = 30000
>> WATCHER::
>> WatchedEvent state:SyncConnected type:None path:null
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Sat, 14 Jul 2018 at 21:01, Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>>> Hi
>>>
>>> This is from remote host
>>>
>>> ./zkCli.sh -server rhes75:3400
>>> *Connecting to rhes75:3400*
>>> 2018-07-14 21:08:41,104 [myid:] - INFO [main:Environment@100] - Client
>>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
>>> 2018-07-14 21:08:41,107 [myid:] - INFO [main:Environment@100] - Client
>>> environment:host.name=rhes564
>>> 2018-07-14 21:08:41,107 [myid:] - INFO [main:Environment@100] - Client
>>> environment:java.version=1.8.0_77
>>> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
>>> environment:java.vendor=Oracle Corporation
>>> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
>>> environment:java.home=/usr/java/jdk1.8.0_77/jre
>>> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
>>> environment:java.class.path=/home/hduser/zookeeper-3.4.6/bin/../build/classes:/home/hduser/zookeeper-3.4.6/bin/../build/lib/*.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/netty-3.7.0.Final.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/log4j-1.2.16.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper-3.4.6/bin/../zookeeper-3.4.6.jar:/home/hduser/zookeeper-3.4.6/bin/../src/java/lib/*.jar:/home/hduser/zookeeper-3.4.6/bin/../conf:
>>> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
>>> environment:java.io.tmpdir=/tmp
>>> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
>>> environment:java.compiler=<NA>
>>> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
>>> environment:os.name=Linux
>>> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
>>> environment:os.arch=amd64
>>> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
>>> environment:os.version=2.6.18-92.el5
>>> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
>>> environment:user.name=hduser
>>> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
>>> environment:user.home=/home/hduser
>>> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
>>> environment:user.dir=/home/hduser/zookeeper-3.4.6/bin
>>> 2018-07-14 21:08:41,111 [myid:] - INFO [main:ZooKeeper@438] -
>>> Initiating client connection, connectString=rhes75:3400
>>> sessionTimeout=30000
>>> watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589e
>>> Welcome to ZooKeeper!
>>> 2018-07-14 21:08:41,131 [myid:] - INFO
>>> [main-SendThread(rhes75:3400):ClientCnxn$SendThread@975] - Opening
>>> socket connection to server rhes75/50.140.197.220:3400. Will not
>>> attempt to authenticate using SASL (unknown error)
>>> JLine support is enabled
>>> 2018-07-14 21:08:41,191 [myid:] - WARN
>>> [main-SendThread(rhes75:3400):ClientCnxn$SendThread@1102] - Session 0x0
>>> for server null, unexpected error, closing socket connection and attempting
>>> reconnect
>>> java.net.ConnectException: Connection refused
>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>> at
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
>>> at
>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
>>> [zk: rhes75:3400(CONNECTING) 0] 2018-07-14 21:08:42,299 [myid:] - INFO
>>> [main-SendThread(rhes75:3400):ClientCnxn$SendThread@975] - Opening
>>> socket connection to server rhes75/50.140.197.220:3400. Will not
>>> attempt to authenticate using SASL (unknown error)
>>> 2018-07-14 21:08:42,300 [myid:] - WARN
>>> [main-SendThread(rhes75:3400):ClientCnxn$SendThread@1102] - Session 0x0
>>> for server null, unexpected error, closing socket connection and attempting
>>> reconnect
>>>
>>> Thanks
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Sat, 14 Jul 2018 at 20:34, M. Manna <ma...@gmail.com> wrote:
>>>
>>>> Can you print out zookeeper admin:
>>>>
>>>> ./zookeeper-shell.sh rhes75:3400
>>>>
>>>> And show us here?
>>>>
>>>> On Sat, 14 Jul 2018, 19:43 Mich Talebzadeh, <mi...@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi,
>>>> >
>>>> > I have created an Ubuntu image docker on a physical host running
>>>> RHES75.
>>>> >
>>>> > I installed both Zookeeper and Kafka binaries in this container and
>>>> they
>>>> > both started OK, I ran my Zookeeper on port 3400 and Kafka running on
>>>> port
>>>> > 12092
>>>> >
>>>> > hduser@bbe7e47d5609: /home/hduser> jps
>>>> > 7026 QuorumPeerMain
>>>> > 7079 Kafka
>>>> >
>>>> > I can connect from contain to any host as long /etc/hosts has the
>>>> relevant
>>>> > host information.
>>>> >
>>>> > I mapped port 3400 and 12092 on the physical host running the
>>>> container as
>>>> > port forwarding.
>>>> >
>>>> > [root@rhes75 ~]# docker ps -a
>>>> > CONTAINER ID IMAGE COMMAND
>>>> > CREATED STATUS PORTS NAMES
>>>> > bbe7e47d5609 ubuntu "bash" 28 hours
>>>> > ago Up 25 hours testdocker
>>>> >
>>>> > I did the mapping of ports as follows:
>>>> >
>>>> > iptables -t nat -A DOCKER -p tcp --dport 4300 -j DNAT --to-destination
>>>> > 172.17.0.2:4300
>>>> > iptables -t nat -A DOCKER -p tcp --dport 12092 -j DNAT
>>>> --to-destination
>>>> > 172.17.0.2:12092
>>>> >
>>>> > So if I go to another host and try to telnet to port 4300 on the
>>>> docker it
>>>> > works
>>>> >
>>>> >
>>>> > hduser@rhes564: /home/hduser/dba/log> telnet rhes75 4300
>>>> > Trying 50.140.197.220...
>>>> > Connected to rhes75 (50.140.197.220).
>>>> > Escape character is '^]'.
>>>> >
>>>> >
>>>> > hduser@rhes564: /home/hduser/dba/log> telnet rhes75 12092
>>>> > Trying 50.140.197.220...
>>>> > Connected to rhes75 (50.140.197.220).
>>>> > Escape character is '^]'.
>>>> >
>>>> >
>>>> > If I stop Zookeeper (running on 4300), the connection will be refused
>>>> >
>>>> > I created a topic called "test" on a remote host as follows
>>>> >
>>>> > ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:4300
>>>> > --replication-factor 1 --partitions 1 --topic test
>>>> >
>>>> > And if I go into docker I can see that topic on localhost:4300
>>>> >
>>>> > ${KAFKA_HOME}/bin/kafka-topics.sh --describe --zookeeper
>>>> *localhost:4300*
>>>> > --topic test
>>>> > Topic:test PartitionCount:1 ReplicationFactor:1
>>>> Configs:
>>>> > Topic: test Partition: 0 Leader: 1 Replicas: 1
>>>> > Isr: 1
>>>> >
>>>> > The problem I see is that I cannot publish messages to the broker on
>>>> this
>>>> > container
>>>> >
>>>> > From remote host I start publishing some test prices
>>>> > cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh
>>>> --broker-list
>>>> > rhes75:12092 --topic test --property "parse.key=true" --property
>>>> > "key.separator=:"
>>>> >
>>>> > I get this error that indicates there is an issue talking to broker
>>>> on the
>>>> > container!
>>>> >
>>>> > [2018-07-14 19:49:22,838] WARN [Producer clientId=console-producer]
>>>> > Connection to node 1 could not be established. Broker may not be
>>>> available.
>>>> > (org.apache.kafka.clients.NetworkClient)
>>>> > [2018-07-14 19:49:22,925] WARN [Producer clientId=console-producer]
>>>> > Connection to node 1 could not be established. Broker may not be
>>>> available.
>>>> > (org.apache.kafka.clients.NetworkClient)
>>>> > [2018-07-14 19:49:23,088] WARN [Producer clientId=console-producer]
>>>> > Connection to node 1 could not be established. Broker may not be
>>>> available.
>>>> > (org.apache.kafka.clients.NetworkClient)
>>>> > [2018-07-14 19:49:23,491] WARN [Producer clientId=console-producer]
>>>> > Connection to node 1 could not be established. Broker may not be
>>>> available.
>>>> > (org.apache.kafka.clients.NetworkClient)
>>>> > [2018-07-14 19:49:24,188] WARN [Producer clientId=console-producer]
>>>> > Connection to node 1 could not be established. Broker may not be
>>>> available.
>>>> > (org.apache.kafka.clients.NetworkClient)
>>>> > [2018-07-14 19:49:25,124] WARN [Producer clientId=console-producer]
>>>> > Connection to node 1 could not be established. Broker may not be
>>>> available.
>>>> > (org.apache.kafka.clients.NetworkClient)
>>>> > [2018-07-14 19:49:25,326] ERROR Error when sending message to topic
>>>> test
>>>> > with key: 54 bytes, value: 12 bytes with error:
>>>> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
>>>> > org.apache.kafka.common.errors.TimeoutException: Expiring 10
>>>> record(s) for
>>>> > test-0: 1542 ms has passed since batch creation plus linger time
>>>> > [2018-07-14 19:49:25,329] ERROR Error when sending message to topic
>>>> test
>>>> > with key: 54 bytes, value: 12 bytes with error:
>>>> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
>>>> > org.apache.kafka.common.errors.TimeoutException: Expiring 10
>>>> record(s) for
>>>> > test-0: 1542 ms has passed since batch creation plus linger time
>>>> > [2018-07-14 19:49:25,329] ERROR Error when sending message to topic
>>>> test
>>>> > with key: 55 bytes, value: 11 bytes with error:
>>>> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
>>>> > org.apache.kafka.common.errors.TimeoutException: Expiring 10
>>>> record(s) for
>>>> > test-0: 1542 ms has passed since batch creation plus linger time
>>>> >
>>>> > I have been working on this for a day or two and have not found any
>>>> > resolution yet!
>>>> >
>>>> > Thanks
>>>> >
>>>> > Dr Mich Talebzadeh
>>>> >
>>>> >
>>>> >
>>>> > LinkedIn *
>>>> >
>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> > <
>>>> >
>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> > >*
>>>> >
>>>> >
>>>> >
>>>> > http://talebzadehmich.wordpress.com
>>>> >
>>>> >
>>>> > *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any
>>>> > loss, damage or destruction of data or any other property which may
>>>> arise
>>>> > from relying on this email's technical content is explicitly
>>>> disclaimed.
>>>> > The author will in no case be liable for any monetary damages arising
>>>> from
>>>> > such loss, damage or destruction.
>>>> >
>>>>
>>>
Re: Accessing Zookepper and Kafka through designated ports in a container
Posted by Mich Talebzadeh <mi...@gmail.com>.
Some additional info from the client side
[zk: rhes75:4300(CONNECTED) 3] *ls /brokers/topics * #Gives the list of
topics
[test, md]
[zk: rhes75:4300(CONNECTED) 5] *ls /brokers/ids*
[1]
[zk: rhes75:4300(CONNECTED) 4] *get /brokers/ids/1 * #detailed info about
broker ID '1'
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://localhost:12092"],"jmx_port":9999,"host":"localhost","timestamp":"1531512017557","port":12092,"version":4}
cZxid = 0x16f
ctime = Fri Jul 13 21:00:17 BST 2018
mZxid = 0x16f
mtime = Fri Jul 13 21:00:17 BST 2018
pZxid = 0x16f
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x164953a10100000
dataLength = 192
numChildren = 0
[zk: rhes75:4300(CONNECTED) 6] *get /controller*
{"version":1,"brokerid":1,"timestamp":"1531512017622"}
cZxid = 0x170
ctime = Fri Jul 13 21:00:17 BST 2018
mZxid = 0x170
mtime = Fri Jul 13 21:00:17 BST 2018
pZxid = 0x170
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x164953a10100000
dataLength = 54
numChildren = 0
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Sat, 14 Jul 2018 at 21:20, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Apologies correction. Got the port number incorrect. It should be 4300
>
> on the docker itself confirming Zookeeper port
>
> *netstat -plten|grep 4300*
> (Not all processes could be identified, non-owned process info
> will not be shown, you would have to be root to see it all.)
> tcp 0 0 0.0.0.0:4300 0.0.0.0:*
> LISTEN 1000 4638251 7026/java
>
> *jps|grep 7026*7026 QuorumPeerMain
>
> On the remote host
>
> hduser@rhes564: /home/hduser/zookeeper-3.4.6/bin> ./zkCli.sh -server
> rhes75:4300
> *Connecting to rhes75:4300*
> 2018-07-14 21:27:35,560 [myid:] - INFO [main:Environment@100] - Client
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
> 2018-07-14 21:27:35,563 [myid:] - INFO [main:Environment@100] - Client
> environment:host.name=rhes564
> 2018-07-14 21:27:35,563 [myid:] - INFO [main:Environment@100] - Client
> environment:java.version=1.8.0_77
> 2018-07-14 21:27:35,565 [myid:] - INFO [main:Environment@100] - Client
> environment:java.vendor=Oracle Corporation
> 2018-07-14 21:27:35,565 [myid:] - INFO [main:Environment@100] - Client
> environment:java.home=/usr/java/jdk1.8.0_77/jre
> 2018-07-14 21:27:35,565 [myid:] - INFO [main:Environment@100] - Client
> environment:java.class.path=/home/hduser/zookeeper-3.4.6/bin/../build/classes:/home/hduser/zookeeper-3.4.6/bin/../build/lib/*.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/netty-3.7.0.Final.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/log4j-1.2.16.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper-3.4.6/bin/../zookeeper-3.4.6.jar:/home/hduser/zookeeper-3.4.6/bin/../src/java/lib/*.jar:/home/hduser/zookeeper-3.4.6/bin/../conf:
> 2018-07-14 21:27:35,565 [myid:] - INFO [main:Environment@100] - Client
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
> environment:java.io.tmpdir=/tmp
> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
> environment:java.compiler=<NA>
> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
> environment:os.name=Linux
> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
> environment:os.arch=amd64
> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
> environment:os.version=2.6.18-92.el5
> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
> environment:user.name=hduser
> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
> environment:user.home=/home/hduser
> 2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
> environment:user.dir=/home/hduser/zookeeper-3.4.6/bin
> 2018-07-14 21:27:35,567 [myid:] - INFO [main:ZooKeeper@438] - Initiating
> client connection, connectString=rhes75:4300 sessionTimeout=30000
> watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589e
> Welcome to ZooKeeper!
> 2018-07-14 21:27:35,588 [myid:] - INFO
> [main-SendThread(rhes75:4300):ClientCnxn$SendThread@975] - Opening socket
> connection to server rhes75/50.140.197.220:4300. Will not attempt to
> authenticate using SASL (unknown error)
> JLine support is enabled
> 2018-07-14 21:27:35,644 [myid:] - INFO
> [main-SendThread(rhes75:4300):ClientCnxn$SendThread@852] - Socket
> connection established to rhes75/50.140.197.220:4300, initiating session
> [zk: rhes75:4300(CONNECTING) 0] 2018-07-14 21:27:35,666 [myid:] - INFO
> [main-SendThread(rhes75:4300):ClientCnxn$SendThread@1235] - Session
> establishment complete on server rhes75/50.140.197.220:4300, sessionid =
> 0x164953a1010000a, negotiated timeout = 30000
> WATCHER::
> WatchedEvent state:SyncConnected type:None path:null
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sat, 14 Jul 2018 at 21:01, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Hi
>>
>> This is from remote host
>>
>> ./zkCli.sh -server rhes75:3400
>> *Connecting to rhes75:3400*
>> 2018-07-14 21:08:41,104 [myid:] - INFO [main:Environment@100] - Client
>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
>> 2018-07-14 21:08:41,107 [myid:] - INFO [main:Environment@100] - Client
>> environment:host.name=rhes564
>> 2018-07-14 21:08:41,107 [myid:] - INFO [main:Environment@100] - Client
>> environment:java.version=1.8.0_77
>> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
>> environment:java.vendor=Oracle Corporation
>> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
>> environment:java.home=/usr/java/jdk1.8.0_77/jre
>> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
>> environment:java.class.path=/home/hduser/zookeeper-3.4.6/bin/../build/classes:/home/hduser/zookeeper-3.4.6/bin/../build/lib/*.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/netty-3.7.0.Final.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/log4j-1.2.16.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper-3.4.6/bin/../zookeeper-3.4.6.jar:/home/hduser/zookeeper-3.4.6/bin/../src/java/lib/*.jar:/home/hduser/zookeeper-3.4.6/bin/../conf:
>> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
>> environment:java.io.tmpdir=/tmp
>> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
>> environment:java.compiler=<NA>
>> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
>> environment:os.name=Linux
>> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
>> environment:os.arch=amd64
>> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
>> environment:os.version=2.6.18-92.el5
>> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
>> environment:user.name=hduser
>> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
>> environment:user.home=/home/hduser
>> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
>> environment:user.dir=/home/hduser/zookeeper-3.4.6/bin
>> 2018-07-14 21:08:41,111 [myid:] - INFO [main:ZooKeeper@438] -
>> Initiating client connection, connectString=rhes75:3400
>> sessionTimeout=30000
>> watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589e
>> Welcome to ZooKeeper!
>> 2018-07-14 21:08:41,131 [myid:] - INFO
>> [main-SendThread(rhes75:3400):ClientCnxn$SendThread@975] - Opening
>> socket connection to server rhes75/50.140.197.220:3400. Will not attempt
>> to authenticate using SASL (unknown error)
>> JLine support is enabled
>> 2018-07-14 21:08:41,191 [myid:] - WARN
>> [main-SendThread(rhes75:3400):ClientCnxn$SendThread@1102] - Session 0x0
>> for server null, unexpected error, closing socket connection and attempting
>> reconnect
>> java.net.ConnectException: Connection refused
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
>> at
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
>> [zk: rhes75:3400(CONNECTING) 0] 2018-07-14 21:08:42,299 [myid:] - INFO
>> [main-SendThread(rhes75:3400):ClientCnxn$SendThread@975] - Opening
>> socket connection to server rhes75/50.140.197.220:3400. Will not attempt
>> to authenticate using SASL (unknown error)
>> 2018-07-14 21:08:42,300 [myid:] - WARN
>> [main-SendThread(rhes75:3400):ClientCnxn$SendThread@1102] - Session 0x0
>> for server null, unexpected error, closing socket connection and attempting
>> reconnect
>>
>> Thanks
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Sat, 14 Jul 2018 at 20:34, M. Manna <ma...@gmail.com> wrote:
>>
>>> Can you print out zookeeper admin:
>>>
>>> ./zookeeper-shell.sh rhes75:3400
>>>
>>> And show us here?
>>>
>>> On Sat, 14 Jul 2018, 19:43 Mich Talebzadeh, <mi...@gmail.com>
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> > I have created an Ubuntu image docker on a physical host running
>>> RHES75.
>>> >
>>> > I installed both Zookeeper and Kafka binaries in this container and
>>> they
>>> > both started OK, I ran my Zookeeper on port 3400 and Kafka running on
>>> port
>>> > 12092
>>> >
>>> > hduser@bbe7e47d5609: /home/hduser> jps
>>> > 7026 QuorumPeerMain
>>> > 7079 Kafka
>>> >
>>> > I can connect from contain to any host as long /etc/hosts has the
>>> relevant
>>> > host information.
>>> >
>>> > I mapped port 3400 and 12092 on the physical host running the
>>> container as
>>> > port forwarding.
>>> >
>>> > [root@rhes75 ~]# docker ps -a
>>> > CONTAINER ID IMAGE COMMAND
>>> > CREATED STATUS PORTS NAMES
>>> > bbe7e47d5609 ubuntu "bash" 28 hours
>>> > ago Up 25 hours testdocker
>>> >
>>> > I did the mapping of ports as follows:
>>> >
>>> > iptables -t nat -A DOCKER -p tcp --dport 4300 -j DNAT --to-destination
>>> > 172.17.0.2:4300
>>> > iptables -t nat -A DOCKER -p tcp --dport 12092 -j DNAT --to-destination
>>> > 172.17.0.2:12092
>>> >
>>> > So if I go to another host and try to telnet to port 4300 on the
>>> docker it
>>> > works
>>> >
>>> >
>>> > hduser@rhes564: /home/hduser/dba/log> telnet rhes75 4300
>>> > Trying 50.140.197.220...
>>> > Connected to rhes75 (50.140.197.220).
>>> > Escape character is '^]'.
>>> >
>>> >
>>> > hduser@rhes564: /home/hduser/dba/log> telnet rhes75 12092
>>> > Trying 50.140.197.220...
>>> > Connected to rhes75 (50.140.197.220).
>>> > Escape character is '^]'.
>>> >
>>> >
>>> > If I stop Zookeeper (running on 4300), the connection will be refused
>>> >
>>> > I created a topic called "test" on a remote host as follows
>>> >
>>> > ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:4300
>>> > --replication-factor 1 --partitions 1 --topic test
>>> >
>>> > And if I go into docker I can see that topic on localhost:4300
>>> >
>>> > ${KAFKA_HOME}/bin/kafka-topics.sh --describe --zookeeper
>>> *localhost:4300*
>>> > --topic test
>>> > Topic:test PartitionCount:1 ReplicationFactor:1
>>> Configs:
>>> > Topic: test Partition: 0 Leader: 1 Replicas: 1
>>> > Isr: 1
>>> >
>>> > The problem I see is that I cannot publish messages to the broker on
>>> this
>>> > container
>>> >
>>> > From remote host I start publishing some test prices
>>> > cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh
>>> --broker-list
>>> > rhes75:12092 --topic test --property "parse.key=true" --property
>>> > "key.separator=:"
>>> >
>>> > I get this error that indicates there is an issue talking to broker on
>>> the
>>> > container!
>>> >
>>> > [2018-07-14 19:49:22,838] WARN [Producer clientId=console-producer]
>>> > Connection to node 1 could not be established. Broker may not be
>>> available.
>>> > (org.apache.kafka.clients.NetworkClient)
>>> > [2018-07-14 19:49:22,925] WARN [Producer clientId=console-producer]
>>> > Connection to node 1 could not be established. Broker may not be
>>> available.
>>> > (org.apache.kafka.clients.NetworkClient)
>>> > [2018-07-14 19:49:23,088] WARN [Producer clientId=console-producer]
>>> > Connection to node 1 could not be established. Broker may not be
>>> available.
>>> > (org.apache.kafka.clients.NetworkClient)
>>> > [2018-07-14 19:49:23,491] WARN [Producer clientId=console-producer]
>>> > Connection to node 1 could not be established. Broker may not be
>>> available.
>>> > (org.apache.kafka.clients.NetworkClient)
>>> > [2018-07-14 19:49:24,188] WARN [Producer clientId=console-producer]
>>> > Connection to node 1 could not be established. Broker may not be
>>> available.
>>> > (org.apache.kafka.clients.NetworkClient)
>>> > [2018-07-14 19:49:25,124] WARN [Producer clientId=console-producer]
>>> > Connection to node 1 could not be established. Broker may not be
>>> available.
>>> > (org.apache.kafka.clients.NetworkClient)
>>> > [2018-07-14 19:49:25,326] ERROR Error when sending message to topic
>>> test
>>> > with key: 54 bytes, value: 12 bytes with error:
>>> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
>>> > org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s)
>>> for
>>> > test-0: 1542 ms has passed since batch creation plus linger time
>>> > [2018-07-14 19:49:25,329] ERROR Error when sending message to topic
>>> test
>>> > with key: 54 bytes, value: 12 bytes with error:
>>> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
>>> > org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s)
>>> for
>>> > test-0: 1542 ms has passed since batch creation plus linger time
>>> > [2018-07-14 19:49:25,329] ERROR Error when sending message to topic
>>> test
>>> > with key: 55 bytes, value: 11 bytes with error:
>>> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
>>> > org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s)
>>> for
>>> > test-0: 1542 ms has passed since batch creation plus linger time
>>> >
>>> > I have been working on this for a day or two and have not found any
>>> > resolution yet!
>>> >
>>> > Thanks
>>> >
>>> > Dr Mich Talebzadeh
>>> >
>>> >
>>> >
>>> > LinkedIn *
>>> >
>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> > <
>>> >
>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> > >*
>>> >
>>> >
>>> >
>>> > http://talebzadehmich.wordpress.com
>>> >
>>> >
>>> > *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any
>>> > loss, damage or destruction of data or any other property which may
>>> arise
>>> > from relying on this email's technical content is explicitly
>>> disclaimed.
>>> > The author will in no case be liable for any monetary damages arising
>>> from
>>> > such loss, damage or destruction.
>>> >
>>>
>>
Re: Accessing Zookepper and Kafka through designated ports in a container
Posted by Mich Talebzadeh <mi...@gmail.com>.
Apologies correction. Got the port number incorrect. It should be 4300
on the docker itself confirming Zookeeper port
*netstat -plten|grep 4300*
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:4300 0.0.0.0:*
LISTEN 1000 4638251 7026/java
*jps|grep 7026*7026 QuorumPeerMain
On the remote host
hduser@rhes564: /home/hduser/zookeeper-3.4.6/bin> ./zkCli.sh -server
rhes75:4300
*Connecting to rhes75:4300*
2018-07-14 21:27:35,560 [myid:] - INFO [main:Environment@100] - Client
environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2018-07-14 21:27:35,563 [myid:] - INFO [main:Environment@100] - Client
environment:host.name=rhes564
2018-07-14 21:27:35,563 [myid:] - INFO [main:Environment@100] - Client
environment:java.version=1.8.0_77
2018-07-14 21:27:35,565 [myid:] - INFO [main:Environment@100] - Client
environment:java.vendor=Oracle Corporation
2018-07-14 21:27:35,565 [myid:] - INFO [main:Environment@100] - Client
environment:java.home=/usr/java/jdk1.8.0_77/jre
2018-07-14 21:27:35,565 [myid:] - INFO [main:Environment@100] - Client
environment:java.class.path=/home/hduser/zookeeper-3.4.6/bin/../build/classes:/home/hduser/zookeeper-3.4.6/bin/../build/lib/*.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/netty-3.7.0.Final.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/log4j-1.2.16.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper-3.4.6/bin/../zookeeper-3.4.6.jar:/home/hduser/zookeeper-3.4.6/bin/../src/java/lib/*.jar:/home/hduser/zookeeper-3.4.6/bin/../conf:
2018-07-14 21:27:35,565 [myid:] - INFO [main:Environment@100] - Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
environment:java.io.tmpdir=/tmp
2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
environment:java.compiler=<NA>
2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
environment:os.name=Linux
2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
environment:os.arch=amd64
2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
environment:os.version=2.6.18-92.el5
2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
environment:user.name=hduser
2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
environment:user.home=/home/hduser
2018-07-14 21:27:35,566 [myid:] - INFO [main:Environment@100] - Client
environment:user.dir=/home/hduser/zookeeper-3.4.6/bin
2018-07-14 21:27:35,567 [myid:] - INFO [main:ZooKeeper@438] - Initiating
client connection, connectString=rhes75:4300 sessionTimeout=30000
watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589e
Welcome to ZooKeeper!
2018-07-14 21:27:35,588 [myid:] - INFO
[main-SendThread(rhes75:4300):ClientCnxn$SendThread@975] - Opening socket
connection to server rhes75/50.140.197.220:4300. Will not attempt to
authenticate using SASL (unknown error)
JLine support is enabled
2018-07-14 21:27:35,644 [myid:] - INFO
[main-SendThread(rhes75:4300):ClientCnxn$SendThread@852] - Socket
connection established to rhes75/50.140.197.220:4300, initiating session
[zk: rhes75:4300(CONNECTING) 0] 2018-07-14 21:27:35,666 [myid:] - INFO
[main-SendThread(rhes75:4300):ClientCnxn$SendThread@1235] - Session
establishment complete on server rhes75/50.140.197.220:4300, sessionid =
0x164953a1010000a, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Sat, 14 Jul 2018 at 21:01, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Hi
>
> This is from remote host
>
> ./zkCli.sh -server rhes75:3400
> *Connecting to rhes75:3400*
> 2018-07-14 21:08:41,104 [myid:] - INFO [main:Environment@100] - Client
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
> 2018-07-14 21:08:41,107 [myid:] - INFO [main:Environment@100] - Client
> environment:host.name=rhes564
> 2018-07-14 21:08:41,107 [myid:] - INFO [main:Environment@100] - Client
> environment:java.version=1.8.0_77
> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
> environment:java.vendor=Oracle Corporation
> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
> environment:java.home=/usr/java/jdk1.8.0_77/jre
> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
> environment:java.class.path=/home/hduser/zookeeper-3.4.6/bin/../build/classes:/home/hduser/zookeeper-3.4.6/bin/../build/lib/*.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/netty-3.7.0.Final.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/log4j-1.2.16.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper-3.4.6/bin/../zookeeper-3.4.6.jar:/home/hduser/zookeeper-3.4.6/bin/../src/java/lib/*.jar:/home/hduser/zookeeper-3.4.6/bin/../conf:
> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
> environment:java.io.tmpdir=/tmp
> 2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
> environment:java.compiler=<NA>
> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
> environment:os.name=Linux
> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
> environment:os.arch=amd64
> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
> environment:os.version=2.6.18-92.el5
> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
> environment:user.name=hduser
> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
> environment:user.home=/home/hduser
> 2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
> environment:user.dir=/home/hduser/zookeeper-3.4.6/bin
> 2018-07-14 21:08:41,111 [myid:] - INFO [main:ZooKeeper@438] - Initiating
> client connection, connectString=rhes75:3400 sessionTimeout=30000
> watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589e
> Welcome to ZooKeeper!
> 2018-07-14 21:08:41,131 [myid:] - INFO
> [main-SendThread(rhes75:3400):ClientCnxn$SendThread@975] - Opening socket
> connection to server rhes75/50.140.197.220:3400. Will not attempt to
> authenticate using SASL (unknown error)
> JLine support is enabled
> 2018-07-14 21:08:41,191 [myid:] - WARN
> [main-SendThread(rhes75:3400):ClientCnxn$SendThread@1102] - Session 0x0
> for server null, unexpected error, closing socket connection and attempting
> reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> [zk: rhes75:3400(CONNECTING) 0] 2018-07-14 21:08:42,299 [myid:] - INFO
> [main-SendThread(rhes75:3400):ClientCnxn$SendThread@975] - Opening socket
> connection to server rhes75/50.140.197.220:3400. Will not attempt to
> authenticate using SASL (unknown error)
> 2018-07-14 21:08:42,300 [myid:] - WARN
> [main-SendThread(rhes75:3400):ClientCnxn$SendThread@1102] - Session 0x0
> for server null, unexpected error, closing socket connection and attempting
> reconnect
>
> Thanks
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sat, 14 Jul 2018 at 20:34, M. Manna <ma...@gmail.com> wrote:
>
>> Can you print out zookeeper admin:
>>
>> ./zookeeper-shell.sh rhes75:3400
>>
>> And show us here?
>>
>> On Sat, 14 Jul 2018, 19:43 Mich Talebzadeh, <mi...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > I have created an Ubuntu image docker on a physical host running RHES75.
>> >
>> > I installed both Zookeeper and Kafka binaries in this container and they
>> > both started OK, I ran my Zookeeper on port 3400 and Kafka running on
>> port
>> > 12092
>> >
>> > hduser@bbe7e47d5609: /home/hduser> jps
>> > 7026 QuorumPeerMain
>> > 7079 Kafka
>> >
>> > I can connect from contain to any host as long /etc/hosts has the
>> relevant
>> > host information.
>> >
>> > I mapped port 3400 and 12092 on the physical host running the container
>> as
>> > port forwarding.
>> >
>> > [root@rhes75 ~]# docker ps -a
>> > CONTAINER ID IMAGE COMMAND
>> > CREATED STATUS PORTS NAMES
>> > bbe7e47d5609 ubuntu "bash" 28 hours
>> > ago Up 25 hours testdocker
>> >
>> > I did the mapping of ports as follows:
>> >
>> > iptables -t nat -A DOCKER -p tcp --dport 4300 -j DNAT --to-destination
>> > 172.17.0.2:4300
>> > iptables -t nat -A DOCKER -p tcp --dport 12092 -j DNAT --to-destination
>> > 172.17.0.2:12092
>> >
>> > So if I go to another host and try to telnet to port 4300 on the docker
>> it
>> > works
>> >
>> >
>> > hduser@rhes564: /home/hduser/dba/log> telnet rhes75 4300
>> > Trying 50.140.197.220...
>> > Connected to rhes75 (50.140.197.220).
>> > Escape character is '^]'.
>> >
>> >
>> > hduser@rhes564: /home/hduser/dba/log> telnet rhes75 12092
>> > Trying 50.140.197.220...
>> > Connected to rhes75 (50.140.197.220).
>> > Escape character is '^]'.
>> >
>> >
>> > If I stop Zookeeper (running on 4300), the connection will be refused
>> >
>> > I created a topic called "test" on a remote host as follows
>> >
>> > ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:4300
>> > --replication-factor 1 --partitions 1 --topic test
>> >
>> > And if I go into docker I can see that topic on localhost:4300
>> >
>> > ${KAFKA_HOME}/bin/kafka-topics.sh --describe --zookeeper
>> *localhost:4300*
>> > --topic test
>> > Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
>> > Topic: test Partition: 0 Leader: 1 Replicas: 1
>> > Isr: 1
>> >
>> > The problem I see is that I cannot publish messages to the broker on
>> this
>> > container
>> >
>> > From remote host I start publishing some test prices
>> > cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh
>> --broker-list
>> > rhes75:12092 --topic test --property "parse.key=true" --property
>> > "key.separator=:"
>> >
>> > I get this error that indicates there is an issue talking to broker on
>> the
>> > container!
>> >
>> > [2018-07-14 19:49:22,838] WARN [Producer clientId=console-producer]
>> > Connection to node 1 could not be established. Broker may not be
>> available.
>> > (org.apache.kafka.clients.NetworkClient)
>> > [2018-07-14 19:49:22,925] WARN [Producer clientId=console-producer]
>> > Connection to node 1 could not be established. Broker may not be
>> available.
>> > (org.apache.kafka.clients.NetworkClient)
>> > [2018-07-14 19:49:23,088] WARN [Producer clientId=console-producer]
>> > Connection to node 1 could not be established. Broker may not be
>> available.
>> > (org.apache.kafka.clients.NetworkClient)
>> > [2018-07-14 19:49:23,491] WARN [Producer clientId=console-producer]
>> > Connection to node 1 could not be established. Broker may not be
>> available.
>> > (org.apache.kafka.clients.NetworkClient)
>> > [2018-07-14 19:49:24,188] WARN [Producer clientId=console-producer]
>> > Connection to node 1 could not be established. Broker may not be
>> available.
>> > (org.apache.kafka.clients.NetworkClient)
>> > [2018-07-14 19:49:25,124] WARN [Producer clientId=console-producer]
>> > Connection to node 1 could not be established. Broker may not be
>> available.
>> > (org.apache.kafka.clients.NetworkClient)
>> > [2018-07-14 19:49:25,326] ERROR Error when sending message to topic test
>> > with key: 54 bytes, value: 12 bytes with error:
>> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
>> > org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s)
>> for
>> > test-0: 1542 ms has passed since batch creation plus linger time
>> > [2018-07-14 19:49:25,329] ERROR Error when sending message to topic test
>> > with key: 54 bytes, value: 12 bytes with error:
>> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
>> > org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s)
>> for
>> > test-0: 1542 ms has passed since batch creation plus linger time
>> > [2018-07-14 19:49:25,329] ERROR Error when sending message to topic test
>> > with key: 55 bytes, value: 11 bytes with error:
>> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
>> > org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s)
>> for
>> > test-0: 1542 ms has passed since batch creation plus linger time
>> >
>> > I have been working on this for a day or two and have not found any
>> > resolution yet!
>> >
>> > Thanks
>> >
>> > Dr Mich Talebzadeh
>> >
>> >
>> >
>> > LinkedIn *
>> >
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> > <
>> >
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> > >*
>> >
>> >
>> >
>> > http://talebzadehmich.wordpress.com
>> >
>> >
>> > *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any
>> > loss, damage or destruction of data or any other property which may
>> arise
>> > from relying on this email's technical content is explicitly disclaimed.
>> > The author will in no case be liable for any monetary damages arising
>> from
>> > such loss, damage or destruction.
>> >
>>
>
Re: Accessing Zookepper and Kafka through designated ports in a container
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi
This is from remote host
./zkCli.sh -server rhes75:3400
*Connecting to rhes75:3400*
2018-07-14 21:08:41,104 [myid:] - INFO [main:Environment@100] - Client
environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2018-07-14 21:08:41,107 [myid:] - INFO [main:Environment@100] - Client
environment:host.name=rhes564
2018-07-14 21:08:41,107 [myid:] - INFO [main:Environment@100] - Client
environment:java.version=1.8.0_77
2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
environment:java.vendor=Oracle Corporation
2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
environment:java.home=/usr/java/jdk1.8.0_77/jre
2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
environment:java.class.path=/home/hduser/zookeeper-3.4.6/bin/../build/classes:/home/hduser/zookeeper-3.4.6/bin/../build/lib/*.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/netty-3.7.0.Final.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/log4j-1.2.16.jar:/home/hduser/zookeeper-3.4.6/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper-3.4.6/bin/../zookeeper-3.4.6.jar:/home/hduser/zookeeper-3.4.6/bin/../src/java/lib/*.jar:/home/hduser/zookeeper-3.4.6/bin/../conf:
2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
environment:java.io.tmpdir=/tmp
2018-07-14 21:08:41,109 [myid:] - INFO [main:Environment@100] - Client
environment:java.compiler=<NA>
2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
environment:os.name=Linux
2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
environment:os.arch=amd64
2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
environment:os.version=2.6.18-92.el5
2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
environment:user.name=hduser
2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
environment:user.home=/home/hduser
2018-07-14 21:08:41,110 [myid:] - INFO [main:Environment@100] - Client
environment:user.dir=/home/hduser/zookeeper-3.4.6/bin
2018-07-14 21:08:41,111 [myid:] - INFO [main:ZooKeeper@438] - Initiating
client connection, connectString=rhes75:3400 sessionTimeout=30000
watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589e
Welcome to ZooKeeper!
2018-07-14 21:08:41,131 [myid:] - INFO
[main-SendThread(rhes75:3400):ClientCnxn$SendThread@975] - Opening socket
connection to server rhes75/50.140.197.220:3400. Will not attempt to
authenticate using SASL (unknown error)
JLine support is enabled
2018-07-14 21:08:41,191 [myid:] - WARN
[main-SendThread(rhes75:3400):ClientCnxn$SendThread@1102] - Session 0x0 for
server null, unexpected error, closing socket connection and attempting
reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[zk: rhes75:3400(CONNECTING) 0] 2018-07-14 21:08:42,299 [myid:] - INFO
[main-SendThread(rhes75:3400):ClientCnxn$SendThread@975] - Opening socket
connection to server rhes75/50.140.197.220:3400. Will not attempt to
authenticate using SASL (unknown error)
2018-07-14 21:08:42,300 [myid:] - WARN
[main-SendThread(rhes75:3400):ClientCnxn$SendThread@1102] - Session 0x0 for
server null, unexpected error, closing socket connection and attempting
reconnect
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Sat, 14 Jul 2018 at 20:34, M. Manna <ma...@gmail.com> wrote:
> Can you print out zookeeper admin:
>
> ./zookeeper-shell.sh rhes75:3400
>
> And show us here?
>
> On Sat, 14 Jul 2018, 19:43 Mich Talebzadeh, <mi...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I have created an Ubuntu image docker on a physical host running RHES75.
> >
> > I installed both Zookeeper and Kafka binaries in this container and they
> > both started OK, I ran my Zookeeper on port 3400 and Kafka running on
> port
> > 12092
> >
> > hduser@bbe7e47d5609: /home/hduser> jps
> > 7026 QuorumPeerMain
> > 7079 Kafka
> >
> > I can connect from contain to any host as long /etc/hosts has the
> relevant
> > host information.
> >
> > I mapped port 3400 and 12092 on the physical host running the container
> as
> > port forwarding.
> >
> > [root@rhes75 ~]# docker ps -a
> > CONTAINER ID IMAGE COMMAND
> > CREATED STATUS PORTS NAMES
> > bbe7e47d5609 ubuntu "bash" 28 hours
> > ago Up 25 hours testdocker
> >
> > I did the mapping of ports as follows:
> >
> > iptables -t nat -A DOCKER -p tcp --dport 4300 -j DNAT --to-destination
> > 172.17.0.2:4300
> > iptables -t nat -A DOCKER -p tcp --dport 12092 -j DNAT --to-destination
> > 172.17.0.2:12092
> >
> > So if I go to another host and try to telnet to port 4300 on the docker
> it
> > works
> >
> >
> > hduser@rhes564: /home/hduser/dba/log> telnet rhes75 4300
> > Trying 50.140.197.220...
> > Connected to rhes75 (50.140.197.220).
> > Escape character is '^]'.
> >
> >
> > hduser@rhes564: /home/hduser/dba/log> telnet rhes75 12092
> > Trying 50.140.197.220...
> > Connected to rhes75 (50.140.197.220).
> > Escape character is '^]'.
> >
> >
> > If I stop Zookeeper (running on 4300), the connection will be refused
> >
> > I created a topic called "test" on a remote host as follows
> >
> > ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:4300
> > --replication-factor 1 --partitions 1 --topic test
> >
> > And if I go into docker I can see that topic on localhost:4300
> >
> > ${KAFKA_HOME}/bin/kafka-topics.sh --describe --zookeeper *localhost:4300*
> > --topic test
> > Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
> > Topic: test Partition: 0 Leader: 1 Replicas: 1
> > Isr: 1
> >
> > The problem I see is that I cannot publish messages to the broker on this
> > container
> >
> > From remote host I start publishing some test prices
> > cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh
> --broker-list
> > rhes75:12092 --topic test --property "parse.key=true" --property
> > "key.separator=:"
> >
> > I get this error that indicates there is an issue talking to broker on
> the
> > container!
> >
> > [2018-07-14 19:49:22,838] WARN [Producer clientId=console-producer]
> > Connection to node 1 could not be established. Broker may not be
> available.
> > (org.apache.kafka.clients.NetworkClient)
> > [2018-07-14 19:49:22,925] WARN [Producer clientId=console-producer]
> > Connection to node 1 could not be established. Broker may not be
> available.
> > (org.apache.kafka.clients.NetworkClient)
> > [2018-07-14 19:49:23,088] WARN [Producer clientId=console-producer]
> > Connection to node 1 could not be established. Broker may not be
> available.
> > (org.apache.kafka.clients.NetworkClient)
> > [2018-07-14 19:49:23,491] WARN [Producer clientId=console-producer]
> > Connection to node 1 could not be established. Broker may not be
> available.
> > (org.apache.kafka.clients.NetworkClient)
> > [2018-07-14 19:49:24,188] WARN [Producer clientId=console-producer]
> > Connection to node 1 could not be established. Broker may not be
> available.
> > (org.apache.kafka.clients.NetworkClient)
> > [2018-07-14 19:49:25,124] WARN [Producer clientId=console-producer]
> > Connection to node 1 could not be established. Broker may not be
> available.
> > (org.apache.kafka.clients.NetworkClient)
> > [2018-07-14 19:49:25,326] ERROR Error when sending message to topic test
> > with key: 54 bytes, value: 12 bytes with error:
> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> > org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s)
> for
> > test-0: 1542 ms has passed since batch creation plus linger time
> > [2018-07-14 19:49:25,329] ERROR Error when sending message to topic test
> > with key: 54 bytes, value: 12 bytes with error:
> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> > org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s)
> for
> > test-0: 1542 ms has passed since batch creation plus linger time
> > [2018-07-14 19:49:25,329] ERROR Error when sending message to topic test
> > with key: 55 bytes, value: 11 bytes with error:
> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> > org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s)
> for
> > test-0: 1542 ms has passed since batch creation plus linger time
> >
> > I have been working on this for a day or two and have not found any
> > resolution yet!
> >
> > Thanks
> >
> > Dr Mich Talebzadeh
> >
> >
> >
> > LinkedIn *
> >
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > <
> >
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > >*
> >
> >
> >
> > http://talebzadehmich.wordpress.com
> >
> >
> > *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> > loss, damage or destruction of data or any other property which may arise
> > from relying on this email's technical content is explicitly disclaimed.
> > The author will in no case be liable for any monetary damages arising
> from
> > such loss, damage or destruction.
> >
>
Re: Accessing Zookepper and Kafka through designated ports in a container
Posted by "M. Manna" <ma...@gmail.com>.
Can you print out zookeeper admin:
./zookeeper-shell.sh rhes75:3400
And show us here?
On Sat, 14 Jul 2018, 19:43 Mich Talebzadeh, <mi...@gmail.com>
wrote:
> Hi,
>
> I have created an Ubuntu image docker on a physical host running RHES75.
>
> I installed both Zookeeper and Kafka binaries in this container and they
> both started OK, I ran my Zookeeper on port 3400 and Kafka running on port
> 12092
>
> hduser@bbe7e47d5609: /home/hduser> jps
> 7026 QuorumPeerMain
> 7079 Kafka
>
> I can connect from contain to any host as long /etc/hosts has the relevant
> host information.
>
> I mapped port 3400 and 12092 on the physical host running the container as
> port forwarding.
>
> [root@rhes75 ~]# docker ps -a
> CONTAINER ID IMAGE COMMAND
> CREATED STATUS PORTS NAMES
> bbe7e47d5609 ubuntu "bash" 28 hours
> ago Up 25 hours testdocker
>
> I did the mapping of ports as follows:
>
> iptables -t nat -A DOCKER -p tcp --dport 4300 -j DNAT --to-destination
> 172.17.0.2:4300
> iptables -t nat -A DOCKER -p tcp --dport 12092 -j DNAT --to-destination
> 172.17.0.2:12092
>
> So if I go to another host and try to telnet to port 4300 on the docker it
> works
>
>
> hduser@rhes564: /home/hduser/dba/log> telnet rhes75 4300
> Trying 50.140.197.220...
> Connected to rhes75 (50.140.197.220).
> Escape character is '^]'.
>
>
> hduser@rhes564: /home/hduser/dba/log> telnet rhes75 12092
> Trying 50.140.197.220...
> Connected to rhes75 (50.140.197.220).
> Escape character is '^]'.
>
>
> If I stop Zookeeper (running on 4300), the connection will be refused
>
> I created a topic called "test" on a remote host as follows
>
> ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:4300
> --replication-factor 1 --partitions 1 --topic test
>
> And if I go into docker I can see that topic on localhost:4300
>
> ${KAFKA_HOME}/bin/kafka-topics.sh --describe --zookeeper *localhost:4300*
> --topic test
> Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
> Topic: test Partition: 0 Leader: 1 Replicas: 1
> Isr: 1
>
> The problem I see is that I cannot publish messages to the broker on this
> container
>
> From remote host I start publishing some test prices
> cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh --broker-list
> rhes75:12092 --topic test --property "parse.key=true" --property
> "key.separator=:"
>
> I get this error that indicates there is an issue talking to broker on the
> container!
>
> [2018-07-14 19:49:22,838] WARN [Producer clientId=console-producer]
> Connection to node 1 could not be established. Broker may not be available.
> (org.apache.kafka.clients.NetworkClient)
> [2018-07-14 19:49:22,925] WARN [Producer clientId=console-producer]
> Connection to node 1 could not be established. Broker may not be available.
> (org.apache.kafka.clients.NetworkClient)
> [2018-07-14 19:49:23,088] WARN [Producer clientId=console-producer]
> Connection to node 1 could not be established. Broker may not be available.
> (org.apache.kafka.clients.NetworkClient)
> [2018-07-14 19:49:23,491] WARN [Producer clientId=console-producer]
> Connection to node 1 could not be established. Broker may not be available.
> (org.apache.kafka.clients.NetworkClient)
> [2018-07-14 19:49:24,188] WARN [Producer clientId=console-producer]
> Connection to node 1 could not be established. Broker may not be available.
> (org.apache.kafka.clients.NetworkClient)
> [2018-07-14 19:49:25,124] WARN [Producer clientId=console-producer]
> Connection to node 1 could not be established. Broker may not be available.
> (org.apache.kafka.clients.NetworkClient)
> [2018-07-14 19:49:25,326] ERROR Error when sending message to topic test
> with key: 54 bytes, value: 12 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s) for
> test-0: 1542 ms has passed since batch creation plus linger time
> [2018-07-14 19:49:25,329] ERROR Error when sending message to topic test
> with key: 54 bytes, value: 12 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s) for
> test-0: 1542 ms has passed since batch creation plus linger time
> [2018-07-14 19:49:25,329] ERROR Error when sending message to topic test
> with key: 55 bytes, value: 11 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s) for
> test-0: 1542 ms has passed since batch creation plus linger time
>
> I have been working on this for a day or two and have not found any
> resolution yet!
>
> Thanks
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn *
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>