You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Zac Harvey <za...@welltok.com> on 2016/11/23 10:07:45 UTC
Messages intermittently get lost
I am playing around with Kafka and have a simple setup:
* 1-node Kafka (Ubuntu) server
* 3-node ZK cluster (each on their own Ubuntu server)
I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that ships with the distribution.
I'd say about 20% of the messages I send via the producer never get consumed by the Scala process (which is running continuously). No errors on either side (producer or consumer): the producer sends, and, nothing...
Any ideas as to what might be going on here, or how I could start troubleshooting?
Thanks!
Setting ZOO_DATADIR
Posted by Martin Gainty <mg...@hotmail.com>.
apparently when ZOO_DATADIR is empty mkdir fubars mkdir ''
when ZOO_DATADIR is empty add this export ZOO_DATADIR="zoo_datadir"
if [ -z "$ZOOPIDFILE" ]; then
ZOO_DATADIR="$(grep "^[[:space:]]*dataDir" "$ZOOCFG" | sed -e 's/.*=//')"
if [ ! -d "$ZOO_DATADIR" ]; then
echo "ZOO_DATADIR="
export ZOO_DATADIR="zoo_datadir"
echo $ZOO_DATADIR
mkdir -p "$ZOO_DATADIR"
fi
HTH
M-
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Thursday, December 1, 2016 11:03 AM
To: Martin Gainty; user@zookeeper.apache.org
Subject: Re: Messages intermittently get lost
Thanks everyone
The script can be found like so:
wget http://apache.spinellicreations.com/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
tar xfz zookeeper-3.4.6.tar.gz
cd zookeeper-3.4.6/bin
zkServer.sh
Let me know if you have any other questions/concerns!
Best,
Zac
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Wednesday, November 30, 2016 7:57:37 PM
To: user@zookeeper.apache.org; Zac Harvey
Subject: Re: Messages intermittently get lost
absolutely..lets get this resolved asap
Zac...can you send me the abending script?
Molte Grazie Flavio!
Martini
______________________________________________
________________________________
From: Flavio Junqueira <fp...@apache.org>
Sent: Wednesday, November 30, 2016 6:41 PM
To: user@zookeeper.apache.org
Cc: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hey Martin,
Do you want to create a jira for this and report the issue?
-Flavio
> On 30 Nov 2016, at 18:33, Martin Gainty <mg...@hotmail.com> wrote:
>
> a shock when a zk script goes fubar
>
>
> zookeeper devs can we get some help for sh zkServer.sh status?
>
>
> Thanks!
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Wednesday, November 30, 2016 10:25 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin, makes sense.
>
>
> When I SSH into all 3 of my ZK nodes and run:
>
>
> sh zkServer.sh status
>
>
> All three of them give me the following output:
>
>
> JMX enabled by default
>
> zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "(" unexpected (expecting "fi")
>
>
> Looks like a bug in the ZK shell script?
>
>
> Best,
>
> Zac
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Tuesday, November 29, 2016 11:18:21 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> we dont know whats causing this intermittent problem..so lets Divide and Conquer each part of this problem individually starting at the source of the data feeds
>
>
> Let us eliminate any potential problem with feeds from external sources
>
>
> Once you verify the zookeeper feeds are 100% reliable lets move onto Kafka
>
>
> Pingback when you have verifiable results from zookeeper feeds
>
>
> Thanks
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Tuesday, November 29, 2016 10:46 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Does anybody have any idea why ZK might be to blame if messages sent by a Kafka producer fail to be received by a Kafka consumer?
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Monday, November 28, 2016 9:07:41 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Thanks Martin, I will look at those links.
>
>
> But you seem to be 100% confident that the problem is with ZooKeeper...can I ask why? What is it about my problem description that makes you think this is an issue with ZooKeeper?
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Friday, November 25, 2016 1:46:28 PM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Friday, November 25, 2016 6:17 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin,
>
>
> My server.properties looks like this:
>
>
> listeners=PLAINTEXT://0.0.0.0:9092
>
> advertised.host.name=<SERVER_PUBLIC_DNS>
>
> broker.id=2
>
> port=9092
>
> num.partitions=4
>
> zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
>
> MG>can you check status for each ZK Node in the quorum?
>
> sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
>
> http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> MG>*greetings from Plimoth Mass*
>
> MG>M
>
>
> num.network.threads=3
>
> num.io.threads=8
>
> socket.send.buffer.bytes=102400
>
> socket.receive.buffer.bytes=102400
>
> log.dirs=/tmp/kafka-logs
>
> num.recovery.threads.per.data.dir=1
>
> log.retention.hours=168
>
> log.segment.bytes=1073741824
>
> log.retention.check.interval.ms=300000
>
> zookeeper.connection.timeout.ms=6000
>
> delete.topic.enable=true
>
> auto.leader.rebalance.enable=true
>
>
> Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK servers, and <SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node that this Kafka is running on.
>
>
> Anything look incorrect to you?
>
>
> And yes, yesterday was a holiday, but there is only work! I'll celebrate one big, long holiday when I'm dead!
>
>
> Thanks for any-and-all input here!
>
>
> Best,
>
> Zac
>
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Thursday, November 24, 2016 9:03:33 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> there is a rumour that today thursday is a holiday?
>
> in server.properties how are you configuring your server?
>
>
> specifically what are these attributes?
>
>
> num.network.threads=
>
>
> num.io.threads=
>
>
> socket.send.buffer.bytes=
>
>
> socket.receive.buffer.bytes=
>
>
> socket.request.max.bytes=
>
>
> num.partitions=
>
>
> num.recovery.threads.per.data.dir=
>
>
> ?
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Thursday, November 24, 2016 7:05 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Anybody?!? This is very disconcerting!
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Wednesday, November 23, 2016 5:07:45 AM
> To: users@kafka.apache.org
> Subject: Messages intermittently get lost
>
> I am playing around with Kafka and have a simple setup:
>
>
> * 1-node Kafka (Ubuntu) server
>
> * 3-node ZK cluster (each on their own Ubuntu server)
>
>
> I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that ships with the distribution.
>
>
> I'd say about 20% of the messages I send via the producer never get consumed by the Scala process (which is running continuously). No errors on either side (producer or consumer): the producer sends, and, nothing...
>
>
> Any ideas as to what might be going on here, or how I could start troubleshooting?
>
>
> Thanks!
Re: Messages intermittently get lost
Posted by Zac Harvey <za...@welltok.com>.
Thanks everyone
The script can be found like so:
wget http://apache.spinellicreations.com/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
tar xfz zookeeper-3.4.6.tar.gz
cd zookeeper-3.4.6/bin
zkServer.sh
Let me know if you have any other questions/concerns!
Best,
Zac
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Wednesday, November 30, 2016 7:57:37 PM
To: user@zookeeper.apache.org; Zac Harvey
Subject: Re: Messages intermittently get lost
absolutely..lets get this resolved asap
Zac...can you send me the abending script?
Molte Grazie Flavio!
Martini
______________________________________________
________________________________
From: Flavio Junqueira <fp...@apache.org>
Sent: Wednesday, November 30, 2016 6:41 PM
To: user@zookeeper.apache.org
Cc: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hey Martin,
Do you want to create a jira for this and report the issue?
-Flavio
> On 30 Nov 2016, at 18:33, Martin Gainty <mg...@hotmail.com> wrote:
>
> a shock when a zk script goes fubar
>
>
> zookeeper devs can we get some help for sh zkServer.sh status?
>
>
> Thanks!
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Wednesday, November 30, 2016 10:25 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin, makes sense.
>
>
> When I SSH into all 3 of my ZK nodes and run:
>
>
> sh zkServer.sh status
>
>
> All three of them give me the following output:
>
>
> JMX enabled by default
>
> zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "(" unexpected (expecting "fi")
>
>
> Looks like a bug in the ZK shell script?
>
>
> Best,
>
> Zac
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Tuesday, November 29, 2016 11:18:21 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> we dont know whats causing this intermittent problem..so lets Divide and Conquer each part of this problem individually starting at the source of the data feeds
>
>
> Let us eliminate any potential problem with feeds from external sources
>
>
> Once you verify the zookeeper feeds are 100% reliable lets move onto Kafka
>
>
> Pingback when you have verifiable results from zookeeper feeds
>
>
> Thanks
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Tuesday, November 29, 2016 10:46 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Does anybody have any idea why ZK might be to blame if messages sent by a Kafka producer fail to be received by a Kafka consumer?
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Monday, November 28, 2016 9:07:41 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Thanks Martin, I will look at those links.
>
>
> But you seem to be 100% confident that the problem is with ZooKeeper...can I ask why? What is it about my problem description that makes you think this is an issue with ZooKeeper?
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Friday, November 25, 2016 1:46:28 PM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Friday, November 25, 2016 6:17 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin,
>
>
> My server.properties looks like this:
>
>
> listeners=PLAINTEXT://0.0.0.0:9092
>
> advertised.host.name=<SERVER_PUBLIC_DNS>
>
> broker.id=2
>
> port=9092
>
> num.partitions=4
>
> zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
>
> MG>can you check status for each ZK Node in the quorum?
>
> sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
>
> http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> www.ibm.com<http://www.ibm.com>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> www.ibm.com<http://www.ibm.com>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> MG>*greetings from Plimoth Mass*
>
> MG>M
>
>
> num.network.threads=3
>
> num.io.threads=8
>
> socket.send.buffer.bytes=102400
>
> socket.receive.buffer.bytes=102400
>
> log.dirs=/tmp/kafka-logs
>
> num.recovery.threads.per.data.dir=1
>
> log.retention.hours=168
>
> log.segment.bytes=1073741824
>
> log.retention.check.interval.ms=300000
>
> zookeeper.connection.timeout.ms=6000
>
> delete.topic.enable=true
>
> auto.leader.rebalance.enable=true
>
>
> Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK servers, and <SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node that this Kafka is running on.
>
>
> Anything look incorrect to you?
>
>
> And yes, yesterday was a holiday, but there is only work! I'll celebrate one big, long holiday when I'm dead!
>
>
> Thanks for any-and-all input here!
>
>
> Best,
>
> Zac
>
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Thursday, November 24, 2016 9:03:33 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> there is a rumour that today thursday is a holiday?
>
> in server.properties how are you configuring your server?
>
>
> specifically what are these attributes?
>
>
> num.network.threads=
>
>
> num.io.threads=
>
>
> socket.send.buffer.bytes=
>
>
> socket.receive.buffer.bytes=
>
>
> socket.request.max.bytes=
>
>
> num.partitions=
>
>
> num.recovery.threads.per.data.dir=
>
>
> ?
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Thursday, November 24, 2016 7:05 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Anybody?!? This is very disconcerting!
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Wednesday, November 23, 2016 5:07:45 AM
> To: users@kafka.apache.org
> Subject: Messages intermittently get lost
>
> I am playing around with Kafka and have a simple setup:
>
>
> * 1-node Kafka (Ubuntu) server
>
> * 3-node ZK cluster (each on their own Ubuntu server)
>
>
> I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that ships with the distribution.
>
>
> I'd say about 20% of the messages I send via the producer never get consumed by the Scala process (which is running continuously). No errors on either side (producer or consumer): the producer sends, and, nothing...
>
>
> Any ideas as to what might be going on here, or how I could start troubleshooting?
>
>
> Thanks!
Re: Messages intermittently get lost
Posted by Martin Gainty <mg...@hotmail.com>.
absolutely..lets get this resolved asap
Zac...can you send me the abending script?
Molte Grazie Flavio!
Martini
______________________________________________
________________________________
From: Flavio Junqueira <fp...@apache.org>
Sent: Wednesday, November 30, 2016 6:41 PM
To: user@zookeeper.apache.org
Cc: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hey Martin,
Do you want to create a jira for this and report the issue?
-Flavio
> On 30 Nov 2016, at 18:33, Martin Gainty <mg...@hotmail.com> wrote:
>
> a shock when a zk script goes fubar
>
>
> zookeeper devs can we get some help for sh zkServer.sh status?
>
>
> Thanks!
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Wednesday, November 30, 2016 10:25 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin, makes sense.
>
>
> When I SSH into all 3 of my ZK nodes and run:
>
>
> sh zkServer.sh status
>
>
> All three of them give me the following output:
>
>
> JMX enabled by default
>
> zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "(" unexpected (expecting "fi")
>
>
> Looks like a bug in the ZK shell script?
>
>
> Best,
>
> Zac
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Tuesday, November 29, 2016 11:18:21 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> we dont know whats causing this intermittent problem..so lets Divide and Conquer each part of this problem individually starting at the source of the data feeds
>
>
> Let us eliminate any potential problem with feeds from external sources
>
>
> Once you verify the zookeeper feeds are 100% reliable lets move onto Kafka
>
>
> Pingback when you have verifiable results from zookeeper feeds
>
>
> Thanks
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Tuesday, November 29, 2016 10:46 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Does anybody have any idea why ZK might be to blame if messages sent by a Kafka producer fail to be received by a Kafka consumer?
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Monday, November 28, 2016 9:07:41 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Thanks Martin, I will look at those links.
>
>
> But you seem to be 100% confident that the problem is with ZooKeeper...can I ask why? What is it about my problem description that makes you think this is an issue with ZooKeeper?
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Friday, November 25, 2016 1:46:28 PM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Friday, November 25, 2016 6:17 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin,
>
>
> My server.properties looks like this:
>
>
> listeners=PLAINTEXT://0.0.0.0:9092
>
> advertised.host.name=<SERVER_PUBLIC_DNS>
>
> broker.id=2
>
> port=9092
>
> num.partitions=4
>
> zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
>
> MG>can you check status for each ZK Node in the quorum?
>
> sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
>
> http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> www.ibm.com<http://www.ibm.com>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> www.ibm.com<http://www.ibm.com>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> MG>*greetings from Plimoth Mass*
>
> MG>M
>
>
> num.network.threads=3
>
> num.io.threads=8
>
> socket.send.buffer.bytes=102400
>
> socket.receive.buffer.bytes=102400
>
> log.dirs=/tmp/kafka-logs
>
> num.recovery.threads.per.data.dir=1
>
> log.retention.hours=168
>
> log.segment.bytes=1073741824
>
> log.retention.check.interval.ms=300000
>
> zookeeper.connection.timeout.ms=6000
>
> delete.topic.enable=true
>
> auto.leader.rebalance.enable=true
>
>
> Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK servers, and <SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node that this Kafka is running on.
>
>
> Anything look incorrect to you?
>
>
> And yes, yesterday was a holiday, but there is only work! I'll celebrate one big, long holiday when I'm dead!
>
>
> Thanks for any-and-all input here!
>
>
> Best,
>
> Zac
>
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Thursday, November 24, 2016 9:03:33 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> there is a rumour that today thursday is a holiday?
>
> in server.properties how are you configuring your server?
>
>
> specifically what are these attributes?
>
>
> num.network.threads=
>
>
> num.io.threads=
>
>
> socket.send.buffer.bytes=
>
>
> socket.receive.buffer.bytes=
>
>
> socket.request.max.bytes=
>
>
> num.partitions=
>
>
> num.recovery.threads.per.data.dir=
>
>
> ?
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Thursday, November 24, 2016 7:05 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Anybody?!? This is very disconcerting!
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Wednesday, November 23, 2016 5:07:45 AM
> To: users@kafka.apache.org
> Subject: Messages intermittently get lost
>
> I am playing around with Kafka and have a simple setup:
>
>
> * 1-node Kafka (Ubuntu) server
>
> * 3-node ZK cluster (each on their own Ubuntu server)
>
>
> I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that ships with the distribution.
>
>
> I'd say about 20% of the messages I send via the producer never get consumed by the Scala process (which is running continuously). No errors on either side (producer or consumer): the producer sends, and, nothing...
>
>
> Any ideas as to what might be going on here, or how I could start troubleshooting?
>
>
> Thanks!
Re: Messages intermittently get lost
Posted by Sudev A C <su...@goibibo.com>.
Hi Zak,
Why don't you try using zookeeper four letter admin tools
<https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#The+Four+Letter+Words>
?
echo stat | nc zookeeper-ip zookeeper-port
echo stat | nc localhost 2181
These commands will tell you the current status of zookeeper client.
https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#The+Four+Letter+Words
Thanks
Sudev
On Thu, 1 Dec 2016 at 5:11 AM, Flavio Junqueira <fp...@apache.org> wrote:
Hey Martin,
Do you want to create a jira for this and report the issue?
-Flavio
> On 30 Nov 2016, at 18:33, Martin Gainty <mg...@hotmail.com> wrote:
>
> a shock when a zk script goes fubar
>
>
> zookeeper devs can we get some help for sh zkServer.sh status?
>
>
> Thanks!
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Wednesday, November 30, 2016 10:25 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin, makes sense.
>
>
> When I SSH into all 3 of my ZK nodes and run:
>
>
> sh zkServer.sh status
>
>
> All three of them give me the following output:
>
>
> JMX enabled by default
>
> zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "("
unexpected (expecting "fi")
>
>
> Looks like a bug in the ZK shell script?
>
>
> Best,
>
> Zac
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Tuesday, November 29, 2016 11:18:21 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> we dont know whats causing this intermittent problem..so lets Divide and
Conquer each part of this problem individually starting at the source of
the data feeds
>
>
> Let us eliminate any potential problem with feeds from external sources
>
>
> Once you verify the zookeeper feeds are 100% reliable lets move onto Kafka
>
>
> Pingback when you have verifiable results from zookeeper feeds
>
>
> Thanks
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Tuesday, November 29, 2016 10:46 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Does anybody have any idea why ZK might be to blame if messages sent by a
Kafka producer fail to be received by a Kafka consumer?
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Monday, November 28, 2016 9:07:41 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Thanks Martin, I will look at those links.
>
>
> But you seem to be 100% confident that the problem is with
ZooKeeper...can I ask why? What is it about my problem description that
makes you think this is an issue with ZooKeeper?
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Friday, November 25, 2016 1:46:28 PM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Friday, November 25, 2016 6:17 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin,
>
>
> My server.properties looks like this:
>
>
> listeners=PLAINTEXT://0.0.0.0:9092
>
> advertised.host.name=<SERVER_PUBLIC_DNS>
>
> broker.id=2
>
> port=9092
>
> num.partitions=4
>
> zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
>
> MG>can you check status for each ZK Node in the quorum?
>
> sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
>
>
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
> ZooKeeper problems and solutions - IBM<
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions - IBM<
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
>
> ZooKeeper problems and solutions - IBM<
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
> [
http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png
]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success
and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
>
>
>
>
> ZooKeeper problems and solutions - IBM<
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
>
> ZooKeeper problems and solutions - IBM<
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions - IBM<
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
>
> ZooKeeper problems and solutions - IBM<
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
> [
http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png
]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success
and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
> [
http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png
]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success
and to creating innovations that matter for the world
>
>
>
> [
http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png
]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> [
http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png
]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success
and to creating innovations that matter for the world
>
>
>
> www.ibm.com<http://www.ibm.com>
> [
http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png
]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success
and to creating innovations that matter for the world
>
>
>
> For more than a century IBM has been dedicated to every client's success
and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
>
>
>
> MG>*greetings from Plimoth Mass*
>
> MG>M
>
>
> num.network.threads=3
>
> num.io.threads=8
>
> socket.send.buffer.bytes=102400
>
> socket.receive.buffer.bytes=102400
>
> log.dirs=/tmp/kafka-logs
>
> num.recovery.threads.per.data.dir=1
>
> log.retention.hours=168
>
> log.segment.bytes=1073741824
>
> log.retention.check.interval.ms=300000
>
> zookeeper.connection.timeout.ms=6000
>
> delete.topic.enable=true
>
> auto.leader.rebalance.enable=true
>
>
> Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK
servers, and <SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node
that this Kafka is running on.
>
>
> Anything look incorrect to you?
>
>
> And yes, yesterday was a holiday, but there is only work! I'll celebrate
one big, long holiday when I'm dead!
>
>
> Thanks for any-and-all input here!
>
>
> Best,
>
> Zac
>
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Thursday, November 24, 2016 9:03:33 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> there is a rumour that today thursday is a holiday?
>
> in server.properties how are you configuring your server?
>
>
> specifically what are these attributes?
>
>
> num.network.threads=
>
>
> num.io.threads=
>
>
> socket.send.buffer.bytes=
>
>
> socket.receive.buffer.bytes=
>
>
> socket.request.max.bytes=
>
>
> num.partitions=
>
>
> num.recovery.threads.per.data.dir=
>
>
> ?
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Thursday, November 24, 2016 7:05 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Anybody?!? This is very disconcerting!
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Wednesday, November 23, 2016 5:07:45 AM
> To: users@kafka.apache.org
> Subject: Messages intermittently get lost
>
> I am playing around with Kafka and have a simple setup:
>
>
> * 1-node Kafka (Ubuntu) server
>
> * 3-node ZK cluster (each on their own Ubuntu server)
>
>
> I have a consumer written in Scala and am using the
kafka-console-producer (v0.10) that ships with the distribution.
>
>
> I'd say about 20% of the messages I send via the producer never get
consumed by the Scala process (which is running continuously). No errors on
either side (producer or consumer): the producer sends, and, nothing...
>
>
> Any ideas as to what might be going on here, or how I could start
troubleshooting?
>
>
> Thanks!
Re: Messages intermittently get lost
Posted by Sudev A C <su...@goibibo.com>.
Hi Zak,
Why don't you try using zookeeper four letter admin tools
<https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#The+Four+Letter+Words>
?
echo stat | nc zookeeper-ip zookeeper-port
echo stat | nc localhost 2181
These commands will tell you the current status of zookeeper client.
https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#The+Four+Letter+Words
Thanks
Sudev
On Thu, 1 Dec 2016 at 5:11 AM, Flavio Junqueira <fp...@apache.org> wrote:
Hey Martin,
Do you want to create a jira for this and report the issue?
-Flavio
> On 30 Nov 2016, at 18:33, Martin Gainty <mg...@hotmail.com> wrote:
>
> a shock when a zk script goes fubar
>
>
> zookeeper devs can we get some help for sh zkServer.sh status?
>
>
> Thanks!
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Wednesday, November 30, 2016 10:25 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin, makes sense.
>
>
> When I SSH into all 3 of my ZK nodes and run:
>
>
> sh zkServer.sh status
>
>
> All three of them give me the following output:
>
>
> JMX enabled by default
>
> zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "("
unexpected (expecting "fi")
>
>
> Looks like a bug in the ZK shell script?
>
>
> Best,
>
> Zac
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Tuesday, November 29, 2016 11:18:21 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> we dont know whats causing this intermittent problem..so lets Divide and
Conquer each part of this problem individually starting at the source of
the data feeds
>
>
> Let us eliminate any potential problem with feeds from external sources
>
>
> Once you verify the zookeeper feeds are 100% reliable lets move onto Kafka
>
>
> Pingback when you have verifiable results from zookeeper feeds
>
>
> Thanks
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Tuesday, November 29, 2016 10:46 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Does anybody have any idea why ZK might be to blame if messages sent by a
Kafka producer fail to be received by a Kafka consumer?
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Monday, November 28, 2016 9:07:41 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Thanks Martin, I will look at those links.
>
>
> But you seem to be 100% confident that the problem is with
ZooKeeper...can I ask why? What is it about my problem description that
makes you think this is an issue with ZooKeeper?
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Friday, November 25, 2016 1:46:28 PM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Friday, November 25, 2016 6:17 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin,
>
>
> My server.properties looks like this:
>
>
> listeners=PLAINTEXT://0.0.0.0:9092
>
> advertised.host.name=<SERVER_PUBLIC_DNS>
>
> broker.id=2
>
> port=9092
>
> num.partitions=4
>
> zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
>
> MG>can you check status for each ZK Node in the quorum?
>
> sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
>
>
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
> ZooKeeper problems and solutions - IBM<
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions - IBM<
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
>
> ZooKeeper problems and solutions - IBM<
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
> [
http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png
]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success
and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
>
>
>
>
> ZooKeeper problems and solutions - IBM<
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
>
> ZooKeeper problems and solutions - IBM<
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions - IBM<
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
>
> ZooKeeper problems and solutions - IBM<
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
> [
http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png
]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success
and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
> [
http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png
]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success
and to creating innovations that matter for the world
>
>
>
> [
http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png
]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> [
http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png
]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success
and to creating innovations that matter for the world
>
>
>
> www.ibm.com<http://www.ibm.com>
> [
http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png
]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success
and to creating innovations that matter for the world
>
>
>
> For more than a century IBM has been dedicated to every client's success
and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
>
>
>
> MG>*greetings from Plimoth Mass*
>
> MG>M
>
>
> num.network.threads=3
>
> num.io.threads=8
>
> socket.send.buffer.bytes=102400
>
> socket.receive.buffer.bytes=102400
>
> log.dirs=/tmp/kafka-logs
>
> num.recovery.threads.per.data.dir=1
>
> log.retention.hours=168
>
> log.segment.bytes=1073741824
>
> log.retention.check.interval.ms=300000
>
> zookeeper.connection.timeout.ms=6000
>
> delete.topic.enable=true
>
> auto.leader.rebalance.enable=true
>
>
> Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK
servers, and <SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node
that this Kafka is running on.
>
>
> Anything look incorrect to you?
>
>
> And yes, yesterday was a holiday, but there is only work! I'll celebrate
one big, long holiday when I'm dead!
>
>
> Thanks for any-and-all input here!
>
>
> Best,
>
> Zac
>
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Thursday, November 24, 2016 9:03:33 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> there is a rumour that today thursday is a holiday?
>
> in server.properties how are you configuring your server?
>
>
> specifically what are these attributes?
>
>
> num.network.threads=
>
>
> num.io.threads=
>
>
> socket.send.buffer.bytes=
>
>
> socket.receive.buffer.bytes=
>
>
> socket.request.max.bytes=
>
>
> num.partitions=
>
>
> num.recovery.threads.per.data.dir=
>
>
> ?
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Thursday, November 24, 2016 7:05 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Anybody?!? This is very disconcerting!
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Wednesday, November 23, 2016 5:07:45 AM
> To: users@kafka.apache.org
> Subject: Messages intermittently get lost
>
> I am playing around with Kafka and have a simple setup:
>
>
> * 1-node Kafka (Ubuntu) server
>
> * 3-node ZK cluster (each on their own Ubuntu server)
>
>
> I have a consumer written in Scala and am using the
kafka-console-producer (v0.10) that ships with the distribution.
>
>
> I'd say about 20% of the messages I send via the producer never get
consumed by the Scala process (which is running continuously). No errors on
either side (producer or consumer): the producer sends, and, nothing...
>
>
> Any ideas as to what might be going on here, or how I could start
troubleshooting?
>
>
> Thanks!
Re: Messages intermittently get lost
Posted by Flavio Junqueira <fp...@apache.org>.
Hey Martin,
Do you want to create a jira for this and report the issue?
-Flavio
> On 30 Nov 2016, at 18:33, Martin Gainty <mg...@hotmail.com> wrote:
>
> a shock when a zk script goes fubar
>
>
> zookeeper devs can we get some help for sh zkServer.sh status?
>
>
> Thanks!
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Wednesday, November 30, 2016 10:25 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin, makes sense.
>
>
> When I SSH into all 3 of my ZK nodes and run:
>
>
> sh zkServer.sh status
>
>
> All three of them give me the following output:
>
>
> JMX enabled by default
>
> zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "(" unexpected (expecting "fi")
>
>
> Looks like a bug in the ZK shell script?
>
>
> Best,
>
> Zac
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Tuesday, November 29, 2016 11:18:21 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> we dont know whats causing this intermittent problem..so lets Divide and Conquer each part of this problem individually starting at the source of the data feeds
>
>
> Let us eliminate any potential problem with feeds from external sources
>
>
> Once you verify the zookeeper feeds are 100% reliable lets move onto Kafka
>
>
> Pingback when you have verifiable results from zookeeper feeds
>
>
> Thanks
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Tuesday, November 29, 2016 10:46 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Does anybody have any idea why ZK might be to blame if messages sent by a Kafka producer fail to be received by a Kafka consumer?
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Monday, November 28, 2016 9:07:41 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Thanks Martin, I will look at those links.
>
>
> But you seem to be 100% confident that the problem is with ZooKeeper...can I ask why? What is it about my problem description that makes you think this is an issue with ZooKeeper?
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Friday, November 25, 2016 1:46:28 PM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Friday, November 25, 2016 6:17 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin,
>
>
> My server.properties looks like this:
>
>
> listeners=PLAINTEXT://0.0.0.0:9092
>
> advertised.host.name=<SERVER_PUBLIC_DNS>
>
> broker.id=2
>
> port=9092
>
> num.partitions=4
>
> zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
>
> MG>can you check status for each ZK Node in the quorum?
>
> sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
>
> http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> MG>*greetings from Plimoth Mass*
>
> MG>M
>
>
> num.network.threads=3
>
> num.io.threads=8
>
> socket.send.buffer.bytes=102400
>
> socket.receive.buffer.bytes=102400
>
> log.dirs=/tmp/kafka-logs
>
> num.recovery.threads.per.data.dir=1
>
> log.retention.hours=168
>
> log.segment.bytes=1073741824
>
> log.retention.check.interval.ms=300000
>
> zookeeper.connection.timeout.ms=6000
>
> delete.topic.enable=true
>
> auto.leader.rebalance.enable=true
>
>
> Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK servers, and <SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node that this Kafka is running on.
>
>
> Anything look incorrect to you?
>
>
> And yes, yesterday was a holiday, but there is only work! I'll celebrate one big, long holiday when I'm dead!
>
>
> Thanks for any-and-all input here!
>
>
> Best,
>
> Zac
>
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Thursday, November 24, 2016 9:03:33 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> there is a rumour that today thursday is a holiday?
>
> in server.properties how are you configuring your server?
>
>
> specifically what are these attributes?
>
>
> num.network.threads=
>
>
> num.io.threads=
>
>
> socket.send.buffer.bytes=
>
>
> socket.receive.buffer.bytes=
>
>
> socket.request.max.bytes=
>
>
> num.partitions=
>
>
> num.recovery.threads.per.data.dir=
>
>
> ?
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Thursday, November 24, 2016 7:05 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Anybody?!? This is very disconcerting!
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Wednesday, November 23, 2016 5:07:45 AM
> To: users@kafka.apache.org
> Subject: Messages intermittently get lost
>
> I am playing around with Kafka and have a simple setup:
>
>
> * 1-node Kafka (Ubuntu) server
>
> * 3-node ZK cluster (each on their own Ubuntu server)
>
>
> I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that ships with the distribution.
>
>
> I'd say about 20% of the messages I send via the producer never get consumed by the Scala process (which is running continuously). No errors on either side (producer or consumer): the producer sends, and, nothing...
>
>
> Any ideas as to what might be going on here, or how I could start troubleshooting?
>
>
> Thanks!
Re: Messages intermittently get lost
Posted by Flavio Junqueira <fp...@apache.org>.
Hey Martin,
Do you want to create a jira for this and report the issue?
-Flavio
> On 30 Nov 2016, at 18:33, Martin Gainty <mg...@hotmail.com> wrote:
>
> a shock when a zk script goes fubar
>
>
> zookeeper devs can we get some help for sh zkServer.sh status?
>
>
> Thanks!
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Wednesday, November 30, 2016 10:25 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin, makes sense.
>
>
> When I SSH into all 3 of my ZK nodes and run:
>
>
> sh zkServer.sh status
>
>
> All three of them give me the following output:
>
>
> JMX enabled by default
>
> zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "(" unexpected (expecting "fi")
>
>
> Looks like a bug in the ZK shell script?
>
>
> Best,
>
> Zac
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Tuesday, November 29, 2016 11:18:21 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> we dont know whats causing this intermittent problem..so lets Divide and Conquer each part of this problem individually starting at the source of the data feeds
>
>
> Let us eliminate any potential problem with feeds from external sources
>
>
> Once you verify the zookeeper feeds are 100% reliable lets move onto Kafka
>
>
> Pingback when you have verifiable results from zookeeper feeds
>
>
> Thanks
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Tuesday, November 29, 2016 10:46 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Does anybody have any idea why ZK might be to blame if messages sent by a Kafka producer fail to be received by a Kafka consumer?
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Monday, November 28, 2016 9:07:41 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Thanks Martin, I will look at those links.
>
>
> But you seem to be 100% confident that the problem is with ZooKeeper...can I ask why? What is it about my problem description that makes you think this is an issue with ZooKeeper?
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Friday, November 25, 2016 1:46:28 PM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Friday, November 25, 2016 6:17 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin,
>
>
> My server.properties looks like this:
>
>
> listeners=PLAINTEXT://0.0.0.0:9092
>
> advertised.host.name=<SERVER_PUBLIC_DNS>
>
> broker.id=2
>
> port=9092
>
> num.partitions=4
>
> zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
>
> MG>can you check status for each ZK Node in the quorum?
>
> sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
>
> http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> MG>*greetings from Plimoth Mass*
>
> MG>M
>
>
> num.network.threads=3
>
> num.io.threads=8
>
> socket.send.buffer.bytes=102400
>
> socket.receive.buffer.bytes=102400
>
> log.dirs=/tmp/kafka-logs
>
> num.recovery.threads.per.data.dir=1
>
> log.retention.hours=168
>
> log.segment.bytes=1073741824
>
> log.retention.check.interval.ms=300000
>
> zookeeper.connection.timeout.ms=6000
>
> delete.topic.enable=true
>
> auto.leader.rebalance.enable=true
>
>
> Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK servers, and <SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node that this Kafka is running on.
>
>
> Anything look incorrect to you?
>
>
> And yes, yesterday was a holiday, but there is only work! I'll celebrate one big, long holiday when I'm dead!
>
>
> Thanks for any-and-all input here!
>
>
> Best,
>
> Zac
>
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> Sent: Thursday, November 24, 2016 9:03:33 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> there is a rumour that today thursday is a holiday?
>
> in server.properties how are you configuring your server?
>
>
> specifically what are these attributes?
>
>
> num.network.threads=
>
>
> num.io.threads=
>
>
> socket.send.buffer.bytes=
>
>
> socket.receive.buffer.bytes=
>
>
> socket.request.max.bytes=
>
>
> num.partitions=
>
>
> num.recovery.threads.per.data.dir=
>
>
> ?
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Thursday, November 24, 2016 7:05 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Anybody?!? This is very disconcerting!
>
> ________________________________
> From: Zac Harvey <za...@welltok.com>
> Sent: Wednesday, November 23, 2016 5:07:45 AM
> To: users@kafka.apache.org
> Subject: Messages intermittently get lost
>
> I am playing around with Kafka and have a simple setup:
>
>
> * 1-node Kafka (Ubuntu) server
>
> * 3-node ZK cluster (each on their own Ubuntu server)
>
>
> I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that ships with the distribution.
>
>
> I'd say about 20% of the messages I send via the producer never get consumed by the Scala process (which is running continuously). No errors on either side (producer or consumer): the producer sends, and, nothing...
>
>
> Any ideas as to what might be going on here, or how I could start troubleshooting?
>
>
> Thanks!
Re: Messages intermittently get lost
Posted by Martin Gainty <mg...@hotmail.com>.
a shock when a zk script goes fubar
zookeeper devs can we get some help for sh zkServer.sh status?
Thanks!
Martin
______________________________________________
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Wednesday, November 30, 2016 10:25 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Martin, makes sense.
When I SSH into all 3 of my ZK nodes and run:
sh zkServer.sh status
All three of them give me the following output:
JMX enabled by default
zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "(" unexpected (expecting "fi")
Looks like a bug in the ZK shell script?
Best,
Zac
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Tuesday, November 29, 2016 11:18:21 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Zach
we dont know whats causing this intermittent problem..so lets Divide and Conquer each part of this problem individually starting at the source of the data feeds
Let us eliminate any potential problem with feeds from external sources
Once you verify the zookeeper feeds are 100% reliable lets move onto Kafka
Pingback when you have verifiable results from zookeeper feeds
Thanks
Martin
______________________________________________
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Tuesday, November 29, 2016 10:46 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Does anybody have any idea why ZK might be to blame if messages sent by a Kafka producer fail to be received by a Kafka consumer?
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Monday, November 28, 2016 9:07:41 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Thanks Martin, I will look at those links.
But you seem to be 100% confident that the problem is with ZooKeeper...can I ask why? What is it about my problem description that makes you think this is an issue with ZooKeeper?
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Friday, November 25, 2016 1:46:28 PM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Friday, November 25, 2016 6:17 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Martin,
My server.properties looks like this:
listeners=PLAINTEXT://0.0.0.0:9092
advertised.host.name=<SERVER_PUBLIC_DNS>
broker.id=2
port=9092
num.partitions=4
zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
MG>can you check status for each ZK Node in the quorum?
sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
MG>*greetings from Plimoth Mass*
MG>M
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
log.dirs=/tmp/kafka-logs
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connection.timeout.ms=6000
delete.topic.enable=true
auto.leader.rebalance.enable=true
Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK servers, and <SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node that this Kafka is running on.
Anything look incorrect to you?
And yes, yesterday was a holiday, but there is only work! I'll celebrate one big, long holiday when I'm dead!
Thanks for any-and-all input here!
Best,
Zac
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Thursday, November 24, 2016 9:03:33 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Zach
there is a rumour that today thursday is a holiday?
in server.properties how are you configuring your server?
specifically what are these attributes?
num.network.threads=
num.io.threads=
socket.send.buffer.bytes=
socket.receive.buffer.bytes=
socket.request.max.bytes=
num.partitions=
num.recovery.threads.per.data.dir=
?
Martin
______________________________________________
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Thursday, November 24, 2016 7:05 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Anybody?!? This is very disconcerting!
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Wednesday, November 23, 2016 5:07:45 AM
To: users@kafka.apache.org
Subject: Messages intermittently get lost
I am playing around with Kafka and have a simple setup:
* 1-node Kafka (Ubuntu) server
* 3-node ZK cluster (each on their own Ubuntu server)
I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that ships with the distribution.
I'd say about 20% of the messages I send via the producer never get consumed by the Scala process (which is running continuously). No errors on either side (producer or consumer): the producer sends, and, nothing...
Any ideas as to what might be going on here, or how I could start troubleshooting?
Thanks!
Re: Messages intermittently get lost
Posted by Martin Gainty <mg...@hotmail.com>.
a shock when a zk script goes fubar
zookeeper devs can we get some help for sh zkServer.sh status?
Thanks!
Martin
______________________________________________
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Wednesday, November 30, 2016 10:25 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Martin, makes sense.
When I SSH into all 3 of my ZK nodes and run:
sh zkServer.sh status
All three of them give me the following output:
JMX enabled by default
zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "(" unexpected (expecting "fi")
Looks like a bug in the ZK shell script?
Best,
Zac
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Tuesday, November 29, 2016 11:18:21 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Zach
we dont know whats causing this intermittent problem..so lets Divide and Conquer each part of this problem individually starting at the source of the data feeds
Let us eliminate any potential problem with feeds from external sources
Once you verify the zookeeper feeds are 100% reliable lets move onto Kafka
Pingback when you have verifiable results from zookeeper feeds
Thanks
Martin
______________________________________________
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Tuesday, November 29, 2016 10:46 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Does anybody have any idea why ZK might be to blame if messages sent by a Kafka producer fail to be received by a Kafka consumer?
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Monday, November 28, 2016 9:07:41 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Thanks Martin, I will look at those links.
But you seem to be 100% confident that the problem is with ZooKeeper...can I ask why? What is it about my problem description that makes you think this is an issue with ZooKeeper?
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Friday, November 25, 2016 1:46:28 PM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Friday, November 25, 2016 6:17 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Martin,
My server.properties looks like this:
listeners=PLAINTEXT://0.0.0.0:9092
advertised.host.name=<SERVER_PUBLIC_DNS>
broker.id=2
port=9092
num.partitions=4
zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
MG>can you check status for each ZK Node in the quorum?
sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
MG>*greetings from Plimoth Mass*
MG>M
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
log.dirs=/tmp/kafka-logs
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connection.timeout.ms=6000
delete.topic.enable=true
auto.leader.rebalance.enable=true
Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK servers, and <SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node that this Kafka is running on.
Anything look incorrect to you?
And yes, yesterday was a holiday, but there is only work! I'll celebrate one big, long holiday when I'm dead!
Thanks for any-and-all input here!
Best,
Zac
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Thursday, November 24, 2016 9:03:33 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Zach
there is a rumour that today thursday is a holiday?
in server.properties how are you configuring your server?
specifically what are these attributes?
num.network.threads=
num.io.threads=
socket.send.buffer.bytes=
socket.receive.buffer.bytes=
socket.request.max.bytes=
num.partitions=
num.recovery.threads.per.data.dir=
?
Martin
______________________________________________
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Thursday, November 24, 2016 7:05 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Anybody?!? This is very disconcerting!
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Wednesday, November 23, 2016 5:07:45 AM
To: users@kafka.apache.org
Subject: Messages intermittently get lost
I am playing around with Kafka and have a simple setup:
* 1-node Kafka (Ubuntu) server
* 3-node ZK cluster (each on their own Ubuntu server)
I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that ships with the distribution.
I'd say about 20% of the messages I send via the producer never get consumed by the Scala process (which is running continuously). No errors on either side (producer or consumer): the producer sends, and, nothing...
Any ideas as to what might be going on here, or how I could start troubleshooting?
Thanks!
Re: Messages intermittently get lost
Posted by Zac Harvey <za...@welltok.com>.
Hi Martin, makes sense.
When I SSH into all 3 of my ZK nodes and run:
sh zkServer.sh status
All three of them give me the following output:
JMX enabled by default
zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "(" unexpected (expecting "fi")
Looks like a bug in the ZK shell script?
Best,
Zac
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Tuesday, November 29, 2016 11:18:21 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Zach
we dont know whats causing this intermittent problem..so lets Divide and Conquer each part of this problem individually starting at the source of the data feeds
Let us eliminate any potential problem with feeds from external sources
Once you verify the zookeeper feeds are 100% reliable lets move onto Kafka
Pingback when you have verifiable results from zookeeper feeds
Thanks
Martin
______________________________________________
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Tuesday, November 29, 2016 10:46 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Does anybody have any idea why ZK might be to blame if messages sent by a Kafka producer fail to be received by a Kafka consumer?
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Monday, November 28, 2016 9:07:41 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Thanks Martin, I will look at those links.
But you seem to be 100% confident that the problem is with ZooKeeper...can I ask why? What is it about my problem description that makes you think this is an issue with ZooKeeper?
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Friday, November 25, 2016 1:46:28 PM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Friday, November 25, 2016 6:17 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Martin,
My server.properties looks like this:
listeners=PLAINTEXT://0.0.0.0:9092
advertised.host.name=<SERVER_PUBLIC_DNS>
broker.id=2
port=9092
num.partitions=4
zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
MG>can you check status for each ZK Node in the quorum?
sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com<http://www.ibm.com>
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com<http://www.ibm.com>
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com<http://www.ibm.com>
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
MG>*greetings from Plimoth Mass*
MG>M
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
log.dirs=/tmp/kafka-logs
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connection.timeout.ms=6000
delete.topic.enable=true
auto.leader.rebalance.enable=true
Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK servers, and <SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node that this Kafka is running on.
Anything look incorrect to you?
And yes, yesterday was a holiday, but there is only work! I'll celebrate one big, long holiday when I'm dead!
Thanks for any-and-all input here!
Best,
Zac
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Thursday, November 24, 2016 9:03:33 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Zach
there is a rumour that today thursday is a holiday?
in server.properties how are you configuring your server?
specifically what are these attributes?
num.network.threads=
num.io.threads=
socket.send.buffer.bytes=
socket.receive.buffer.bytes=
socket.request.max.bytes=
num.partitions=
num.recovery.threads.per.data.dir=
?
Martin
______________________________________________
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Thursday, November 24, 2016 7:05 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Anybody?!? This is very disconcerting!
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Wednesday, November 23, 2016 5:07:45 AM
To: users@kafka.apache.org
Subject: Messages intermittently get lost
I am playing around with Kafka and have a simple setup:
* 1-node Kafka (Ubuntu) server
* 3-node ZK cluster (each on their own Ubuntu server)
I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that ships with the distribution.
I'd say about 20% of the messages I send via the producer never get consumed by the Scala process (which is running continuously). No errors on either side (producer or consumer): the producer sends, and, nothing...
Any ideas as to what might be going on here, or how I could start troubleshooting?
Thanks!
Re: Messages intermittently get lost
Posted by Martin Gainty <mg...@hotmail.com>.
Hi Zach
we dont know whats causing this intermittent problem..so lets Divide and Conquer each part of this problem individually starting at the source of the data feeds
Let us eliminate any potential problem with feeds from external sources
Once you verify the zookeeper feeds are 100% reliable lets move onto Kafka
Pingback when you have verifiable results from zookeeper feeds
Thanks
Martin
______________________________________________
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Tuesday, November 29, 2016 10:46 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Does anybody have any idea why ZK might be to blame if messages sent by a Kafka producer fail to be received by a Kafka consumer?
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Monday, November 28, 2016 9:07:41 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Thanks Martin, I will look at those links.
But you seem to be 100% confident that the problem is with ZooKeeper...can I ask why? What is it about my problem description that makes you think this is an issue with ZooKeeper?
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Friday, November 25, 2016 1:46:28 PM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Friday, November 25, 2016 6:17 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Martin,
My server.properties looks like this:
listeners=PLAINTEXT://0.0.0.0:9092
advertised.host.name=<SERVER_PUBLIC_DNS>
broker.id=2
port=9092
num.partitions=4
zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
MG>can you check status for each ZK Node in the quorum?
sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations that matter for the world
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
MG>*greetings from Plimoth Mass*
MG>M
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
log.dirs=/tmp/kafka-logs
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connection.timeout.ms=6000
delete.topic.enable=true
auto.leader.rebalance.enable=true
Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK servers, and <SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node that this Kafka is running on.
Anything look incorrect to you?
And yes, yesterday was a holiday, but there is only work! I'll celebrate one big, long holiday when I'm dead!
Thanks for any-and-all input here!
Best,
Zac
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Thursday, November 24, 2016 9:03:33 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Zach
there is a rumour that today thursday is a holiday?
in server.properties how are you configuring your server?
specifically what are these attributes?
num.network.threads=
num.io.threads=
socket.send.buffer.bytes=
socket.receive.buffer.bytes=
socket.request.max.bytes=
num.partitions=
num.recovery.threads.per.data.dir=
?
Martin
______________________________________________
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Thursday, November 24, 2016 7:05 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Anybody?!? This is very disconcerting!
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Wednesday, November 23, 2016 5:07:45 AM
To: users@kafka.apache.org
Subject: Messages intermittently get lost
I am playing around with Kafka and have a simple setup:
* 1-node Kafka (Ubuntu) server
* 3-node ZK cluster (each on their own Ubuntu server)
I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that ships with the distribution.
I'd say about 20% of the messages I send via the producer never get consumed by the Scala process (which is running continuously). No errors on either side (producer or consumer): the producer sends, and, nothing...
Any ideas as to what might be going on here, or how I could start troubleshooting?
Thanks!
Re: Messages intermittently get lost
Posted by Zac Harvey <za...@welltok.com>.
Does anybody have any idea why ZK might be to blame if messages sent by a Kafka producer fail to be received by a Kafka consumer?
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Monday, November 28, 2016 9:07:41 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Thanks Martin, I will look at those links.
But you seem to be 100% confident that the problem is with ZooKeeper...can I ask why? What is it about my problem description that makes you think this is an issue with ZooKeeper?
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Friday, November 25, 2016 1:46:28 PM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Friday, November 25, 2016 6:17 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Martin,
My server.properties looks like this:
listeners=PLAINTEXT://0.0.0.0:9092
advertised.host.name=<SERVER_PUBLIC_DNS>
broker.id=2
port=9092
num.partitions=4
zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
MG>can you check status for each ZK Node in the quorum?
sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com<http://www.ibm.com>
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
MG>*greetings from Plimoth Mass*
MG>M
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
log.dirs=/tmp/kafka-logs
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connection.timeout.ms=6000
delete.topic.enable=true
auto.leader.rebalance.enable=true
Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK servers, and <SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node that this Kafka is running on.
Anything look incorrect to you?
And yes, yesterday was a holiday, but there is only work! I'll celebrate one big, long holiday when I'm dead!
Thanks for any-and-all input here!
Best,
Zac
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Thursday, November 24, 2016 9:03:33 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Zach
there is a rumour that today thursday is a holiday?
in server.properties how are you configuring your server?
specifically what are these attributes?
num.network.threads=
num.io.threads=
socket.send.buffer.bytes=
socket.receive.buffer.bytes=
socket.request.max.bytes=
num.partitions=
num.recovery.threads.per.data.dir=
?
Martin
______________________________________________
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Thursday, November 24, 2016 7:05 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Anybody?!? This is very disconcerting!
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Wednesday, November 23, 2016 5:07:45 AM
To: users@kafka.apache.org
Subject: Messages intermittently get lost
I am playing around with Kafka and have a simple setup:
* 1-node Kafka (Ubuntu) server
* 3-node ZK cluster (each on their own Ubuntu server)
I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that ships with the distribution.
I'd say about 20% of the messages I send via the producer never get consumed by the Scala process (which is running continuously). No errors on either side (producer or consumer): the producer sends, and, nothing...
Any ideas as to what might be going on here, or how I could start troubleshooting?
Thanks!
Re: Messages intermittently get lost
Posted by Zac Harvey <za...@welltok.com>.
Thanks Martin, I will look at those links.
But you seem to be 100% confident that the problem is with ZooKeeper...can I ask why? What is it about my problem description that makes you think this is an issue with ZooKeeper?
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Friday, November 25, 2016 1:46:28 PM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Friday, November 25, 2016 6:17 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Martin,
My server.properties looks like this:
listeners=PLAINTEXT://0.0.0.0:9092
advertised.host.name=<SERVER_PUBLIC_DNS>
broker.id=2
port=9092
num.partitions=4
zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
MG>can you check status for each ZK Node in the quorum?
sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com<http://www.ibm.com>
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
MG>*greetings from Plimoth Mass*
MG>M
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
log.dirs=/tmp/kafka-logs
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connection.timeout.ms=6000
delete.topic.enable=true
auto.leader.rebalance.enable=true
Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK servers, and <SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node that this Kafka is running on.
Anything look incorrect to you?
And yes, yesterday was a holiday, but there is only work! I'll celebrate one big, long holiday when I'm dead!
Thanks for any-and-all input here!
Best,
Zac
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Thursday, November 24, 2016 9:03:33 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Zach
there is a rumour that today thursday is a holiday?
in server.properties how are you configuring your server?
specifically what are these attributes?
num.network.threads=
num.io.threads=
socket.send.buffer.bytes=
socket.receive.buffer.bytes=
socket.request.max.bytes=
num.partitions=
num.recovery.threads.per.data.dir=
?
Martin
______________________________________________
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Thursday, November 24, 2016 7:05 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Anybody?!? This is very disconcerting!
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Wednesday, November 23, 2016 5:07:45 AM
To: users@kafka.apache.org
Subject: Messages intermittently get lost
I am playing around with Kafka and have a simple setup:
* 1-node Kafka (Ubuntu) server
* 3-node ZK cluster (each on their own Ubuntu server)
I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that ships with the distribution.
I'd say about 20% of the messages I send via the producer never get consumed by the Scala process (which is running continuously). No errors on either side (producer or consumer): the producer sends, and, nothing...
Any ideas as to what might be going on here, or how I could start troubleshooting?
Thanks!
Re: Messages intermittently get lost
Posted by Martin Gainty <mg...@hotmail.com>.
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Friday, November 25, 2016 6:17 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Martin,
My server.properties looks like this:
listeners=PLAINTEXT://0.0.0.0:9092
advertised.host.name=<SERVER_PUBLIC_DNS>
broker.id=2
port=9092
num.partitions=4
zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
MG>can you check status for each ZK Node in the quorum?
sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
MG>*greetings from Plimoth Mass*
MG>M
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
log.dirs=/tmp/kafka-logs
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connection.timeout.ms=6000
delete.topic.enable=true
auto.leader.rebalance.enable=true
Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK servers, and <SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node that this Kafka is running on.
Anything look incorrect to you?
And yes, yesterday was a holiday, but there is only work! I'll celebrate one big, long holiday when I'm dead!
Thanks for any-and-all input here!
Best,
Zac
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Thursday, November 24, 2016 9:03:33 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Zach
there is a rumour that today thursday is a holiday?
in server.properties how are you configuring your server?
specifically what are these attributes?
num.network.threads=
num.io.threads=
socket.send.buffer.bytes=
socket.receive.buffer.bytes=
socket.request.max.bytes=
num.partitions=
num.recovery.threads.per.data.dir=
?
Martin
______________________________________________
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Thursday, November 24, 2016 7:05 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Anybody?!? This is very disconcerting!
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Wednesday, November 23, 2016 5:07:45 AM
To: users@kafka.apache.org
Subject: Messages intermittently get lost
I am playing around with Kafka and have a simple setup:
* 1-node Kafka (Ubuntu) server
* 3-node ZK cluster (each on their own Ubuntu server)
I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that ships with the distribution.
I'd say about 20% of the messages I send via the producer never get consumed by the Scala process (which is running continuously). No errors on either side (producer or consumer): the producer sends, and, nothing...
Any ideas as to what might be going on here, or how I could start troubleshooting?
Thanks!
Re: Messages intermittently get lost
Posted by Zac Harvey <za...@welltok.com>.
Hi Martin,
My server.properties looks like this:
listeners=PLAINTEXT://0.0.0.0:9092
advertised.host.name=<SERVER_PUBLIC_DNS>
broker.id=2
port=9092
num.partitions=4
zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
log.dirs=/tmp/kafka-logs
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connection.timeout.ms=6000
delete.topic.enable=true
auto.leader.rebalance.enable=true
Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK servers, and <SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node that this Kafka is running on.
Anything look incorrect to you?
And yes, yesterday was a holiday, but there is only work! I'll celebrate one big, long holiday when I'm dead!
Thanks for any-and-all input here!
Best,
Zac
________________________________
From: Martin Gainty <mg...@hotmail.com>
Sent: Thursday, November 24, 2016 9:03:33 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Hi Zach
there is a rumour that today thursday is a holiday?
in server.properties how are you configuring your server?
specifically what are these attributes?
num.network.threads=
num.io.threads=
socket.send.buffer.bytes=
socket.receive.buffer.bytes=
socket.request.max.bytes=
num.partitions=
num.recovery.threads.per.data.dir=
?
Martin
______________________________________________
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Thursday, November 24, 2016 7:05 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Anybody?!? This is very disconcerting!
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Wednesday, November 23, 2016 5:07:45 AM
To: users@kafka.apache.org
Subject: Messages intermittently get lost
I am playing around with Kafka and have a simple setup:
* 1-node Kafka (Ubuntu) server
* 3-node ZK cluster (each on their own Ubuntu server)
I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that ships with the distribution.
I'd say about 20% of the messages I send via the producer never get consumed by the Scala process (which is running continuously). No errors on either side (producer or consumer): the producer sends, and, nothing...
Any ideas as to what might be going on here, or how I could start troubleshooting?
Thanks!
Re: Messages intermittently get lost
Posted by Martin Gainty <mg...@hotmail.com>.
Hi Zach
there is a rumour that today thursday is a holiday?
in server.properties how are you configuring your server?
specifically what are these attributes?
num.network.threads=
num.io.threads=
socket.send.buffer.bytes=
socket.receive.buffer.bytes=
socket.request.max.bytes=
num.partitions=
num.recovery.threads.per.data.dir=
?
Martin
______________________________________________
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Thursday, November 24, 2016 7:05 AM
To: users@kafka.apache.org
Subject: Re: Messages intermittently get lost
Anybody?!? This is very disconcerting!
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Wednesday, November 23, 2016 5:07:45 AM
To: users@kafka.apache.org
Subject: Messages intermittently get lost
I am playing around with Kafka and have a simple setup:
* 1-node Kafka (Ubuntu) server
* 3-node ZK cluster (each on their own Ubuntu server)
I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that ships with the distribution.
I'd say about 20% of the messages I send via the producer never get consumed by the Scala process (which is running continuously). No errors on either side (producer or consumer): the producer sends, and, nothing...
Any ideas as to what might be going on here, or how I could start troubleshooting?
Thanks!
Re: Messages intermittently get lost
Posted by Zac Harvey <za...@welltok.com>.
Anybody?!? This is very disconcerting!
________________________________
From: Zac Harvey <za...@welltok.com>
Sent: Wednesday, November 23, 2016 5:07:45 AM
To: users@kafka.apache.org
Subject: Messages intermittently get lost
I am playing around with Kafka and have a simple setup:
* 1-node Kafka (Ubuntu) server
* 3-node ZK cluster (each on their own Ubuntu server)
I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that ships with the distribution.
I'd say about 20% of the messages I send via the producer never get consumed by the Scala process (which is running continuously). No errors on either side (producer or consumer): the producer sends, and, nothing...
Any ideas as to what might be going on here, or how I could start troubleshooting?
Thanks!