You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by 최재완 <jw...@naver.com> on 2016/11/22 02:14:37 UTC
kafka spout consume same data every restart topology
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">Hi
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">I'm currently developing storm version up project
0.9.6->1.0.2
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">I changed my project configuration and source like below
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">1. storm-core and storm-kafka version change 0.9.6 to
1.0.2 and kafka_2.10 0.8.2.2 in pom.xml.
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">2. change package path
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">- backtype -> org.apache
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">- storm.kafka -> org.apache.storm.kafka
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">3. Change serializing code byte[] to ByteBuffer in Custom
Scheme(I used AVRO class)
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">I started topology and my spout consume data.
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">But when I restarted my topology, read same data that I
restarted before and read it.
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">I didn't change zkRoot value and id in SpoutConfig.(I
think it means kafka group id)
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">Eventhough it occurred both local and server mode.
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">How can I fix it? Please Help!
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">
<span lang="EN-US" style="font-size:12.0pt;font-family:"굴림",sans-serif;mso-bidi-font-family:Gulim;
mso-font-kerning:0pt">Thanks and regards<span lang="EN-US" style="font-size:11.0pt;mso-ascii-font-family:"Malgun Gothic";mso-fareast-font-family:
"Malgun Gothic";mso-hansi-font-family:"Malgun Gothic";mso-bidi-font-family:
Gulim;mso-font-kerning:0pt">
Re: kafka spout consume same data every restart topology
Posted by 최재완 <jw...@naver.com>.
Yes, I already set auto commit true(default value is true).
Topology for storm 0.9.6 version is executing already.
In storm 0.9.6, I made spout like below. But spout read data continuously every restart.
SpoutConfig spoutConf = new SpoutConfig(zkHosts, topicWithUnitServiceCode,
groupWithPrefix, UUID.randomUUID().toString());
But up to 1.0.2, spout consume read same data every restart.
So I changed and fixed id parameter to "test", but occur samely.
Any changed in SpoutConfig and KafkaSpout?
-----Original Message-----
From: <smwangf@tsmc.com>
To: <user@storm.apache.org>;
Cc:
Sent: 2016-11-22 (화) 11:23:53
Subject: Re: kafka spout consume same data every restart topology
Hi.
You should commit the message when you finish the process or set the Kafka
auto commit to true.
There is a kafka offset setting which will only read the latest message.
auto.offset.reset = latest.
---------------------------------------------------------------------------
TSMC PROPERTY
This email communication (and any attachments) is proprietary information
for the sole use of its
intended recipient. Any unauthorized review, use or distribution by anyone
other than the intended
recipient is strictly prohibited. If you are not the intended recipient,
please notify the sender by
replying to this email, and then delete this email and any copies of it
immediately. Thank you.
---------------------------------------------------------------------------
Re: kafka spout consume same data every restart topology
Posted by sm...@tsmc.com.
Hi.
You should commit the message when you finish the process or set the Kafka
auto commit to true.
There is a kafka offset setting which will only read the latest message.
auto.offset.reset = latest.
---------------------------------------------------------------------------
TSMC PROPERTY
This email communication (and any attachments) is proprietary information
for the sole use of its
intended recipient. Any unauthorized review, use or distribution by anyone
other than the intended
recipient is strictly prohibited. If you are not the intended recipient,
please notify the sender by
replying to this email, and then delete this email and any copies of it
immediately. Thank you.
---------------------------------------------------------------------------