You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@storm.apache.org by Santosh Pingale <pi...@gmail.com> on 2015/10/25 07:16:48 UTC

Storm Offsets in Zookeeper Automatically Rolling Back

I am facing a very weird issue at the moment.

I have a storm topology which uses Kafka Spout and it stores offsets in ZK.

Kafka topic sometimes do not receive messages for a brief period of time
and the topology catches up with the Kafka offsets.

But after sometime, offset stored in ZK change to a lesser number
indicating topology is failing behind. I need to restart the topology to
make it catch up again. I am not able to understand whats going on. This is
very strange. I believe there is no other topology running with same ZK
root.

Storm Version: 0.9.5
ZK Version: 3.4.6
Kafka Version : 0.8.1.1.
Storm-Kafka Version: 0.9.5

Your help is appreciated.

Re: Storm Offsets in Zookeeper Automatically Rolling Back

Posted by Santosh Pingale <pi...@gmail.com>.

Craig and Harsha: Those were the points I was following while
troubleshooting the issue!

I will check with my colleagues whether someone uses the same zk root.

Spout id is unique across each topology but same across all the deployments
of same topology
I have useStartOffsetTimeIfOffsetOutOfRange set to true.

I haven't observed this issue again, so it might be some one else using the
same consumer group.

On Sun, Oct 25, 2015 at 9:03 PM, Harsha <st...@harsha.io> wrote:

> Santosh,
>            Make sure your SpoutConfig(BrokerHosts hosts, String topic,
>            String zkRoot, String id); "String id" is static value and
>            unique across the topologies. If you killing & re-deploying
>            the topology the value shouldn't be changing. Apart from that
>            depending on which version you are using forceFromStart or
>            ignoreZkOffsets set to false.
>
> -Harsha
>
> On Sun, Oct 25, 2015, at 06:18 AM, Craig Charleton wrote:
> > Keep in mind that Zookeeper stores the Kafka offsets as they relate to
> > the consumer group, not overall. Maybe you have another component running
> > under the same consumer group or you could experiment with running your
> > topology with a different Kafka consumer group.
> >
> > Sent from my iPhone
> >
> > > On Oct 25, 2015, at 2:16 AM, Santosh Pingale <pi...@gmail.com>
> wrote:
> > >
> > > --089e01184bf256960a0522e7cb6c
> > > Content-Type: text/plain; charset=UTF-8
> > >
> > > I am facing a very weird issue at the moment.
> > >
> > > I have a storm topology which uses Kafka Spout and it stores offsets
> in ZK.
> > >
> > > Kafka topic sometimes do not receive messages for a brief period of
> time
> > > and the topology catches up with the Kafka offsets.
> > >
> > > But after sometime, offset stored in ZK change to a lesser number
> > > indicating topology is failing behind. I need to restart the topology
> to
> > > make it catch up again. I am not able to understand whats going on.
> This is
> > > very strange. I believe there is no other topology running with same ZK
> > > root.
> > >
> > > Storm Version: 0.9.5
> > > ZK Version: 3.4.6
> > > Kafka Version : 0.8.1.1.
> > > Storm-Kafka Version: 0.9.5
> > >
> > > Your help is appreciated.
> > >
> > > --089e01184bf256960a0522e7cb6c
> > > Content-Type: text/html; charset=UTF-8
> > > Content-Transfer-Encoding: quoted-printable
> > >
> > > <div dir=3D"ltr"><div>I am facing a very weird issue at the
> moment.</div><d=
> > > iv><br></div><div>I have a storm topology which uses Kafka Spout and
> it sto=
> > > res offsets in ZK.</div><div><br></div><div>Kafka topic sometimes do
> not re=
> > > ceive messages for a brief period of time and the topology catches up
> with =
> > > the Kafka offsets.=C2=A0</div><div><br></div><div>But after sometime,
> offse=
> > > t stored in ZK change to a lesser number indicating topology is
> failing beh=
> > > ind. I need to restart the topology to make it catch up again. I am
> not abl=
> > > e to understand whats going on. This is very strange. I believe there
> is no=
> > > other topology running with same ZK
> root.</div><div><br></div><div>Storm V=
> > > ersion: 0.9.5</div><div>ZK Version: 3.4.6</div><div>Kafka Version :
> 0.8.1.1=
> > > .</div><div>Storm-Kafka Version: 0.9.5</div><div><br></div><div>Your
> help i=
> > > s appreciated.</div></div>
> > >
> > > --089e01184bf256960a0522e7cb6c--
>

Re: Storm Offsets in Zookeeper Automatically Rolling Back

Posted by Harsha <st...@harsha.io>.

Santosh,
           Make sure your SpoutConfig(BrokerHosts hosts, String topic,
           String zkRoot, String id); "String id" is static value and
           unique across the topologies. If you killing & re-deploying
           the topology the value shouldn't be changing. Apart from that
           depending on which version you are using forceFromStart or
           ignoreZkOffsets set to false.

-Harsha

On Sun, Oct 25, 2015, at 06:18 AM, Craig Charleton wrote:
> Keep in mind that Zookeeper stores the Kafka offsets as they relate to
> the consumer group, not overall. Maybe you have another component running
> under the same consumer group or you could experiment with running your
> topology with a different Kafka consumer group. 
> 
> Sent from my iPhone
> 
> > On Oct 25, 2015, at 2:16 AM, Santosh Pingale <pi...@gmail.com> wrote:
> > 
> > --089e01184bf256960a0522e7cb6c
> > Content-Type: text/plain; charset=UTF-8
> > 
> > I am facing a very weird issue at the moment.
> > 
> > I have a storm topology which uses Kafka Spout and it stores offsets in ZK.
> > 
> > Kafka topic sometimes do not receive messages for a brief period of time
> > and the topology catches up with the Kafka offsets.
> > 
> > But after sometime, offset stored in ZK change to a lesser number
> > indicating topology is failing behind. I need to restart the topology to
> > make it catch up again. I am not able to understand whats going on. This is
> > very strange. I believe there is no other topology running with same ZK
> > root.
> > 
> > Storm Version: 0.9.5
> > ZK Version: 3.4.6
> > Kafka Version : 0.8.1.1.
> > Storm-Kafka Version: 0.9.5
> > 
> > Your help is appreciated.
> > 
> > --089e01184bf256960a0522e7cb6c
> > Content-Type: text/html; charset=UTF-8
> > Content-Transfer-Encoding: quoted-printable
> > 
> > <div dir=3D"ltr"><div>I am facing a very weird issue at the moment.</div><d=
> > iv><br></div><div>I have a storm topology which uses Kafka Spout and it sto=
> > res offsets in ZK.</div><div><br></div><div>Kafka topic sometimes do not re=
> > ceive messages for a brief period of time and the topology catches up with =
> > the Kafka offsets.=C2=A0</div><div><br></div><div>But after sometime, offse=
> > t stored in ZK change to a lesser number indicating topology is failing beh=
> > ind. I need to restart the topology to make it catch up again. I am not abl=
> > e to understand whats going on. This is very strange. I believe there is no=
> > other topology running with same ZK root.</div><div><br></div><div>Storm V=
> > ersion: 0.9.5</div><div>ZK Version: 3.4.6</div><div>Kafka Version : 0.8.1.1=
> > .</div><div>Storm-Kafka Version: 0.9.5</div><div><br></div><div>Your help i=
> > s appreciated.</div></div>
> > 
> > --089e01184bf256960a0522e7cb6c--

Re: Storm Offsets in Zookeeper Automatically Rolling Back

Posted by Craig Charleton <cr...@gmail.com>.

Keep in mind that Zookeeper stores the Kafka offsets as they relate to the consumer group, not overall. Maybe you have another component running under the same consumer group or you could experiment with running your topology with a different Kafka consumer group. 

Sent from my iPhone

> On Oct 25, 2015, at 2:16 AM, Santosh Pingale <pi...@gmail.com> wrote:
> 
> --089e01184bf256960a0522e7cb6c
> Content-Type: text/plain; charset=UTF-8
> 
> I am facing a very weird issue at the moment.
> 
> I have a storm topology which uses Kafka Spout and it stores offsets in ZK.
> 
> Kafka topic sometimes do not receive messages for a brief period of time
> and the topology catches up with the Kafka offsets.
> 
> But after sometime, offset stored in ZK change to a lesser number
> indicating topology is failing behind. I need to restart the topology to
> make it catch up again. I am not able to understand whats going on. This is
> very strange. I believe there is no other topology running with same ZK
> root.
> 
> Storm Version: 0.9.5
> ZK Version: 3.4.6
> Kafka Version : 0.8.1.1.
> Storm-Kafka Version: 0.9.5
> 
> Your help is appreciated.
> 
> --089e01184bf256960a0522e7cb6c
> Content-Type: text/html; charset=UTF-8
> Content-Transfer-Encoding: quoted-printable
> 
> <div dir=3D"ltr"><div>I am facing a very weird issue at the moment.</div><d=
> iv><br></div><div>I have a storm topology which uses Kafka Spout and it sto=
> res offsets in ZK.</div><div><br></div><div>Kafka topic sometimes do not re=
> ceive messages for a brief period of time and the topology catches up with =
> the Kafka offsets.=C2=A0</div><div><br></div><div>But after sometime, offse=
> t stored in ZK change to a lesser number indicating topology is failing beh=
> ind. I need to restart the topology to make it catch up again. I am not abl=
> e to understand whats going on. This is very strange. I believe there is no=
> other topology running with same ZK root.</div><div><br></div><div>Storm V=
> ersion: 0.9.5</div><div>ZK Version: 3.4.6</div><div>Kafka Version : 0.8.1.1=
> .</div><div>Storm-Kafka Version: 0.9.5</div><div><br></div><div>Your help i=
> s appreciated.</div></div>
> 
> --089e01184bf256960a0522e7cb6c--