Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

[Chris Neal <cw...@gmail.com>]

Hi all,

I have a weird situation here.  I have deleted a few topics on my 0.8.1.1
cluster (old, I know...).  The deletes succeeded according to the
controller.log:

[2017-07-20 16:40:31,175] INFO [TopicChangeListener on Controller 1]: New
topics: [Set()], deleted topics:
[Set(perf_doorway-supplier-adapter-uat_raw)], new partition replica
assignment [Map()]
(kafka.controller.PartitionStateMachine$TopicChangeListener)
[2017-07-20 16:40:33,507] INFO [TopicChangeListener on Controller 1]: New
topics: [Set()], deleted topics:
[Set(perf_doorway-supplier-scheduler-uat_raw)], new partition replica
assignment [Map()]
(kafka.controller.PartitionStateMachine$TopicChangeListener)
[2017-07-20 16:40:36,504] INFO [TopicChangeListener on Controller 1]: New
topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)], new
partition replica assignment [Map()]
(kafka.controller.PartitionStateMachine$TopicChangeListener)
[2017-07-20 16:40:38,290] INFO [TopicChangeListener on Controller 1]: New
topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)], new
partition replica assignment [Map()]
(kafka.controller.PartitionStateMachine$TopicChangeListener)

I query Zookeeper and the path is not there under /brokers/topics as well.

But, one of the nodes in my cluster continues to try and use them:

[2017-07-20 17:04:36,723] ERROR Conditional update of path
/brokers/topics/perf_doorway-supplier-scheduler-uat_raw/partitions/3/state
with data
{"controller_epoch":34,"leader":1,"version":1,"leader_epoch":2,"isr":[1,0]}
and expected version 69 failed due to
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for
/brokers/topics/perf_doorway-supplier-scheduler-uat_raw/partitions/3/state
(kafka.utils.ZkUtils$)
[2017-07-20 17:04:36,723] INFO Partition
[perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached zkVersion
[69] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition)
[2017-07-20 17:04:36,723] INFO Partition
[perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached zkVersion
[69] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition)
[2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] on
broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] from 1,0
to 1 (kafka.cluster.Partition)
[2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] on
broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] from 1,0
to 1 (kafka.cluster.Partition)
[2017-07-20 17:04:36,765] ERROR Conditional update of path
/brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data
{"controller_epoch":34,"leader":1,"version":1,"leader_epoch":2,"isr":[1]}
and expected version 70 failed due to
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/2/state
(kafka.utils.ZkUtils$)
[2017-07-20 17:04:36,765] ERROR Conditional update of path
/brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data
{"controller_epoch":34,"leader":1,"version":1,"leader_epoch":2,"isr":[1]}
and expected version 70 failed due to
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/2/state
(kafka.utils.ZkUtils$)
[2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2] on
broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip
updating ISR (kafka.cluster.Partition)
[2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2] on
broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip
updating ISR (kafka.cluster.Partition)
[2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1] on
broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from 1,0
to 1 (kafka.cluster.Partition)
[2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1] on
broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from 1,0
to 1 (kafka.cluster.Partition)
[2017-07-20 17:04:36,988] ERROR Conditional update of path
/brokers/topics/perf_gocontent-uat_raw/partitions/1/state with data
{"controller_epoch":34,"leader":1,"version":1,"leader_epoch":4,"isr":[1]}
and expected version 90 failed due to
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
(kafka.utils.ZkUtils$)
[2017-07-20 17:04:36,988] ERROR Conditional update of path
/brokers/topics/perf_gocontent-uat_raw/partitions/1/state with data
{"controller_epoch":34,"leader":1,"version":1,"leader_epoch":4,"isr":[1]}
and expected version 90 failed due to
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
(kafka.utils.ZkUtils$)
[2017-07-20 17:04:36,988] INFO Partition [perf_gocontent-uat_raw,1] on
broker 1: Cached zkVersion [90] not equal to that in zookeeper, skip
updating ISR (kafka.cluster.Partition)
[2017-07-20 17:04:36,988] INFO Partition [perf_gocontent-uat_raw,1] on
broker 1: Cached zkVersion [90] not equal to that in zookeeper, skip
updating ISR (kafka.cluster.Partition)

I've tried a rolling restart of the cluster to see if that fixed it, but it
did not.

Can someone please help me out here?  I'm not sure how I can get things
back in sync.

Thank you so much for your time.
Chris

Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

[Chris Neal <cw...@gmail.com>]

Gotcha.  Thanks again.  Will post back once I've tried this with an update!

Chris

On Fri, Jul 21, 2017 at 1:12 PM, Carl Haferd <ch...@groupon.com.invalid>
wrote:

> I would recommend allowing each broker enough time to catch-up before
> starting the next, but this may be less of a concern if the entire cluster
> is being brought down and then started from scratch.  To automate, we poll
> until the Kafka process binds to its configured port (9092), and then once
> all brokers are up we also poll until there are no longer any
> under-replicated partitions before performing further maintenance.
>
> Carl
>
> On Fri, Jul 21, 2017 at 10:50 AM, Chris Neal <cw...@gmail.com> wrote:
>
> > Thanks Carl.
> >
> > Always fun to do this stuff in production... ;)
> >
> > Appreciate the input.  I'll try a full cycle and see how that works.
> >
> > In your opinion, if I stop all brokers and all Zookeeper nodes, then
> > restart all Zookeepers...at that point can I start both brokers at the
> same
> > time, or should I let one broker fully start and read all the unflushed
> > segments from disk before starting the second broker?
> >
> > Again, many thanks.
> > Chris
> >
> >
> > On Fri, Jul 21, 2017 at 12:13 PM, Carl Haferd
> <chaferd@groupon.com.invalid
> > >
> > wrote:
> >
> > > I have encountered similar difficulties in a test environment and it
> may
> > be
> > > necessary to stop the Kafka process on each broker and take Zookeeper
> > > offline before removing the files and zookeeper paths.  Otherwise there
> > may
> > > be a race condition between brokers which could cause the cluster to
> > retain
> > > information for the topic.
> > >
> > > Carl
> > >
> > > On Fri, Jul 21, 2017 at 9:06 AM, Chris Neal <cw...@gmail.com> wrote:
> > >
> > > > Welp.  Surprisingly, that did not fix the problem. :(
> > > >
> > > > I cleaned out all the entries for these topics from /config/topics,
> and
> > > > removed the logs from the file system for those topics, and the
> > messages
> > > > are still flying by in the server.log file.
> > > >
> > > > Also, more concerning, when I was looking through the log files for
> the
> > > > other broker in the cluster, I noticed the same type of message for a
> > > topic
> > > > that should actually be there:
> > > >
> > > > [2017-07-21 16:03:29,140] ERROR Conditional update of path
> > > > /brokers/topics/perf_dstorage_raw/partitions/4/state with data
> > > > {"controller_epoch":34,"leader":0,"version":1,"leader_
> > > epoch":0,"isr":[0]}
> > > > and expected version 0 failed due to
> > > > org.apache.zookeeper.KeeperException$BadVersionException:
> > > KeeperErrorCode
> > > > =
> > > > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/4/state
> > > > (kafka.utils.ZkUtils$)
> > > > [2017-07-21 16:03:29,142] ERROR Conditional update of path
> > > > /brokers/topics/perf_dstorage_raw/partitions/0/state with data
> > > > {"controller_epoch":34,"leader":0,"version":1,"leader_
> > > epoch":0,"isr":[0]}
> > > > and expected version 0 failed due to
> > > > org.apache.zookeeper.KeeperException$BadVersionException:
> > > KeeperErrorCode
> > > > =
> > > > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state
> > > > (kafka.utils.ZkUtils$)
> > > > [2017-07-21 16:03:29,142] ERROR Conditional update of path
> > > > /brokers/topics/perf_dstorage_raw/partitions/0/state with data
> > > > {"controller_epoch":34,"leader":0,"version":1,"leader_
> > > epoch":0,"isr":[0]}
> > > > and expected version 0 failed due to
> > > > org.apache.zookeeper.KeeperException$BadVersionException:
> > > KeeperErrorCode
> > > > =
> > > > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state
> > > > (kafka.utils.ZkUtils$)
> > > >
> > > > So, the issue is not isolated to just these
> "should-have-been-removed"
> > > > topics, unfortunately.
> > > >
> > > > Really appreciate the input so far everyone.  Still looking though
> for
> > a
> > > > solution.  Many thanks. :)
> > > >
> > > > Chris
> > > >
> > > > On Fri, Jul 21, 2017 at 10:58 AM, M. Manna <ma...@gmail.com>
> wrote:
> > > >
> > > > > Just to add (in case the platoform is Windows)
> > > > >
> > > > > For Windows based cluster implementation, log/topic cleanup doesn't
> > > work
> > > > > out of the box. Users are more or less aware of it, and doing their
> > own
> > > > > maintenance as workaround.
> > > > >  If you have issues on Topic deletion not working properly on
> Windows
> > > > (i.e.
> > > > > with topic deletion enable and all other settings). then you have
> to
> > > > > manually delete the files.
> > > > >
> > > > >
> > > > > On 21 July 2017 at 16:53, Chris Neal <cw...@gmail.com> wrote:
> > > > >
> > > > > > @Carl,
> > > > > >
> > > > > > There is nothing under /admin/delete_topics other than
> > > > > >
> > > > > > []
> > > > > >
> > > > > > And nothing under /admin other than delete_topics :)
> > > > > >
> > > > > > The topics DO exist, however, under /config/topics!  We may be on
> > to
> > > > > > something.  I will remove them here and see if that clears it up.
> > > > > >
> > > > > > Thanks so much for all the help!
> > > > > > Chris
> > > > > >
> > > > > > On Thu, Jul 20, 2017 at 10:37 PM, Chris Neal <cw...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Thanks again for the replies.  VERY much appreciated.  I'll
> check
> > > > both
> > > > > > > /admin/delete_topics and /config/topics.
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > On Thu, Jul 20, 2017 at 9:22 PM, Carl Haferd
> > > > > <chaferd@groupon.com.invalid
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> If delete normally works, there would hopefully be some log
> > > entries
> > > > > when
> > > > > > >> it
> > > > > > >> fails.  Are there any unusual zookeeper entries in the
> > > > > > >> /admin/delete_topics
> > > > > > >> path or in the other /admin folders?
> > > > > > >>
> > > > > > >> Does the topic name still exist in zookeeper under
> > /config/topics?
> > > > If
> > > > > > so,
> > > > > > >> that should probably deleted as well.
> > > > > > >>
> > > > > > >> Carl
> > > > > > >>
> > > > > > >> On Thu, Jul 20, 2017 at 6:42 PM, Chris Neal <cwneal@gmail.com
> >
> > > > wrote:
> > > > > > >>
> > > > > > >> > Delete is definitely there.  The delete worked fine, based
> on
> > > the
> > > > > fact
> > > > > > >> that
> > > > > > >> > there is nothing in Zookeeper, and that the controller
> > reported
> > > > that
> > > > > > the
> > > > > > >> > delete was successful, it's just something seems to have
> > gotten
> > > > out
> > > > > of
> > > > > > >> > sync.
> > > > > > >> >
> > > > > > >> > delete.topic.enabled is true.  I've successfully deleted
> > topics
> > > in
> > > > > the
> > > > > > >> > past, so I know it *should* work. :)
> > > > > > >> >
> > > > > > >> > I also had already checked in Zookeeper, and there is no
> > > directory
> > > > > for
> > > > > > >> the
> > > > > > >> > topics under /brokers/topics....  Very strange indeed.
> > > > > > >> >
> > > > > > >> > If I just remove the log directories from the filesystem, is
> > > that
> > > > > > >> enough to
> > > > > > >> > get the broker to stop asking about the topics?  I would
> guess
> > > > there
> > > > > > >> would
> > > > > > >> > need to be more than just that, but I could be wrong.
> > > > > > >> >
> > > > > > >> > Thanks guys for the suggestions though!
> > > > > > >> >
> > > > > > >> > On Thu, Jul 20, 2017 at 8:19 PM, Stephen Powis <
> > > > > spowis@salesforce.com
> > > > > > >
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > I could be totally wrong, but I seem to recall that delete
> > > > wasn't
> > > > > > >> fully
> > > > > > >> > > implemented in 0.8.x?
> > > > > > >> > >
> > > > > > >> > > On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd
> > > > > > >> > <chaferd@groupon.com.invalid
> > > > > > >> > > >
> > > > > > >> > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Chris,
> > > > > > >> > > >
> > > > > > >> > > > You could first check to make sure that
> > delete.topic.enable
> > > is
> > > > > > true
> > > > > > >> and
> > > > > > >> > > try
> > > > > > >> > > > deleting again if not.  If that doesn't work with
> 0.8.1.1
> > > you
> > > > > > might
> > > > > > >> > need
> > > > > > >> > > to
> > > > > > >> > > > manually remove the topic's log files from the
> configured
> > > > > log.dirs
> > > > > > >> > folder
> > > > > > >> > > > on each broker in addition to removing the topic's
> > zookeeper
> > > > > path.
> > > > > > >> > > >
> > > > > > >> > > > Carl
> > > > > > >> > > >
> > > > > > >> > > > On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal <
> > > > cwneal@gmail.com>
> > > > > > >> wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Hi all,
> > > > > > >> > > > >
> > > > > > >> > > > > I have a weird situation here.  I have deleted a few
> > > topics
> > > > on
> > > > > > my
> > > > > > >> > > 0.8.1.1
> > > > > > >> > > > > cluster (old, I know...).  The deletes succeeded
> > according
> > > > to
> > > > > > the
> > > > > > >> > > > > controller.log:
> > > > > > >> > > > >
> > > > > > >> > > > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on
> > > > > > Controller
> > > > > > >> 1]:
> > > > > > >> > > New
> > > > > > >> > > > > topics: [Set()], deleted topics:
> > > > > > >> > > > > [Set(perf_doorway-supplier-adapter-uat_raw)], new
> > > partition
> > > > > > >> replica
> > > > > > >> > > > > assignment [Map()]
> > > > > > >> > > > > (kafka.controller.PartitionStateMachine$
> > > > TopicChangeListener)
> > > > > > >> > > > > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on
> > > > > > Controller
> > > > > > >> 1]:
> > > > > > >> > > New
> > > > > > >> > > > > topics: [Set()], deleted topics:
> > > > > > >> > > > > [Set(perf_doorway-supplier-scheduler-uat_raw)], new
> > > > partition
> > > > > > >> > replica
> > > > > > >> > > > > assignment [Map()]
> > > > > > >> > > > > (kafka.controller.PartitionStateMachine$
> > > > TopicChangeListener)
> > > > > > >> > > > > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on
> > > > > > Controller
> > > > > > >> 1]:
> > > > > > >> > > New
> > > > > > >> > > > > topics: [Set()], deleted topics:
> > > > > [Set(perf_gocontent-uat_raw)],
> > > > > > >> new
> > > > > > >> > > > > partition replica assignment [Map()]
> > > > > > >> > > > > (kafka.controller.PartitionStateMachine$
> > > > TopicChangeListener)
> > > > > > >> > > > > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on
> > > > > > Controller
> > > > > > >> 1]:
> > > > > > >> > > New
> > > > > > >> > > > > topics: [Set()], deleted topics:
> > > > > [Set(perf_goplatform-uat_raw)]
> > > > > > ,
> > > > > > >> new
> > > > > > >> > > > > partition replica assignment [Map()]
> > > > > > >> > > > > (kafka.controller.PartitionStateMachine$
> > > > TopicChangeListener)
> > > > > > >> > > > >
> > > > > > >> > > > > I query Zookeeper and the path is not there under
> > > > > > /brokers/topics
> > > > > > >> as
> > > > > > >> > > > well.
> > > > > > >> > > > >
> > > > > > >> > > > > But, one of the nodes in my cluster continues to try
> and
> > > use
> > > > > > them:
> > > > > > >> > > > >
> > > > > > >> > > > > [2017-07-20 17:04:36,723] ERROR Conditional update of
> > path
> > > > > > >> > > > > /brokers/topics/perf_doorway-
> > supplier-scheduler-uat_raw/
> > > > > > >> > > > partitions/3/state
> > > > > > >> > > > > with data
> > > > > > >> > > > > {"controller_epoch":34,"
> leader":1,"version":1,"leader_
> > > > > > >> > > > > epoch":2,"isr":[1,0]}
> > > > > > >> > > > > and expected version 69 failed due to
> > > > > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > > > > >> > KeeperErrorCode
> > > > > > >> > > =
> > > > > > >> > > > > NoNode for
> > > > > > >> > > > > /brokers/topics/perf_doorway-
> > supplier-scheduler-uat_raw/
> > > > > > >> > > > partitions/3/state
> > > > > > >> > > > > (kafka.utils.ZkUtils$)
> > > > > > >> > > > > [2017-07-20 17:04:36,723] INFO Partition
> > > > > > >> > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker
> > 1:
> > > > > Cached
> > > > > > >> > > > zkVersion
> > > > > > >> > > > > [69] not equal to that in zookeeper, skip updating ISR
> > > > > > >> > > > > (kafka.cluster.Partition)
> > > > > > >> > > > > [2017-07-20 17:04:36,723] INFO Partition
> > > > > > >> > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker
> > 1:
> > > > > Cached
> > > > > > >> > > > zkVersion
> > > > > > >> > > > > [69] not equal to that in zookeeper, skip updating ISR
> > > > > > >> > > > > (kafka.cluster.Partition)
> > > > > > >> > > > > [2017-07-20 17:04:36,764] INFO Partition
> > > > > > >> [perf_goplatform-uat_raw,2]
> > > > > > >> > on
> > > > > > >> > > > > broker 1: Shrinking ISR for partition
> > > > > > [perf_goplatform-uat_raw,2]
> > > > > > >> > from
> > > > > > >> > > > 1,0
> > > > > > >> > > > > to 1 (kafka.cluster.Partition)
> > > > > > >> > > > > [2017-07-20 17:04:36,764] INFO Partition
> > > > > > >> [perf_goplatform-uat_raw,2]
> > > > > > >> > on
> > > > > > >> > > > > broker 1: Shrinking ISR for partition
> > > > > > [perf_goplatform-uat_raw,2]
> > > > > > >> > from
> > > > > > >> > > > 1,0
> > > > > > >> > > > > to 1 (kafka.cluster.Partition)
> > > > > > >> > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of
> > path
> > > > > > >> > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/
> > > 2/state
> > > > > with
> > > > > > >> data
> > > > > > >> > > > > {"controller_epoch":34,"
> leader":1,"version":1,"leader_
> > > > > > >> > > > epoch":2,"isr":[1]}
> > > > > > >> > > > > and expected version 70 failed due to
> > > > > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > > > > >> > KeeperErrorCode
> > > > > > >> > > =
> > > > > > >> > > > > NoNode for /brokers/topics/perf_
> > > > > goplatform-uat_raw/partitions/
> > > > > > >> > 2/state
> > > > > > >> > > > > (kafka.utils.ZkUtils$)
> > > > > > >> > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of
> > path
> > > > > > >> > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/
> > > 2/state
> > > > > with
> > > > > > >> data
> > > > > > >> > > > > {"controller_epoch":34,"
> leader":1,"version":1,"leader_
> > > > > > >> > > > epoch":2,"isr":[1]}
> > > > > > >> > > > > and expected version 70 failed due to
> > > > > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > > > > >> > KeeperErrorCode
> > > > > > >> > > =
> > > > > > >> > > > > NoNode for /brokers/topics/perf_
> > > > > goplatform-uat_raw/partitions/
> > > > > > >> > 2/state
> > > > > > >> > > > > (kafka.utils.ZkUtils$)
> > > > > > >> > > > > [2017-07-20 17:04:36,765] INFO Partition
> > > > > > >> [perf_goplatform-uat_raw,2]
> > > > > > >> > on
> > > > > > >> > > > > broker 1: Cached zkVersion [70] not equal to that in
> > > > > zookeeper,
> > > > > > >> skip
> > > > > > >> > > > > updating ISR (kafka.cluster.Partition)
> > > > > > >> > > > > [2017-07-20 17:04:36,765] INFO Partition
> > > > > > >> [perf_goplatform-uat_raw,2]
> > > > > > >> > on
> > > > > > >> > > > > broker 1: Cached zkVersion [70] not equal to that in
> > > > > zookeeper,
> > > > > > >> skip
> > > > > > >> > > > > updating ISR (kafka.cluster.Partition)
> > > > > > >> > > > > [2017-07-20 17:04:36,981] INFO Partition
> > > > > > >> [perf_gocontent-uat_raw,1]
> > > > > > >> > on
> > > > > > >> > > > > broker 1: Shrinking ISR for partition
> > > > > [perf_gocontent-uat_raw,1]
> > > > > > >> from
> > > > > > >> > > 1,0
> > > > > > >> > > > > to 1 (kafka.cluster.Partition)
> > > > > > >> > > > > [2017-07-20 17:04:36,981] INFO Partition
> > > > > > >> [perf_gocontent-uat_raw,1]
> > > > > > >> > on
> > > > > > >> > > > > broker 1: Shrinking ISR for partition
> > > > > [perf_gocontent-uat_raw,1]
> > > > > > >> from
> > > > > > >> > > 1,0
> > > > > > >> > > > > to 1 (kafka.cluster.Partition)
> > > > > > >> > > > > [2017-07-20 17:04:36,988] ERROR Conditional update of
> > path
> > > > > > >> > > > > /brokers/topics/perf_gocontent-uat_raw/partitions/
> > 1/state
> > > > > with
> > > > > > >> data
> > > > > > >> > > > > {"controller_epoch":34,"
> leader":1,"version":1,"leader_
> > > > > > >> > > > epoch":4,"isr":[1]}
> > > > > > >> > > > > and expected version 90 failed due to
> > > > > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > > > > >> > KeeperErrorCode
> > > > > > >> > > =
> > > > > > >> > > > > NoNode for /brokers/topics/perf_gocontent
> > > > > > >> -uat_raw/partitions/1/state
> > > > > > >> > > > > (kafka.utils.ZkUtils$)
> > > > > > >> > > > > [2017-07-20 17:04:36,988] ERROR Conditional update of
> > path
> > > > > > >> > > > > /brokers/topics/perf_gocontent-uat_raw/partitions/
> > 1/state
> > > > > with
> > > > > > >> data
> > > > > > >> > > > > {"controller_epoch":34,"
> leader":1,"version":1,"leader_
> > > > > > >> > > > epoch":4,"isr":[1]}
> > > > > > >> > > > > and expected version 90 failed due to
> > > > > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > > > > >> > KeeperErrorCode
> > > > > > >> > > =
> > > > > > >> > > > > NoNode for /brokers/topics/perf_gocontent
> > > > > > >> -uat_raw/partitions/1/state
> > > > > > >> > > > > (kafka.utils.ZkUtils$)
> > > > > > >> > > > > [2017-07-20 17:04:36,988] INFO Partition
> > > > > > >> [perf_gocontent-uat_raw,1]
> > > > > > >> > on
> > > > > > >> > > > > broker 1: Cached zkVersion [90] not equal to that in
> > > > > zookeeper,
> > > > > > >> skip
> > > > > > >> > > > > updating ISR (kafka.cluster.Partition)
> > > > > > >> > > > > [2017-07-20 17:04:36,988] INFO Partition
> > > > > > >> [perf_gocontent-uat_raw,1]
> > > > > > >> > on
> > > > > > >> > > > > broker 1: Cached zkVersion [90] not equal to that in
> > > > > zookeeper,
> > > > > > >> skip
> > > > > > >> > > > > updating ISR (kafka.cluster.Partition)
> > > > > > >> > > > >
> > > > > > >> > > > > I've tried a rolling restart of the cluster to see if
> > that
> > > > > fixed
> > > > > > >> it,
> > > > > > >> > > but
> > > > > > >> > > > it
> > > > > > >> > > > > did not.
> > > > > > >> > > > >
> > > > > > >> > > > > Can someone please help me out here?  I'm not sure
> how I
> > > can
> > > > > get
> > > > > > >> > things
> > > > > > >> > > > > back in sync.
> > > > > > >> > > > >
> > > > > > >> > > > > Thank you so much for your time.
> > > > > > >> > > > > Chris
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

[Carl Haferd <ch...@groupon.com.INVALID>]

I would recommend allowing each broker enough time to catch-up before
starting the next, but this may be less of a concern if the entire cluster
is being brought down and then started from scratch.  To automate, we poll
until the Kafka process binds to its configured port (9092), and then once
all brokers are up we also poll until there are no longer any
under-replicated partitions before performing further maintenance.

Carl

On Fri, Jul 21, 2017 at 10:50 AM, Chris Neal <cw...@gmail.com> wrote:

> Thanks Carl.
>
> Always fun to do this stuff in production... ;)
>
> Appreciate the input.  I'll try a full cycle and see how that works.
>
> In your opinion, if I stop all brokers and all Zookeeper nodes, then
> restart all Zookeepers...at that point can I start both brokers at the same
> time, or should I let one broker fully start and read all the unflushed
> segments from disk before starting the second broker?
>
> Again, many thanks.
> Chris
>
>
> On Fri, Jul 21, 2017 at 12:13 PM, Carl Haferd <chaferd@groupon.com.invalid
> >
> wrote:
>
> > I have encountered similar difficulties in a test environment and it may
> be
> > necessary to stop the Kafka process on each broker and take Zookeeper
> > offline before removing the files and zookeeper paths.  Otherwise there
> may
> > be a race condition between brokers which could cause the cluster to
> retain
> > information for the topic.
> >
> > Carl
> >
> > On Fri, Jul 21, 2017 at 9:06 AM, Chris Neal <cw...@gmail.com> wrote:
> >
> > > Welp.  Surprisingly, that did not fix the problem. :(
> > >
> > > I cleaned out all the entries for these topics from /config/topics, and
> > > removed the logs from the file system for those topics, and the
> messages
> > > are still flying by in the server.log file.
> > >
> > > Also, more concerning, when I was looking through the log files for the
> > > other broker in the cluster, I noticed the same type of message for a
> > topic
> > > that should actually be there:
> > >
> > > [2017-07-21 16:03:29,140] ERROR Conditional update of path
> > > /brokers/topics/perf_dstorage_raw/partitions/4/state with data
> > > {"controller_epoch":34,"leader":0,"version":1,"leader_
> > epoch":0,"isr":[0]}
> > > and expected version 0 failed due to
> > > org.apache.zookeeper.KeeperException$BadVersionException:
> > KeeperErrorCode
> > > =
> > > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/4/state
> > > (kafka.utils.ZkUtils$)
> > > [2017-07-21 16:03:29,142] ERROR Conditional update of path
> > > /brokers/topics/perf_dstorage_raw/partitions/0/state with data
> > > {"controller_epoch":34,"leader":0,"version":1,"leader_
> > epoch":0,"isr":[0]}
> > > and expected version 0 failed due to
> > > org.apache.zookeeper.KeeperException$BadVersionException:
> > KeeperErrorCode
> > > =
> > > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state
> > > (kafka.utils.ZkUtils$)
> > > [2017-07-21 16:03:29,142] ERROR Conditional update of path
> > > /brokers/topics/perf_dstorage_raw/partitions/0/state with data
> > > {"controller_epoch":34,"leader":0,"version":1,"leader_
> > epoch":0,"isr":[0]}
> > > and expected version 0 failed due to
> > > org.apache.zookeeper.KeeperException$BadVersionException:
> > KeeperErrorCode
> > > =
> > > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state
> > > (kafka.utils.ZkUtils$)
> > >
> > > So, the issue is not isolated to just these "should-have-been-removed"
> > > topics, unfortunately.
> > >
> > > Really appreciate the input so far everyone.  Still looking though for
> a
> > > solution.  Many thanks. :)
> > >
> > > Chris
> > >
> > > On Fri, Jul 21, 2017 at 10:58 AM, M. Manna <ma...@gmail.com> wrote:
> > >
> > > > Just to add (in case the platoform is Windows)
> > > >
> > > > For Windows based cluster implementation, log/topic cleanup doesn't
> > work
> > > > out of the box. Users are more or less aware of it, and doing their
> own
> > > > maintenance as workaround.
> > > >  If you have issues on Topic deletion not working properly on Windows
> > > (i.e.
> > > > with topic deletion enable and all other settings). then you have to
> > > > manually delete the files.
> > > >
> > > >
> > > > On 21 July 2017 at 16:53, Chris Neal <cw...@gmail.com> wrote:
> > > >
> > > > > @Carl,
> > > > >
> > > > > There is nothing under /admin/delete_topics other than
> > > > >
> > > > > []
> > > > >
> > > > > And nothing under /admin other than delete_topics :)
> > > > >
> > > > > The topics DO exist, however, under /config/topics!  We may be on
> to
> > > > > something.  I will remove them here and see if that clears it up.
> > > > >
> > > > > Thanks so much for all the help!
> > > > > Chris
> > > > >
> > > > > On Thu, Jul 20, 2017 at 10:37 PM, Chris Neal <cw...@gmail.com>
> > wrote:
> > > > >
> > > > > > Thanks again for the replies.  VERY much appreciated.  I'll check
> > > both
> > > > > > /admin/delete_topics and /config/topics.
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Thu, Jul 20, 2017 at 9:22 PM, Carl Haferd
> > > > <chaferd@groupon.com.invalid
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> If delete normally works, there would hopefully be some log
> > entries
> > > > when
> > > > > >> it
> > > > > >> fails.  Are there any unusual zookeeper entries in the
> > > > > >> /admin/delete_topics
> > > > > >> path or in the other /admin folders?
> > > > > >>
> > > > > >> Does the topic name still exist in zookeeper under
> /config/topics?
> > > If
> > > > > so,
> > > > > >> that should probably deleted as well.
> > > > > >>
> > > > > >> Carl
> > > > > >>
> > > > > >> On Thu, Jul 20, 2017 at 6:42 PM, Chris Neal <cw...@gmail.com>
> > > wrote:
> > > > > >>
> > > > > >> > Delete is definitely there.  The delete worked fine, based on
> > the
> > > > fact
> > > > > >> that
> > > > > >> > there is nothing in Zookeeper, and that the controller
> reported
> > > that
> > > > > the
> > > > > >> > delete was successful, it's just something seems to have
> gotten
> > > out
> > > > of
> > > > > >> > sync.
> > > > > >> >
> > > > > >> > delete.topic.enabled is true.  I've successfully deleted
> topics
> > in
> > > > the
> > > > > >> > past, so I know it *should* work. :)
> > > > > >> >
> > > > > >> > I also had already checked in Zookeeper, and there is no
> > directory
> > > > for
> > > > > >> the
> > > > > >> > topics under /brokers/topics....  Very strange indeed.
> > > > > >> >
> > > > > >> > If I just remove the log directories from the filesystem, is
> > that
> > > > > >> enough to
> > > > > >> > get the broker to stop asking about the topics?  I would guess
> > > there
> > > > > >> would
> > > > > >> > need to be more than just that, but I could be wrong.
> > > > > >> >
> > > > > >> > Thanks guys for the suggestions though!
> > > > > >> >
> > > > > >> > On Thu, Jul 20, 2017 at 8:19 PM, Stephen Powis <
> > > > spowis@salesforce.com
> > > > > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > I could be totally wrong, but I seem to recall that delete
> > > wasn't
> > > > > >> fully
> > > > > >> > > implemented in 0.8.x?
> > > > > >> > >
> > > > > >> > > On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd
> > > > > >> > <chaferd@groupon.com.invalid
> > > > > >> > > >
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > Chris,
> > > > > >> > > >
> > > > > >> > > > You could first check to make sure that
> delete.topic.enable
> > is
> > > > > true
> > > > > >> and
> > > > > >> > > try
> > > > > >> > > > deleting again if not.  If that doesn't work with 0.8.1.1
> > you
> > > > > might
> > > > > >> > need
> > > > > >> > > to
> > > > > >> > > > manually remove the topic's log files from the configured
> > > > log.dirs
> > > > > >> > folder
> > > > > >> > > > on each broker in addition to removing the topic's
> zookeeper
> > > > path.
> > > > > >> > > >
> > > > > >> > > > Carl
> > > > > >> > > >
> > > > > >> > > > On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal <
> > > cwneal@gmail.com>
> > > > > >> wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi all,
> > > > > >> > > > >
> > > > > >> > > > > I have a weird situation here.  I have deleted a few
> > topics
> > > on
> > > > > my
> > > > > >> > > 0.8.1.1
> > > > > >> > > > > cluster (old, I know...).  The deletes succeeded
> according
> > > to
> > > > > the
> > > > > >> > > > > controller.log:
> > > > > >> > > > >
> > > > > >> > > > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on
> > > > > Controller
> > > > > >> 1]:
> > > > > >> > > New
> > > > > >> > > > > topics: [Set()], deleted topics:
> > > > > >> > > > > [Set(perf_doorway-supplier-adapter-uat_raw)], new
> > partition
> > > > > >> replica
> > > > > >> > > > > assignment [Map()]
> > > > > >> > > > > (kafka.controller.PartitionStateMachine$
> > > TopicChangeListener)
> > > > > >> > > > > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on
> > > > > Controller
> > > > > >> 1]:
> > > > > >> > > New
> > > > > >> > > > > topics: [Set()], deleted topics:
> > > > > >> > > > > [Set(perf_doorway-supplier-scheduler-uat_raw)], new
> > > partition
> > > > > >> > replica
> > > > > >> > > > > assignment [Map()]
> > > > > >> > > > > (kafka.controller.PartitionStateMachine$
> > > TopicChangeListener)
> > > > > >> > > > > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on
> > > > > Controller
> > > > > >> 1]:
> > > > > >> > > New
> > > > > >> > > > > topics: [Set()], deleted topics:
> > > > [Set(perf_gocontent-uat_raw)],
> > > > > >> new
> > > > > >> > > > > partition replica assignment [Map()]
> > > > > >> > > > > (kafka.controller.PartitionStateMachine$
> > > TopicChangeListener)
> > > > > >> > > > > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on
> > > > > Controller
> > > > > >> 1]:
> > > > > >> > > New
> > > > > >> > > > > topics: [Set()], deleted topics:
> > > > [Set(perf_goplatform-uat_raw)]
> > > > > ,
> > > > > >> new
> > > > > >> > > > > partition replica assignment [Map()]
> > > > > >> > > > > (kafka.controller.PartitionStateMachine$
> > > TopicChangeListener)
> > > > > >> > > > >
> > > > > >> > > > > I query Zookeeper and the path is not there under
> > > > > /brokers/topics
> > > > > >> as
> > > > > >> > > > well.
> > > > > >> > > > >
> > > > > >> > > > > But, one of the nodes in my cluster continues to try and
> > use
> > > > > them:
> > > > > >> > > > >
> > > > > >> > > > > [2017-07-20 17:04:36,723] ERROR Conditional update of
> path
> > > > > >> > > > > /brokers/topics/perf_doorway-
> supplier-scheduler-uat_raw/
> > > > > >> > > > partitions/3/state
> > > > > >> > > > > with data
> > > > > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > > > >> > > > > epoch":2,"isr":[1,0]}
> > > > > >> > > > > and expected version 69 failed due to
> > > > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > > > >> > KeeperErrorCode
> > > > > >> > > =
> > > > > >> > > > > NoNode for
> > > > > >> > > > > /brokers/topics/perf_doorway-
> supplier-scheduler-uat_raw/
> > > > > >> > > > partitions/3/state
> > > > > >> > > > > (kafka.utils.ZkUtils$)
> > > > > >> > > > > [2017-07-20 17:04:36,723] INFO Partition
> > > > > >> > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker
> 1:
> > > > Cached
> > > > > >> > > > zkVersion
> > > > > >> > > > > [69] not equal to that in zookeeper, skip updating ISR
> > > > > >> > > > > (kafka.cluster.Partition)
> > > > > >> > > > > [2017-07-20 17:04:36,723] INFO Partition
> > > > > >> > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker
> 1:
> > > > Cached
> > > > > >> > > > zkVersion
> > > > > >> > > > > [69] not equal to that in zookeeper, skip updating ISR
> > > > > >> > > > > (kafka.cluster.Partition)
> > > > > >> > > > > [2017-07-20 17:04:36,764] INFO Partition
> > > > > >> [perf_goplatform-uat_raw,2]
> > > > > >> > on
> > > > > >> > > > > broker 1: Shrinking ISR for partition
> > > > > [perf_goplatform-uat_raw,2]
> > > > > >> > from
> > > > > >> > > > 1,0
> > > > > >> > > > > to 1 (kafka.cluster.Partition)
> > > > > >> > > > > [2017-07-20 17:04:36,764] INFO Partition
> > > > > >> [perf_goplatform-uat_raw,2]
> > > > > >> > on
> > > > > >> > > > > broker 1: Shrinking ISR for partition
> > > > > [perf_goplatform-uat_raw,2]
> > > > > >> > from
> > > > > >> > > > 1,0
> > > > > >> > > > > to 1 (kafka.cluster.Partition)
> > > > > >> > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of
> path
> > > > > >> > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/
> > 2/state
> > > > with
> > > > > >> data
> > > > > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > > > >> > > > epoch":2,"isr":[1]}
> > > > > >> > > > > and expected version 70 failed due to
> > > > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > > > >> > KeeperErrorCode
> > > > > >> > > =
> > > > > >> > > > > NoNode for /brokers/topics/perf_
> > > > goplatform-uat_raw/partitions/
> > > > > >> > 2/state
> > > > > >> > > > > (kafka.utils.ZkUtils$)
> > > > > >> > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of
> path
> > > > > >> > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/
> > 2/state
> > > > with
> > > > > >> data
> > > > > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > > > >> > > > epoch":2,"isr":[1]}
> > > > > >> > > > > and expected version 70 failed due to
> > > > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > > > >> > KeeperErrorCode
> > > > > >> > > =
> > > > > >> > > > > NoNode for /brokers/topics/perf_
> > > > goplatform-uat_raw/partitions/
> > > > > >> > 2/state
> > > > > >> > > > > (kafka.utils.ZkUtils$)
> > > > > >> > > > > [2017-07-20 17:04:36,765] INFO Partition
> > > > > >> [perf_goplatform-uat_raw,2]
> > > > > >> > on
> > > > > >> > > > > broker 1: Cached zkVersion [70] not equal to that in
> > > > zookeeper,
> > > > > >> skip
> > > > > >> > > > > updating ISR (kafka.cluster.Partition)
> > > > > >> > > > > [2017-07-20 17:04:36,765] INFO Partition
> > > > > >> [perf_goplatform-uat_raw,2]
> > > > > >> > on
> > > > > >> > > > > broker 1: Cached zkVersion [70] not equal to that in
> > > > zookeeper,
> > > > > >> skip
> > > > > >> > > > > updating ISR (kafka.cluster.Partition)
> > > > > >> > > > > [2017-07-20 17:04:36,981] INFO Partition
> > > > > >> [perf_gocontent-uat_raw,1]
> > > > > >> > on
> > > > > >> > > > > broker 1: Shrinking ISR for partition
> > > > [perf_gocontent-uat_raw,1]
> > > > > >> from
> > > > > >> > > 1,0
> > > > > >> > > > > to 1 (kafka.cluster.Partition)
> > > > > >> > > > > [2017-07-20 17:04:36,981] INFO Partition
> > > > > >> [perf_gocontent-uat_raw,1]
> > > > > >> > on
> > > > > >> > > > > broker 1: Shrinking ISR for partition
> > > > [perf_gocontent-uat_raw,1]
> > > > > >> from
> > > > > >> > > 1,0
> > > > > >> > > > > to 1 (kafka.cluster.Partition)
> > > > > >> > > > > [2017-07-20 17:04:36,988] ERROR Conditional update of
> path
> > > > > >> > > > > /brokers/topics/perf_gocontent-uat_raw/partitions/
> 1/state
> > > > with
> > > > > >> data
> > > > > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > > > >> > > > epoch":4,"isr":[1]}
> > > > > >> > > > > and expected version 90 failed due to
> > > > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > > > >> > KeeperErrorCode
> > > > > >> > > =
> > > > > >> > > > > NoNode for /brokers/topics/perf_gocontent
> > > > > >> -uat_raw/partitions/1/state
> > > > > >> > > > > (kafka.utils.ZkUtils$)
> > > > > >> > > > > [2017-07-20 17:04:36,988] ERROR Conditional update of
> path
> > > > > >> > > > > /brokers/topics/perf_gocontent-uat_raw/partitions/
> 1/state
> > > > with
> > > > > >> data
> > > > > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > > > >> > > > epoch":4,"isr":[1]}
> > > > > >> > > > > and expected version 90 failed due to
> > > > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > > > >> > KeeperErrorCode
> > > > > >> > > =
> > > > > >> > > > > NoNode for /brokers/topics/perf_gocontent
> > > > > >> -uat_raw/partitions/1/state
> > > > > >> > > > > (kafka.utils.ZkUtils$)
> > > > > >> > > > > [2017-07-20 17:04:36,988] INFO Partition
> > > > > >> [perf_gocontent-uat_raw,1]
> > > > > >> > on
> > > > > >> > > > > broker 1: Cached zkVersion [90] not equal to that in
> > > > zookeeper,
> > > > > >> skip
> > > > > >> > > > > updating ISR (kafka.cluster.Partition)
> > > > > >> > > > > [2017-07-20 17:04:36,988] INFO Partition
> > > > > >> [perf_gocontent-uat_raw,1]
> > > > > >> > on
> > > > > >> > > > > broker 1: Cached zkVersion [90] not equal to that in
> > > > zookeeper,
> > > > > >> skip
> > > > > >> > > > > updating ISR (kafka.cluster.Partition)
> > > > > >> > > > >
> > > > > >> > > > > I've tried a rolling restart of the cluster to see if
> that
> > > > fixed
> > > > > >> it,
> > > > > >> > > but
> > > > > >> > > > it
> > > > > >> > > > > did not.
> > > > > >> > > > >
> > > > > >> > > > > Can someone please help me out here?  I'm not sure how I
> > can
> > > > get
> > > > > >> > things
> > > > > >> > > > > back in sync.
> > > > > >> > > > >
> > > > > >> > > > > Thank you so much for your time.
> > > > > >> > > > > Chris
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

[Chris Neal <cw...@gmail.com>]

Thanks Carl.

Always fun to do this stuff in production... ;)

Appreciate the input.  I'll try a full cycle and see how that works.

In your opinion, if I stop all brokers and all Zookeeper nodes, then
restart all Zookeepers...at that point can I start both brokers at the same
time, or should I let one broker fully start and read all the unflushed
segments from disk before starting the second broker?

Again, many thanks.
Chris


On Fri, Jul 21, 2017 at 12:13 PM, Carl Haferd <ch...@groupon.com.invalid>
wrote:

> I have encountered similar difficulties in a test environment and it may be
> necessary to stop the Kafka process on each broker and take Zookeeper
> offline before removing the files and zookeeper paths.  Otherwise there may
> be a race condition between brokers which could cause the cluster to retain
> information for the topic.
>
> Carl
>
> On Fri, Jul 21, 2017 at 9:06 AM, Chris Neal <cw...@gmail.com> wrote:
>
> > Welp.  Surprisingly, that did not fix the problem. :(
> >
> > I cleaned out all the entries for these topics from /config/topics, and
> > removed the logs from the file system for those topics, and the messages
> > are still flying by in the server.log file.
> >
> > Also, more concerning, when I was looking through the log files for the
> > other broker in the cluster, I noticed the same type of message for a
> topic
> > that should actually be there:
> >
> > [2017-07-21 16:03:29,140] ERROR Conditional update of path
> > /brokers/topics/perf_dstorage_raw/partitions/4/state with data
> > {"controller_epoch":34,"leader":0,"version":1,"leader_
> epoch":0,"isr":[0]}
> > and expected version 0 failed due to
> > org.apache.zookeeper.KeeperException$BadVersionException:
> KeeperErrorCode
> > =
> > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/4/state
> > (kafka.utils.ZkUtils$)
> > [2017-07-21 16:03:29,142] ERROR Conditional update of path
> > /brokers/topics/perf_dstorage_raw/partitions/0/state with data
> > {"controller_epoch":34,"leader":0,"version":1,"leader_
> epoch":0,"isr":[0]}
> > and expected version 0 failed due to
> > org.apache.zookeeper.KeeperException$BadVersionException:
> KeeperErrorCode
> > =
> > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state
> > (kafka.utils.ZkUtils$)
> > [2017-07-21 16:03:29,142] ERROR Conditional update of path
> > /brokers/topics/perf_dstorage_raw/partitions/0/state with data
> > {"controller_epoch":34,"leader":0,"version":1,"leader_
> epoch":0,"isr":[0]}
> > and expected version 0 failed due to
> > org.apache.zookeeper.KeeperException$BadVersionException:
> KeeperErrorCode
> > =
> > BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state
> > (kafka.utils.ZkUtils$)
> >
> > So, the issue is not isolated to just these "should-have-been-removed"
> > topics, unfortunately.
> >
> > Really appreciate the input so far everyone.  Still looking though for a
> > solution.  Many thanks. :)
> >
> > Chris
> >
> > On Fri, Jul 21, 2017 at 10:58 AM, M. Manna <ma...@gmail.com> wrote:
> >
> > > Just to add (in case the platoform is Windows)
> > >
> > > For Windows based cluster implementation, log/topic cleanup doesn't
> work
> > > out of the box. Users are more or less aware of it, and doing their own
> > > maintenance as workaround.
> > >  If you have issues on Topic deletion not working properly on Windows
> > (i.e.
> > > with topic deletion enable and all other settings). then you have to
> > > manually delete the files.
> > >
> > >
> > > On 21 July 2017 at 16:53, Chris Neal <cw...@gmail.com> wrote:
> > >
> > > > @Carl,
> > > >
> > > > There is nothing under /admin/delete_topics other than
> > > >
> > > > []
> > > >
> > > > And nothing under /admin other than delete_topics :)
> > > >
> > > > The topics DO exist, however, under /config/topics!  We may be on to
> > > > something.  I will remove them here and see if that clears it up.
> > > >
> > > > Thanks so much for all the help!
> > > > Chris
> > > >
> > > > On Thu, Jul 20, 2017 at 10:37 PM, Chris Neal <cw...@gmail.com>
> wrote:
> > > >
> > > > > Thanks again for the replies.  VERY much appreciated.  I'll check
> > both
> > > > > /admin/delete_topics and /config/topics.
> > > > >
> > > > > Chris
> > > > >
> > > > > On Thu, Jul 20, 2017 at 9:22 PM, Carl Haferd
> > > <chaferd@groupon.com.invalid
> > > > >
> > > > > wrote:
> > > > >
> > > > >> If delete normally works, there would hopefully be some log
> entries
> > > when
> > > > >> it
> > > > >> fails.  Are there any unusual zookeeper entries in the
> > > > >> /admin/delete_topics
> > > > >> path or in the other /admin folders?
> > > > >>
> > > > >> Does the topic name still exist in zookeeper under /config/topics?
> > If
> > > > so,
> > > > >> that should probably deleted as well.
> > > > >>
> > > > >> Carl
> > > > >>
> > > > >> On Thu, Jul 20, 2017 at 6:42 PM, Chris Neal <cw...@gmail.com>
> > wrote:
> > > > >>
> > > > >> > Delete is definitely there.  The delete worked fine, based on
> the
> > > fact
> > > > >> that
> > > > >> > there is nothing in Zookeeper, and that the controller reported
> > that
> > > > the
> > > > >> > delete was successful, it's just something seems to have gotten
> > out
> > > of
> > > > >> > sync.
> > > > >> >
> > > > >> > delete.topic.enabled is true.  I've successfully deleted topics
> in
> > > the
> > > > >> > past, so I know it *should* work. :)
> > > > >> >
> > > > >> > I also had already checked in Zookeeper, and there is no
> directory
> > > for
> > > > >> the
> > > > >> > topics under /brokers/topics....  Very strange indeed.
> > > > >> >
> > > > >> > If I just remove the log directories from the filesystem, is
> that
> > > > >> enough to
> > > > >> > get the broker to stop asking about the topics?  I would guess
> > there
> > > > >> would
> > > > >> > need to be more than just that, but I could be wrong.
> > > > >> >
> > > > >> > Thanks guys for the suggestions though!
> > > > >> >
> > > > >> > On Thu, Jul 20, 2017 at 8:19 PM, Stephen Powis <
> > > spowis@salesforce.com
> > > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > I could be totally wrong, but I seem to recall that delete
> > wasn't
> > > > >> fully
> > > > >> > > implemented in 0.8.x?
> > > > >> > >
> > > > >> > > On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd
> > > > >> > <chaferd@groupon.com.invalid
> > > > >> > > >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Chris,
> > > > >> > > >
> > > > >> > > > You could first check to make sure that delete.topic.enable
> is
> > > > true
> > > > >> and
> > > > >> > > try
> > > > >> > > > deleting again if not.  If that doesn't work with 0.8.1.1
> you
> > > > might
> > > > >> > need
> > > > >> > > to
> > > > >> > > > manually remove the topic's log files from the configured
> > > log.dirs
> > > > >> > folder
> > > > >> > > > on each broker in addition to removing the topic's zookeeper
> > > path.
> > > > >> > > >
> > > > >> > > > Carl
> > > > >> > > >
> > > > >> > > > On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal <
> > cwneal@gmail.com>
> > > > >> wrote:
> > > > >> > > >
> > > > >> > > > > Hi all,
> > > > >> > > > >
> > > > >> > > > > I have a weird situation here.  I have deleted a few
> topics
> > on
> > > > my
> > > > >> > > 0.8.1.1
> > > > >> > > > > cluster (old, I know...).  The deletes succeeded according
> > to
> > > > the
> > > > >> > > > > controller.log:
> > > > >> > > > >
> > > > >> > > > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on
> > > > Controller
> > > > >> 1]:
> > > > >> > > New
> > > > >> > > > > topics: [Set()], deleted topics:
> > > > >> > > > > [Set(perf_doorway-supplier-adapter-uat_raw)], new
> partition
> > > > >> replica
> > > > >> > > > > assignment [Map()]
> > > > >> > > > > (kafka.controller.PartitionStateMachine$
> > TopicChangeListener)
> > > > >> > > > > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on
> > > > Controller
> > > > >> 1]:
> > > > >> > > New
> > > > >> > > > > topics: [Set()], deleted topics:
> > > > >> > > > > [Set(perf_doorway-supplier-scheduler-uat_raw)], new
> > partition
> > > > >> > replica
> > > > >> > > > > assignment [Map()]
> > > > >> > > > > (kafka.controller.PartitionStateMachine$
> > TopicChangeListener)
> > > > >> > > > > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on
> > > > Controller
> > > > >> 1]:
> > > > >> > > New
> > > > >> > > > > topics: [Set()], deleted topics:
> > > [Set(perf_gocontent-uat_raw)],
> > > > >> new
> > > > >> > > > > partition replica assignment [Map()]
> > > > >> > > > > (kafka.controller.PartitionStateMachine$
> > TopicChangeListener)
> > > > >> > > > > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on
> > > > Controller
> > > > >> 1]:
> > > > >> > > New
> > > > >> > > > > topics: [Set()], deleted topics:
> > > [Set(perf_goplatform-uat_raw)]
> > > > ,
> > > > >> new
> > > > >> > > > > partition replica assignment [Map()]
> > > > >> > > > > (kafka.controller.PartitionStateMachine$
> > TopicChangeListener)
> > > > >> > > > >
> > > > >> > > > > I query Zookeeper and the path is not there under
> > > > /brokers/topics
> > > > >> as
> > > > >> > > > well.
> > > > >> > > > >
> > > > >> > > > > But, one of the nodes in my cluster continues to try and
> use
> > > > them:
> > > > >> > > > >
> > > > >> > > > > [2017-07-20 17:04:36,723] ERROR Conditional update of path
> > > > >> > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
> > > > >> > > > partitions/3/state
> > > > >> > > > > with data
> > > > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > > >> > > > > epoch":2,"isr":[1,0]}
> > > > >> > > > > and expected version 69 failed due to
> > > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > > >> > KeeperErrorCode
> > > > >> > > =
> > > > >> > > > > NoNode for
> > > > >> > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
> > > > >> > > > partitions/3/state
> > > > >> > > > > (kafka.utils.ZkUtils$)
> > > > >> > > > > [2017-07-20 17:04:36,723] INFO Partition
> > > > >> > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1:
> > > Cached
> > > > >> > > > zkVersion
> > > > >> > > > > [69] not equal to that in zookeeper, skip updating ISR
> > > > >> > > > > (kafka.cluster.Partition)
> > > > >> > > > > [2017-07-20 17:04:36,723] INFO Partition
> > > > >> > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1:
> > > Cached
> > > > >> > > > zkVersion
> > > > >> > > > > [69] not equal to that in zookeeper, skip updating ISR
> > > > >> > > > > (kafka.cluster.Partition)
> > > > >> > > > > [2017-07-20 17:04:36,764] INFO Partition
> > > > >> [perf_goplatform-uat_raw,2]
> > > > >> > on
> > > > >> > > > > broker 1: Shrinking ISR for partition
> > > > [perf_goplatform-uat_raw,2]
> > > > >> > from
> > > > >> > > > 1,0
> > > > >> > > > > to 1 (kafka.cluster.Partition)
> > > > >> > > > > [2017-07-20 17:04:36,764] INFO Partition
> > > > >> [perf_goplatform-uat_raw,2]
> > > > >> > on
> > > > >> > > > > broker 1: Shrinking ISR for partition
> > > > [perf_goplatform-uat_raw,2]
> > > > >> > from
> > > > >> > > > 1,0
> > > > >> > > > > to 1 (kafka.cluster.Partition)
> > > > >> > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path
> > > > >> > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/
> 2/state
> > > with
> > > > >> data
> > > > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > > >> > > > epoch":2,"isr":[1]}
> > > > >> > > > > and expected version 70 failed due to
> > > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > > >> > KeeperErrorCode
> > > > >> > > =
> > > > >> > > > > NoNode for /brokers/topics/perf_
> > > goplatform-uat_raw/partitions/
> > > > >> > 2/state
> > > > >> > > > > (kafka.utils.ZkUtils$)
> > > > >> > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path
> > > > >> > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/
> 2/state
> > > with
> > > > >> data
> > > > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > > >> > > > epoch":2,"isr":[1]}
> > > > >> > > > > and expected version 70 failed due to
> > > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > > >> > KeeperErrorCode
> > > > >> > > =
> > > > >> > > > > NoNode for /brokers/topics/perf_
> > > goplatform-uat_raw/partitions/
> > > > >> > 2/state
> > > > >> > > > > (kafka.utils.ZkUtils$)
> > > > >> > > > > [2017-07-20 17:04:36,765] INFO Partition
> > > > >> [perf_goplatform-uat_raw,2]
> > > > >> > on
> > > > >> > > > > broker 1: Cached zkVersion [70] not equal to that in
> > > zookeeper,
> > > > >> skip
> > > > >> > > > > updating ISR (kafka.cluster.Partition)
> > > > >> > > > > [2017-07-20 17:04:36,765] INFO Partition
> > > > >> [perf_goplatform-uat_raw,2]
> > > > >> > on
> > > > >> > > > > broker 1: Cached zkVersion [70] not equal to that in
> > > zookeeper,
> > > > >> skip
> > > > >> > > > > updating ISR (kafka.cluster.Partition)
> > > > >> > > > > [2017-07-20 17:04:36,981] INFO Partition
> > > > >> [perf_gocontent-uat_raw,1]
> > > > >> > on
> > > > >> > > > > broker 1: Shrinking ISR for partition
> > > [perf_gocontent-uat_raw,1]
> > > > >> from
> > > > >> > > 1,0
> > > > >> > > > > to 1 (kafka.cluster.Partition)
> > > > >> > > > > [2017-07-20 17:04:36,981] INFO Partition
> > > > >> [perf_gocontent-uat_raw,1]
> > > > >> > on
> > > > >> > > > > broker 1: Shrinking ISR for partition
> > > [perf_gocontent-uat_raw,1]
> > > > >> from
> > > > >> > > 1,0
> > > > >> > > > > to 1 (kafka.cluster.Partition)
> > > > >> > > > > [2017-07-20 17:04:36,988] ERROR Conditional update of path
> > > > >> > > > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
> > > with
> > > > >> data
> > > > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > > >> > > > epoch":4,"isr":[1]}
> > > > >> > > > > and expected version 90 failed due to
> > > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > > >> > KeeperErrorCode
> > > > >> > > =
> > > > >> > > > > NoNode for /brokers/topics/perf_gocontent
> > > > >> -uat_raw/partitions/1/state
> > > > >> > > > > (kafka.utils.ZkUtils$)
> > > > >> > > > > [2017-07-20 17:04:36,988] ERROR Conditional update of path
> > > > >> > > > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
> > > with
> > > > >> data
> > > > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > > >> > > > epoch":4,"isr":[1]}
> > > > >> > > > > and expected version 90 failed due to
> > > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > > >> > KeeperErrorCode
> > > > >> > > =
> > > > >> > > > > NoNode for /brokers/topics/perf_gocontent
> > > > >> -uat_raw/partitions/1/state
> > > > >> > > > > (kafka.utils.ZkUtils$)
> > > > >> > > > > [2017-07-20 17:04:36,988] INFO Partition
> > > > >> [perf_gocontent-uat_raw,1]
> > > > >> > on
> > > > >> > > > > broker 1: Cached zkVersion [90] not equal to that in
> > > zookeeper,
> > > > >> skip
> > > > >> > > > > updating ISR (kafka.cluster.Partition)
> > > > >> > > > > [2017-07-20 17:04:36,988] INFO Partition
> > > > >> [perf_gocontent-uat_raw,1]
> > > > >> > on
> > > > >> > > > > broker 1: Cached zkVersion [90] not equal to that in
> > > zookeeper,
> > > > >> skip
> > > > >> > > > > updating ISR (kafka.cluster.Partition)
> > > > >> > > > >
> > > > >> > > > > I've tried a rolling restart of the cluster to see if that
> > > fixed
> > > > >> it,
> > > > >> > > but
> > > > >> > > > it
> > > > >> > > > > did not.
> > > > >> > > > >
> > > > >> > > > > Can someone please help me out here?  I'm not sure how I
> can
> > > get
> > > > >> > things
> > > > >> > > > > back in sync.
> > > > >> > > > >
> > > > >> > > > > Thank you so much for your time.
> > > > >> > > > > Chris
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

[Carl Haferd <ch...@groupon.com.INVALID>]

I have encountered similar difficulties in a test environment and it may be
necessary to stop the Kafka process on each broker and take Zookeeper
offline before removing the files and zookeeper paths.  Otherwise there may
be a race condition between brokers which could cause the cluster to retain
information for the topic.

Carl

On Fri, Jul 21, 2017 at 9:06 AM, Chris Neal <cw...@gmail.com> wrote:

> Welp.  Surprisingly, that did not fix the problem. :(
>
> I cleaned out all the entries for these topics from /config/topics, and
> removed the logs from the file system for those topics, and the messages
> are still flying by in the server.log file.
>
> Also, more concerning, when I was looking through the log files for the
> other broker in the cluster, I noticed the same type of message for a topic
> that should actually be there:
>
> [2017-07-21 16:03:29,140] ERROR Conditional update of path
> /brokers/topics/perf_dstorage_raw/partitions/4/state with data
> {"controller_epoch":34,"leader":0,"version":1,"leader_epoch":0,"isr":[0]}
> and expected version 0 failed due to
> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode
> =
> BadVersion for /brokers/topics/perf_dstorage_raw/partitions/4/state
> (kafka.utils.ZkUtils$)
> [2017-07-21 16:03:29,142] ERROR Conditional update of path
> /brokers/topics/perf_dstorage_raw/partitions/0/state with data
> {"controller_epoch":34,"leader":0,"version":1,"leader_epoch":0,"isr":[0]}
> and expected version 0 failed due to
> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode
> =
> BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state
> (kafka.utils.ZkUtils$)
> [2017-07-21 16:03:29,142] ERROR Conditional update of path
> /brokers/topics/perf_dstorage_raw/partitions/0/state with data
> {"controller_epoch":34,"leader":0,"version":1,"leader_epoch":0,"isr":[0]}
> and expected version 0 failed due to
> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode
> =
> BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state
> (kafka.utils.ZkUtils$)
>
> So, the issue is not isolated to just these "should-have-been-removed"
> topics, unfortunately.
>
> Really appreciate the input so far everyone.  Still looking though for a
> solution.  Many thanks. :)
>
> Chris
>
> On Fri, Jul 21, 2017 at 10:58 AM, M. Manna <ma...@gmail.com> wrote:
>
> > Just to add (in case the platoform is Windows)
> >
> > For Windows based cluster implementation, log/topic cleanup doesn't work
> > out of the box. Users are more or less aware of it, and doing their own
> > maintenance as workaround.
> >  If you have issues on Topic deletion not working properly on Windows
> (i.e.
> > with topic deletion enable and all other settings). then you have to
> > manually delete the files.
> >
> >
> > On 21 July 2017 at 16:53, Chris Neal <cw...@gmail.com> wrote:
> >
> > > @Carl,
> > >
> > > There is nothing under /admin/delete_topics other than
> > >
> > > []
> > >
> > > And nothing under /admin other than delete_topics :)
> > >
> > > The topics DO exist, however, under /config/topics!  We may be on to
> > > something.  I will remove them here and see if that clears it up.
> > >
> > > Thanks so much for all the help!
> > > Chris
> > >
> > > On Thu, Jul 20, 2017 at 10:37 PM, Chris Neal <cw...@gmail.com> wrote:
> > >
> > > > Thanks again for the replies.  VERY much appreciated.  I'll check
> both
> > > > /admin/delete_topics and /config/topics.
> > > >
> > > > Chris
> > > >
> > > > On Thu, Jul 20, 2017 at 9:22 PM, Carl Haferd
> > <chaferd@groupon.com.invalid
> > > >
> > > > wrote:
> > > >
> > > >> If delete normally works, there would hopefully be some log entries
> > when
> > > >> it
> > > >> fails.  Are there any unusual zookeeper entries in the
> > > >> /admin/delete_topics
> > > >> path or in the other /admin folders?
> > > >>
> > > >> Does the topic name still exist in zookeeper under /config/topics?
> If
> > > so,
> > > >> that should probably deleted as well.
> > > >>
> > > >> Carl
> > > >>
> > > >> On Thu, Jul 20, 2017 at 6:42 PM, Chris Neal <cw...@gmail.com>
> wrote:
> > > >>
> > > >> > Delete is definitely there.  The delete worked fine, based on the
> > fact
> > > >> that
> > > >> > there is nothing in Zookeeper, and that the controller reported
> that
> > > the
> > > >> > delete was successful, it's just something seems to have gotten
> out
> > of
> > > >> > sync.
> > > >> >
> > > >> > delete.topic.enabled is true.  I've successfully deleted topics in
> > the
> > > >> > past, so I know it *should* work. :)
> > > >> >
> > > >> > I also had already checked in Zookeeper, and there is no directory
> > for
> > > >> the
> > > >> > topics under /brokers/topics....  Very strange indeed.
> > > >> >
> > > >> > If I just remove the log directories from the filesystem, is that
> > > >> enough to
> > > >> > get the broker to stop asking about the topics?  I would guess
> there
> > > >> would
> > > >> > need to be more than just that, but I could be wrong.
> > > >> >
> > > >> > Thanks guys for the suggestions though!
> > > >> >
> > > >> > On Thu, Jul 20, 2017 at 8:19 PM, Stephen Powis <
> > spowis@salesforce.com
> > > >
> > > >> > wrote:
> > > >> >
> > > >> > > I could be totally wrong, but I seem to recall that delete
> wasn't
> > > >> fully
> > > >> > > implemented in 0.8.x?
> > > >> > >
> > > >> > > On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd
> > > >> > <chaferd@groupon.com.invalid
> > > >> > > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Chris,
> > > >> > > >
> > > >> > > > You could first check to make sure that delete.topic.enable is
> > > true
> > > >> and
> > > >> > > try
> > > >> > > > deleting again if not.  If that doesn't work with 0.8.1.1 you
> > > might
> > > >> > need
> > > >> > > to
> > > >> > > > manually remove the topic's log files from the configured
> > log.dirs
> > > >> > folder
> > > >> > > > on each broker in addition to removing the topic's zookeeper
> > path.
> > > >> > > >
> > > >> > > > Carl
> > > >> > > >
> > > >> > > > On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal <
> cwneal@gmail.com>
> > > >> wrote:
> > > >> > > >
> > > >> > > > > Hi all,
> > > >> > > > >
> > > >> > > > > I have a weird situation here.  I have deleted a few topics
> on
> > > my
> > > >> > > 0.8.1.1
> > > >> > > > > cluster (old, I know...).  The deletes succeeded according
> to
> > > the
> > > >> > > > > controller.log:
> > > >> > > > >
> > > >> > > > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on
> > > Controller
> > > >> 1]:
> > > >> > > New
> > > >> > > > > topics: [Set()], deleted topics:
> > > >> > > > > [Set(perf_doorway-supplier-adapter-uat_raw)], new partition
> > > >> replica
> > > >> > > > > assignment [Map()]
> > > >> > > > > (kafka.controller.PartitionStateMachine$
> TopicChangeListener)
> > > >> > > > > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on
> > > Controller
> > > >> 1]:
> > > >> > > New
> > > >> > > > > topics: [Set()], deleted topics:
> > > >> > > > > [Set(perf_doorway-supplier-scheduler-uat_raw)], new
> partition
> > > >> > replica
> > > >> > > > > assignment [Map()]
> > > >> > > > > (kafka.controller.PartitionStateMachine$
> TopicChangeListener)
> > > >> > > > > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on
> > > Controller
> > > >> 1]:
> > > >> > > New
> > > >> > > > > topics: [Set()], deleted topics:
> > [Set(perf_gocontent-uat_raw)],
> > > >> new
> > > >> > > > > partition replica assignment [Map()]
> > > >> > > > > (kafka.controller.PartitionStateMachine$
> TopicChangeListener)
> > > >> > > > > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on
> > > Controller
> > > >> 1]:
> > > >> > > New
> > > >> > > > > topics: [Set()], deleted topics:
> > [Set(perf_goplatform-uat_raw)]
> > > ,
> > > >> new
> > > >> > > > > partition replica assignment [Map()]
> > > >> > > > > (kafka.controller.PartitionStateMachine$
> TopicChangeListener)
> > > >> > > > >
> > > >> > > > > I query Zookeeper and the path is not there under
> > > /brokers/topics
> > > >> as
> > > >> > > > well.
> > > >> > > > >
> > > >> > > > > But, one of the nodes in my cluster continues to try and use
> > > them:
> > > >> > > > >
> > > >> > > > > [2017-07-20 17:04:36,723] ERROR Conditional update of path
> > > >> > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
> > > >> > > > partitions/3/state
> > > >> > > > > with data
> > > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > >> > > > > epoch":2,"isr":[1,0]}
> > > >> > > > > and expected version 69 failed due to
> > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > >> > KeeperErrorCode
> > > >> > > =
> > > >> > > > > NoNode for
> > > >> > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
> > > >> > > > partitions/3/state
> > > >> > > > > (kafka.utils.ZkUtils$)
> > > >> > > > > [2017-07-20 17:04:36,723] INFO Partition
> > > >> > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1:
> > Cached
> > > >> > > > zkVersion
> > > >> > > > > [69] not equal to that in zookeeper, skip updating ISR
> > > >> > > > > (kafka.cluster.Partition)
> > > >> > > > > [2017-07-20 17:04:36,723] INFO Partition
> > > >> > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1:
> > Cached
> > > >> > > > zkVersion
> > > >> > > > > [69] not equal to that in zookeeper, skip updating ISR
> > > >> > > > > (kafka.cluster.Partition)
> > > >> > > > > [2017-07-20 17:04:36,764] INFO Partition
> > > >> [perf_goplatform-uat_raw,2]
> > > >> > on
> > > >> > > > > broker 1: Shrinking ISR for partition
> > > [perf_goplatform-uat_raw,2]
> > > >> > from
> > > >> > > > 1,0
> > > >> > > > > to 1 (kafka.cluster.Partition)
> > > >> > > > > [2017-07-20 17:04:36,764] INFO Partition
> > > >> [perf_goplatform-uat_raw,2]
> > > >> > on
> > > >> > > > > broker 1: Shrinking ISR for partition
> > > [perf_goplatform-uat_raw,2]
> > > >> > from
> > > >> > > > 1,0
> > > >> > > > > to 1 (kafka.cluster.Partition)
> > > >> > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path
> > > >> > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state
> > with
> > > >> data
> > > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > >> > > > epoch":2,"isr":[1]}
> > > >> > > > > and expected version 70 failed due to
> > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > >> > KeeperErrorCode
> > > >> > > =
> > > >> > > > > NoNode for /brokers/topics/perf_
> > goplatform-uat_raw/partitions/
> > > >> > 2/state
> > > >> > > > > (kafka.utils.ZkUtils$)
> > > >> > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path
> > > >> > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state
> > with
> > > >> data
> > > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > >> > > > epoch":2,"isr":[1]}
> > > >> > > > > and expected version 70 failed due to
> > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > >> > KeeperErrorCode
> > > >> > > =
> > > >> > > > > NoNode for /brokers/topics/perf_
> > goplatform-uat_raw/partitions/
> > > >> > 2/state
> > > >> > > > > (kafka.utils.ZkUtils$)
> > > >> > > > > [2017-07-20 17:04:36,765] INFO Partition
> > > >> [perf_goplatform-uat_raw,2]
> > > >> > on
> > > >> > > > > broker 1: Cached zkVersion [70] not equal to that in
> > zookeeper,
> > > >> skip
> > > >> > > > > updating ISR (kafka.cluster.Partition)
> > > >> > > > > [2017-07-20 17:04:36,765] INFO Partition
> > > >> [perf_goplatform-uat_raw,2]
> > > >> > on
> > > >> > > > > broker 1: Cached zkVersion [70] not equal to that in
> > zookeeper,
> > > >> skip
> > > >> > > > > updating ISR (kafka.cluster.Partition)
> > > >> > > > > [2017-07-20 17:04:36,981] INFO Partition
> > > >> [perf_gocontent-uat_raw,1]
> > > >> > on
> > > >> > > > > broker 1: Shrinking ISR for partition
> > [perf_gocontent-uat_raw,1]
> > > >> from
> > > >> > > 1,0
> > > >> > > > > to 1 (kafka.cluster.Partition)
> > > >> > > > > [2017-07-20 17:04:36,981] INFO Partition
> > > >> [perf_gocontent-uat_raw,1]
> > > >> > on
> > > >> > > > > broker 1: Shrinking ISR for partition
> > [perf_gocontent-uat_raw,1]
> > > >> from
> > > >> > > 1,0
> > > >> > > > > to 1 (kafka.cluster.Partition)
> > > >> > > > > [2017-07-20 17:04:36,988] ERROR Conditional update of path
> > > >> > > > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
> > with
> > > >> data
> > > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > >> > > > epoch":4,"isr":[1]}
> > > >> > > > > and expected version 90 failed due to
> > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > >> > KeeperErrorCode
> > > >> > > =
> > > >> > > > > NoNode for /brokers/topics/perf_gocontent
> > > >> -uat_raw/partitions/1/state
> > > >> > > > > (kafka.utils.ZkUtils$)
> > > >> > > > > [2017-07-20 17:04:36,988] ERROR Conditional update of path
> > > >> > > > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
> > with
> > > >> data
> > > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > >> > > > epoch":4,"isr":[1]}
> > > >> > > > > and expected version 90 failed due to
> > > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > >> > KeeperErrorCode
> > > >> > > =
> > > >> > > > > NoNode for /brokers/topics/perf_gocontent
> > > >> -uat_raw/partitions/1/state
> > > >> > > > > (kafka.utils.ZkUtils$)
> > > >> > > > > [2017-07-20 17:04:36,988] INFO Partition
> > > >> [perf_gocontent-uat_raw,1]
> > > >> > on
> > > >> > > > > broker 1: Cached zkVersion [90] not equal to that in
> > zookeeper,
> > > >> skip
> > > >> > > > > updating ISR (kafka.cluster.Partition)
> > > >> > > > > [2017-07-20 17:04:36,988] INFO Partition
> > > >> [perf_gocontent-uat_raw,1]
> > > >> > on
> > > >> > > > > broker 1: Cached zkVersion [90] not equal to that in
> > zookeeper,
> > > >> skip
> > > >> > > > > updating ISR (kafka.cluster.Partition)
> > > >> > > > >
> > > >> > > > > I've tried a rolling restart of the cluster to see if that
> > fixed
> > > >> it,
> > > >> > > but
> > > >> > > > it
> > > >> > > > > did not.
> > > >> > > > >
> > > >> > > > > Can someone please help me out here?  I'm not sure how I can
> > get
> > > >> > things
> > > >> > > > > back in sync.
> > > >> > > > >
> > > >> > > > > Thank you so much for your time.
> > > >> > > > > Chris
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

[Chris Neal <cw...@gmail.com>]

Welp.  Surprisingly, that did not fix the problem. :(

I cleaned out all the entries for these topics from /config/topics, and
removed the logs from the file system for those topics, and the messages
are still flying by in the server.log file.

Also, more concerning, when I was looking through the log files for the
other broker in the cluster, I noticed the same type of message for a topic
that should actually be there:

[2017-07-21 16:03:29,140] ERROR Conditional update of path
/brokers/topics/perf_dstorage_raw/partitions/4/state with data
{"controller_epoch":34,"leader":0,"version":1,"leader_epoch":0,"isr":[0]}
and expected version 0 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/perf_dstorage_raw/partitions/4/state
(kafka.utils.ZkUtils$)
[2017-07-21 16:03:29,142] ERROR Conditional update of path
/brokers/topics/perf_dstorage_raw/partitions/0/state with data
{"controller_epoch":34,"leader":0,"version":1,"leader_epoch":0,"isr":[0]}
and expected version 0 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state
(kafka.utils.ZkUtils$)
[2017-07-21 16:03:29,142] ERROR Conditional update of path
/brokers/topics/perf_dstorage_raw/partitions/0/state with data
{"controller_epoch":34,"leader":0,"version":1,"leader_epoch":0,"isr":[0]}
and expected version 0 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/perf_dstorage_raw/partitions/0/state
(kafka.utils.ZkUtils$)

So, the issue is not isolated to just these "should-have-been-removed"
topics, unfortunately.

Really appreciate the input so far everyone.  Still looking though for a
solution.  Many thanks. :)

Chris

On Fri, Jul 21, 2017 at 10:58 AM, M. Manna <ma...@gmail.com> wrote:

> Just to add (in case the platoform is Windows)
>
> For Windows based cluster implementation, log/topic cleanup doesn't work
> out of the box. Users are more or less aware of it, and doing their own
> maintenance as workaround.
>  If you have issues on Topic deletion not working properly on Windows (i.e.
> with topic deletion enable and all other settings). then you have to
> manually delete the files.
>
>
> On 21 July 2017 at 16:53, Chris Neal <cw...@gmail.com> wrote:
>
> > @Carl,
> >
> > There is nothing under /admin/delete_topics other than
> >
> > []
> >
> > And nothing under /admin other than delete_topics :)
> >
> > The topics DO exist, however, under /config/topics!  We may be on to
> > something.  I will remove them here and see if that clears it up.
> >
> > Thanks so much for all the help!
> > Chris
> >
> > On Thu, Jul 20, 2017 at 10:37 PM, Chris Neal <cw...@gmail.com> wrote:
> >
> > > Thanks again for the replies.  VERY much appreciated.  I'll check both
> > > /admin/delete_topics and /config/topics.
> > >
> > > Chris
> > >
> > > On Thu, Jul 20, 2017 at 9:22 PM, Carl Haferd
> <chaferd@groupon.com.invalid
> > >
> > > wrote:
> > >
> > >> If delete normally works, there would hopefully be some log entries
> when
> > >> it
> > >> fails.  Are there any unusual zookeeper entries in the
> > >> /admin/delete_topics
> > >> path or in the other /admin folders?
> > >>
> > >> Does the topic name still exist in zookeeper under /config/topics?  If
> > so,
> > >> that should probably deleted as well.
> > >>
> > >> Carl
> > >>
> > >> On Thu, Jul 20, 2017 at 6:42 PM, Chris Neal <cw...@gmail.com> wrote:
> > >>
> > >> > Delete is definitely there.  The delete worked fine, based on the
> fact
> > >> that
> > >> > there is nothing in Zookeeper, and that the controller reported that
> > the
> > >> > delete was successful, it's just something seems to have gotten out
> of
> > >> > sync.
> > >> >
> > >> > delete.topic.enabled is true.  I've successfully deleted topics in
> the
> > >> > past, so I know it *should* work. :)
> > >> >
> > >> > I also had already checked in Zookeeper, and there is no directory
> for
> > >> the
> > >> > topics under /brokers/topics....  Very strange indeed.
> > >> >
> > >> > If I just remove the log directories from the filesystem, is that
> > >> enough to
> > >> > get the broker to stop asking about the topics?  I would guess there
> > >> would
> > >> > need to be more than just that, but I could be wrong.
> > >> >
> > >> > Thanks guys for the suggestions though!
> > >> >
> > >> > On Thu, Jul 20, 2017 at 8:19 PM, Stephen Powis <
> spowis@salesforce.com
> > >
> > >> > wrote:
> > >> >
> > >> > > I could be totally wrong, but I seem to recall that delete wasn't
> > >> fully
> > >> > > implemented in 0.8.x?
> > >> > >
> > >> > > On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd
> > >> > <chaferd@groupon.com.invalid
> > >> > > >
> > >> > > wrote:
> > >> > >
> > >> > > > Chris,
> > >> > > >
> > >> > > > You could first check to make sure that delete.topic.enable is
> > true
> > >> and
> > >> > > try
> > >> > > > deleting again if not.  If that doesn't work with 0.8.1.1 you
> > might
> > >> > need
> > >> > > to
> > >> > > > manually remove the topic's log files from the configured
> log.dirs
> > >> > folder
> > >> > > > on each broker in addition to removing the topic's zookeeper
> path.
> > >> > > >
> > >> > > > Carl
> > >> > > >
> > >> > > > On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal <cw...@gmail.com>
> > >> wrote:
> > >> > > >
> > >> > > > > Hi all,
> > >> > > > >
> > >> > > > > I have a weird situation here.  I have deleted a few topics on
> > my
> > >> > > 0.8.1.1
> > >> > > > > cluster (old, I know...).  The deletes succeeded according to
> > the
> > >> > > > > controller.log:
> > >> > > > >
> > >> > > > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on
> > Controller
> > >> 1]:
> > >> > > New
> > >> > > > > topics: [Set()], deleted topics:
> > >> > > > > [Set(perf_doorway-supplier-adapter-uat_raw)], new partition
> > >> replica
> > >> > > > > assignment [Map()]
> > >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > >> > > > > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on
> > Controller
> > >> 1]:
> > >> > > New
> > >> > > > > topics: [Set()], deleted topics:
> > >> > > > > [Set(perf_doorway-supplier-scheduler-uat_raw)], new partition
> > >> > replica
> > >> > > > > assignment [Map()]
> > >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > >> > > > > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on
> > Controller
> > >> 1]:
> > >> > > New
> > >> > > > > topics: [Set()], deleted topics:
> [Set(perf_gocontent-uat_raw)],
> > >> new
> > >> > > > > partition replica assignment [Map()]
> > >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > >> > > > > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on
> > Controller
> > >> 1]:
> > >> > > New
> > >> > > > > topics: [Set()], deleted topics:
> [Set(perf_goplatform-uat_raw)]
> > ,
> > >> new
> > >> > > > > partition replica assignment [Map()]
> > >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > >> > > > >
> > >> > > > > I query Zookeeper and the path is not there under
> > /brokers/topics
> > >> as
> > >> > > > well.
> > >> > > > >
> > >> > > > > But, one of the nodes in my cluster continues to try and use
> > them:
> > >> > > > >
> > >> > > > > [2017-07-20 17:04:36,723] ERROR Conditional update of path
> > >> > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
> > >> > > > partitions/3/state
> > >> > > > > with data
> > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > >> > > > > epoch":2,"isr":[1,0]}
> > >> > > > > and expected version 69 failed due to
> > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > >> > KeeperErrorCode
> > >> > > =
> > >> > > > > NoNode for
> > >> > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
> > >> > > > partitions/3/state
> > >> > > > > (kafka.utils.ZkUtils$)
> > >> > > > > [2017-07-20 17:04:36,723] INFO Partition
> > >> > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1:
> Cached
> > >> > > > zkVersion
> > >> > > > > [69] not equal to that in zookeeper, skip updating ISR
> > >> > > > > (kafka.cluster.Partition)
> > >> > > > > [2017-07-20 17:04:36,723] INFO Partition
> > >> > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1:
> Cached
> > >> > > > zkVersion
> > >> > > > > [69] not equal to that in zookeeper, skip updating ISR
> > >> > > > > (kafka.cluster.Partition)
> > >> > > > > [2017-07-20 17:04:36,764] INFO Partition
> > >> [perf_goplatform-uat_raw,2]
> > >> > on
> > >> > > > > broker 1: Shrinking ISR for partition
> > [perf_goplatform-uat_raw,2]
> > >> > from
> > >> > > > 1,0
> > >> > > > > to 1 (kafka.cluster.Partition)
> > >> > > > > [2017-07-20 17:04:36,764] INFO Partition
> > >> [perf_goplatform-uat_raw,2]
> > >> > on
> > >> > > > > broker 1: Shrinking ISR for partition
> > [perf_goplatform-uat_raw,2]
> > >> > from
> > >> > > > 1,0
> > >> > > > > to 1 (kafka.cluster.Partition)
> > >> > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path
> > >> > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state
> with
> > >> data
> > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > >> > > > epoch":2,"isr":[1]}
> > >> > > > > and expected version 70 failed due to
> > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > >> > KeeperErrorCode
> > >> > > =
> > >> > > > > NoNode for /brokers/topics/perf_
> goplatform-uat_raw/partitions/
> > >> > 2/state
> > >> > > > > (kafka.utils.ZkUtils$)
> > >> > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path
> > >> > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state
> with
> > >> data
> > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > >> > > > epoch":2,"isr":[1]}
> > >> > > > > and expected version 70 failed due to
> > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > >> > KeeperErrorCode
> > >> > > =
> > >> > > > > NoNode for /brokers/topics/perf_
> goplatform-uat_raw/partitions/
> > >> > 2/state
> > >> > > > > (kafka.utils.ZkUtils$)
> > >> > > > > [2017-07-20 17:04:36,765] INFO Partition
> > >> [perf_goplatform-uat_raw,2]
> > >> > on
> > >> > > > > broker 1: Cached zkVersion [70] not equal to that in
> zookeeper,
> > >> skip
> > >> > > > > updating ISR (kafka.cluster.Partition)
> > >> > > > > [2017-07-20 17:04:36,765] INFO Partition
> > >> [perf_goplatform-uat_raw,2]
> > >> > on
> > >> > > > > broker 1: Cached zkVersion [70] not equal to that in
> zookeeper,
> > >> skip
> > >> > > > > updating ISR (kafka.cluster.Partition)
> > >> > > > > [2017-07-20 17:04:36,981] INFO Partition
> > >> [perf_gocontent-uat_raw,1]
> > >> > on
> > >> > > > > broker 1: Shrinking ISR for partition
> [perf_gocontent-uat_raw,1]
> > >> from
> > >> > > 1,0
> > >> > > > > to 1 (kafka.cluster.Partition)
> > >> > > > > [2017-07-20 17:04:36,981] INFO Partition
> > >> [perf_gocontent-uat_raw,1]
> > >> > on
> > >> > > > > broker 1: Shrinking ISR for partition
> [perf_gocontent-uat_raw,1]
> > >> from
> > >> > > 1,0
> > >> > > > > to 1 (kafka.cluster.Partition)
> > >> > > > > [2017-07-20 17:04:36,988] ERROR Conditional update of path
> > >> > > > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
> with
> > >> data
> > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > >> > > > epoch":4,"isr":[1]}
> > >> > > > > and expected version 90 failed due to
> > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > >> > KeeperErrorCode
> > >> > > =
> > >> > > > > NoNode for /brokers/topics/perf_gocontent
> > >> -uat_raw/partitions/1/state
> > >> > > > > (kafka.utils.ZkUtils$)
> > >> > > > > [2017-07-20 17:04:36,988] ERROR Conditional update of path
> > >> > > > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
> with
> > >> data
> > >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > >> > > > epoch":4,"isr":[1]}
> > >> > > > > and expected version 90 failed due to
> > >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > >> > KeeperErrorCode
> > >> > > =
> > >> > > > > NoNode for /brokers/topics/perf_gocontent
> > >> -uat_raw/partitions/1/state
> > >> > > > > (kafka.utils.ZkUtils$)
> > >> > > > > [2017-07-20 17:04:36,988] INFO Partition
> > >> [perf_gocontent-uat_raw,1]
> > >> > on
> > >> > > > > broker 1: Cached zkVersion [90] not equal to that in
> zookeeper,
> > >> skip
> > >> > > > > updating ISR (kafka.cluster.Partition)
> > >> > > > > [2017-07-20 17:04:36,988] INFO Partition
> > >> [perf_gocontent-uat_raw,1]
> > >> > on
> > >> > > > > broker 1: Cached zkVersion [90] not equal to that in
> zookeeper,
> > >> skip
> > >> > > > > updating ISR (kafka.cluster.Partition)
> > >> > > > >
> > >> > > > > I've tried a rolling restart of the cluster to see if that
> fixed
> > >> it,
> > >> > > but
> > >> > > > it
> > >> > > > > did not.
> > >> > > > >
> > >> > > > > Can someone please help me out here?  I'm not sure how I can
> get
> > >> > things
> > >> > > > > back in sync.
> > >> > > > >
> > >> > > > > Thank you so much for your time.
> > >> > > > > Chris
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

["M. Manna" <ma...@gmail.com>]

Just to add (in case the platoform is Windows)

For Windows based cluster implementation, log/topic cleanup doesn't work
out of the box. Users are more or less aware of it, and doing their own
maintenance as workaround.
 If you have issues on Topic deletion not working properly on Windows (i.e.
with topic deletion enable and all other settings). then you have to
manually delete the files.


On 21 July 2017 at 16:53, Chris Neal <cw...@gmail.com> wrote:

> @Carl,
>
> There is nothing under /admin/delete_topics other than
>
> []
>
> And nothing under /admin other than delete_topics :)
>
> The topics DO exist, however, under /config/topics!  We may be on to
> something.  I will remove them here and see if that clears it up.
>
> Thanks so much for all the help!
> Chris
>
> On Thu, Jul 20, 2017 at 10:37 PM, Chris Neal <cw...@gmail.com> wrote:
>
> > Thanks again for the replies.  VERY much appreciated.  I'll check both
> > /admin/delete_topics and /config/topics.
> >
> > Chris
> >
> > On Thu, Jul 20, 2017 at 9:22 PM, Carl Haferd <chaferd@groupon.com.invalid
> >
> > wrote:
> >
> >> If delete normally works, there would hopefully be some log entries when
> >> it
> >> fails.  Are there any unusual zookeeper entries in the
> >> /admin/delete_topics
> >> path or in the other /admin folders?
> >>
> >> Does the topic name still exist in zookeeper under /config/topics?  If
> so,
> >> that should probably deleted as well.
> >>
> >> Carl
> >>
> >> On Thu, Jul 20, 2017 at 6:42 PM, Chris Neal <cw...@gmail.com> wrote:
> >>
> >> > Delete is definitely there.  The delete worked fine, based on the fact
> >> that
> >> > there is nothing in Zookeeper, and that the controller reported that
> the
> >> > delete was successful, it's just something seems to have gotten out of
> >> > sync.
> >> >
> >> > delete.topic.enabled is true.  I've successfully deleted topics in the
> >> > past, so I know it *should* work. :)
> >> >
> >> > I also had already checked in Zookeeper, and there is no directory for
> >> the
> >> > topics under /brokers/topics....  Very strange indeed.
> >> >
> >> > If I just remove the log directories from the filesystem, is that
> >> enough to
> >> > get the broker to stop asking about the topics?  I would guess there
> >> would
> >> > need to be more than just that, but I could be wrong.
> >> >
> >> > Thanks guys for the suggestions though!
> >> >
> >> > On Thu, Jul 20, 2017 at 8:19 PM, Stephen Powis <spowis@salesforce.com
> >
> >> > wrote:
> >> >
> >> > > I could be totally wrong, but I seem to recall that delete wasn't
> >> fully
> >> > > implemented in 0.8.x?
> >> > >
> >> > > On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd
> >> > <chaferd@groupon.com.invalid
> >> > > >
> >> > > wrote:
> >> > >
> >> > > > Chris,
> >> > > >
> >> > > > You could first check to make sure that delete.topic.enable is
> true
> >> and
> >> > > try
> >> > > > deleting again if not.  If that doesn't work with 0.8.1.1 you
> might
> >> > need
> >> > > to
> >> > > > manually remove the topic's log files from the configured log.dirs
> >> > folder
> >> > > > on each broker in addition to removing the topic's zookeeper path.
> >> > > >
> >> > > > Carl
> >> > > >
> >> > > > On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal <cw...@gmail.com>
> >> wrote:
> >> > > >
> >> > > > > Hi all,
> >> > > > >
> >> > > > > I have a weird situation here.  I have deleted a few topics on
> my
> >> > > 0.8.1.1
> >> > > > > cluster (old, I know...).  The deletes succeeded according to
> the
> >> > > > > controller.log:
> >> > > > >
> >> > > > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on
> Controller
> >> 1]:
> >> > > New
> >> > > > > topics: [Set()], deleted topics:
> >> > > > > [Set(perf_doorway-supplier-adapter-uat_raw)], new partition
> >> replica
> >> > > > > assignment [Map()]
> >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> >> > > > > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on
> Controller
> >> 1]:
> >> > > New
> >> > > > > topics: [Set()], deleted topics:
> >> > > > > [Set(perf_doorway-supplier-scheduler-uat_raw)], new partition
> >> > replica
> >> > > > > assignment [Map()]
> >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> >> > > > > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on
> Controller
> >> 1]:
> >> > > New
> >> > > > > topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)],
> >> new
> >> > > > > partition replica assignment [Map()]
> >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> >> > > > > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on
> Controller
> >> 1]:
> >> > > New
> >> > > > > topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)]
> ,
> >> new
> >> > > > > partition replica assignment [Map()]
> >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> >> > > > >
> >> > > > > I query Zookeeper and the path is not there under
> /brokers/topics
> >> as
> >> > > > well.
> >> > > > >
> >> > > > > But, one of the nodes in my cluster continues to try and use
> them:
> >> > > > >
> >> > > > > [2017-07-20 17:04:36,723] ERROR Conditional update of path
> >> > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
> >> > > > partitions/3/state
> >> > > > > with data
> >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> >> > > > > epoch":2,"isr":[1,0]}
> >> > > > > and expected version 69 failed due to
> >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> >> > KeeperErrorCode
> >> > > =
> >> > > > > NoNode for
> >> > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
> >> > > > partitions/3/state
> >> > > > > (kafka.utils.ZkUtils$)
> >> > > > > [2017-07-20 17:04:36,723] INFO Partition
> >> > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached
> >> > > > zkVersion
> >> > > > > [69] not equal to that in zookeeper, skip updating ISR
> >> > > > > (kafka.cluster.Partition)
> >> > > > > [2017-07-20 17:04:36,723] INFO Partition
> >> > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached
> >> > > > zkVersion
> >> > > > > [69] not equal to that in zookeeper, skip updating ISR
> >> > > > > (kafka.cluster.Partition)
> >> > > > > [2017-07-20 17:04:36,764] INFO Partition
> >> [perf_goplatform-uat_raw,2]
> >> > on
> >> > > > > broker 1: Shrinking ISR for partition
> [perf_goplatform-uat_raw,2]
> >> > from
> >> > > > 1,0
> >> > > > > to 1 (kafka.cluster.Partition)
> >> > > > > [2017-07-20 17:04:36,764] INFO Partition
> >> [perf_goplatform-uat_raw,2]
> >> > on
> >> > > > > broker 1: Shrinking ISR for partition
> [perf_goplatform-uat_raw,2]
> >> > from
> >> > > > 1,0
> >> > > > > to 1 (kafka.cluster.Partition)
> >> > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path
> >> > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with
> >> data
> >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> >> > > > epoch":2,"isr":[1]}
> >> > > > > and expected version 70 failed due to
> >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> >> > KeeperErrorCode
> >> > > =
> >> > > > > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/
> >> > 2/state
> >> > > > > (kafka.utils.ZkUtils$)
> >> > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path
> >> > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with
> >> data
> >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> >> > > > epoch":2,"isr":[1]}
> >> > > > > and expected version 70 failed due to
> >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> >> > KeeperErrorCode
> >> > > =
> >> > > > > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/
> >> > 2/state
> >> > > > > (kafka.utils.ZkUtils$)
> >> > > > > [2017-07-20 17:04:36,765] INFO Partition
> >> [perf_goplatform-uat_raw,2]
> >> > on
> >> > > > > broker 1: Cached zkVersion [70] not equal to that in zookeeper,
> >> skip
> >> > > > > updating ISR (kafka.cluster.Partition)
> >> > > > > [2017-07-20 17:04:36,765] INFO Partition
> >> [perf_goplatform-uat_raw,2]
> >> > on
> >> > > > > broker 1: Cached zkVersion [70] not equal to that in zookeeper,
> >> skip
> >> > > > > updating ISR (kafka.cluster.Partition)
> >> > > > > [2017-07-20 17:04:36,981] INFO Partition
> >> [perf_gocontent-uat_raw,1]
> >> > on
> >> > > > > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1]
> >> from
> >> > > 1,0
> >> > > > > to 1 (kafka.cluster.Partition)
> >> > > > > [2017-07-20 17:04:36,981] INFO Partition
> >> [perf_gocontent-uat_raw,1]
> >> > on
> >> > > > > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1]
> >> from
> >> > > 1,0
> >> > > > > to 1 (kafka.cluster.Partition)
> >> > > > > [2017-07-20 17:04:36,988] ERROR Conditional update of path
> >> > > > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with
> >> data
> >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> >> > > > epoch":4,"isr":[1]}
> >> > > > > and expected version 90 failed due to
> >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> >> > KeeperErrorCode
> >> > > =
> >> > > > > NoNode for /brokers/topics/perf_gocontent
> >> -uat_raw/partitions/1/state
> >> > > > > (kafka.utils.ZkUtils$)
> >> > > > > [2017-07-20 17:04:36,988] ERROR Conditional update of path
> >> > > > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with
> >> data
> >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> >> > > > epoch":4,"isr":[1]}
> >> > > > > and expected version 90 failed due to
> >> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> >> > KeeperErrorCode
> >> > > =
> >> > > > > NoNode for /brokers/topics/perf_gocontent
> >> -uat_raw/partitions/1/state
> >> > > > > (kafka.utils.ZkUtils$)
> >> > > > > [2017-07-20 17:04:36,988] INFO Partition
> >> [perf_gocontent-uat_raw,1]
> >> > on
> >> > > > > broker 1: Cached zkVersion [90] not equal to that in zookeeper,
> >> skip
> >> > > > > updating ISR (kafka.cluster.Partition)
> >> > > > > [2017-07-20 17:04:36,988] INFO Partition
> >> [perf_gocontent-uat_raw,1]
> >> > on
> >> > > > > broker 1: Cached zkVersion [90] not equal to that in zookeeper,
> >> skip
> >> > > > > updating ISR (kafka.cluster.Partition)
> >> > > > >
> >> > > > > I've tried a rolling restart of the cluster to see if that fixed
> >> it,
> >> > > but
> >> > > > it
> >> > > > > did not.
> >> > > > >
> >> > > > > Can someone please help me out here?  I'm not sure how I can get
> >> > things
> >> > > > > back in sync.
> >> > > > >
> >> > > > > Thank you so much for your time.
> >> > > > > Chris
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

[Chris Neal <cw...@gmail.com>]

@Carl,

There is nothing under /admin/delete_topics other than

[]

And nothing under /admin other than delete_topics :)

The topics DO exist, however, under /config/topics!  We may be on to
something.  I will remove them here and see if that clears it up.

Thanks so much for all the help!
Chris

On Thu, Jul 20, 2017 at 10:37 PM, Chris Neal <cw...@gmail.com> wrote:

> Thanks again for the replies.  VERY much appreciated.  I'll check both
> /admin/delete_topics and /config/topics.
>
> Chris
>
> On Thu, Jul 20, 2017 at 9:22 PM, Carl Haferd <ch...@groupon.com.invalid>
> wrote:
>
>> If delete normally works, there would hopefully be some log entries when
>> it
>> fails.  Are there any unusual zookeeper entries in the
>> /admin/delete_topics
>> path or in the other /admin folders?
>>
>> Does the topic name still exist in zookeeper under /config/topics?  If so,
>> that should probably deleted as well.
>>
>> Carl
>>
>> On Thu, Jul 20, 2017 at 6:42 PM, Chris Neal <cw...@gmail.com> wrote:
>>
>> > Delete is definitely there.  The delete worked fine, based on the fact
>> that
>> > there is nothing in Zookeeper, and that the controller reported that the
>> > delete was successful, it's just something seems to have gotten out of
>> > sync.
>> >
>> > delete.topic.enabled is true.  I've successfully deleted topics in the
>> > past, so I know it *should* work. :)
>> >
>> > I also had already checked in Zookeeper, and there is no directory for
>> the
>> > topics under /brokers/topics....  Very strange indeed.
>> >
>> > If I just remove the log directories from the filesystem, is that
>> enough to
>> > get the broker to stop asking about the topics?  I would guess there
>> would
>> > need to be more than just that, but I could be wrong.
>> >
>> > Thanks guys for the suggestions though!
>> >
>> > On Thu, Jul 20, 2017 at 8:19 PM, Stephen Powis <sp...@salesforce.com>
>> > wrote:
>> >
>> > > I could be totally wrong, but I seem to recall that delete wasn't
>> fully
>> > > implemented in 0.8.x?
>> > >
>> > > On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd
>> > <chaferd@groupon.com.invalid
>> > > >
>> > > wrote:
>> > >
>> > > > Chris,
>> > > >
>> > > > You could first check to make sure that delete.topic.enable is true
>> and
>> > > try
>> > > > deleting again if not.  If that doesn't work with 0.8.1.1 you might
>> > need
>> > > to
>> > > > manually remove the topic's log files from the configured log.dirs
>> > folder
>> > > > on each broker in addition to removing the topic's zookeeper path.
>> > > >
>> > > > Carl
>> > > >
>> > > > On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal <cw...@gmail.com>
>> wrote:
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > I have a weird situation here.  I have deleted a few topics on my
>> > > 0.8.1.1
>> > > > > cluster (old, I know...).  The deletes succeeded according to the
>> > > > > controller.log:
>> > > > >
>> > > > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on Controller
>> 1]:
>> > > New
>> > > > > topics: [Set()], deleted topics:
>> > > > > [Set(perf_doorway-supplier-adapter-uat_raw)], new partition
>> replica
>> > > > > assignment [Map()]
>> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
>> > > > > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on Controller
>> 1]:
>> > > New
>> > > > > topics: [Set()], deleted topics:
>> > > > > [Set(perf_doorway-supplier-scheduler-uat_raw)], new partition
>> > replica
>> > > > > assignment [Map()]
>> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
>> > > > > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on Controller
>> 1]:
>> > > New
>> > > > > topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)],
>> new
>> > > > > partition replica assignment [Map()]
>> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
>> > > > > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on Controller
>> 1]:
>> > > New
>> > > > > topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)],
>> new
>> > > > > partition replica assignment [Map()]
>> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
>> > > > >
>> > > > > I query Zookeeper and the path is not there under /brokers/topics
>> as
>> > > > well.
>> > > > >
>> > > > > But, one of the nodes in my cluster continues to try and use them:
>> > > > >
>> > > > > [2017-07-20 17:04:36,723] ERROR Conditional update of path
>> > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
>> > > > partitions/3/state
>> > > > > with data
>> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
>> > > > > epoch":2,"isr":[1,0]}
>> > > > > and expected version 69 failed due to
>> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
>> > KeeperErrorCode
>> > > =
>> > > > > NoNode for
>> > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
>> > > > partitions/3/state
>> > > > > (kafka.utils.ZkUtils$)
>> > > > > [2017-07-20 17:04:36,723] INFO Partition
>> > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached
>> > > > zkVersion
>> > > > > [69] not equal to that in zookeeper, skip updating ISR
>> > > > > (kafka.cluster.Partition)
>> > > > > [2017-07-20 17:04:36,723] INFO Partition
>> > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached
>> > > > zkVersion
>> > > > > [69] not equal to that in zookeeper, skip updating ISR
>> > > > > (kafka.cluster.Partition)
>> > > > > [2017-07-20 17:04:36,764] INFO Partition
>> [perf_goplatform-uat_raw,2]
>> > on
>> > > > > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2]
>> > from
>> > > > 1,0
>> > > > > to 1 (kafka.cluster.Partition)
>> > > > > [2017-07-20 17:04:36,764] INFO Partition
>> [perf_goplatform-uat_raw,2]
>> > on
>> > > > > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2]
>> > from
>> > > > 1,0
>> > > > > to 1 (kafka.cluster.Partition)
>> > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path
>> > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with
>> data
>> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
>> > > > epoch":2,"isr":[1]}
>> > > > > and expected version 70 failed due to
>> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
>> > KeeperErrorCode
>> > > =
>> > > > > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/
>> > 2/state
>> > > > > (kafka.utils.ZkUtils$)
>> > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path
>> > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with
>> data
>> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
>> > > > epoch":2,"isr":[1]}
>> > > > > and expected version 70 failed due to
>> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
>> > KeeperErrorCode
>> > > =
>> > > > > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/
>> > 2/state
>> > > > > (kafka.utils.ZkUtils$)
>> > > > > [2017-07-20 17:04:36,765] INFO Partition
>> [perf_goplatform-uat_raw,2]
>> > on
>> > > > > broker 1: Cached zkVersion [70] not equal to that in zookeeper,
>> skip
>> > > > > updating ISR (kafka.cluster.Partition)
>> > > > > [2017-07-20 17:04:36,765] INFO Partition
>> [perf_goplatform-uat_raw,2]
>> > on
>> > > > > broker 1: Cached zkVersion [70] not equal to that in zookeeper,
>> skip
>> > > > > updating ISR (kafka.cluster.Partition)
>> > > > > [2017-07-20 17:04:36,981] INFO Partition
>> [perf_gocontent-uat_raw,1]
>> > on
>> > > > > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1]
>> from
>> > > 1,0
>> > > > > to 1 (kafka.cluster.Partition)
>> > > > > [2017-07-20 17:04:36,981] INFO Partition
>> [perf_gocontent-uat_raw,1]
>> > on
>> > > > > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1]
>> from
>> > > 1,0
>> > > > > to 1 (kafka.cluster.Partition)
>> > > > > [2017-07-20 17:04:36,988] ERROR Conditional update of path
>> > > > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with
>> data
>> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
>> > > > epoch":4,"isr":[1]}
>> > > > > and expected version 90 failed due to
>> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
>> > KeeperErrorCode
>> > > =
>> > > > > NoNode for /brokers/topics/perf_gocontent
>> -uat_raw/partitions/1/state
>> > > > > (kafka.utils.ZkUtils$)
>> > > > > [2017-07-20 17:04:36,988] ERROR Conditional update of path
>> > > > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with
>> data
>> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
>> > > > epoch":4,"isr":[1]}
>> > > > > and expected version 90 failed due to
>> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
>> > KeeperErrorCode
>> > > =
>> > > > > NoNode for /brokers/topics/perf_gocontent
>> -uat_raw/partitions/1/state
>> > > > > (kafka.utils.ZkUtils$)
>> > > > > [2017-07-20 17:04:36,988] INFO Partition
>> [perf_gocontent-uat_raw,1]
>> > on
>> > > > > broker 1: Cached zkVersion [90] not equal to that in zookeeper,
>> skip
>> > > > > updating ISR (kafka.cluster.Partition)
>> > > > > [2017-07-20 17:04:36,988] INFO Partition
>> [perf_gocontent-uat_raw,1]
>> > on
>> > > > > broker 1: Cached zkVersion [90] not equal to that in zookeeper,
>> skip
>> > > > > updating ISR (kafka.cluster.Partition)
>> > > > >
>> > > > > I've tried a rolling restart of the cluster to see if that fixed
>> it,
>> > > but
>> > > > it
>> > > > > did not.
>> > > > >
>> > > > > Can someone please help me out here?  I'm not sure how I can get
>> > things
>> > > > > back in sync.
>> > > > >
>> > > > > Thank you so much for your time.
>> > > > > Chris
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

[Chris Neal <cw...@gmail.com>]

Thanks again for the replies.  VERY much appreciated.  I'll check both
/admin/delete_topics and /config/topics.

Chris

On Thu, Jul 20, 2017 at 9:22 PM, Carl Haferd <ch...@groupon.com.invalid>
wrote:

> If delete normally works, there would hopefully be some log entries when it
> fails.  Are there any unusual zookeeper entries in the /admin/delete_topics
> path or in the other /admin folders?
>
> Does the topic name still exist in zookeeper under /config/topics?  If so,
> that should probably deleted as well.
>
> Carl
>
> On Thu, Jul 20, 2017 at 6:42 PM, Chris Neal <cw...@gmail.com> wrote:
>
> > Delete is definitely there.  The delete worked fine, based on the fact
> that
> > there is nothing in Zookeeper, and that the controller reported that the
> > delete was successful, it's just something seems to have gotten out of
> > sync.
> >
> > delete.topic.enabled is true.  I've successfully deleted topics in the
> > past, so I know it *should* work. :)
> >
> > I also had already checked in Zookeeper, and there is no directory for
> the
> > topics under /brokers/topics....  Very strange indeed.
> >
> > If I just remove the log directories from the filesystem, is that enough
> to
> > get the broker to stop asking about the topics?  I would guess there
> would
> > need to be more than just that, but I could be wrong.
> >
> > Thanks guys for the suggestions though!
> >
> > On Thu, Jul 20, 2017 at 8:19 PM, Stephen Powis <sp...@salesforce.com>
> > wrote:
> >
> > > I could be totally wrong, but I seem to recall that delete wasn't fully
> > > implemented in 0.8.x?
> > >
> > > On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd
> > <chaferd@groupon.com.invalid
> > > >
> > > wrote:
> > >
> > > > Chris,
> > > >
> > > > You could first check to make sure that delete.topic.enable is true
> and
> > > try
> > > > deleting again if not.  If that doesn't work with 0.8.1.1 you might
> > need
> > > to
> > > > manually remove the topic's log files from the configured log.dirs
> > folder
> > > > on each broker in addition to removing the topic's zookeeper path.
> > > >
> > > > Carl
> > > >
> > > > On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal <cw...@gmail.com>
> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I have a weird situation here.  I have deleted a few topics on my
> > > 0.8.1.1
> > > > > cluster (old, I know...).  The deletes succeeded according to the
> > > > > controller.log:
> > > > >
> > > > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on Controller
> 1]:
> > > New
> > > > > topics: [Set()], deleted topics:
> > > > > [Set(perf_doorway-supplier-adapter-uat_raw)], new partition
> replica
> > > > > assignment [Map()]
> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on Controller
> 1]:
> > > New
> > > > > topics: [Set()], deleted topics:
> > > > > [Set(perf_doorway-supplier-scheduler-uat_raw)], new partition
> > replica
> > > > > assignment [Map()]
> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on Controller
> 1]:
> > > New
> > > > > topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)], new
> > > > > partition replica assignment [Map()]
> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on Controller
> 1]:
> > > New
> > > > > topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)],
> new
> > > > > partition replica assignment [Map()]
> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > >
> > > > > I query Zookeeper and the path is not there under /brokers/topics
> as
> > > > well.
> > > > >
> > > > > But, one of the nodes in my cluster continues to try and use them:
> > > > >
> > > > > [2017-07-20 17:04:36,723] ERROR Conditional update of path
> > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
> > > > partitions/3/state
> > > > > with data
> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > > > epoch":2,"isr":[1,0]}
> > > > > and expected version 69 failed due to
> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > KeeperErrorCode
> > > =
> > > > > NoNode for
> > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
> > > > partitions/3/state
> > > > > (kafka.utils.ZkUtils$)
> > > > > [2017-07-20 17:04:36,723] INFO Partition
> > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached
> > > > zkVersion
> > > > > [69] not equal to that in zookeeper, skip updating ISR
> > > > > (kafka.cluster.Partition)
> > > > > [2017-07-20 17:04:36,723] INFO Partition
> > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached
> > > > zkVersion
> > > > > [69] not equal to that in zookeeper, skip updating ISR
> > > > > (kafka.cluster.Partition)
> > > > > [2017-07-20 17:04:36,764] INFO Partition
> [perf_goplatform-uat_raw,2]
> > on
> > > > > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2]
> > from
> > > > 1,0
> > > > > to 1 (kafka.cluster.Partition)
> > > > > [2017-07-20 17:04:36,764] INFO Partition
> [perf_goplatform-uat_raw,2]
> > on
> > > > > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2]
> > from
> > > > 1,0
> > > > > to 1 (kafka.cluster.Partition)
> > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path
> > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with
> data
> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > > epoch":2,"isr":[1]}
> > > > > and expected version 70 failed due to
> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > KeeperErrorCode
> > > =
> > > > > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/
> > 2/state
> > > > > (kafka.utils.ZkUtils$)
> > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path
> > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with
> data
> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > > epoch":2,"isr":[1]}
> > > > > and expected version 70 failed due to
> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > KeeperErrorCode
> > > =
> > > > > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/
> > 2/state
> > > > > (kafka.utils.ZkUtils$)
> > > > > [2017-07-20 17:04:36,765] INFO Partition
> [perf_goplatform-uat_raw,2]
> > on
> > > > > broker 1: Cached zkVersion [70] not equal to that in zookeeper,
> skip
> > > > > updating ISR (kafka.cluster.Partition)
> > > > > [2017-07-20 17:04:36,765] INFO Partition
> [perf_goplatform-uat_raw,2]
> > on
> > > > > broker 1: Cached zkVersion [70] not equal to that in zookeeper,
> skip
> > > > > updating ISR (kafka.cluster.Partition)
> > > > > [2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1]
> > on
> > > > > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1]
> from
> > > 1,0
> > > > > to 1 (kafka.cluster.Partition)
> > > > > [2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1]
> > on
> > > > > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1]
> from
> > > 1,0
> > > > > to 1 (kafka.cluster.Partition)
> > > > > [2017-07-20 17:04:36,988] ERROR Conditional update of path
> > > > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with
> data
> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > > epoch":4,"isr":[1]}
> > > > > and expected version 90 failed due to
> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > KeeperErrorCode
> > > =
> > > > > NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/
> 1/state
> > > > > (kafka.utils.ZkUtils$)
> > > > > [2017-07-20 17:04:36,988] ERROR Conditional update of path
> > > > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with
> data
> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > > epoch":4,"isr":[1]}
> > > > > and expected version 90 failed due to
> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > KeeperErrorCode
> > > =
> > > > > NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/
> 1/state
> > > > > (kafka.utils.ZkUtils$)
> > > > > [2017-07-20 17:04:36,988] INFO Partition [perf_gocontent-uat_raw,1]
> > on
> > > > > broker 1: Cached zkVersion [90] not equal to that in zookeeper,
> skip
> > > > > updating ISR (kafka.cluster.Partition)
> > > > > [2017-07-20 17:04:36,988] INFO Partition [perf_gocontent-uat_raw,1]
> > on
> > > > > broker 1: Cached zkVersion [90] not equal to that in zookeeper,
> skip
> > > > > updating ISR (kafka.cluster.Partition)
> > > > >
> > > > > I've tried a rolling restart of the cluster to see if that fixed
> it,
> > > but
> > > > it
> > > > > did not.
> > > > >
> > > > > Can someone please help me out here?  I'm not sure how I can get
> > things
> > > > > back in sync.
> > > > >
> > > > > Thank you so much for your time.
> > > > > Chris
> > > > >
> > > >
> > >
> >
>

Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

[Carl Haferd <ch...@groupon.com.INVALID>]

If delete normally works, there would hopefully be some log entries when it
fails.  Are there any unusual zookeeper entries in the /admin/delete_topics
path or in the other /admin folders?

Does the topic name still exist in zookeeper under /config/topics?  If so,
that should probably deleted as well.

Carl

On Thu, Jul 20, 2017 at 6:42 PM, Chris Neal <cw...@gmail.com> wrote:

> Delete is definitely there.  The delete worked fine, based on the fact that
> there is nothing in Zookeeper, and that the controller reported that the
> delete was successful, it's just something seems to have gotten out of
> sync.
>
> delete.topic.enabled is true.  I've successfully deleted topics in the
> past, so I know it *should* work. :)
>
> I also had already checked in Zookeeper, and there is no directory for the
> topics under /brokers/topics....  Very strange indeed.
>
> If I just remove the log directories from the filesystem, is that enough to
> get the broker to stop asking about the topics?  I would guess there would
> need to be more than just that, but I could be wrong.
>
> Thanks guys for the suggestions though!
>
> On Thu, Jul 20, 2017 at 8:19 PM, Stephen Powis <sp...@salesforce.com>
> wrote:
>
> > I could be totally wrong, but I seem to recall that delete wasn't fully
> > implemented in 0.8.x?
> >
> > On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd
> <chaferd@groupon.com.invalid
> > >
> > wrote:
> >
> > > Chris,
> > >
> > > You could first check to make sure that delete.topic.enable is true and
> > try
> > > deleting again if not.  If that doesn't work with 0.8.1.1 you might
> need
> > to
> > > manually remove the topic's log files from the configured log.dirs
> folder
> > > on each broker in addition to removing the topic's zookeeper path.
> > >
> > > Carl
> > >
> > > On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal <cw...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have a weird situation here.  I have deleted a few topics on my
> > 0.8.1.1
> > > > cluster (old, I know...).  The deletes succeeded according to the
> > > > controller.log:
> > > >
> > > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on Controller 1]:
> > New
> > > > topics: [Set()], deleted topics:
> > > > [Set(perf_doorway-supplier-adapter-uat_raw)], new partition replica
> > > > assignment [Map()]
> > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on Controller 1]:
> > New
> > > > topics: [Set()], deleted topics:
> > > > [Set(perf_doorway-supplier-scheduler-uat_raw)], new partition
> replica
> > > > assignment [Map()]
> > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on Controller 1]:
> > New
> > > > topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)], new
> > > > partition replica assignment [Map()]
> > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on Controller 1]:
> > New
> > > > topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)], new
> > > > partition replica assignment [Map()]
> > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > >
> > > > I query Zookeeper and the path is not there under /brokers/topics as
> > > well.
> > > >
> > > > But, one of the nodes in my cluster continues to try and use them:
> > > >
> > > > [2017-07-20 17:04:36,723] ERROR Conditional update of path
> > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
> > > partitions/3/state
> > > > with data
> > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > > epoch":2,"isr":[1,0]}
> > > > and expected version 69 failed due to
> > > > org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode
> > =
> > > > NoNode for
> > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
> > > partitions/3/state
> > > > (kafka.utils.ZkUtils$)
> > > > [2017-07-20 17:04:36,723] INFO Partition
> > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached
> > > zkVersion
> > > > [69] not equal to that in zookeeper, skip updating ISR
> > > > (kafka.cluster.Partition)
> > > > [2017-07-20 17:04:36,723] INFO Partition
> > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached
> > > zkVersion
> > > > [69] not equal to that in zookeeper, skip updating ISR
> > > > (kafka.cluster.Partition)
> > > > [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2]
> on
> > > > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2]
> from
> > > 1,0
> > > > to 1 (kafka.cluster.Partition)
> > > > [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2]
> on
> > > > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2]
> from
> > > 1,0
> > > > to 1 (kafka.cluster.Partition)
> > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path
> > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data
> > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > epoch":2,"isr":[1]}
> > > > and expected version 70 failed due to
> > > > org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode
> > =
> > > > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/
> 2/state
> > > > (kafka.utils.ZkUtils$)
> > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path
> > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data
> > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > epoch":2,"isr":[1]}
> > > > and expected version 70 failed due to
> > > > org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode
> > =
> > > > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/
> 2/state
> > > > (kafka.utils.ZkUtils$)
> > > > [2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2]
> on
> > > > broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip
> > > > updating ISR (kafka.cluster.Partition)
> > > > [2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2]
> on
> > > > broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip
> > > > updating ISR (kafka.cluster.Partition)
> > > > [2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1]
> on
> > > > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from
> > 1,0
> > > > to 1 (kafka.cluster.Partition)
> > > > [2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1]
> on
> > > > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from
> > 1,0
> > > > to 1 (kafka.cluster.Partition)
> > > > [2017-07-20 17:04:36,988] ERROR Conditional update of path
> > > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with data
> > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > epoch":4,"isr":[1]}
> > > > and expected version 90 failed due to
> > > > org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode
> > =
> > > > NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
> > > > (kafka.utils.ZkUtils$)
> > > > [2017-07-20 17:04:36,988] ERROR Conditional update of path
> > > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with data
> > > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > epoch":4,"isr":[1]}
> > > > and expected version 90 failed due to
> > > > org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode
> > =
> > > > NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
> > > > (kafka.utils.ZkUtils$)
> > > > [2017-07-20 17:04:36,988] INFO Partition [perf_gocontent-uat_raw,1]
> on
> > > > broker 1: Cached zkVersion [90] not equal to that in zookeeper, skip
> > > > updating ISR (kafka.cluster.Partition)
> > > > [2017-07-20 17:04:36,988] INFO Partition [perf_gocontent-uat_raw,1]
> on
> > > > broker 1: Cached zkVersion [90] not equal to that in zookeeper, skip
> > > > updating ISR (kafka.cluster.Partition)
> > > >
> > > > I've tried a rolling restart of the cluster to see if that fixed it,
> > but
> > > it
> > > > did not.
> > > >
> > > > Can someone please help me out here?  I'm not sure how I can get
> things
> > > > back in sync.
> > > >
> > > > Thank you so much for your time.
> > > > Chris
> > > >
> > >
> >
>

Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

[Chris Neal <cw...@gmail.com>]

Delete is definitely there.  The delete worked fine, based on the fact that
there is nothing in Zookeeper, and that the controller reported that the
delete was successful, it's just something seems to have gotten out of
sync.

delete.topic.enabled is true.  I've successfully deleted topics in the
past, so I know it *should* work. :)

I also had already checked in Zookeeper, and there is no directory for the
topics under /brokers/topics....  Very strange indeed.

If I just remove the log directories from the filesystem, is that enough to
get the broker to stop asking about the topics?  I would guess there would
need to be more than just that, but I could be wrong.

Thanks guys for the suggestions though!

On Thu, Jul 20, 2017 at 8:19 PM, Stephen Powis <sp...@salesforce.com>
wrote:

> I could be totally wrong, but I seem to recall that delete wasn't fully
> implemented in 0.8.x?
>
> On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd <chaferd@groupon.com.invalid
> >
> wrote:
>
> > Chris,
> >
> > You could first check to make sure that delete.topic.enable is true and
> try
> > deleting again if not.  If that doesn't work with 0.8.1.1 you might need
> to
> > manually remove the topic's log files from the configured log.dirs folder
> > on each broker in addition to removing the topic's zookeeper path.
> >
> > Carl
> >
> > On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal <cw...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > I have a weird situation here.  I have deleted a few topics on my
> 0.8.1.1
> > > cluster (old, I know...).  The deletes succeeded according to the
> > > controller.log:
> > >
> > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on Controller 1]:
> New
> > > topics: [Set()], deleted topics:
> > > [Set(perf_doorway-supplier-adapter-uat_raw)], new partition replica
> > > assignment [Map()]
> > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on Controller 1]:
> New
> > > topics: [Set()], deleted topics:
> > > [Set(perf_doorway-supplier-scheduler-uat_raw)], new partition replica
> > > assignment [Map()]
> > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on Controller 1]:
> New
> > > topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)], new
> > > partition replica assignment [Map()]
> > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on Controller 1]:
> New
> > > topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)], new
> > > partition replica assignment [Map()]
> > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > >
> > > I query Zookeeper and the path is not there under /brokers/topics as
> > well.
> > >
> > > But, one of the nodes in my cluster continues to try and use them:
> > >
> > > [2017-07-20 17:04:36,723] ERROR Conditional update of path
> > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
> > partitions/3/state
> > > with data
> > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > > epoch":2,"isr":[1,0]}
> > > and expected version 69 failed due to
> > > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode
> =
> > > NoNode for
> > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
> > partitions/3/state
> > > (kafka.utils.ZkUtils$)
> > > [2017-07-20 17:04:36,723] INFO Partition
> > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached
> > zkVersion
> > > [69] not equal to that in zookeeper, skip updating ISR
> > > (kafka.cluster.Partition)
> > > [2017-07-20 17:04:36,723] INFO Partition
> > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached
> > zkVersion
> > > [69] not equal to that in zookeeper, skip updating ISR
> > > (kafka.cluster.Partition)
> > > [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] on
> > > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] from
> > 1,0
> > > to 1 (kafka.cluster.Partition)
> > > [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] on
> > > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] from
> > 1,0
> > > to 1 (kafka.cluster.Partition)
> > > [2017-07-20 17:04:36,765] ERROR Conditional update of path
> > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data
> > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > epoch":2,"isr":[1]}
> > > and expected version 70 failed due to
> > > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode
> =
> > > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/2/state
> > > (kafka.utils.ZkUtils$)
> > > [2017-07-20 17:04:36,765] ERROR Conditional update of path
> > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data
> > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > epoch":2,"isr":[1]}
> > > and expected version 70 failed due to
> > > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode
> =
> > > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/2/state
> > > (kafka.utils.ZkUtils$)
> > > [2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2] on
> > > broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip
> > > updating ISR (kafka.cluster.Partition)
> > > [2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2] on
> > > broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip
> > > updating ISR (kafka.cluster.Partition)
> > > [2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1] on
> > > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from
> 1,0
> > > to 1 (kafka.cluster.Partition)
> > > [2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1] on
> > > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from
> 1,0
> > > to 1 (kafka.cluster.Partition)
> > > [2017-07-20 17:04:36,988] ERROR Conditional update of path
> > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with data
> > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > epoch":4,"isr":[1]}
> > > and expected version 90 failed due to
> > > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode
> =
> > > NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
> > > (kafka.utils.ZkUtils$)
> > > [2017-07-20 17:04:36,988] ERROR Conditional update of path
> > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with data
> > > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > epoch":4,"isr":[1]}
> > > and expected version 90 failed due to
> > > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode
> =
> > > NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
> > > (kafka.utils.ZkUtils$)
> > > [2017-07-20 17:04:36,988] INFO Partition [perf_gocontent-uat_raw,1] on
> > > broker 1: Cached zkVersion [90] not equal to that in zookeeper, skip
> > > updating ISR (kafka.cluster.Partition)
> > > [2017-07-20 17:04:36,988] INFO Partition [perf_gocontent-uat_raw,1] on
> > > broker 1: Cached zkVersion [90] not equal to that in zookeeper, skip
> > > updating ISR (kafka.cluster.Partition)
> > >
> > > I've tried a rolling restart of the cluster to see if that fixed it,
> but
> > it
> > > did not.
> > >
> > > Can someone please help me out here?  I'm not sure how I can get things
> > > back in sync.
> > >
> > > Thank you so much for your time.
> > > Chris
> > >
> >
>

Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

[Stephen Powis <sp...@salesforce.com>]

I could be totally wrong, but I seem to recall that delete wasn't fully
implemented in 0.8.x?

On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd <ch...@groupon.com.invalid>
wrote:

> Chris,
>
> You could first check to make sure that delete.topic.enable is true and try
> deleting again if not.  If that doesn't work with 0.8.1.1 you might need to
> manually remove the topic's log files from the configured log.dirs folder
> on each broker in addition to removing the topic's zookeeper path.
>
> Carl
>
> On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal <cw...@gmail.com> wrote:
>
> > Hi all,
> >
> > I have a weird situation here.  I have deleted a few topics on my 0.8.1.1
> > cluster (old, I know...).  The deletes succeeded according to the
> > controller.log:
> >
> > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on Controller 1]: New
> > topics: [Set()], deleted topics:
> > [Set(perf_doorway-supplier-adapter-uat_raw)], new partition replica
> > assignment [Map()]
> > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on Controller 1]: New
> > topics: [Set()], deleted topics:
> > [Set(perf_doorway-supplier-scheduler-uat_raw)], new partition replica
> > assignment [Map()]
> > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on Controller 1]: New
> > topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)], new
> > partition replica assignment [Map()]
> > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on Controller 1]: New
> > topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)], new
> > partition replica assignment [Map()]
> > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> >
> > I query Zookeeper and the path is not there under /brokers/topics as
> well.
> >
> > But, one of the nodes in my cluster continues to try and use them:
> >
> > [2017-07-20 17:04:36,723] ERROR Conditional update of path
> > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
> partitions/3/state
> > with data
> > {"controller_epoch":34,"leader":1,"version":1,"leader_
> > epoch":2,"isr":[1,0]}
> > and expected version 69 failed due to
> > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> > NoNode for
> > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/
> partitions/3/state
> > (kafka.utils.ZkUtils$)
> > [2017-07-20 17:04:36,723] INFO Partition
> > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached
> zkVersion
> > [69] not equal to that in zookeeper, skip updating ISR
> > (kafka.cluster.Partition)
> > [2017-07-20 17:04:36,723] INFO Partition
> > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached
> zkVersion
> > [69] not equal to that in zookeeper, skip updating ISR
> > (kafka.cluster.Partition)
> > [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] on
> > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] from
> 1,0
> > to 1 (kafka.cluster.Partition)
> > [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] on
> > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] from
> 1,0
> > to 1 (kafka.cluster.Partition)
> > [2017-07-20 17:04:36,765] ERROR Conditional update of path
> > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data
> > {"controller_epoch":34,"leader":1,"version":1,"leader_
> epoch":2,"isr":[1]}
> > and expected version 70 failed due to
> > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/2/state
> > (kafka.utils.ZkUtils$)
> > [2017-07-20 17:04:36,765] ERROR Conditional update of path
> > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data
> > {"controller_epoch":34,"leader":1,"version":1,"leader_
> epoch":2,"isr":[1]}
> > and expected version 70 failed due to
> > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/2/state
> > (kafka.utils.ZkUtils$)
> > [2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2] on
> > broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip
> > updating ISR (kafka.cluster.Partition)
> > [2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2] on
> > broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip
> > updating ISR (kafka.cluster.Partition)
> > [2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1] on
> > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from 1,0
> > to 1 (kafka.cluster.Partition)
> > [2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1] on
> > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from 1,0
> > to 1 (kafka.cluster.Partition)
> > [2017-07-20 17:04:36,988] ERROR Conditional update of path
> > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with data
> > {"controller_epoch":34,"leader":1,"version":1,"leader_
> epoch":4,"isr":[1]}
> > and expected version 90 failed due to
> > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> > NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
> > (kafka.utils.ZkUtils$)
> > [2017-07-20 17:04:36,988] ERROR Conditional update of path
> > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with data
> > {"controller_epoch":34,"leader":1,"version":1,"leader_
> epoch":4,"isr":[1]}
> > and expected version 90 failed due to
> > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> > NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
> > (kafka.utils.ZkUtils$)
> > [2017-07-20 17:04:36,988] INFO Partition [perf_gocontent-uat_raw,1] on
> > broker 1: Cached zkVersion [90] not equal to that in zookeeper, skip
> > updating ISR (kafka.cluster.Partition)
> > [2017-07-20 17:04:36,988] INFO Partition [perf_gocontent-uat_raw,1] on
> > broker 1: Cached zkVersion [90] not equal to that in zookeeper, skip
> > updating ISR (kafka.cluster.Partition)
> >
> > I've tried a rolling restart of the cluster to see if that fixed it, but
> it
> > did not.
> >
> > Can someone please help me out here?  I'm not sure how I can get things
> > back in sync.
> >
> > Thank you so much for your time.
> > Chris
> >
>

Re: Help please: Topics deleted, but Kafka continues to try and sync them with Zookeeper

[Carl Haferd <ch...@groupon.com.INVALID>]

Chris,

You could first check to make sure that delete.topic.enable is true and try
deleting again if not.  If that doesn't work with 0.8.1.1 you might need to
manually remove the topic's log files from the configured log.dirs folder
on each broker in addition to removing the topic's zookeeper path.

Carl

On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal <cw...@gmail.com> wrote:

> Hi all,
>
> I have a weird situation here.  I have deleted a few topics on my 0.8.1.1
> cluster (old, I know...).  The deletes succeeded according to the
> controller.log:
>
> [2017-07-20 16:40:31,175] INFO [TopicChangeListener on Controller 1]: New
> topics: [Set()], deleted topics:
> [Set(perf_doorway-supplier-adapter-uat_raw)], new partition replica
> assignment [Map()]
> (kafka.controller.PartitionStateMachine$TopicChangeListener)
> [2017-07-20 16:40:33,507] INFO [TopicChangeListener on Controller 1]: New
> topics: [Set()], deleted topics:
> [Set(perf_doorway-supplier-scheduler-uat_raw)], new partition replica
> assignment [Map()]
> (kafka.controller.PartitionStateMachine$TopicChangeListener)
> [2017-07-20 16:40:36,504] INFO [TopicChangeListener on Controller 1]: New
> topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)], new
> partition replica assignment [Map()]
> (kafka.controller.PartitionStateMachine$TopicChangeListener)
> [2017-07-20 16:40:38,290] INFO [TopicChangeListener on Controller 1]: New
> topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)], new
> partition replica assignment [Map()]
> (kafka.controller.PartitionStateMachine$TopicChangeListener)
>
> I query Zookeeper and the path is not there under /brokers/topics as well.
>
> But, one of the nodes in my cluster continues to try and use them:
>
> [2017-07-20 17:04:36,723] ERROR Conditional update of path
> /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/partitions/3/state
> with data
> {"controller_epoch":34,"leader":1,"version":1,"leader_
> epoch":2,"isr":[1,0]}
> and expected version 69 failed due to
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for
> /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/partitions/3/state
> (kafka.utils.ZkUtils$)
> [2017-07-20 17:04:36,723] INFO Partition
> [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached zkVersion
> [69] not equal to that in zookeeper, skip updating ISR
> (kafka.cluster.Partition)
> [2017-07-20 17:04:36,723] INFO Partition
> [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached zkVersion
> [69] not equal to that in zookeeper, skip updating ISR
> (kafka.cluster.Partition)
> [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] on
> broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] from 1,0
> to 1 (kafka.cluster.Partition)
> [2017-07-20 17:04:36,764] INFO Partition [perf_goplatform-uat_raw,2] on
> broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] from 1,0
> to 1 (kafka.cluster.Partition)
> [2017-07-20 17:04:36,765] ERROR Conditional update of path
> /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data
> {"controller_epoch":34,"leader":1,"version":1,"leader_epoch":2,"isr":[1]}
> and expected version 70 failed due to
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/2/state
> (kafka.utils.ZkUtils$)
> [2017-07-20 17:04:36,765] ERROR Conditional update of path
> /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with data
> {"controller_epoch":34,"leader":1,"version":1,"leader_epoch":2,"isr":[1]}
> and expected version 70 failed due to
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/2/state
> (kafka.utils.ZkUtils$)
> [2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2] on
> broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip
> updating ISR (kafka.cluster.Partition)
> [2017-07-20 17:04:36,765] INFO Partition [perf_goplatform-uat_raw,2] on
> broker 1: Cached zkVersion [70] not equal to that in zookeeper, skip
> updating ISR (kafka.cluster.Partition)
> [2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1] on
> broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from 1,0
> to 1 (kafka.cluster.Partition)
> [2017-07-20 17:04:36,981] INFO Partition [perf_gocontent-uat_raw,1] on
> broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] from 1,0
> to 1 (kafka.cluster.Partition)
> [2017-07-20 17:04:36,988] ERROR Conditional update of path
> /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with data
> {"controller_epoch":34,"leader":1,"version":1,"leader_epoch":4,"isr":[1]}
> and expected version 90 failed due to
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
> (kafka.utils.ZkUtils$)
> [2017-07-20 17:04:36,988] ERROR Conditional update of path
> /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with data
> {"controller_epoch":34,"leader":1,"version":1,"leader_epoch":4,"isr":[1]}
> and expected version 90 failed due to
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for /brokers/topics/perf_gocontent-uat_raw/partitions/1/state
> (kafka.utils.ZkUtils$)
> [2017-07-20 17:04:36,988] INFO Partition [perf_gocontent-uat_raw,1] on
> broker 1: Cached zkVersion [90] not equal to that in zookeeper, skip
> updating ISR (kafka.cluster.Partition)
> [2017-07-20 17:04:36,988] INFO Partition [perf_gocontent-uat_raw,1] on
> broker 1: Cached zkVersion [90] not equal to that in zookeeper, skip
> updating ISR (kafka.cluster.Partition)
>
> I've tried a rolling restart of the cluster to see if that fixed it, but it
> did not.
>
> Can someone please help me out here?  I'm not sure how I can get things
> back in sync.
>
> Thank you so much for your time.
> Chris
>