kafka_2.10-0.8.1.1:: Need help regarding a split configuration situation between kafka brokers and topic metadata stored in zookeeper

[Jagbir Hooda <jh...@gmail.com>]

Hi there,

Wondering if anyone ran into a unique situation where zookeeper seems
to have the topic metadata, but broker doesn't have the corresponding
log file

Below is what we noticed in zookeeper:

------------------------------------------------------------------
kafka@kafka-3:~$ /opt/kafka/kafka_2.10-0.8.1.1/bin/zookeeper-shell.sh
localhost:2181 get /brokers/topics/T_60036/partitions/0/state
Connecting to localhost:2181

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
{"controller_epoch":6,"leader":1,"version":1,"leader_epoch":0,"isr":[1,2]}
cZxid = 0x80013308c
ctime = Wed Jun 06 04:55:37 UTC 2018
mZxid = 0x80013308c
mtime = Wed Jun 06 04:55:37 UTC 2018
pZxid = 0x80013308c
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 74
numChildren = 0

kafka@kafka-3:~$ /opt/kafka/kafka_2.10-0.8.1.1/bin/zookeeper-shell.sh
localhost:2181 get /config/topics/T_60036
Connecting to localhost:2181

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
{"version":1,"config":{}}
cZxid = 0x800132992
ctime = Wed Jun 06 04:55:13 UTC 2018
mZxid = 0x800132992
mtime = Wed Jun 06 04:55:13 UTC 2018
pZxid = 0x800132992
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 25
numChildren = 0
======================================

But there are no log files for this topic:

------------------------------------------------------------------
kafka@kafka-3:~$ ls -l /var/kafka/topics/T_60036*
ls: cannot access /var/kafka/topics/T_60036*: No such file or directory
======================================

Since in this version of kafka there is no topic deletion utility,
will it be OK to delete zookeeper entries ("/config/topics/T_60036",
"/brokers/topics/T_60036") from zookeeper without restarting or
jeopardizing the cluster. Right now our producer keeps getting
"kafka.common.FailedToSendMessageException" exception due to above
split configuration.

I really appreciate your help if you can share your insights.

Jsh

Notes:
----------------------------------
Version: kafka_2.10-0.8.1.1
Cluster Configuration: 4 kafka brokers + 4 zookeeper
Topic Partiton: 1
Topic Replica: 1
----------------------------------