Kafka cluster broken / com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath

[Charles Bueche <cb...@gmail.com>]

Hello all,

[sorry if this is a duplicate, not sure why my 1st attempt didn't come
through]

I do have two Kafka clusters in action, test and prod. The two are
formed by 3 nodes each, are independent and run their own zookeeper
setups. My prod cluster is running fine. My test cluster is half-broken
and I'm struggling to fix it. I could wipe it but I prefer to understand
what's wrong and fix it.

I'm not sure what broke my test cluster. I had several network
disconnections / split-brains but Kafka always recovered fine. The
reasons for the network issues are independent and still being
investigated (layer-2 storms, etc).

So I upgraded my zookeeper and kafka to the latest versions and when
trying to rebalance a topic across brokers I started to notice the
problems. Not sure really when they started, before or after the upgrade.

I ran the upgrade as for the official doc (rolling upgrade, moving up
inter.broker.protocol.version and log.message.format.version gradually).

------------------------------------------------------
cat /etc/os-release
NAME="SLES"
VERSION="12-SP4"
VERSION_ID="12.4"
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP4"
ID="sles"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:12:sp4"

------------------------------------------------------
versions :
- zookeeper 3.5.6
- kafka 2.12-2.3.1

------------------------------------------------------

many rapid log entries on brokers 1 & 2 (we have 1, 2, 3)

[2019-12-05 09:56:54,967] ERROR [KafkaApi-1] Number of alive brokers '0'
does not meet the required replication factor '1' for the offsets topic
(configured via 'offsets.topic.replication.factor'). This error can be
ignored if the cluster is starting up and not all brokers are up yet.
(kafka.server.KafkaApis)

------------------------------------------------------
java is java-1_8_0-openjdk-1.8.0.222-27.35.2.x86_64 from SLES. I have
tried Oracle Java jdk1.8.0_231 with the same issue.

------------------------------------------------------
when trying to see a reassignment I have this very suspect error :

root@vmgato701a01:/appl/kafka/bin # ./kafka-reassign-partitions.sh
--zookeeper $ZKLIST --reassignment-json-file /tmp/r7.json --verify

Status of partition reassignment:
Partitions reassignment failed due to
com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer;
java.lang.AbstractMethodError:
com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer;
        at
com.fasterxml.jackson.databind.ext.OptionalHandlerFactory.findDeserializer(OptionalHandlerFactory.java:122)
        at
com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findOptionalStdDeserializer(BasicDeserializerFactory.java:1589)
        at
com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findDefaultDeserializer(BasicDeserializerFactory.java:1812)
        at
com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.findStdDeserializer(BeanDeserializerFactory.java:161)
        at
com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:125)
        at
com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:411)
        at
com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
        at
com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
        at
com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
        at
com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
        at
com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:477)
        at
com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4178)
        at
com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3997)
        at
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3079)
        at kafka.utils.Json$.parseBytesAs(Json.scala:73)
        at kafka.zk.ReassignPartitionsZNode$.decode(ZkData.scala:407)
        at
kafka.zk.KafkaZkClient.getPartitionReassignment(KafkaZkClient.scala:795)
        at
kafka.admin.ReassignPartitionsCommand$.checkIfPartitionReassignmentSucceeded(ReassignPartitionsCommand.scala:355)
        at
kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:97)
        at
kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:90)
        at
kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:61)
        at
kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)

Help appreciated.
Charles