Replication Issue with Samza / Kafka

[Louis Calisi <lc...@tripadvisor.com>]

Hi:

I’ve been a quite fan of Kafka / Samza for about a year now. The project I’ve been working for TripAdvisor is about to soft launch soon. Lately we’ve noticed replication error coming from Kafka (0.8.2.0). Samza is exclusively using this Kafka cluster, minus a small application that’s feeding it data. The version of Samza is 0.9.1.

It feels like we’re stumbling on the same issue described here: https://issues.apache.org/jira/browse/KAFKA-2143. I attempted to apply the patch but it was written for a version of Kafka greater than then one we’re using (and recommended by Samza).

Here’s an example:

2015-12-18 02:31:35,573 ERROR kafka.server.ReplicaManager: [Replica Manager on Broker 230]: Error when processing fetch request for partition [Stream_<label>_Prod,9] offset 78214693 from consumer with correlation id 114263. Possible cause: Request for offset 78214693 but we only have log segments in the range 79642852 to 128218996.

We don’t see any errors in the Samza logs till much later.

2015-12-18 04:56:35 ERROR consumer.ConsumerIterator:97 - consumed offset: 76451107 doesn't match fetch offset: 76741109 for Stream_<label>_Prod:6: fetched offset = 76741629: consumed offset = 76451107; Consumer may lose data

We have 3 Kafka nodes and 5 samza nodes. Topics have a 2X replication. There are a total of 60 topics. 10 of these topics carry the bulk of the data.

I’m charting Kafka’s under replicated partitions graph and it  show only 4 partitions max are under replicated. This happens briefly when we’re flushing out stale data.

Ironically, the replication errors seem to happen when the system is under low activity. Our application is very bursty. Our front end servers roll logs every hour. Yesterday for example we were replaying old logs at full speed for 8 hours without issues. Once we caught up and the application went into "burst mode” then we started to see the errors.

Has anyone seen issues like this? I checked the samza mailing list for the past 3 months and didn’t see anything indicating it. I’m hoping it’s the way we configured Kafka? Our settings are below. Thanks in advance for any help.

Regards, Louis Calisi


auto.create.topics.enable=true
auto.leader.rebalance.enable=true
broker.id=230
controlled.shutdown.enable=true
default.replication.factor=2
delete.topic.enable=true
jmx_port=XXXX
kafka.http.metrics.host=0.0.0.0
kafka.http.metrics.port=XXXXX
kafka.log4j.dir=/var/log/kafka
log.dirs=/data/2/kafka/data,/data/3/kafka/data,/data/4/kafka/data,/data/5/kafka/data,/data/6/kafka/data,/data/7/kafka/data,/data/8/kafka/data,/data/9/kafka/data,/data/10/kafka/data,/data/11/kafka/data,/data/12/kafka/data
log.retention.bytes=-1
log.retention.hours=12
log.roll.hours=24
log.segment.bytes=1073741824
max.connections.per.ip=400
message.max.bytes=1000000
min.insync.replicas=2
num.partitions=1
port=XXXX
replica.fetch.max.bytes=1048576
replica.lag.max.messages=4000
unclean.leader.election.enable=false
zookeeper.session.timeout.ms=30000
num.io.threads=12
num.network.threads=8
log.cleaner.enable=true
log.cleaner.threads=3
offsets.storage=kafka
dual.commit.enabled=false
zookeeper.connect=…..
kafka.metrics.reporters=nl.techop.kafka.KafkaHttpMetricsReporter