0.9 KafkaConsumer Memory Usage

[noah <ia...@gmail.com>]

I'm using 0.9.0.1 consumers on 0.9.0.1 brokers. In a single Java service,
we have 4 producers and 5 consumers. They are all KafkaProducer and
KafkaConsumer instances (the new consumer.)

Since the 0.9 upgrade, this service is now OOMing after a being up for a
few minutes. Heap dumps show >80MB of objects related to topic and
partition metadata (hundreds of thousands of org.apache.kafka.common.Node
objects.)

Digging through the heap, I see references to all sorts of topics in the
cluster that this service is not producing to or consuming from. We are not
using any pattern matching on topics.

Unusual things about this service and cluster:
* Smallish heap (was fine with 128mb on the 0.8 consumer, but even bumping
up to 256mb we still OOM on 0.9)
* We have a large number of dev/test topics on this cluster (several
hundred) and thousands of partitions as a result. The service is only
concerned with ~15 of these topics.

It seems pretty likely that our issue is that metadata for all partitions
is being cached locally. Is there a way to keep that from happening? Or are
there other known memory leak issues that could be causing us to OOM?