Does subscription by pattern work properly?

[Skip Montanaro <sk...@gmail.com>]

I'm running kafka_2.11-1.0.0 on my (test) server and version 1.3.5 of
the kafka-python client library (available from PyPI). Subscribing
using patterns seems not to work. I always have to enumerate a set of
topics, and given that the topics are created dynamically as different
kinds of content arrive at my virtual front door, I'm guaranteed to
eventually miss something.

I don't know enough about the ins and outs of Kafka to know where to
look for clues. I tried tailing server.log.2018-01-03-11 for a bit,
but saw no obvious indication that it logs anything. (Maybe I need to
crank up the verbosity?)

As an example, this works:

import kafka
consumer = kafka.KafkaConsumer(bootstrap_servers="server:9092",
group_id="skip-test")
consumer.subscribe(topics=["topic1", "topic2", ...])
consumer.next()

but this fails, the next() call hanging indefinitely:

import kafka
consumer = kafka.KafkaConsumer(bootstrap_servers="server:9092",
group_id="skip-test")
consumer.subscribe(pattern="^topic")
consumer.next()

The only way I got pattern subscription to work is by anchoring the
tail end of the pattern. This wasn't sufficient:

^topic.*

I needed to add the $ as well so the pattern was anchored at both ends:

^topic.*$

I don't remember seeing any sort of pattern restrictions in my reading
of the docs. Shouldn't any old regular expression do the trick?

Any clues/documentation pointers appreciated.

Skip Montanaro