Limits of REST interface to kafka

[Tim Smith <se...@gmail.com>]

At the outset, this isn't about challenging the work that has been done,
primarily by folks @ Confluence for wrapping kafka in a REST API. Clearly,
there is a use case for a REST Service and they rose up to the challenge.

That said, I am trying to evaluate the limitations of a REST service around
kafka, if any, specifically for a Spark Streaming app as a consumer?

The reason I ask because I have been asked to move my Spark Streaming app,
that's a native kafka consumer, to a REST service that provides security
(auth/encryption). And, I am trying to figure out what requirements should
I place on the REST service if I am give up my direct access to the kafka
cluster?

With Spark Streaming, what I have learned is that you need to maintain a
certain ratio between Spark executors and kafka partitions to balance the
processing load evenly on the app. The consumer group re-balancing in kafka
works really well to transition a load off a dead Spark executor to an idle
one. Not sure, how these features and other features that I use almost
unknowingly in kafka translate to a REST service like the kafka-rest
provides?

More broadly, are there features in kafka that inherently run into
limitations of the stateless/REST model?

Thanks,

Tim