Re: Colocating Kafka Connect on Kafka Broker

[Ewen Cheslack-Postava <ew...@confluent.io>]

Generally we discourage colocating services with Kafka. Kafka relies
heavily on the page cache. It's generally light on CPU (except maybe if it
has to recompress messages), but may not play well with other services.

For very light installations, colocating some services (e.g. both ZK and
Kafka), might work fine. For most production settings, you'll want to keep
them separate unless you're *very* sure that they won't interact poorly.

-Ewen

On Wed, Jun 29, 2016 at 1:19 AM, Kristoffer Sjögren <st...@gmail.com>
wrote:

> Hi
>
> We want to use Kafka Connect to copy data to HDFS (using
> kafka-connect-hdfs) in parquet format and was wondering if its a good
> idea to collocate distributed Kafka Connect 1-1 on Kafka Brokers?
>
> Considering the parquet indexing process would steal (a lot of / too
> much?) computing resources from the brokers. The alternative is to
> deploy on separate hardware (1-1?).
>
> Experiences in this area?
>
> Cheers,
> -Kristoffer
>



-- 
Thanks,
Ewen