Distribute data load across tasks in a connector

[Sreejith S <sr...@gmail.com>]

Hi All,

I understand that Kafka connect is not meant for static/batch processing
scenarios.

I am trying to accommodate static/batch load also with Kafka so that i dont
want to introduce additional layers in my architecture for doing batch.

I created a FTP connector , and it works well with small files. But the
moment when i have large files in FTP server, the classic problem of Out of
Memory Java Heap is coming.

My question is , is there any way in connect framework to distribute the
data load across tasks ?

Any pointers on this please ?

Thank you

Sreejith.