RE: Pattern to create Task with dependencies (DI)

[Petr Novak <os...@gmail.com>]

Hello,

Nobody has experience with Kafka Connect tasks with external dependencies?

 

Thanks,

Petr

 

From: Petr Novak [mailto:oss.mlists@gmail.com] 
Sent: 23. února 2017 14:48
To: users@kafka.apache.org
Subject: Pattern to create Task with dependencies (DI)

 

Hello,

it seems that KConnect take control over creating task instance and requires
no-arg constructor. What is the recommended pattern when I need to create a
task which has dependency e.g. on some db client and I want to be able to
pass in mock in tests, preferable through constructor?

In Java I probably have to divide actual task implementation into separate
class and use method forwarding from task factory class. I assume that this
factory, whose class would be passed to KConnect, requires to extend
abstract class whose “this” I have to pass to implementing class as another
dependency. I can’t figure out anything less ugly.

 

In Scala it can be done elegantly through mixin composition which is
invisible to KConnect. 

 

Why KConnect does take control from me how tasks are created? Why KConnect
doesn’t accept factory class on which it would call no-arg create method
which would return instance of my task so that I can control how it is
created and which dependencies are created. Or instead of getTaskClass() to
have createTask() on Connector which returns task instance.

 

Many thanks for advice,

Petr

Re: Pattern to create Task with dependencies (DI)

[Mathieu Fenniak <ma...@replicon.com>]

Hey Petr,

I have the same issue.  But I just cope with it; I wire up default
dependencies directly in the connector and task constructors, expose them
through properties, and modify them to refer to mocks in my unit tests.

It's not a great approach, but it is simple.

Why KConnect does take control from me how tasks are created?  ... Or
> instead of getTaskClass() to have createTask() on Connector which returns
> task instance.


The problem is that in distributed mode, Kafka Connect will be creating
connectors on "arbitrary" nodes, and then very likely creating tasks on
nodes where the connector is not instantiated.  This is my understanding of
why Kafka Connect takes the control away from you.

I'm not saying it can't be done better, but it's definitely not without
reason. :-)

Mathieu


On Mon, Mar 13, 2017 at 4:42 AM, Petr Novak <os...@gmail.com> wrote:

> Hello,
>
> Nobody has experience with Kafka Connect tasks with external dependencies?
>
>
>
> Thanks,
>
> Petr
>
>
>
> From: Petr Novak [mailto:oss.mlists@gmail.com]
> Sent: 23. února 2017 14:48
> To: users@kafka.apache.org
> Subject: Pattern to create Task with dependencies (DI)
>
>
>
> Hello,
>
> it seems that KConnect take control over creating task instance and
> requires
> no-arg constructor. What is the recommended pattern when I need to create a
> task which has dependency e.g. on some db client and I want to be able to
> pass in mock in tests, preferable through constructor?
>
> In Java I probably have to divide actual task implementation into separate
> class and use method forwarding from task factory class. I assume that this
> factory, whose class would be passed to KConnect, requires to extend
> abstract class whose “this” I have to pass to implementing class as another
> dependency. I can’t figure out anything less ugly.
>
>
>
> In Scala it can be done elegantly through mixin composition which is
> invisible to KConnect.
>
>
>
> Why KConnect does take control from me how tasks are created? Why KConnect
> doesn’t accept factory class on which it would call no-arg create method
> which would return instance of my task so that I can control how it is
> created and which dependencies are created. Or instead of getTaskClass() to
> have createTask() on Connector which returns task instance.
>
>
>
> Many thanks for advice,
>
> Petr
>
>