Kafka connector

[Tushar Sudhakar Jee <tu...@levyx.com>]

Hello Sir/Ma'am,
I was trying to write a simple case of using kafka connector. My setup
involves using three nodes N1,N2 and N3.
N1 is the source and N2, N3 are the sink nodes in my case.
I am writing data to a text file(say input.txt) on Node N1 and using the
standalone kafka connector I wish to see a text file with content similar
to input.txt on the nodes N2 and N3.

I am using the REST API to make changes in* topic name, file name and
tasks.max*.

However, during the experiments I ran I was unable to get a complete copy
of the input.txt on both nodes(N2 and N3) at the *same time. *
*Also tuning the value of tasks.max on nodes (N2 and N3) for the sink
connector decided on which node data would be sent. *

So, my question is whether I am wrong in expecting such an outcome?
If so then what should I be expecting as a result of the experiment?
If not then how do I get my desired outcome?


Regards,

-- 

*Tushar Sudhakar Jee *| Software Engineer

c *424.535.8225* | tushar@levyx.com <wi...@levyx.com>

[image: Levyx-Logo-Final 9%]

49 Discovery, Suite #220

Irvine, CA 92618

*www.levyx.com <http://www.levyx.com/>*

-- 


Levyx | 49 Discovery, Suite #220 | Irvine, CA 92618 | www.levyx.com

This email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please let us know by e-mail reply 
and delete it from your system; copying this message or disclosing its 
contents to anyone is strictly prohibited.

Re: Kafka connector

[Hans Jespersen <ha...@confluent.io>]

If you want both N2 and N3 to get all the same messages (rather than each
getting an exclusive partitioned subset of the data) then you need to
configure N2 and N3 to be in unique Kafka consumer groups which I believe
is driven off the "name" of the N2 and N3 connectors. Make sure N2 and N3
have different names.

-hans

/**
 * Hans Jespersen, Principal Systems Engineer, Confluent Inc.
 * hans@confluent.io (650)924-2670
 */

On Thu, Apr 6, 2017 at 4:26 PM, Tushar Sudhakar Jee <tu...@levyx.com>
wrote:

> Hello Sir/Ma'am,
> I was trying to write a simple case of using kafka connector. My setup
> involves using three nodes N1,N2 and N3.
> N1 is the source and N2, N3 are the sink nodes in my case.
> I am writing data to a text file(say input.txt) on Node N1 and using the
> standalone kafka connector I wish to see a text file with content similar
> to input.txt on the nodes N2 and N3.
>
> I am using the REST API to make changes in* topic name, file name and
> tasks.max*.
>
> However, during the experiments I ran I was unable to get a complete copy
> of the input.txt on both nodes(N2 and N3) at the *same time. *
> *Also tuning the value of tasks.max on nodes (N2 and N3) for the sink
> connector decided on which node data would be sent. *
>
> So, my question is whether I am wrong in expecting such an outcome?
> If so then what should I be expecting as a result of the experiment?
> If not then how do I get my desired outcome?
>
>
> Regards,
>
> --
>
> *Tushar Sudhakar Jee *| Software Engineer
>
> c *424.535.8225 <(424)%20535-8225>* | tushar@levyx.com <wi...@levyx.com>
>
> [image: Levyx-Logo-Final 9%]
>
> 49 Discovery, Suite #220
>
> Irvine, CA 92618
>
> *www.levyx.com <http://www.levyx.com/>*
>
>
> Levyx | 49 Discovery, Suite #220 | Irvine, CA 92618 | www.levyx.com
>
> This email and any files transmitted with it are confidential and intended
> solely for the use of the individual or entity to whom they are addressed.
> If you have received this email in error please let us know by e-mail
> reply and delete it from your system; copying this message or disclosing
> its contents to anyone is strictly prohibited.
>