You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by dhanuka ranasinghe <dh...@gmail.com> on 2016/08/31 15:04:54 UTC
Kafka Source Connector tasks run only in same node
Hi All,
I'm new to Kafka and I created TCP Kafka Source connector [1] and deployed
in Kafka Connect Cluster in distributed mode. There is two node/instance of
connectors in Kafka Connector cluster.
But out of them only in one node run *tasks* even though it shows
connector in both nodes active and running status.
In here I have used RxNetty server, so what happened is netty server up in
one node/host but tasks are running in other node/host
Could you please let me know the issue with this? Thanks in advance.
node==host
[1] https://github.com/dhanuka84/kafka-connect-tcp
cheers,
Dhanuka
--
Nothing Impossible,Creativity is more important than knowledge.
Re: Kafka Source Connector tasks run only in same node
Posted by Shikhar Bhushan <sh...@confluent.io>.
Hi Dhanuka,
For something like a TCP connector where you have a dependency on the
placement, the standalone mode makes more sense. With distributed mode you
can't have guarantees of what instances your connector's tasks will be
distributed to.
Best,
Shikhar
On Wed, Aug 31, 2016 at 8:05 AM dhanuka ranasinghe <
dhanuka.priyanath@gmail.com> wrote:
> Hi All,
>
> I'm new to Kafka and I created TCP Kafka Source connector [1] and deployed
> in Kafka Connect Cluster in distributed mode. There is two node/instance of
> connectors in Kafka Connector cluster.
>
> But out of them only in one node run *tasks* even though it shows
> connector in both nodes active and running status.
>
> In here I have used RxNetty server, so what happened is netty server up in
> one node/host but tasks are running in other node/host
>
> Could you please let me know the issue with this? Thanks in advance.
>
> node==host
>
>
> [1] https://github.com/dhanuka84/kafka-connect-tcp
>
> cheers,
> Dhanuka
>
> --
> Nothing Impossible,Creativity is more important than knowledge.
>
Re: Kafka Source Connector tasks run only in same node
Posted by dhanuka ranasinghe <dh...@gmail.com>.
Please note *zookeeper* running on only in *10.72.133.195
<http://10.72.133.195:8084>*
On Thu, Sep 1, 2016 at 9:49 AM, dhanuka ranasinghe <
dhanuka.priyanath@gmail.com> wrote:
> As you can see, connector/worker and task run in different hosts
>
> On Thu, Sep 1, 2016 at 9:48 AM, dhanuka ranasinghe <
> dhanuka.priyanath@gmail.com> wrote:
>
>> kafka connector status
>>
>> {"name":"socket-connector","connector":{"state":"RUNNING","worker_id":"*10.72.133.194:8084
>> <http://10.72.133.194:8084>*"},"tasks":[{"state":"RUNNING"
>> ,"id":0,"worker_id":"*10.72.133.195:8084 <http://10.72.133.195:8084>*"}]}
>>
>> On Wed, Aug 31, 2016 at 8:34 PM, dhanuka ranasinghe <
>> dhanuka.priyanath@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> I'm new to Kafka and I created TCP Kafka Source connector [1] and
>>> deployed in Kafka Connect Cluster in distributed mode. There is two
>>> node/instance of connectors in Kafka Connector cluster.
>>>
>>> But out of them only in one node run *tasks* even though it shows
>>> connector in both nodes active and running status.
>>>
>>> In here I have used RxNetty server, so what happened is netty server up
>>> in one node/host but tasks are running in other node/host
>>>
>>> Could you please let me know the issue with this? Thanks in advance.
>>>
>>> node==host
>>>
>>>
>>> [1] https://github.com/dhanuka84/kafka-connect-tcp
>>>
>>> cheers,
>>> Dhanuka
>>>
>>> --
>>> Nothing Impossible,Creativity is more important than knowledge.
>>>
>>
>>
>>
>> --
>> Nothing Impossible,Creativity is more important than knowledge.
>>
>
>
>
> --
> Nothing Impossible,Creativity is more important than knowledge.
>
--
Nothing Impossible,Creativity is more important than knowledge.
Re: Kafka Source Connector tasks run only in same node
Posted by dhanuka ranasinghe <dh...@gmail.com>.
As you can see, connector/worker and task run in different hosts
On Thu, Sep 1, 2016 at 9:48 AM, dhanuka ranasinghe <
dhanuka.priyanath@gmail.com> wrote:
> kafka connector status
>
> {"name":"socket-connector","connector":{"state":"RUNNING","worker_id":"*10.72.133.194:8084
> <http://10.72.133.194:8084>*"},"tasks":[{"state":"
> RUNNING","id":0,"worker_id":"*10.72.133.195:8084
> <http://10.72.133.195:8084>*"}]}
>
> On Wed, Aug 31, 2016 at 8:34 PM, dhanuka ranasinghe <
> dhanuka.priyanath@gmail.com> wrote:
>
>> Hi All,
>>
>> I'm new to Kafka and I created TCP Kafka Source connector [1] and
>> deployed in Kafka Connect Cluster in distributed mode. There is two
>> node/instance of connectors in Kafka Connector cluster.
>>
>> But out of them only in one node run *tasks* even though it shows
>> connector in both nodes active and running status.
>>
>> In here I have used RxNetty server, so what happened is netty server up
>> in one node/host but tasks are running in other node/host
>>
>> Could you please let me know the issue with this? Thanks in advance.
>>
>> node==host
>>
>>
>> [1] https://github.com/dhanuka84/kafka-connect-tcp
>>
>> cheers,
>> Dhanuka
>>
>> --
>> Nothing Impossible,Creativity is more important than knowledge.
>>
>
>
>
> --
> Nothing Impossible,Creativity is more important than knowledge.
>
--
Nothing Impossible,Creativity is more important than knowledge.
Re: Kafka Source Connector tasks run only in same node
Posted by dhanuka ranasinghe <dh...@gmail.com>.
kafka connector status
{"name":"socket-connector","connector":{"state":"RUNNING","worker_id":"*10.72.133.194:8084
<http://10.72.133.194:8084>*
"},"tasks":[{"state":"RUNNING","id":0,"worker_id":"*10.72.133.195:8084
<http://10.72.133.195:8084>*"}]}
On Wed, Aug 31, 2016 at 8:34 PM, dhanuka ranasinghe <
dhanuka.priyanath@gmail.com> wrote:
> Hi All,
>
> I'm new to Kafka and I created TCP Kafka Source connector [1] and deployed
> in Kafka Connect Cluster in distributed mode. There is two node/instance of
> connectors in Kafka Connector cluster.
>
> But out of them only in one node run *tasks* even though it shows
> connector in both nodes active and running status.
>
> In here I have used RxNetty server, so what happened is netty server up
> in one node/host but tasks are running in other node/host
>
> Could you please let me know the issue with this? Thanks in advance.
>
> node==host
>
>
> [1] https://github.com/dhanuka84/kafka-connect-tcp
>
> cheers,
> Dhanuka
>
> --
> Nothing Impossible,Creativity is more important than knowledge.
>
--
Nothing Impossible,Creativity is more important than knowledge.