You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Yu Watanabe <yu...@gmail.com> on 2020/05/28 03:11:50 UTC
How to manually start ingesting in kafka source connector ?
Dear community .
I would like to ask question related to source connector in kafka
connect (2.4.0) .
Is there a way to manually start source connector after registering to
kafka connect ?
Looking at the document , I found PAUSE API ,
https://docs.confluent.io/current/connect/references/restapi.html#put--connectors-(string-name)-pause
however, could not find set initial state for individual tasks in
connector properties ..
https://docs.confluent.io/current/connect/managing/configuring.html
I appreciate if I could get some help.
Best Regards,
Yu Watanabe
--
Yu Watanabe
linkedin: www.linkedin.com/in/yuwatanabe1/
twitter: twitter.com/yuwtennis
Re: How to manually start ingesting in kafka source connector ?
Posted by Yu Watanabe <yu...@gmail.com>.
Thank you for the link.
In my environment I use strimzi (0.17.0).
I realized that connector offset is created when kafka connect start
up . So before creating connector , I am able to set offset for Kafka
Connect.
In addition to you link ,scenario in below link also helped my idea.
https://www.confluent.jp/blog/kafka-connect-deep-dive-jdbc-source-connector/#starting-table-capture
Thanks,
Yu
On Thu, May 28, 2020 at 9:40 PM Robin Moffatt <ro...@confluent.io> wrote:
>
> You could look at
> https://rmoff.net/2019/08/15/reset-kafka-connect-source-connector-offsets/ and
> experiment with creating the connector elsewhere to see if you can pre-empt
> the key value that Kafka Connect will use when writing the offsets, and so
> do your list 2 - 1 - 3 instead
>
>
> --
>
> Robin Moffatt | Senior Developer Advocate | robin@confluent.io | @rmoff
>
>
> On Thu, 28 May 2020 at 10:12, Yu Watanabe <yu...@gmail.com> wrote:
>
> > Robin
> >
> > Thank you for the reply.
> >
> > Any way to not automatically start after creating connector ?
> >
> > I am trying to find a way to change connector offset as described in
> > below link before starting connector ..
> >
> >
> > https://www.confluent.jp/blog/kafka-connect-deep-dive-jdbc-source-connector/#starting-table-capture
> >
> > Steps I want to do will be
> >
> > 1. Create jdbc connector
> > 2. Change connector offset
> > 3. Start connector
> >
> > Thanks,
> > Yu
> >
> > On Thu, May 28, 2020 at 6:01 PM Robin Moffatt <ro...@confluent.io> wrote:
> > >
> > > When you create the connector, it will start.
> > >
> > >
> > > --
> > >
> > > Robin Moffatt | Senior Developer Advocate | robin@confluent.io | @rmoff
> > >
> > >
> > > On Thu, 28 May 2020 at 04:12, Yu Watanabe <yu...@gmail.com> wrote:
> > >
> > > > Dear community .
> > > >
> > > > I would like to ask question related to source connector in kafka
> > > > connect (2.4.0) .
> > > >
> > > > Is there a way to manually start source connector after registering to
> > > > kafka connect ?
> > > >
> > > > Looking at the document , I found PAUSE API ,
> > > >
> > > >
> > > >
> > https://docs.confluent.io/current/connect/references/restapi.html#put--connectors-(string-name)-pause
> > > >
> > > > however, could not find set initial state for individual tasks in
> > > > connector properties ..
> > > >
> > > > https://docs.confluent.io/current/connect/managing/configuring.html
> > > >
> > > > I appreciate if I could get some help.
> > > >
> > > > Best Regards,
> > > > Yu Watanabe
> > > >
> > > > --
> > > > Yu Watanabe
> > > >
> > > > linkedin: www.linkedin.com/in/yuwatanabe1/
> > > > twitter: twitter.com/yuwtennis
> > > >
> >
> >
> >
> > --
> > Yu Watanabe
> >
> > linkedin: www.linkedin.com/in/yuwatanabe1/
> > twitter: twitter.com/yuwtennis
> >
--
Yu Watanabe
linkedin: www.linkedin.com/in/yuwatanabe1/
twitter: twitter.com/yuwtennis
Re: How to manually start ingesting in kafka source connector ?
Posted by Robin Moffatt <ro...@confluent.io>.
You could look at
https://rmoff.net/2019/08/15/reset-kafka-connect-source-connector-offsets/ and
experiment with creating the connector elsewhere to see if you can pre-empt
the key value that Kafka Connect will use when writing the offsets, and so
do your list 2 - 1 - 3 instead
--
Robin Moffatt | Senior Developer Advocate | robin@confluent.io | @rmoff
On Thu, 28 May 2020 at 10:12, Yu Watanabe <yu...@gmail.com> wrote:
> Robin
>
> Thank you for the reply.
>
> Any way to not automatically start after creating connector ?
>
> I am trying to find a way to change connector offset as described in
> below link before starting connector ..
>
>
> https://www.confluent.jp/blog/kafka-connect-deep-dive-jdbc-source-connector/#starting-table-capture
>
> Steps I want to do will be
>
> 1. Create jdbc connector
> 2. Change connector offset
> 3. Start connector
>
> Thanks,
> Yu
>
> On Thu, May 28, 2020 at 6:01 PM Robin Moffatt <ro...@confluent.io> wrote:
> >
> > When you create the connector, it will start.
> >
> >
> > --
> >
> > Robin Moffatt | Senior Developer Advocate | robin@confluent.io | @rmoff
> >
> >
> > On Thu, 28 May 2020 at 04:12, Yu Watanabe <yu...@gmail.com> wrote:
> >
> > > Dear community .
> > >
> > > I would like to ask question related to source connector in kafka
> > > connect (2.4.0) .
> > >
> > > Is there a way to manually start source connector after registering to
> > > kafka connect ?
> > >
> > > Looking at the document , I found PAUSE API ,
> > >
> > >
> > >
> https://docs.confluent.io/current/connect/references/restapi.html#put--connectors-(string-name)-pause
> > >
> > > however, could not find set initial state for individual tasks in
> > > connector properties ..
> > >
> > > https://docs.confluent.io/current/connect/managing/configuring.html
> > >
> > > I appreciate if I could get some help.
> > >
> > > Best Regards,
> > > Yu Watanabe
> > >
> > > --
> > > Yu Watanabe
> > >
> > > linkedin: www.linkedin.com/in/yuwatanabe1/
> > > twitter: twitter.com/yuwtennis
> > >
>
>
>
> --
> Yu Watanabe
>
> linkedin: www.linkedin.com/in/yuwatanabe1/
> twitter: twitter.com/yuwtennis
>
Re: How to manually start ingesting in kafka source connector ?
Posted by Yu Watanabe <yu...@gmail.com>.
Robin
Thank you for the reply.
Any way to not automatically start after creating connector ?
I am trying to find a way to change connector offset as described in
below link before starting connector ..
https://www.confluent.jp/blog/kafka-connect-deep-dive-jdbc-source-connector/#starting-table-capture
Steps I want to do will be
1. Create jdbc connector
2. Change connector offset
3. Start connector
Thanks,
Yu
On Thu, May 28, 2020 at 6:01 PM Robin Moffatt <ro...@confluent.io> wrote:
>
> When you create the connector, it will start.
>
>
> --
>
> Robin Moffatt | Senior Developer Advocate | robin@confluent.io | @rmoff
>
>
> On Thu, 28 May 2020 at 04:12, Yu Watanabe <yu...@gmail.com> wrote:
>
> > Dear community .
> >
> > I would like to ask question related to source connector in kafka
> > connect (2.4.0) .
> >
> > Is there a way to manually start source connector after registering to
> > kafka connect ?
> >
> > Looking at the document , I found PAUSE API ,
> >
> >
> > https://docs.confluent.io/current/connect/references/restapi.html#put--connectors-(string-name)-pause
> >
> > however, could not find set initial state for individual tasks in
> > connector properties ..
> >
> > https://docs.confluent.io/current/connect/managing/configuring.html
> >
> > I appreciate if I could get some help.
> >
> > Best Regards,
> > Yu Watanabe
> >
> > --
> > Yu Watanabe
> >
> > linkedin: www.linkedin.com/in/yuwatanabe1/
> > twitter: twitter.com/yuwtennis
> >
--
Yu Watanabe
linkedin: www.linkedin.com/in/yuwatanabe1/
twitter: twitter.com/yuwtennis
Re: How to manually start ingesting in kafka source connector ?
Posted by Robin Moffatt <ro...@confluent.io>.
When you create the connector, it will start.
--
Robin Moffatt | Senior Developer Advocate | robin@confluent.io | @rmoff
On Thu, 28 May 2020 at 04:12, Yu Watanabe <yu...@gmail.com> wrote:
> Dear community .
>
> I would like to ask question related to source connector in kafka
> connect (2.4.0) .
>
> Is there a way to manually start source connector after registering to
> kafka connect ?
>
> Looking at the document , I found PAUSE API ,
>
>
> https://docs.confluent.io/current/connect/references/restapi.html#put--connectors-(string-name)-pause
>
> however, could not find set initial state for individual tasks in
> connector properties ..
>
> https://docs.confluent.io/current/connect/managing/configuring.html
>
> I appreciate if I could get some help.
>
> Best Regards,
> Yu Watanabe
>
> --
> Yu Watanabe
>
> linkedin: www.linkedin.com/in/yuwatanabe1/
> twitter: twitter.com/yuwtennis
>