You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@seatunnel.apache.org by Guangdong Liu <li...@gmail.com> on 2023/08/10 04:39:03 UTC
[DISCUSS]Remove scheduler in connectors
Hi everyone,
At present, many connectors push data to the destination through
asynchronous threads, like Jdbc,InfluxDB,IoTDB,StarRocks. This way there
will be some problems,
1.The way of asynchronous submission is the same as the function of
checkpoint, we can complete asynchronous submission through checkpoint
2.Asynchronous threads may not be closed in exceptional cases and the
connector cannot exit. Refer to
https://github.com/apache/seatunnel/issues/4736
Therefore, I suggest removing the asynchronous submission operations in
these connectors and replacing them with checkpoints, like
https://github.com/apache/seatunnel/pull/5168
Looking forward to your feedback, thanks!
--
Best Regards
------------
Liugddx
liugddx@gmail.com
Re: [DISCUSS]Remove scheduler in connectors
Posted by Guangdong Liu <li...@gmail.com>.
Of course.
--
Best Regards
------------
Liugddx
liugddx@gmail.com
JUN GAO <ga...@apache.org> 于2023年8月10日周四 15:27写道：
> +1, Can you create an issue and make it in RoadMap?
>
>
> Jia Fan <fa...@qq.com.invalid> 于2023年8月10日周四 13:36写道：
>
> > Thanks Guangdong for open this discussion.
> >
> > +1 from my side.
> >
> > ________________________
> >
> > Jia Fan
> >
> >
> >
> > > 2023年8月10日 12:39，Guangdong Liu <li...@gmail.com> 写道：
> > >
> > > Hi everyone,
> > >
> > > At present, many connectors push data to the destination through
> > > asynchronous threads, like Jdbc,InfluxDB,IoTDB,StarRocks. This way
> there
> > > will be some problems,
> > >
> > > 1.The way of asynchronous submission is the same as the function of
> > > checkpoint, we can complete asynchronous submission through checkpoint
> > > 2.Asynchronous threads may not be closed in exceptional cases and the
> > > connector cannot exit. Refer to
> > > https://github.com/apache/seatunnel/issues/4736
> > >
> > > Therefore, I suggest removing the asynchronous submission operations in
> > > these connectors and replacing them with checkpoints, like
> > > https://github.com/apache/seatunnel/pull/5168
> > >
> > > Looking forward to your feedback, thanks!
> > >
> > > --
> > >
> > > Best Regards
> > >
> > > ------------
> > >
> > > Liugddx
> > > liugddx@gmail.com
> >
> >
>
> --
>
> Best Regards
>
> ------------
>
> EricJoy2048
> gaojun2048@gmail.com
>
Re: [DISCUSS]Remove scheduler in connectors
Posted by JUN GAO <ga...@apache.org>.
+1, Can you create an issue and make it in RoadMap?
Jia Fan <fa...@qq.com.invalid> 于2023年8月10日周四 13:36写道：
> Thanks Guangdong for open this discussion.
>
> +1 from my side.
>
> ________________________
>
> Jia Fan
>
>
>
> > 2023年8月10日 12:39，Guangdong Liu <li...@gmail.com> 写道：
> >
> > Hi everyone,
> >
> > At present, many connectors push data to the destination through
> > asynchronous threads, like Jdbc,InfluxDB,IoTDB,StarRocks. This way there
> > will be some problems,
> >
> > 1.The way of asynchronous submission is the same as the function of
> > checkpoint, we can complete asynchronous submission through checkpoint
> > 2.Asynchronous threads may not be closed in exceptional cases and the
> > connector cannot exit. Refer to
> > https://github.com/apache/seatunnel/issues/4736
> >
> > Therefore, I suggest removing the asynchronous submission operations in
> > these connectors and replacing them with checkpoints, like
> > https://github.com/apache/seatunnel/pull/5168
> >
> > Looking forward to your feedback, thanks!
> >
> > --
> >
> > Best Regards
> >
> > ------------
> >
> > Liugddx
> > liugddx@gmail.com
>
>
--
Best Regards
------------
EricJoy2048
gaojun2048@gmail.com
Re: [DISCUSS]Remove scheduler in connectors
Posted by Jia Fan <fa...@qq.com.INVALID>.
Thanks Guangdong for open this discussion.
+1 from my side.
________________________
Jia Fan
> 2023年8月10日 12:39，Guangdong Liu <li...@gmail.com> 写道：
>
> Hi everyone,
>
> At present, many connectors push data to the destination through
> asynchronous threads, like Jdbc,InfluxDB,IoTDB,StarRocks. This way there
> will be some problems,
>
> 1.The way of asynchronous submission is the same as the function of
> checkpoint, we can complete asynchronous submission through checkpoint
> 2.Asynchronous threads may not be closed in exceptional cases and the
> connector cannot exit. Refer to
> https://github.com/apache/seatunnel/issues/4736
>
> Therefore, I suggest removing the asynchronous submission operations in
> these connectors and replacing them with checkpoints, like
> https://github.com/apache/seatunnel/pull/5168
>
> Looking forward to your feedback, thanks!
>
> --
>
> Best Regards
>
> ------------
>
> Liugddx
> liugddx@gmail.com