You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/04/12 02:35:47 UTC

[GitHub] [rocketmq-connect] sunxiaojian opened a new issue, #85: SourceTask 和 SinkTask API的优化建议

sunxiaojian opened a new issue, #85:
URL: https://github.com/apache/rocketmq-connect/issues/85

   1. API版本 0.1.2
   
   2. 问题一：SourceTask和SinkTask API中建议把 init、validate、start合并起来比较好，
   原因是，在用新的api构造插件的时候发现，由于不太清楚init，validate，start的执行顺序，所以就查了一下源码，困惑的是这个调用顺序是怎么样的，但其实写插件的人可能不太关心这个，所以方法多了反而增加了写插件的成本，建议只提供strat接口给用户实现，把validate可以放在配置类加载时实现，init可以留给写插件的人自己抽象；
   
   3.问题二：task、connector、transform都实现了Component 接口，接口中透出了validate方法但实际发现task调用中是没有用到的，如下图:
          
   ![image](https://user-images.githubusercontent.com/6446530/162867517-e0cefff9-938a-4a70-9c9e-5675519463e4.png)
   
     这样会造成每个实现的sourceTask和sinkTask会有个空方法，造成疑惑，是否可以在SinkTask和SourceTask抽象类中隐去
   
       
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] odbozhou commented on issue #85: SourceTask and SinkTask API Optimization suggestions

Posted by GitBox <gi...@apache.org>.
odbozhou commented on issue #85:
URL: https://github.com/apache/rocketmq-connect/issues/85#issuecomment-1096039896

   > 
   https://github.com/openmessaging/openconnect
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] odbozhou commented on issue #85: SourceTask and SinkTask API Optimization suggestions

Posted by GitBox <gi...@apache.org>.
odbozhou commented on issue #85:
URL: https://github.com/apache/rocketmq-connect/issues/85#issuecomment-1096041045

   > 关于问题二，可以把validate 空方法 放到 SinkTask和 SourceTask 抽象类中吧，这样在实现这个抽象方法的时候就不会有空方法了，像这样，不用使用者必须实现 ![image](https://user-images.githubusercontent.com/6446530/162880255-7e16288b-de13-496b-8225-8c1cdb0d7839.png)
   
   方便的化，可以把你对connect api的想法提pr到这个仓库吗？
   https://github.com/openmessaging/openconnect


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] lizhiboo commented on issue #85: SourceTask and SinkTask API Optimization suggestions

Posted by GitBox <gi...@apache.org>.
lizhiboo commented on issue #85:
URL: https://github.com/apache/rocketmq-connect/issues/85#issuecomment-1158459741

   IMO, SourceTask also needs commit(successMessages) and rollback(failedMessages) interfaces. Besides, we can add a plugin-in mechanism in connect flow for custom logic, such as exception handling for producing message to RocketMQ in SourceConnector.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] sunxiaojian commented on issue #85: SourceTask and SinkTask API Optimization suggestions

Posted by GitBox <gi...@apache.org>.
sunxiaojian commented on issue #85:
URL: https://github.com/apache/rocketmq-connect/issues/85#issuecomment-1095988751

   关于问题二，可以把validate 空方法 放到 SinkTask和 SourceTask 抽象类中吧，这样在实现这个抽象方法的时候就不会有空方法了，像这样，不用使用者必须实现
   ![image](https://user-images.githubusercontent.com/6446530/162880255-7e16288b-de13-496b-8225-8c1cdb0d7839.png)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] odbozhou commented on issue #85: SourceTask and SinkTask API Optimization suggestions

Posted by GitBox <gi...@apache.org>.
odbozhou commented on issue #85:
URL: https://github.com/apache/rocketmq-connect/issues/85#issuecomment-1323322518

   merged


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] odbozhou commented on issue #85: SourceTask 和 SinkTask API的优化建议

Posted by GitBox <gi...@apache.org>.
odbozhou commented on issue #85:
URL: https://github.com/apache/rocketmq-connect/issues/85#issuecomment-1095944099

   问题1：目前的想法是这样的，validate校验配置，init接受配置，然后start的时候传递上下文信息。这可能是站在熟悉connector的设想。的确在刚接触connector的情况下会有一些困扰，增加了学习成本。我任务适当的调整是可以减少一些困扰的。我的想法是，init和start还是有必要分开切必要的的，这样也比较好理解，init可以用来初始化上线文，配置， 然后再start拿到启动。validate可以作为抽象类中的可选实现而不是接口的必要实现，这样可能对实现着更加友好一些。
   问题2：现在的继承关系确实会产生你说的问题，有些实现不会用到，但必须有空实现。你有具体的一些的建议吗？


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] sunxiaojian commented on issue #85: SourceTask and SinkTask API Optimization suggestions

Posted by GitBox <gi...@apache.org>.
sunxiaojian commented on issue #85:
URL: https://github.com/apache/rocketmq-connect/issues/85#issuecomment-1098675922

   问题一描述：
   
   
   版本  0.1.1  api的定义
   
   
   
   版本0.1.2 api定义
   
   
   差异点： 0.1.1版本是 start是必须实现的，0.1.2版本是init是必须实现的，
   问题描述：如果定位是init是初始化配置，那start应该是初始化offset等逻辑的；现在start是不必实现的，对于不了解api的插件扩展者来说可能就不会实现了，这样可能会把初始化客户端和jdbc连接的操作在init里面全部实现了，但是由于在WorkerSinkTask(WorkerSourceTask一样)中的调用顺序是下面如图，在init里面是无法拿到父类SourceTask和SinkTask中的 context 变量的，由于调用顺序的原因，这个值在init的时候应该是空，因为调用顺序造成在init使用这个变量无法实现，这样可能就无法拿到offset等信息，所以实现start方法就成为了必须，但是父类又实现了这个start方法赋值了context，本来是必须要实现的现在做成了可实现可不实现，所以就感觉比较矛盾，所以建议还是0.1.1 api 会比较好一点
   
   
   问题二描述：
   
   pause和 resume 是不是就不要对客户开放了，插件的编写只需要了解获取数据和offset的逻辑，不应该处理任务的暂停和重启的逻辑，不知这个地方是怎么考虑的？
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] sunxiaojian commented on issue #85: SourceTask and SinkTask API Optimization suggestions

Posted by GitBox <gi...@apache.org>.
sunxiaojian commented on issue #85:
URL: https://github.com/apache/rocketmq-connect/issues/85#issuecomment-1096214342

   ok，


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] odbozhou closed issue #85: SourceTask and SinkTask API Optimization suggestions

Posted by GitBox <gi...@apache.org>.
odbozhou closed issue #85: SourceTask and SinkTask  API Optimization suggestions
URL: https://github.com/apache/rocketmq-connect/issues/85


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org