You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@eventmesh.apache.org by "Ish1yu (via GitHub)" <gi...@apache.org> on 2024/03/15 09:10:00 UTC

[PR] [ISSUE #4787]integration connector (eventmesh)

Ish1yu opened a new pull request, #4790:
URL: https://github.com/apache/eventmesh/pull/4790

   Fixes https://github.com/apache/eventmesh/issues/4781
   
   ########
   UnMergeable!!! its demo ,不能合并这只是用于讨论
   ########
   
   Motivation
   
   Integrate connector with minimal modification
   
   Very simple integration using connecter
   
   Modifications
   
   Integrating 'connecter' means that 'connecter' has to have another running mode, and integrating into 'runtime' has some problems to solve
   集成 `connecter` ,意味着 `connecter` 要有另一种运行模式,集成到 `runtime` 中有一些问题需要解决
   --------------------------------------------------------------------------------------------------------------------------------
   The first problem that needs to be solved is that the main method is not called, but `connecter` is a plug-in module, and it is troublesome to hard-code main once for each plug-in. So the method I think of is that all the `connecter` module's startup class inherits a `EmbeddableConnector` interface, and uses the `Eventmesh-SPI` to load, so that the changes are brought
   `EmbeddableConnector`
   All startup classes in `Connector` (in this case, `FileConnectServer`)
   All files referred to by `SPI` in `Connector`
   首先需要解决的问题就是 `main` 方法没有被调用,但是 `connecter` 是插件化的模块,为每一个插件都硬编码一次 `main` 这样很麻烦,所以我想到的方法是所有 `connecter` 模块的启动类都继承一个`EmbeddableConnector` 接口,使用 `Eventmesh` 的 `SPI` 进行加载,这样带来的修改就有
   `EmbeddableConnector`
   所有 `Connector` 中的启动类(这个例子中的启动类是 `FileConnectServer` )
   所有 `Connector` 中 `SPI` 涉及的文件
   --------------------------------------------------------------------------------------------------------------------------------
   Second, the original interaction between `connector` and `runtime` uses `tcp protocol,` and the purpose of integration is to reduce serialization and communication consumption. Therefore, interaction needs to be realized in other ways. The method I think is to use the form of service. connector itself only has two roles, `sink` and `source`, which are publish and subscribe in terms of function, so this service needs to use publish and subscribe functions. Meanwhile, in order to preserve the original structure and design to the greatest extent, the connector uses a `tcp adapter` (the adapter does not implement network communication in the middle). Instead, the service is called.) So the design changes are
   IInnerPubSubService Indicates the interface of the service
   InnerPubSubService implementation
   IntegrationCloudEventTCPClientAdapter TCP adapter
   Application must have a static variable holding the service
   The SinkWorker passes the service to the tcp adapter
   SourceWorker transfers the service tcp adapter
   其次需要解决的就是原有 `connector` 和 `runtime` 之间的交互使用的是 `tcp协议` 而集成的目的就是减少序列化和通信消耗,所以需要通过其他方式对交互进行实现,我想到的方法是使用服务的形式, `connector` 本身其实只有两种角色,`sink` 和` source` ,从功能来说就是发布和订阅,那么这个服务就需要用发布和订阅的功能即可,同时为了最大程度保留原有结构和设计,使用tcp适配器(适配器中间不在实现网络通信,而是对服务进行调用)所以设计的修改有
   IInnerPubSubService 服务的接口
   InnerPubSubService 服务的实现
   IntegrationCloudEventTCPClientAdapter tcp适配器
   Application 要有静态变量持有服务
   SinkWorker 传递服务给tcp适配器
   SourceWorker 传递服务tcp适配器
   --------------------------------------------------------------------------------------------------------------------------------
   Unfinished changes:
   The implementation of InnerPubSubService
   未完成的修改:
   `InnerPubSubService` 的具体实现
   --------------------------------------------------------------------------------------------------------------------------------
   达成的效果:
   只要 `gradle` 引用了对应的 `connector`  插件就会自动处理 内嵌 `connector` 相关逻辑 
   Results achieved:
   The built-in `connector` logic is automatically handled whenever `gradle` references the corresponding ` connector` plug-in
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org


Re: [PR] [ISSUE #4787]integration connector (eventmesh)

Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on PR #4790:
URL: https://github.com/apache/eventmesh/pull/4790#issuecomment-2003847072

   > Sorry, I have a bad cold these days, so I think it may be necessary to use a form like file lock. After connector is activated, the Worker will generate a file named Connector-instance-mian at a fixed location. When the connector is booted either integrally or separately, it will get an error that cannot be started if the file exists (init timing), and delete the file when the connector stops (shutdown timing). The only drawback is the need to manually delete this file if the connector stops unexpectedly (such as when the process is killed). 
   不好意思,这几天重感冒,那我想可能需要使用比如类似文件锁的形式,当connector启动后Worker在某一固定位置生成一个名为connector-instance-mian类名的文件,当无论以集成的方式还是单独启动都会去获取这个文件如果文件存在那么就报错不可以启动(init时机),当connector停止时删除这个文件(shutdown时机),唯一的缺陷是当connector意外停止时需要手动删除这个文件(比如进程被杀了)
   
   I see in the issue that you have raised a lot of queries about the community's planning for this part of the demand, so let's leave this minor detail aside for now. I'll keep an eye on these queries as well https://github.com/apache/eventmesh/issues/4787#issuecomment-2003288218 . 
   
   我看到issue中您提了很多关于社区对这部分需求的规划的疑问,所以这个次要的细节问题暂不考虑吧。对于这些疑问(https://github.com/apache/eventmesh/issues/4787#issuecomment-2003288218 ),我也会继续关注。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org


Re: [PR] [ISSUE #4787]integration connector (eventmesh)

Posted by "Pil0tXia (via GitHub)" <gi...@apache.org>.
Pil0tXia commented on PR #4790:
URL: https://github.com/apache/eventmesh/pull/4790#issuecomment-2041454260

   @Ish1yu 
   
   Now we have a detailed issue describing the connector integration design, which may greatly help your subsequent work of this PR. So any communication with community is welcomed~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org


Re: [PR] [ISSUE #4787]integration connector (eventmesh)

Posted by "Ish1yu (via GitHub)" <gi...@apache.org>.
Ish1yu commented on PR #4790:
URL: https://github.com/apache/eventmesh/pull/4790#issuecomment-2002753310

   Sorry, I have a bad cold these days, so I think it may be necessary to use a form like file lock. After connector is activated, the Worker will generate a file named Connector-instance-mian at a fixed location. When the connector is booted either integrally or separately, it will get an error that cannot be started if the file exists (init timing), and delete the file when the connector stops (shutdown timing). The only drawback is the need to manually delete this file if the connector stops unexpectedly (such as when the process is killed).
   不好意思,这几天重感冒,那我想可能需要使用比如类似文件锁的形式,当connector启动后Worker在某一固定位置生成一个名为connector-instance-mian类名的文件,当无论以集成的方式还是单独启动都会去获取这个文件如果文件存在那么就报错不可以启动(init时机),当connector停止时删除这个文件(shutdown时机),唯一的缺陷是当connector意外停止时需要手动删除这个文件(比如进程被杀了)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org


Re: [PR] [ISSUE #4787]integration connector (eventmesh)

Posted by "Ish1yu (via GitHub)" <gi...@apache.org>.
Ish1yu closed pull request #4790: [ISSUE #4787]integration connector
URL: https://github.com/apache/eventmesh/pull/4790


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org