You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@eventmesh.apache.org by GitBox <gi...@apache.org> on 2021/12/30 02:59:19 UTC
[GitHub] [incubator-eventmesh] activemumu opened a new issue #676: [Feature] Event Streaming
activemumu opened a new issue #676:
URL: https://github.com/apache/incubator-eventmesh/issues/676
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/eventmesh/issues?q=is%3Aissue) and found no similar issues.
### Feature Request
### EventMesh-Camel-SpringCloudStream
In order to implement event stream processing, we will integrate Camel and Spring Cloud Stream in EventMesh. This document provides the basic architecture of the stream processing system in EventMesh.
**Spring-cloud-stream**
Spring Cloud Stream is a framework for building message-driven microservices. Spring Cloud Stream provides an opinionated configuration of message brokers, introducing the concepts of persistent pub/sub semantics, consumer groups and partitions across several middleware vendors. This opinionated configuration provides the basis to create stream processing applications.
The two components of Spring Cloud Stream and Spring Cloud Binder always appear in pairs. The application interacts with the binder object in Spring Cloud Stream through inputs or outputs. Binding is performed through configuration, and the binder object of Spring Cloud Stream is responsible for interacting with the message middleware, such as Kafka, RabbitMQ and RocketMQ. So, we only need to figure out how to interact with Spring Cloud Stream to facilitate the use of message-driven methods. At its most basic, Stream comes with three predefined interfaces: Source, Processor and Sink. These interfaces have a combination of input and output bindings. Normally, for incoming messages, the sink and input channels are binding. For the message to be sent, the source and output channels are binding.Processor can process incoming messages and then send them to other applications. The channel for sending messages can be abstracted as `MessageChannel`:
- Producers/consumers rely on message media to transfer information content (Message)
- The message must go through a specific channel (`MessageChannel`)
The architectures of messaging middleware such as RabbitMQ and RocketMQ are usually different. The differences of these middleware have caused some difficulties in actual project development. The migration of middleware based on business needs is undoubtedly disastrous. At this time Spring Cloud Stream provides a way of decoupling. Through a unified channel, the application does not need to consider various message middleware implementations. The middleware we use as an example in the architecture is RocketMQ.
**Apache Camel**
Spring Cloud Stream provides personalized automation configuration implementations for some vendors' messaging middleware products. Usually, we can use Spring Integration to connect to the message broker middleware to achieve message event-driven. But based on the characteristics of Source, Processor and Sink, we can also have the power and convenience of Spring Cloud Stream combined with the simple, yet powerful ecosystem of Apache Camel.
Apache Camel abstracts a set of message interaction models for application integration scenarios, and accesses third-party systems through components. Apache Camel provides domain-specific languages (DSL) such as message routing and message conversion, which greatly reduces the difficulty of integrated application development. The application node information that needs to be integrated is defined by URI, and users can use DSL to quickly compile message routing rules according to business needs without paying attention to the details of the integration protocol. At the same time, the core library of Apache Camel is very small and can be easily integrated with other systems.
The innermost part in the picture is Camel. In the specific implementation, we can provide some Routes defined in the DSL language. These Routes define the direction of message sending. It can be sent from one topic to another or multiple topics. The code format is mainly as follows:
`from("rocketmq:topic1?namesrvAddr=localhost:9876&consumerGroup=consumer")`
`.to("rocketmq:topic2?namesrvAddr=localhost:9876&producerGroup=producer");`
Message filter processors are an important function in Camel. They are processors for message communication, which can be used for message format conversion or other transformations. The processor in Spring Cloud Stream handle with the message between input channel and output channel in binder. The processors in Apache Camel are used in each route, and multiple processors can be used in one route.
**Middleware**
The camel spring cloud stream system mentioned above can use a variety of messaging middleware such as RabbitMQ and Kafka. Here, combined with EventMesh, we use RocketMQ for message delivery.
RocketMQ cluster deployment usually includes four parts: Name server, Broker, Producer and Consumer, and supports distributed transaction messages. RocketMQ will receive messages from Spring Cloud Stream in the channel, and deliver the message.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org
[GitHub] [incubator-eventmesh] ruanwenjun commented on issue #676: [Feature] Event Streaming
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #676:
URL: https://github.com/apache/incubator-eventmesh/issues/676#issuecomment-1004117745
@msdhinesh-geek What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org
[GitHub] [incubator-eventmesh] qqeasonchen commented on issue #676: [Feature] Event Streaming
Posted by GitBox <gi...@apache.org>.
qqeasonchen commented on issue #676:
URL: https://github.com/apache/incubator-eventmesh/issues/676#issuecomment-1002981815
@activemumu good job, just go ahead and we can launch a discussion if neccessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org
[GitHub] [incubator-eventmesh] github-actions[bot] commented on issue #676: [Feature] Event Streaming
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #676:
URL: https://github.com/apache/incubator-eventmesh/issues/676#issuecomment-1002851362
Welcome to the Apache EventMesh (incubating) community!!
We are glad that you are contributing by opening this issue. :D
Please make sure to include all the relevant context.
We will be here shortly.
If you are interested in contributing to our project, please let us know!
You can check out our contributing guide on [contributing to EventMesh](https://github.com/apache/incubator-eventmesh/blob/develop/CONTRIBUTING.md).
Want to get closer to the community?
WeChat Group：
Mailing Lists:
| Name | Description |Subscribe |Unsubscribe|Archive
| ---- | ---- |---- | ---- | ---- |
|Users |User support and questions mailing list| [Subscribe](mailto:users-subscribe@eventmesh.incubator.apache.org) |[Unsubscribe](mailto:users-unsubscribe@eventmesh.incubator.apache.org) |[Mail Archives](https://lists.apache.org/list.html?users@eventmesh.apache.org)|
|Development |Development related discussions| [Subscribe](mailto:dev-subscribe@eventmesh.incubator.apache.org) |[Unsubscribe](mailto:dev-unsubscribe@eventmesh.incubator.apache.org) |[Mail Archives](https://lists.apache.org/list.html?dev@eventmesh.apache.org)|
|Commits |All commits to repositories| [Subscribe](mailto:commits-subscribe@eventmesh.incubator.apache.org) |[Unsubscribe](mailto:commits-unsubscribe@eventmesh.incubator.apache.org) |[Mail Archives](https://lists.apache.org/list.html?commits@eventmesh.apache.org)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org