You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@eventmesh.apache.org by "epiao55 (via GitHub)" <gi...@apache.org> on 2023/02/24 16:32:58 UTC
[GitHub] [incubator-eventmesh] epiao55 opened a new issue, #3263: [Feature] The filtering feature of events
epiao55 opened a new issue, #3263:
URL: https://github.com/apache/incubator-eventmesh/issues/3263
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/eventmesh/issues?q=is%3Aissue) and found no similar issues.
### Feature Request
I would like to contribute a filtering feature for events. Here is my rough design idea.
First, when creating a subscription, add a new json String field 'filter' to the SubscriptionItem to describe the specific filtering rules.for example
```json
{
"and": [{
"field": "specversion",
"operator": "EQ",
"value": "1.0"
}, {
"or": [{
"field": "id",
"operator": "EQ",
"value": "9aeb0fdf-c01e-0131-0922-9eb54906e209"
}, {
"field": "email",
"operator": "CN",
"value": "example"
}]
}, {
"field": "data.contentType",
"operator": "SW",
"value": "image/png"
}]
}
```
When a consumer consumes, check the 'filter' field and convert it into a 'FilterGroup'. Then, generate a combined predicate based on the rules and provide the filtering results.
The logical relationship between the rules is described by 'and' and 'or'. The 'field' field describes the field to be checked, the 'operator' describes the specific rule, and the 'value' describes the comparison data for the rule.
The value of the field field can be a nested field in the event, expressed as a hierarchical relationship using '.'.
and This is the filtering process.
```mermaid
sequenceDiagram
participant Consumer
participant FilterGroup
participant SubscriptionItem
participant SubscriptionItem
Consumer->>+FilterGroup: Transform event to JSON
FilterGroup->>SubscriptionItem: Get filter rules JSON
FilterGroup->>FilterGroup: Parse filter rules JSON
FilterGroup->>FilterGroup: Generate combined predicate
FilterGroup->>+FilterRule: Predicate rules
Note over FilterRule: Loop for all rules
FilterRule->>FilterRule: Rule 1 apply
FilterRule->>FilterRule: Rule 2 apply
FilterRule->>FilterRule: Rule 3 apply
Note over FilterRule: End of loop
FilterRule->>-FilterGroup: Allowed or denied
alt FilterGroup allowed or denied
FilterGroup->>-Consumer: Allowed or denied
else FilterGroup throws exception
FilterGroup->>Consumer: Apply default strategy
end
```
And provide optional default policy to handle filtering exceptions for allowing or denying access.
I have initially designed the following specific types of rules:
| operator | class
| --- | --- |
| **EQ** | EqualFilterRule
| **NE** | NotEqualFilterRule
| **GT** | GreaterThanFilterRule
| **GE** | GreaterThanOrEqualFilterRule
| **LT** | LessThanFilterRule
| **LE** | LessThanOrEqualFilterRule
| BT | BetweenFilterRule
| **CN** | ContainsFilterRule
| NC | NotContainsFilterRule
| **SW** | StartsWithFilterRule |
| **EW** | EndsWithFilterRule
| **RE** | RegexFilterRule
| IN | InFilterRule
| NIN | NotInFilterRule
| EX | ExistsFilterRule
| NEX | NotExistsFilterRule
| NOT | NotFilterRule
| NL | NullFilterRule
| NN | NotNullFilterRule
| EQT | EpochTimeFilterRule
| GTT | GreaterThanTimeFilterRule
| LTT | LessThanTimeFilterRule
| BT | BetweenTimeFilterRule
I have started working on the implementation of this code. Please consider my approach, which can also be applied to similar transformation features.
The bolded rules have been implemented. Rules can be continuously extended by implementing the interface.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org
[GitHub] [incubator-eventmesh] xwm1992 commented on issue #3263: [Feature] The filtering feature of events
Posted by "xwm1992 (via GitHub)" <gi...@apache.org>.
xwm1992 commented on issue #3263:
URL: https://github.com/apache/incubator-eventmesh/issues/3263#issuecomment-1445288098
> > @epiao55 Hi, first of all, thank you for your design of the Event Filter scheme. You can refer to the code of [eventmesh-filters](https://github.com/apache/incubator-eventmesh/tree/eventmesh-filters) branch. Your scheme seems to apply the event filter rule configuration and processing filtering from the consumer sdk side when subscribing. We hope to be able to use it from the EventMesh API Layer design and implement filter and transformer capabilities in EventMesh runtime. We can arrange a meeting to discuss the design plan and talk about the specific design ideas.
> > @xwm1992 Thank you for your reply, I will first understand the existing design of this part. And How can I contact you
you can contact with the eventmesh wechat helper, and this design is similar to the design of zuul 2.0. If you have a better design reference, you can also put it forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org
[GitHub] [incubator-eventmesh] epiao55 commented on issue #3263: [Feature] The filtering feature of events
Posted by "epiao55 (via GitHub)" <gi...@apache.org>.
epiao55 commented on issue #3263:
URL: https://github.com/apache/incubator-eventmesh/issues/3263#issuecomment-1445272739
> @epiao55 Hi, first of all, thank you for your design of the Event Filter scheme. You can refer to the code of [eventmesh-filters](https://github.com/apache/incubator-eventmesh/tree/eventmesh-filters) branch. Your scheme seems to apply the event filter rule configuration and processing filtering from the consumer sdk side when subscribing. We hope to be able to use it from the EventMesh API Layer design and implement filter and transformer capabilities in EventMesh runtime. We can arrange a meeting to discuss the design plan and talk about the specific design ideas.
@xwm1992 Thank you for your reply, I will first understand the existing design of this part. And How can I contact you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org
[GitHub] [incubator-eventmesh] qqeasonchen commented on issue #3263: [Feature] The filtering feature of events
Posted by "qqeasonchen (via GitHub)" <gi...@apache.org>.
qqeasonchen commented on issue #3263:
URL: https://github.com/apache/incubator-eventmesh/issues/3263#issuecomment-1445994890
@epiao55 good idea!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org
[GitHub] [incubator-eventmesh] xwm1992 commented on issue #3263: [Feature] The filtering feature of events
Posted by "xwm1992 (via GitHub)" <gi...@apache.org>.
xwm1992 commented on issue #3263:
URL: https://github.com/apache/incubator-eventmesh/issues/3263#issuecomment-1445268655
@epiao55 Hi,
first of all, thank you for your design of the Event Filter scheme. You can refer to the code of [eventmesh-filters](https://github.com/apache/incubator-eventmesh/tree/eventmesh-filters) branch. Your scheme seems to apply the event filter rule configuration and processing filtering from the consumer sdk side when subscribing. We hope to be able to use it from the EventMesh API Layer design and implement filter and transformer capabilities in EventMesh runtime. We can arrange a meeting to discuss the design plan and talk about the specific design ideas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org