You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by Dominik Riemer <ri...@apache.org> on 2020/10/26 21:11:12 UTC
Improvements to the Siddhi wrapper
Hi all,
Branislav has recently asked for list support in the Siddhi wrapper (e.g.,
filter a list for contained items).
I took the opportunity to extend and improve the Siddhi wrapper in some
ways:
- Support for lists (see example processors at [1, 2])
- Possibility to create a new Siddhi processor from a single class (no
controller and parameter class needed anymore, there is also a prototype of
the same structure available for the standalone wrapper
(StreamPipesDataProcessor)
- I added a simple object model to create Siddhi EPL statements instead of
the previous string-based approach. This should the creation of Siddhi
processors less error-prone. The object model is still rather incomplete and
undocumented, but it should be easy to extend in the future - the code is at
[3]
- Improved code structure of the whole wrapper to make it easier to extend
in the future
What do you think? Any feedback is welcome!
Also @Grainier, as you are the Siddhi expert, in case you have any ideas for
improvements, let me know!
Dominik
[1]
https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampi
pes-processors-filters-siddhi/src/main/java/org/apache/streampipes/processor
s/siddhi/listfilter/ListFilter.java
[2]
https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampi
pes-processors-filters-siddhi/src/main/java/org/apache/streampipes/processor
s/siddhi/listcollector/ListCollector.java
[3]
https://github.com/apache/incubator-streampipes/tree/dev/streampipes-wrapper
-siddhi/src/main/java/org/apache/streampipes/wrapper/siddhi/query
Re: Improvements to the Siddhi wrapper
Posted by Grainier Perera <gr...@apache.org>.
Sorry for the late reply. The improved wrapper looks really clean,
intuitive and easy to work with. With this, it'll be easier to expose any
builtin siddhi extensions. Great work Dominik.
However, I think we shouldn't expose all the Siddhi extensions unless they
are most commonly used or required by the community. Because otherwise,
it'll clutter the list of SP processors.
Grainier Perera.
On Tue, 27 Oct 2020 at 02:41, Dominik Riemer <ri...@apache.org> wrote:
> Hi all,
>
> Branislav has recently asked for list support in the Siddhi wrapper (e.g.,
> filter a list for contained items).
> I took the opportunity to extend and improve the Siddhi wrapper in some
> ways:
>
> - Support for lists (see example processors at [1, 2])
> - Possibility to create a new Siddhi processor from a single class (no
> controller and parameter class needed anymore, there is also a prototype of
> the same structure available for the standalone wrapper
> (StreamPipesDataProcessor)
> - I added a simple object model to create Siddhi EPL statements instead of
> the previous string-based approach. This should the creation of Siddhi
> processors less error-prone. The object model is still rather incomplete
> and
> undocumented, but it should be easy to extend in the future - the code is
> at
> [3]
> - Improved code structure of the whole wrapper to make it easier to extend
> in the future
>
> What do you think? Any feedback is welcome!
>
> Also @Grainier, as you are the Siddhi expert, in case you have any ideas
> for
> improvements, let me know!
>
> Dominik
>
> [1]
>
> https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampi
>
> pes-processors-filters-siddhi/src/main/java/org/apache/streampipes/processor
> s/siddhi/listfilter/ListFilter.java
> <https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampipes-processors-filters-siddhi/src/main/java/org/apache/streampipes/processors/siddhi/listfilter/ListFilter.java>
> [2]
>
> https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampi
>
> pes-processors-filters-siddhi/src/main/java/org/apache/streampipes/processor
> s/siddhi/listcollector/ListCollector.java
> <https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampipes-processors-filters-siddhi/src/main/java/org/apache/streampipes/processors/siddhi/listcollector/ListCollector.java>
> [3]
>
> https://github.com/apache/incubator-streampipes/tree/dev/streampipes-wrapper
> -siddhi/src/main/java/org/apache/streampipes/wrapper/siddhi/query
> <https://github.com/apache/incubator-streampipes/tree/dev/streampipes-wrapper-siddhi/src/main/java/org/apache/streampipes/wrapper/siddhi/query>
>
>
>
>
>
Re: Improvements to the Siddhi wrapper
Posted by Branislav Jovicic <br...@nissatech.com>.
Greetings, all,
Sorry for the late reply.
First of all, I want to thank Dominik for his help and all of you for your effort in creating better wrapper for Siddhi.
In general, I really like the changes that were made - It is easier to create StreamPipes element that utilizes Siddhi and to write queries.
Also, queries with simple lists now work properly.
However, I stumbled upon few problems:
* In the StreamPipes Editor, when I connect element with Siddhi query to the rest of the pipeline requiredPropertyWithUnaryMapping is not rendered properly. It shows that element's mapping Label name, instead of available Event properties of element to which it was connected [1].
[cid:17983475-1bd9-4d29-a3e7-a98c677681ca] [1]
*
Function makeEventType of EventTypeGenerator gets called for every property of an Event that gets received in element with Siddhi query, regardless of whether it was required or not. I do not think that should be the case (this may be the case for every other element as well, but I have not checked). Consequently, this function gets called for complex list that is not even required by this element and it throws an exception, because it cannot work with complex lists. (I know that in onInvocation function of ElementController you can remove these 'unwanted' properties, but how can I achieve that with current implementation of Siddhi wrapper?)
* The implementation of simple lists in Siddhi wrapper works well. But there are only two implemented functions for lists (contains and collect). I believe it would be good to implement other commonly used list functions in Siddhi, such as size(), add(), remove(), etc. In addition, having support for complex lists would be good, as well
* If I noticed correctly, the function count() is not supported by the object model of Siddhi EPL in this version (e.g., "from ... select count() as numOfOccurrences insert into ..."). It would be good to go over basic functions/features of Siddhi EPL and make sure that they get implemented first.
In addition, Siddhi has a lot of functions/features and extensions regarding Complex Event Processing and covering all of them in StreamPipes may be exhausting and, generally, not desired. Is there a plan to integrate other functions in object model (regarding lists and, possibly, other extensions) and which ones? Or, will we ('developer' users) need to implement them by ourselves (somehow?) in case we need them?
Let me know if you have any questions.
Kindest regards,
Branislav
________________________________
From: Dominik Riemer <ri...@apache.org>
Sent: Monday, October 26, 2020 22:11
To: dev@streampipes.apache.org <de...@streampipes.apache.org>
Subject: Improvements to the Siddhi wrapper
Hi all,
Branislav has recently asked for list support in the Siddhi wrapper (e.g.,
filter a list for contained items).
I took the opportunity to extend and improve the Siddhi wrapper in some
ways:
- Support for lists (see example processors at [1, 2])
- Possibility to create a new Siddhi processor from a single class (no
controller and parameter class needed anymore, there is also a prototype of
the same structure available for the standalone wrapper
(StreamPipesDataProcessor)
- I added a simple object model to create Siddhi EPL statements instead of
the previous string-based approach. This should the creation of Siddhi
processors less error-prone. The object model is still rather incomplete and
undocumented, but it should be easy to extend in the future - the code is at
[3]
- Improved code structure of the whole wrapper to make it easier to extend
in the future
What do you think? Any feedback is welcome!
Also @Grainier, as you are the Siddhi expert, in case you have any ideas for
improvements, let me know!
Dominik
[1]
https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampi
pes-processors-filters-siddhi/src/main/java/org/apache/streampipes/processor
s/siddhi/listfilter/ListFilter.java
[2]
https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampi
pes-processors-filters-siddhi/src/main/java/org/apache/streampipes/processor
s/siddhi/listcollector/ListCollector.java
[3]
https://github.com/apache/incubator-streampipes/tree/dev/streampipes-wrapper
-siddhi/src/main/java/org/apache/streampipes/wrapper/siddhi/query