You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by Grainier Perera <gr...@apache.org> on 2020/05/31 14:48:18 UTC
PE to rate-limit events
Hi all,
As per the discussion on [1], I've implemented a PE to rate-limit events.
For now, it supports the following configurations, and I believe it covers
most rate-limit use cases. What do you think?
- window type: time/length
- window size.
- event selection: first/last/all
- event grouping: by field (for each unique field value, it'll create a
separate window)
Issue: https://issues.apache.org/jira/browse/STREAMPIPES-147
PR: https://github.com/apache/incubator-streampipes-extensions/pull/18
[1]
https://lists.apache.org/thread.html/r836afda725bab338d81174c3f37b425f3cdd95c7b8699b1841d93170%40%3Cdev.streampipes.apache.org%3E
Grainier Perera.
Re: PE to rate-limit events
Posted by Grainier Perera <gr...@apache.org>.
@Philipp,
I didn't notice that earlier. Thank you for adding that to
'streampipes-pipeline-elements-all' module. :)
@Patrick,
+1 to the idea of adding the window concept to the sdk/core. As you
mentioned since aggregations and even sequences/patterns can leverage
window concepts, it'll be really useful to have it at that level. I'll go
through the sdk/core and see how window concept can be generify to be used
at that level.
Grainier Perera.
On Wed, 3 Jun 2020 at 20:45, Patrick Wiener <wi...@apache.org> wrote:
> Hi Grainier,
>
> indeed a very nice extension to the toolbox.
>
> What I was thinking of is the following.
>
> Since you basically implemented window-semantic in plain java this is
> something we could need for various pipeline elements, e.g. a lightweight
> aggregation component in Java other than in Flink.
>
> So maybe it is a good idea to add such capabilities straight to the
> SDK/engine
> so that in case it is needed, pipeline element developers can leverage it.
>
> What do you guys think?
>
> Patrick
>
> > Am 03.06.2020 um 14:14 schrieb Philipp Zehnder <ze...@apache.org>:
> >
> > Hi Grainier,
> >
> > very cool component, especially the cron window is a very good idea.
> > I have to admit, I have to get used to the syntax ;) but the feature is
> super useful and this is a good way to make this component very flexible.
> >
> > By the way, in case you didn’t see it so far. In the extension project
> there is a module 'streampipes-pipeline-elements-all' and there I also
> registered your new components.
> > We use this project for the StreamPipes ‘lite’ version. All single host
> pipeline elements are registered there. Then we just have to start a single
> container and reduce the memory footprint for StreamPipes.
> >
> > Philipp
> >
> >
> >> On 1. Jun 2020, at 14:59, Grainier Perera <gr...@apache.org> wrote:
> >>
> >> Thanks Dominik. I've also added Cron window support with RateLimiting.
> So
> >> it gives the flexibility of having various time triggers instead of
> fixed
> >> intervals. What do you think?
> >>
> >> Issue: https://issues.apache.org/jira/browse/STREAMPIPES-151
> >> PR: https://github.com/apache/incubator-streampipes-extensions/pull/20
> >>
> >> Grainier Perera.
> >>
> >>
> >> On Mon, 1 Jun 2020 at 03:11, Dominik Riemer <ri...@apache.org> wrote:
> >>
> >>> This is super cool!
> >>> That's a component we should definitely highlight in the next release
> as
> >>> it can be used together with many other pipeline elements.
> >>>
> >>> Btw, I also moved the pipeline element overview from the documentation
> to
> >>> the main website, so that it's easier for users to discover available
> >>> pipeline elements and made the list searchable:
> >>> https://streampipes.apache.org/pipeline-elements.html
> >>>
> >>> Can't wait to add the new elements to this list after the next release
> 😊
> >>>
> >>> Dominik
> >>>
> >>> -----Original Message-----
> >>> From: Grainier Perera <gr...@apache.org>
> >>> Sent: Sunday, May 31, 2020 4:48 PM
> >>> To: dev@streampipes.apache.org
> >>> Subject: PE to rate-limit events
> >>>
> >>> Hi all,
> >>>
> >>> As per the discussion on [1], I've implemented a PE to rate-limit
> events.
> >>> For now, it supports the following configurations, and I believe it
> covers
> >>> most rate-limit use cases. What do you think?
> >>>
> >>> - window type: time/length
> >>> - window size.
> >>> - event selection: first/last/all
> >>> - event grouping: by field (for each unique field value, it'll create
> a
> >>> separate window)
> >>>
> >>> Issue: https://issues.apache.org/jira/browse/STREAMPIPES-147
> >>> PR: https://github.com/apache/incubator-streampipes-extensions/pull/18
> >>> [1]
> >>>
> >>>
> https://lists.apache.org/thread.html/r836afda725bab338d81174c3f37b425f3cdd95c7b8699b1841d93170%40%3Cdev.streampipes.apache.org%3E
> >>>
> >>> Grainier Perera.
> >>>
> >>>
> >
> >
>
>
Re: PE to rate-limit events
Posted by Patrick Wiener <wi...@apache.org>.
Hi Grainier,
indeed a very nice extension to the toolbox.
What I was thinking of is the following.
Since you basically implemented window-semantic in plain java this is
something we could need for various pipeline elements, e.g. a lightweight
aggregation component in Java other than in Flink.
So maybe it is a good idea to add such capabilities straight to the SDK/engine
so that in case it is needed, pipeline element developers can leverage it.
What do you guys think?
Patrick
> Am 03.06.2020 um 14:14 schrieb Philipp Zehnder <ze...@apache.org>:
>
> Hi Grainier,
>
> very cool component, especially the cron window is a very good idea.
> I have to admit, I have to get used to the syntax ;) but the feature is super useful and this is a good way to make this component very flexible.
>
> By the way, in case you didn’t see it so far. In the extension project there is a module 'streampipes-pipeline-elements-all' and there I also registered your new components.
> We use this project for the StreamPipes ‘lite’ version. All single host pipeline elements are registered there. Then we just have to start a single container and reduce the memory footprint for StreamPipes.
>
> Philipp
>
>
>> On 1. Jun 2020, at 14:59, Grainier Perera <gr...@apache.org> wrote:
>>
>> Thanks Dominik. I've also added Cron window support with RateLimiting. So
>> it gives the flexibility of having various time triggers instead of fixed
>> intervals. What do you think?
>>
>> Issue: https://issues.apache.org/jira/browse/STREAMPIPES-151
>> PR: https://github.com/apache/incubator-streampipes-extensions/pull/20
>>
>> Grainier Perera.
>>
>>
>> On Mon, 1 Jun 2020 at 03:11, Dominik Riemer <ri...@apache.org> wrote:
>>
>>> This is super cool!
>>> That's a component we should definitely highlight in the next release as
>>> it can be used together with many other pipeline elements.
>>>
>>> Btw, I also moved the pipeline element overview from the documentation to
>>> the main website, so that it's easier for users to discover available
>>> pipeline elements and made the list searchable:
>>> https://streampipes.apache.org/pipeline-elements.html
>>>
>>> Can't wait to add the new elements to this list after the next release 😊
>>>
>>> Dominik
>>>
>>> -----Original Message-----
>>> From: Grainier Perera <gr...@apache.org>
>>> Sent: Sunday, May 31, 2020 4:48 PM
>>> To: dev@streampipes.apache.org
>>> Subject: PE to rate-limit events
>>>
>>> Hi all,
>>>
>>> As per the discussion on [1], I've implemented a PE to rate-limit events.
>>> For now, it supports the following configurations, and I believe it covers
>>> most rate-limit use cases. What do you think?
>>>
>>> - window type: time/length
>>> - window size.
>>> - event selection: first/last/all
>>> - event grouping: by field (for each unique field value, it'll create a
>>> separate window)
>>>
>>> Issue: https://issues.apache.org/jira/browse/STREAMPIPES-147
>>> PR: https://github.com/apache/incubator-streampipes-extensions/pull/18
>>> [1]
>>>
>>> https://lists.apache.org/thread.html/r836afda725bab338d81174c3f37b425f3cdd95c7b8699b1841d93170%40%3Cdev.streampipes.apache.org%3E
>>>
>>> Grainier Perera.
>>>
>>>
>
>
Re: PE to rate-limit events
Posted by Philipp Zehnder <ze...@apache.org>.
Hi Grainier,
very cool component, especially the cron window is a very good idea.
I have to admit, I have to get used to the syntax ;) but the feature is super useful and this is a good way to make this component very flexible.
By the way, in case you didn’t see it so far. In the extension project there is a module 'streampipes-pipeline-elements-all' and there I also registered your new components.
We use this project for the StreamPipes ‘lite’ version. All single host pipeline elements are registered there. Then we just have to start a single container and reduce the memory footprint for StreamPipes.
Philipp
> On 1. Jun 2020, at 14:59, Grainier Perera <gr...@apache.org> wrote:
>
> Thanks Dominik. I've also added Cron window support with RateLimiting. So
> it gives the flexibility of having various time triggers instead of fixed
> intervals. What do you think?
>
> Issue: https://issues.apache.org/jira/browse/STREAMPIPES-151
> PR: https://github.com/apache/incubator-streampipes-extensions/pull/20
>
> Grainier Perera.
>
>
> On Mon, 1 Jun 2020 at 03:11, Dominik Riemer <ri...@apache.org> wrote:
>
>> This is super cool!
>> That's a component we should definitely highlight in the next release as
>> it can be used together with many other pipeline elements.
>>
>> Btw, I also moved the pipeline element overview from the documentation to
>> the main website, so that it's easier for users to discover available
>> pipeline elements and made the list searchable:
>> https://streampipes.apache.org/pipeline-elements.html
>>
>> Can't wait to add the new elements to this list after the next release 😊
>>
>> Dominik
>>
>> -----Original Message-----
>> From: Grainier Perera <gr...@apache.org>
>> Sent: Sunday, May 31, 2020 4:48 PM
>> To: dev@streampipes.apache.org
>> Subject: PE to rate-limit events
>>
>> Hi all,
>>
>> As per the discussion on [1], I've implemented a PE to rate-limit events.
>> For now, it supports the following configurations, and I believe it covers
>> most rate-limit use cases. What do you think?
>>
>> - window type: time/length
>> - window size.
>> - event selection: first/last/all
>> - event grouping: by field (for each unique field value, it'll create a
>> separate window)
>>
>> Issue: https://issues.apache.org/jira/browse/STREAMPIPES-147
>> PR: https://github.com/apache/incubator-streampipes-extensions/pull/18
>> [1]
>>
>> https://lists.apache.org/thread.html/r836afda725bab338d81174c3f37b425f3cdd95c7b8699b1841d93170%40%3Cdev.streampipes.apache.org%3E
>>
>> Grainier Perera.
>>
>>
Re: PE to rate-limit events
Posted by Grainier Perera <gr...@apache.org>.
Thanks Dominik. I've also added Cron window support with RateLimiting. So
it gives the flexibility of having various time triggers instead of fixed
intervals. What do you think?
Issue: https://issues.apache.org/jira/browse/STREAMPIPES-151
PR: https://github.com/apache/incubator-streampipes-extensions/pull/20
Grainier Perera.
On Mon, 1 Jun 2020 at 03:11, Dominik Riemer <ri...@apache.org> wrote:
> This is super cool!
> That's a component we should definitely highlight in the next release as
> it can be used together with many other pipeline elements.
>
> Btw, I also moved the pipeline element overview from the documentation to
> the main website, so that it's easier for users to discover available
> pipeline elements and made the list searchable:
> https://streampipes.apache.org/pipeline-elements.html
>
> Can't wait to add the new elements to this list after the next release 😊
>
> Dominik
>
> -----Original Message-----
> From: Grainier Perera <gr...@apache.org>
> Sent: Sunday, May 31, 2020 4:48 PM
> To: dev@streampipes.apache.org
> Subject: PE to rate-limit events
>
> Hi all,
>
> As per the discussion on [1], I've implemented a PE to rate-limit events.
> For now, it supports the following configurations, and I believe it covers
> most rate-limit use cases. What do you think?
>
> - window type: time/length
> - window size.
> - event selection: first/last/all
> - event grouping: by field (for each unique field value, it'll create a
> separate window)
>
> Issue: https://issues.apache.org/jira/browse/STREAMPIPES-147
> PR: https://github.com/apache/incubator-streampipes-extensions/pull/18
> [1]
>
> https://lists.apache.org/thread.html/r836afda725bab338d81174c3f37b425f3cdd95c7b8699b1841d93170%40%3Cdev.streampipes.apache.org%3E
>
> Grainier Perera.
>
>
RE: PE to rate-limit events
Posted by Dominik Riemer <ri...@apache.org>.
This is super cool!
That's a component we should definitely highlight in the next release as it can be used together with many other pipeline elements.
Btw, I also moved the pipeline element overview from the documentation to the main website, so that it's easier for users to discover available pipeline elements and made the list searchable:
https://streampipes.apache.org/pipeline-elements.html
Can't wait to add the new elements to this list after the next release 😊
Dominik
-----Original Message-----
From: Grainier Perera <gr...@apache.org>
Sent: Sunday, May 31, 2020 4:48 PM
To: dev@streampipes.apache.org
Subject: PE to rate-limit events
Hi all,
As per the discussion on [1], I've implemented a PE to rate-limit events.
For now, it supports the following configurations, and I believe it covers most rate-limit use cases. What do you think?
- window type: time/length
- window size.
- event selection: first/last/all
- event grouping: by field (for each unique field value, it'll create a
separate window)
Issue: https://issues.apache.org/jira/browse/STREAMPIPES-147
PR: https://github.com/apache/incubator-streampipes-extensions/pull/18
[1]
https://lists.apache.org/thread.html/r836afda725bab338d81174c3f37b425f3cdd95c7b8699b1841d93170%40%3Cdev.streampipes.apache.org%3E
Grainier Perera.