You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by PedroMrChaves <pe...@gmail.com> on 2016/11/03 17:30:21 UTC
Re: What is the best way to load/add patterns dynamically (at
runtime) with Flink?
Hello,
Your tip was very helpful and I took a similar approach.
I have something like this:
class Processor extends RichCoFlatMapFunction<Event, Rule, String> {
public void flatMap1(Event event, Collector<String> out) {
process(event,out); // run the javscript (rules) against the
incoming events
}
public void flatMap2(Rule rule , Collector<String> out) {
// We add the rule to the list of existing rules
addNewRule(rule)
}
}
But know I face a new challenge, I don't have access to the windowed
constructs of flink and I can't dynamically create new window aggregations
inside the flatMap. At least not that I know of.
Did you face a similar problem? Any Ideas?
Thank you and regards,
Pedro Chaves
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/What-is-the-best-way-to-load-add-patterns-dynamically-at-runtime-with-Flink-tp9461p9876.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: What is the best way to load/add patterns dynamically (at
runtime) with Flink?
Posted by PedroMrChaves <pe...@gmail.com>.
The best answer I can give you is the one given in the post. Currently,
there is no way of dynamically changing the patterns.
The only way would be to dive into Flink's core code and change the way
operators are shipped to the cluster.
On Thu, Nov 24, 2016 at 3:34 PM, kaelumania [via Apache Flink User Mailing
List archive.] <ml...@n4.nabble.com> wrote:
> I also found this really interesting post
>
> http://stackoverflow.com/questions/40018199/flink-and-
> dynamic-templates-recognition
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/What-is-the-best-way-to-load-add-patterns-
> dynamically-at-runtime-with-Flink-tp9461p10326.html
> To unsubscribe from What is the best way to load/add patterns dynamically
> (at runtime) with Flink?, click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=9461&code=cGVkcm8ubXIuY2hhdmVzQGdtYWlsLmNvbXw5NDYxfDE1MjI4MzU3MDg=>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
-----
Best Regards,
Pedro Chaves
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/What-is-the-best-way-to-load-add-patterns-dynamically-at-runtime-with-Flink-tp9461p10328.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: What is the best way to load/add patterns dynamically (at runtime) with Flink?
Posted by Stephan Epping <st...@zweitag.de>.
I also found this really interesting post
http://stackoverflow.com/questions/40018199/flink-and-dynamic-templates-recognition <http://stackoverflow.com/questions/40018199/flink-and-dynamic-templates-recognition>
Re: What is the best way to load/add patterns dynamically (at
runtime) with Flink?
Posted by kaelumania <st...@zweitag.de>.
Hey,
the javascript solution seems very limited. Is there a solution with
compiling new patterns to native Flink CEP Patterns and add them to a stream
dynamically?
best Stephan
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/What-is-the-best-way-to-load-add-patterns-dynamically-at-runtime-with-Flink-tp9461p10325.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: What is the best way to load/add patterns dynamically (at
runtime) with Flink?
Posted by Aljoscha Krettek <al...@apache.org>.
Hi Pedro,
yes, I was more or less suggesting a similar approach to the one taken by
King. In code, it would look somewhat like this:
DataStream<T> input = ...;
DataStream<Tuple2<T, DynamicWindow>> withMyWindows = input.map(new
AssignMyWindows())
withMyWindows
.keyBy(...)
.window(new DynamicWindowAssigner())
...
where this is DynamicWindowAssigner:
public class DynamicWindowAssigner<T> extends WindowAssigner<Tuple2<T,
DynamicWindow>, TimeWindow> {
@Override
public Collection<TimeWindow> assignWindows(Tuple2<T,
DynamicWindow> element, long timestamp, WindowAssignerContext context) {
// extract some info from the dynamic window
return Collections.singletonList(new TimeWindow(..., ...)); // <-
this can also be a custom window
}
...
}
Cheers,
Aljoscha
On Thu, 3 Nov 2016 at 23:47 PedroMrChaves <pe...@gmail.com> wrote:
> Hi,
>
> Thank you for the response.
>
> Can you give me an example?
> I'm new to Flink and I still don't understand all the constructs.
>
> I also read this article
> https://techblog.king.com/rbea-scalable-real-time-analytics-king/. They
> use a similar approach, but am still not understanding how assign windows.
>
> Regards,
> Pedro Chaves
>
>
>
> On Thu, Nov 3, 2016 at 6:02 PM, Aljoscha Krettek [via Apache Flink User
> Mailing List archive.] <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9891&i=0>> wrote:
>
> Hi Pedro,
> you can have dynamic windows by assigning the windows to elements in your
> Processor (so you would need to extend that type to have a field for the
> window). Then, you can write a custom WindowAssigner that will simply get
> the window from an event and assign that as the internal window.
>
> Please let me know if you need more details.
>
> Cheers,
> Aljoscha
>
> On Thu, 3 Nov 2016 at 18:40 PedroMrChaves <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9882&i=0>> wrote:
>
> Hello,
>
> Your tip was very helpful and I took a similar approach.
>
> I have something like this:
> class Processor extends RichCoFlatMapFunction<Event, Rule, String> {
> public void flatMap1(Event event, Collector<String> out) {
> process(event,out); // run the javscript (rules) against the
> incoming events
> }
>
> public void flatMap2(Rule rule , Collector<String> out) {
> // We add the rule to the list of existing rules
> addNewRule(rule)
> }
> }
>
> But know I face a new challenge, I don't have access to the windowed
> constructs of flink and I can't dynamically create new window aggregations
> inside the flatMap. At least not that I know of.
>
> Did you face a similar problem? Any Ideas?
>
> Thank you and regards,
> Pedro Chaves
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/What-is-the-best-way-to-load-add-patterns-dynamically-at-runtime-with-Flink-tp9461p9876.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/What-is-the-best-way-to-load-add-patterns-dynamically-at-runtime-with-Flink-tp9461p9882.html
> To unsubscribe from What is the best way to load/add patterns dynamically
> (at runtime) with Flink?, click here.
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
> ------------------------------
> View this message in context: Re: What is the best way to load/add
> patterns dynamically (at runtime) with Flink?
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/What-is-the-best-way-to-load-add-patterns-dynamically-at-runtime-with-Flink-tp9461p9891.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com.
>
Re: What is the best way to load/add patterns dynamically (at
runtime) with Flink?
Posted by PedroMrChaves <pe...@gmail.com>.
Hi,
Thank you for the response.
Can you give me an example?
I'm new to Flink and I still don't understand all the constructs.
I also read this article
https://techblog.king.com/rbea-scalable-real-time-analytics-king/. They use
a similar approach, but am still not understanding how assign windows.
Regards,
Pedro Chaves
On Thu, Nov 3, 2016 at 6:02 PM, Aljoscha Krettek [via Apache Flink User
Mailing List archive.] <ml...@n4.nabble.com> wrote:
> Hi Pedro,
> you can have dynamic windows by assigning the windows to elements in your
> Processor (so you would need to extend that type to have a field for the
> window). Then, you can write a custom WindowAssigner that will simply get
> the window from an event and assign that as the internal window.
>
> Please let me know if you need more details.
>
> Cheers,
> Aljoscha
>
> On Thu, 3 Nov 2016 at 18:40 PedroMrChaves <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9882&i=0>> wrote:
>
>> Hello,
>>
>> Your tip was very helpful and I took a similar approach.
>>
>> I have something like this:
>> class Processor extends RichCoFlatMapFunction<Event, Rule, String> {
>> public void flatMap1(Event event, Collector<String> out) {
>> process(event,out); // run the javscript (rules) against the
>> incoming events
>> }
>>
>> public void flatMap2(Rule rule , Collector<String> out) {
>> // We add the rule to the list of existing rules
>> addNewRule(rule)
>> }
>> }
>>
>> But know I face a new challenge, I don't have access to the windowed
>> constructs of flink and I can't dynamically create new window aggregations
>> inside the flatMap. At least not that I know of.
>>
>> Did you face a similar problem? Any Ideas?
>>
>> Thank you and regards,
>> Pedro Chaves
>>
>>
>>
>> --
>> View this message in context: http://apache-flink-user-
>> mailing-list-archive.2336050.n4.nabble.com/What-is-the-
>> best-way-to-load-add-patterns-dynamically-at-runtime-with-
>> Flink-tp9461p9876.html
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive at Nabble.com.
>>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/What-is-the-best-way-to-load-add-patterns-
> dynamically-at-runtime-with-Flink-tp9461p9882.html
> To unsubscribe from What is the best way to load/add patterns dynamically
> (at runtime) with Flink?, click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=9461&code=cGVkcm8ubXIuY2hhdmVzQGdtYWlsLmNvbXw5NDYxfDE1MjI4MzU3MDg=>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/What-is-the-best-way-to-load-add-patterns-dynamically-at-runtime-with-Flink-tp9461p9891.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: What is the best way to load/add patterns dynamically (at
runtime) with Flink?
Posted by Aljoscha Krettek <al...@apache.org>.
Hi Pedro,
you can have dynamic windows by assigning the windows to elements in your
Processor (so you would need to extend that type to have a field for the
window). Then, you can write a custom WindowAssigner that will simply get
the window from an event and assign that as the internal window.
Please let me know if you need more details.
Cheers,
Aljoscha
On Thu, 3 Nov 2016 at 18:40 PedroMrChaves <pe...@gmail.com> wrote:
> Hello,
>
> Your tip was very helpful and I took a similar approach.
>
> I have something like this:
> class Processor extends RichCoFlatMapFunction<Event, Rule, String> {
> public void flatMap1(Event event, Collector<String> out) {
> process(event,out); // run the javscript (rules) against the
> incoming events
> }
>
> public void flatMap2(Rule rule , Collector<String> out) {
> // We add the rule to the list of existing rules
> addNewRule(rule)
> }
> }
>
> But know I face a new challenge, I don't have access to the windowed
> constructs of flink and I can't dynamically create new window aggregations
> inside the flatMap. At least not that I know of.
>
> Did you face a similar problem? Any Ideas?
>
> Thank you and regards,
> Pedro Chaves
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/What-is-the-best-way-to-load-add-patterns-dynamically-at-runtime-with-Flink-tp9461p9876.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>