You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@fluo.apache.org by Keith Turner <ke...@deenlo.com> on 2017/02/01 00:43:02 UTC
Re: third party service to poll Fluo for absence of event
On Tue, Jan 31, 2017 at 3:50 PM, Meier, Caleb <Ca...@parsons.com> wrote:
> Hello,
>
> I’m looking into using Fluo to develop an event based notification system that incrementally generates events of increasing complexity. The one issue that I’m running into is how to handle the non-event event. That is, Fluo (as I understand it) is not well-suited to handle the following request: “generate a notification if no events of a given type have occurred within the last 24 hours”. This is because it is a push based notification framework that only generates notifications when things actually happen. So the question is, has anyone looked into developing a service for generating notifications at regular intervals (even if something doesn’t happen) that works with Fluo? I’m toying with the idea of creating some sort of Twill application that tells Fluo to wake up at regular intervals to generate a notification about the set of events falling within the given time window. Before doing this I just wanted to make sure that something like this does not already exist, and I also want to get a sense of how bad an idea it is to delegate some of the logic of this periodic notification service to Fluo. Would it be better to separate out the temporal portion of my notification request to be processed entirely outside of Fluo to avoid transactional overhead?
>
At one point in time I was thinking of delayed notifications, but I
can't remember what my specific use case was. I am not sure if this is
good idea or not.
How many event types are there in your use case? Just curious about
the order of magnitude. Like is it ~100 event types and millions of
events?
Just to make sure I understand your use case, are you interested in
finding which event types were not updated in the last 24hrs? Or are
you interested in finding which events were not updated in the last
24hrs? If none of these, would you be able to describe the use case
you want with an example?
> Caleb A. Meier, Ph.D.
> Software Engineer II ♦ Analyst
> Parsons Corporation
> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
> Office: (703)797-3066
> Caleb.Meier@Parsons.com<ma...@Parsons.com> ♦ www.parsons.com<https://webportal.parsons.com/,DanaInfo=www.parsons.com+>
>
Re: third party service to poll Fluo for absence of event
Posted by Keith Turner <ke...@deenlo.com>.
On Wed, Feb 1, 2017 at 9:53 AM, Meier, Caleb <Ca...@parsons.com> wrote:
> Hey Keith,
>
> Right now the project is in its infancy so I don't have concrete examples of how events are distributed, but I think we would like everything to scale as needed. Right now, the user is in charge of how events are modeled and how many different event types are included.
>
> Regarding the use cases that you mentioned, I think we would be more interested in find which event types haven't been updated or haven't occurred within the last 24 hours. I'm not ruling out the possibility of tracking updates to an individual event though.
>
> More generally, we're interested in issuing periodic notifications. The non-event event is kind of a canonical example. Another use case would be "generate a notification if more than 100 events of a given type have occurred within the last 24 hours", and the user specifies that they want a notification to be generated every hour. In the case that 1000 events happen in the first hour and then no other events occur during that 24 hour period, 1000 notifications would be generated by Fluo during the first hour, but then nothing would happen the remaining 23 hours. We would need some service (that possibly utilizes an observer if that makes sense) to periodically detect how many events have occurred within the specified window and generate a notification. So in this case, every hour for the next 23 hours, something would check and find that 1000 events occurred within the window and generate a notification.
>
> Did that clarify things?
Yeah. I feel like this could be broken into two parts.
* A many to few update recipe with short circuiting.
* Something that periodically scans the "few" row/cols and possibly
injects a new action.
Below is an example to show what I mean by short circuiting.
1. Processing event 100934 with event type X
2. Update the last update hour for event type X only if its value is
less than current hour
The Collision Free Map (CFM) could be used for these many to few
updates. However, it lacks the short circuit optimization. It will
always queue the updates, even when the applied update will not change
the current value.
The external scan service periodically scan the few event type last
update times and inject new notifications. This mechanism could use
transactions to avoid problems from accidentally running multiple
external services.
> ________________________________________
> From: Keith Turner <ke...@deenlo.com>
> Sent: Tuesday, January 31, 2017 6:43 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: third party service to poll Fluo for absence of event
>
> On Tue, Jan 31, 2017 at 3:50 PM, Meier, Caleb <Ca...@parsons.com> wrote:
>> Hello,
>>
>> I’m looking into using Fluo to develop an event based notification system that incrementally generates events of increasing complexity. The one issue that I’m running into is how to handle the non-event event. That is, Fluo (as I understand it) is not well-suited to handle the following request: “generate a notification if no events of a given type have occurred within the last 24 hours”. This is because it is a push based notification framework that only generates notifications when things actually happen. So the question is, has anyone looked into developing a service for generating notifications at regular intervals (even if something doesn’t happen) that works with Fluo? I’m toying with the idea of creating some sort of Twill application that tells Fluo to wake up at regular intervals to generate a notification about the set of events falling within the given time window. Before doing this I just wanted to make sure that something like this does not already exist, and I also want to get a sense of how bad an idea it is to delegate some of the logic of this periodic notification service to Fluo. Would it be better to separate out the temporal portion of my notification request to be processed entirely outside of Fluo to avoid transactional overhead?
>>
>
> At one point in time I was thinking of delayed notifications, but I
> can't remember what my specific use case was. I am not sure if this is
> good idea or not.
>
> How many event types are there in your use case? Just curious about
> the order of magnitude. Like is it ~100 event types and millions of
> events?
>
> Just to make sure I understand your use case, are you interested in
> finding which event types were not updated in the last 24hrs? Or are
> you interested in finding which events were not updated in the last
> 24hrs? If none of these, would you be able to describe the use case
> you want with an example?
>
>> Caleb A. Meier, Ph.D.
>> Software Engineer II ♦ Analyst
>> Parsons Corporation
>> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
>> Office: (703)797-3066
>> Caleb.Meier@Parsons.com<ma...@Parsons.com> ♦ www.parsons.com<http://www.parsons.com/>
>>
Re: third party service to poll Fluo for absence of event
Posted by "Meier, Caleb" <Ca...@parsons.com>.
Hey Keith,
Right now the project is in its infancy so I don't have concrete examples of how events are distributed, but I think we would like everything to scale as needed. Right now, the user is in charge of how events are modeled and how many different event types are included.
Regarding the use cases that you mentioned, I think we would be more interested in find which event types haven't been updated or haven't occurred within the last 24 hours. I'm not ruling out the possibility of tracking updates to an individual event though.
More generally, we're interested in issuing periodic notifications. The non-event event is kind of a canonical example. Another use case would be "generate a notification if more than 100 events of a given type have occurred within the last 24 hours", and the user specifies that they want a notification to be generated every hour. In the case that 1000 events happen in the first hour and then no other events occur during that 24 hour period, 1000 notifications would be generated by Fluo during the first hour, but then nothing would happen the remaining 23 hours. We would need some service (that possibly utilizes an observer if that makes sense) to periodically detect how many events have occurred within the specified window and generate a notification. So in this case, every hour for the next 23 hours, something would check and find that 1000 events occurred within the window and generate a notification.
Did that clarify things?
________________________________________
From: Keith Turner <ke...@deenlo.com>
Sent: Tuesday, January 31, 2017 6:43 PM
To: dev@fluo.incubator.apache.org
Subject: Re: third party service to poll Fluo for absence of event
On Tue, Jan 31, 2017 at 3:50 PM, Meier, Caleb <Ca...@parsons.com> wrote:
> Hello,
>
> I’m looking into using Fluo to develop an event based notification system that incrementally generates events of increasing complexity. The one issue that I’m running into is how to handle the non-event event. That is, Fluo (as I understand it) is not well-suited to handle the following request: “generate a notification if no events of a given type have occurred within the last 24 hours”. This is because it is a push based notification framework that only generates notifications when things actually happen. So the question is, has anyone looked into developing a service for generating notifications at regular intervals (even if something doesn’t happen) that works with Fluo? I’m toying with the idea of creating some sort of Twill application that tells Fluo to wake up at regular intervals to generate a notification about the set of events falling within the given time window. Before doing this I just wanted to make sure that something like this does not already exist, and I also want to get a sense of how bad an idea it is to delegate some of the logic of this periodic notification service to Fluo. Would it be better to separate out the temporal portion of my notification request to be processed entirely outside of Fluo to avoid transactional overhead?
>
At one point in time I was thinking of delayed notifications, but I
can't remember what my specific use case was. I am not sure if this is
good idea or not.
How many event types are there in your use case? Just curious about
the order of magnitude. Like is it ~100 event types and millions of
events?
Just to make sure I understand your use case, are you interested in
finding which event types were not updated in the last 24hrs? Or are
you interested in finding which events were not updated in the last
24hrs? If none of these, would you be able to describe the use case
you want with an example?
> Caleb A. Meier, Ph.D.
> Software Engineer II ♦ Analyst
> Parsons Corporation
> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
> Office: (703)797-3066
> Caleb.Meier@Parsons.com<ma...@Parsons.com> ♦ www.parsons.com<http://www.parsons.com/>
>