You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@fluo.apache.org by "Meier, Caleb" <Ca...@parsons.com> on 2017/01/31 20:50:42 UTC
third party service to poll Fluo for absence of event
Hello,
I’m looking into using Fluo to develop an event based notification system that incrementally generates events of increasing complexity. The one issue that I’m running into is how to handle the non-event event. That is, Fluo (as I understand it) is not well-suited to handle the following request: “generate a notification if no events of a given type have occurred within the last 24 hours”. This is because it is a push based notification framework that only generates notifications when things actually happen. So the question is, has anyone looked into developing a service for generating notifications at regular intervals (even if something doesn’t happen) that works with Fluo? I’m toying with the idea of creating some sort of Twill application that tells Fluo to wake up at regular intervals to generate a notification about the set of events falling within the given time window. Before doing this I just wanted to make sure that something like this does not already exist, and I also want to get a sense of how bad an idea it is to delegate some of the logic of this periodic notification service to Fluo. Would it be better to separate out the temporal portion of my notification request to be processed entirely outside of Fluo to avoid transactional overhead?
Caleb A. Meier, Ph.D.
Software Engineer II ♦ Analyst
Parsons Corporation
1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
Office: (703)797-3066
Caleb.Meier@Parsons.com<ma...@Parsons.com> ♦ www.parsons.com<https://webportal.parsons.com/,DanaInfo=www.parsons.com+>
Re: third party service to poll Fluo for absence of event
Posted by Keith Turner <ke...@deenlo.com>.
On Wed, Feb 1, 2017 at 9:53 AM, Meier, Caleb <Ca...@parsons.com> wrote:
> Hey Keith,
>
> Right now the project is in its infancy so I don't have concrete examples of how events are distributed, but I think we would like everything to scale as needed. Right now, the user is in charge of how events are modeled and how many different event types are included.
>
> Regarding the use cases that you mentioned, I think we would be more interested in find which event types haven't been updated or haven't occurred within the last 24 hours. I'm not ruling out the possibility of tracking updates to an individual event though.
>
> More generally, we're interested in issuing periodic notifications. The non-event event is kind of a canonical example. Another use case would be "generate a notification if more than 100 events of a given type have occurred within the last 24 hours", and the user specifies that they want a notification to be generated every hour. In the case that 1000 events happen in the first hour and then no other events occur during that 24 hour period, 1000 notifications would be generated by Fluo during the first hour, but then nothing would happen the remaining 23 hours. We would need some service (that possibly utilizes an observer if that makes sense) to periodically detect how many events have occurred within the specified window and generate a notification. So in this case, every hour for the next 23 hours, something would check and find that 1000 events occurred within the window and generate a notification.
>
> Did that clarify things?
Yeah. I feel like this could be broken into two parts.
* A many to few update recipe with short circuiting.
* Something that periodically scans the "few" row/cols and possibly
injects a new action.
Below is an example to show what I mean by short circuiting.
1. Processing event 100934 with event type X
2. Update the last update hour for event type X only if its value is
less than current hour
The Collision Free Map (CFM) could be used for these many to few
updates. However, it lacks the short circuit optimization. It will
always queue the updates, even when the applied update will not change
the current value.
The external scan service periodically scan the few event type last
update times and inject new notifications. This mechanism could use
transactions to avoid problems from accidentally running multiple
external services.
> ________________________________________
> From: Keith Turner <ke...@deenlo.com>
> Sent: Tuesday, January 31, 2017 6:43 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: third party service to poll Fluo for absence of event
>
> On Tue, Jan 31, 2017 at 3:50 PM, Meier, Caleb <Ca...@parsons.com> wrote:
>> Hello,
>>
>> I’m looking into using Fluo to develop an event based notification system that incrementally generates events of increasing complexity. The one issue that I’m running into is how to handle the non-event event. That is, Fluo (as I understand it) is not well-suited to handle the following request: “generate a notification if no events of a given type have occurred within the last 24 hours”. This is because it is a push based notification framework that only generates notifications when things actually happen. So the question is, has anyone looked into developing a service for generating notifications at regular intervals (even if something doesn’t happen) that works with Fluo? I’m toying with the idea of creating some sort of Twill application that tells Fluo to wake up at regular intervals to generate a notification about the set of events falling within the given time window. Before doing this I just wanted to make sure that something like this does not already exist, and I also want to get a sense of how bad an idea it is to delegate some of the logic of this periodic notification service to Fluo. Would it be better to separate out the temporal portion of my notification request to be processed entirely outside of Fluo to avoid transactional overhead?
>>
>
> At one point in time I was thinking of delayed notifications, but I
> can't remember what my specific use case was. I am not sure if this is
> good idea or not.
>
> How many event types are there in your use case? Just curious about
> the order of magnitude. Like is it ~100 event types and millions of
> events?
>
> Just to make sure I understand your use case, are you interested in
> finding which event types were not updated in the last 24hrs? Or are
> you interested in finding which events were not updated in the last
> 24hrs? If none of these, would you be able to describe the use case
> you want with an example?
>
>> Caleb A. Meier, Ph.D.
>> Software Engineer II ♦ Analyst
>> Parsons Corporation
>> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
>> Office: (703)797-3066
>> Caleb.Meier@Parsons.com<ma...@Parsons.com> ♦ www.parsons.com<http://www.parsons.com/>
>>
Re: third party service to poll Fluo for absence of event
Posted by "Meier, Caleb" <Ca...@parsons.com>.
Hey Keith,
Right now the project is in its infancy so I don't have concrete examples of how events are distributed, but I think we would like everything to scale as needed. Right now, the user is in charge of how events are modeled and how many different event types are included.
Regarding the use cases that you mentioned, I think we would be more interested in find which event types haven't been updated or haven't occurred within the last 24 hours. I'm not ruling out the possibility of tracking updates to an individual event though.
More generally, we're interested in issuing periodic notifications. The non-event event is kind of a canonical example. Another use case would be "generate a notification if more than 100 events of a given type have occurred within the last 24 hours", and the user specifies that they want a notification to be generated every hour. In the case that 1000 events happen in the first hour and then no other events occur during that 24 hour period, 1000 notifications would be generated by Fluo during the first hour, but then nothing would happen the remaining 23 hours. We would need some service (that possibly utilizes an observer if that makes sense) to periodically detect how many events have occurred within the specified window and generate a notification. So in this case, every hour for the next 23 hours, something would check and find that 1000 events occurred within the window and generate a notification.
Did that clarify things?
________________________________________
From: Keith Turner <ke...@deenlo.com>
Sent: Tuesday, January 31, 2017 6:43 PM
To: dev@fluo.incubator.apache.org
Subject: Re: third party service to poll Fluo for absence of event
On Tue, Jan 31, 2017 at 3:50 PM, Meier, Caleb <Ca...@parsons.com> wrote:
> Hello,
>
> I’m looking into using Fluo to develop an event based notification system that incrementally generates events of increasing complexity. The one issue that I’m running into is how to handle the non-event event. That is, Fluo (as I understand it) is not well-suited to handle the following request: “generate a notification if no events of a given type have occurred within the last 24 hours”. This is because it is a push based notification framework that only generates notifications when things actually happen. So the question is, has anyone looked into developing a service for generating notifications at regular intervals (even if something doesn’t happen) that works with Fluo? I’m toying with the idea of creating some sort of Twill application that tells Fluo to wake up at regular intervals to generate a notification about the set of events falling within the given time window. Before doing this I just wanted to make sure that something like this does not already exist, and I also want to get a sense of how bad an idea it is to delegate some of the logic of this periodic notification service to Fluo. Would it be better to separate out the temporal portion of my notification request to be processed entirely outside of Fluo to avoid transactional overhead?
>
At one point in time I was thinking of delayed notifications, but I
can't remember what my specific use case was. I am not sure if this is
good idea or not.
How many event types are there in your use case? Just curious about
the order of magnitude. Like is it ~100 event types and millions of
events?
Just to make sure I understand your use case, are you interested in
finding which event types were not updated in the last 24hrs? Or are
you interested in finding which events were not updated in the last
24hrs? If none of these, would you be able to describe the use case
you want with an example?
> Caleb A. Meier, Ph.D.
> Software Engineer II ♦ Analyst
> Parsons Corporation
> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
> Office: (703)797-3066
> Caleb.Meier@Parsons.com<ma...@Parsons.com> ♦ www.parsons.com<http://www.parsons.com/>
>
Re: third party service to poll Fluo for absence of event
Posted by Keith Turner <ke...@deenlo.com>.
On Tue, Jan 31, 2017 at 3:50 PM, Meier, Caleb <Ca...@parsons.com> wrote:
> Hello,
>
> I’m looking into using Fluo to develop an event based notification system that incrementally generates events of increasing complexity. The one issue that I’m running into is how to handle the non-event event. That is, Fluo (as I understand it) is not well-suited to handle the following request: “generate a notification if no events of a given type have occurred within the last 24 hours”. This is because it is a push based notification framework that only generates notifications when things actually happen. So the question is, has anyone looked into developing a service for generating notifications at regular intervals (even if something doesn’t happen) that works with Fluo? I’m toying with the idea of creating some sort of Twill application that tells Fluo to wake up at regular intervals to generate a notification about the set of events falling within the given time window. Before doing this I just wanted to make sure that something like this does not already exist, and I also want to get a sense of how bad an idea it is to delegate some of the logic of this periodic notification service to Fluo. Would it be better to separate out the temporal portion of my notification request to be processed entirely outside of Fluo to avoid transactional overhead?
>
At one point in time I was thinking of delayed notifications, but I
can't remember what my specific use case was. I am not sure if this is
good idea or not.
How many event types are there in your use case? Just curious about
the order of magnitude. Like is it ~100 event types and millions of
events?
Just to make sure I understand your use case, are you interested in
finding which event types were not updated in the last 24hrs? Or are
you interested in finding which events were not updated in the last
24hrs? If none of these, would you be able to describe the use case
you want with an example?
> Caleb A. Meier, Ph.D.
> Software Engineer II ♦ Analyst
> Parsons Corporation
> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
> Office: (703)797-3066
> Caleb.Meier@Parsons.com<ma...@Parsons.com> ♦ www.parsons.com<https://webportal.parsons.com/,DanaInfo=www.parsons.com+>
>
RE: third party service to poll Fluo for absence of event
Posted by "Meier, Caleb" <Ca...@parsons.com>.
Thanks a lot Keith. That was really helpful. I'll trying tinkering with the fluo.impl.ScanTask.maxSleep property to see if I can liven things up a bit.
Caleb A. Meier, Ph.D.
Software Engineer II ♦ Analyst
Parsons Corporation
1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
Office: (703)797-3066
Caleb.Meier@Parsons.com ♦ www.parsons.com
-----Original Message-----
From: Keith Turner [mailto:keith@deenlo.com]
Sent: Thursday, February 16, 2017 6:02 PM
To: dev@fluo.incubator.apache.org
Subject: Re: third party service to poll Fluo for absence of event
On Thu, Feb 16, 2017 at 12:08 PM, Meier, Caleb <Ca...@parsons.com> wrote:
> Quick question. How timely is Fluo when it comes to processing notifications? If there are enough workers, will a notification be processed in a timely manner after writing to the observed column? Earlier we had a discussion about a periodic query service.
> If I write a notification to issue a periodic query to Fluo, can I expect that my Observer will process that query fairly quickly (provided there are enough workers)?
Fluo keeps track of the last time it saw a notification in a tablet and exponentially increases the scan period for that tablet when it keeps seeing nothing. The increase is up to a configurable maximum.
The following code does the backoff.
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dfluo_blob_rel_fluo-2D1.0.0-2Dincubating_modules_core_src_main_java_org_apache_fluo_core_worker_finder_hash_TabletData.java&d=CwIFaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=pSG3R4ixSmXTc6ylLxMgs2ZWPSv1UbQN6qOvmuscQIk&s=Kt-fnyzs5et-oFglhm59HR9yOWLcQYWraarJY3h1vnM&e=
The following is where it gets the max sleep time.
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dfluo_blob_rel_fluo-2D1.0.0-2Dincubating_modules_core_src_main_java_org_apache_fluo_core_worker_finder_hash_ScanTask.java-23L72&d=CwIFaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=pSG3R4ixSmXTc6ylLxMgs2ZWPSv1UbQN6qOvmuscQIk&s=-Pp1MzlArxkS1JFEeJDP79fAOtthoQmDLl8HW-P8XsM&e=
Looking at the following, the default max sleep time for a tablet is 5
minutes. If I expanded the constant correctly, this can be changed
by setting fluo.impl.ScanTask.maxSleep. Note, impl properties are not part of the public API and are subject to change when the implementation changes.
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dfluo_blob_rel_fluo-2D1.0.0-2Dincubating_modules_core_src_main_java_org_apache_fluo_core_impl_FluoConfigurationImpl.java-23L37&d=CwIFaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=pSG3R4ixSmXTc6ylLxMgs2ZWPSv1UbQN6qOvmuscQIk&s=GxvVrABsKcsKx_E_NPwxWqibuzTd8uSJFiOV5Saf2jE&e=
Also, when Fluo has notifications queued, it will wait till the queue size halves before scanning again for notifications. So if 10,000 notifications were queued, then scanning for notifications would not happen until the queue size was 5,000 or less. The following code shows where that happens. I noticed while looking for the max scan sleep code.
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dfluo_blob_rel_fluo-2D1.0.0-2Dincubating_modules_core_src_main_java_org_apache_fluo_core_worker_finder_hash_ScanTask.java-23L85&d=CwIFaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=pSG3R4ixSmXTc6ylLxMgs2ZWPSv1UbQN6qOvmuscQIk&s=jnFRdNJK4RxHd3qjLcWZJIQTXY1_sHjVhNEebg5GWA4&e=
>
> Caleb A. Meier, Ph.D.
> Software Engineer II ♦ Analyst
> Parsons Corporation
> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
> Office: (703)797-3066
> Caleb.Meier@Parsons.com ♦ www.parsons.com
>
> -----Original Message-----
> From: Keith Turner [mailto:keith@deenlo.com]
> Sent: Wednesday, February 01, 2017 11:03 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: third party service to poll Fluo for absence of event
>
> On Wed, Feb 1, 2017 at 9:54 PM, Christopher <ct...@apache.org> wrote:
>> On Wed, Feb 1, 2017 at 10:04 AM Meier, Caleb
>> <Ca...@parsons.com>
>> wrote:
>>
>>> Yeah, this seems pretty reasonable to me. I guess it then boils
>>> down to the nitty gritty of do I store results in Fluo and have my
>>> service query Fluo (I think you guys actually advise against that in
>>> your documentation), or export results and then have the service
>>> query some external index that I am exporting to.
>>>
>>>
>> I'm not sure we advise against it, so much as recognize that it may
>> not be suitable for certain use cases and may not meet query
>> performance expectations (
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__fluo.apache.org_docs_fluo-2Drecipes_1.0.0-2Dincubating_export-2Dqueue_&d=CwIFaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=zqJSJTFo90FyUVCiF79uq3P0FHnxr0MLFKbsPsHGgyk&s=spmwJN_FBTO6TBBT2dne8sbE7MRMrlhz8lLPpfPZBbs&e= ).
>>
>
> I would advise against querying Fluo for low latency queries.
> However, this external service thats checking a few stats within Fluo and injecting new notifications probably does not care about latency.
>
> The reason Fluo is not geared towards low latency is that it does lazy
> recovery of failed transactions. Failed transactions are not cleaned
> up until something tries to read the data, which could significantly delay reads.
>
>> In any case, your observer need not write the final "last occurrence"
>> entries into a Fluo table. It could write them anywhere.
>>
>>
>>> Regarding timestamps, does the oracle server provide actual
>>> timestamps or just logical timestamps? That is, could I use the
>>> timestamps that the server provides to define some sort of now()
>>> function to obtain the current time to compare with the times of incoming events?
>>>
>>
>> Just logical time, and it delivers batches to limit locking, so it
>> can appear to jump ahead spontaneously. I'm not sure the OracleServer
>> is suitable for this purpose. What level of precision are you going for?
>> It might be enough to just run NTP, if you don't need more precision
>> than "within seconds".
>>
>>
>>> ________________________________________
>>> From: Christopher <ct...@apache.org>
>>> Sent: Tuesday, January 31, 2017 5:08 PM
>>> To: dev@fluo.incubator.apache.org
>>> Subject: Re: third party service to poll Fluo for absence of event
>>>
>>> You could write an observer which rolls up timestamps from all the
>>> events you are concerned about, and puts the most recent event
>>> timestamp into a centralized place, which you could poll. If there
>>> is no ingest of these events, then the last timestamp in this
>>> central place will exceed some threshold and the poller could detect that and trigger additional actions.
>>>
>>> On Tue, Jan 31, 2017 at 3:51 PM Meier, Caleb
>>> <Ca...@parsons.com>
>>> wrote:
>>>
>>> > Hello,
>>> >
>>> > I’m looking into using Fluo to develop an event based notification
>>> > system that incrementally generates events of increasing
>>> > complexity. The one issue that I’m running into is how to handle
>>> > the non-event event. That
>>> is,
>>> > Fluo (as I understand it) is not well-suited to handle the
>>> > following
>>> > request: “generate a notification if no events of a given type
>>> > have occurred within the last 24 hours”. This is because it is a
>>> > push based notification framework that only generates
>>> > notifications when things actually happen. So the question is,
>>> > has anyone looked into developing a service for generating
>>> > notifications at regular intervals (even if something doesn’t happen) that works with Fluo?
>>> > I’m toying with the idea of creating some sort of Twill
>>> > application that tells Fluo to wake up at regular intervals to
>>> > generate a notification about the set of events falling within the
>>> > given time window. Before doing this I just wanted to make sure
>>> > that something like this does not already exist, and I also
>>> want
>>> > to get a sense of how bad an idea it is to delegate some of the logic of
>>> > this periodic notification service to Fluo. Would it be better to
>>> > separate out the temporal portion of my notification request to be
>>> > processed entirely outside of Fluo to avoid transactional overhead?
>>> >
>>> > Caleb A. Meier, Ph.D.
>>> > Software Engineer II ♦ Analyst
>>> > Parsons Corporation
>>> > 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
>>> > Office: (703)797-3066 <(703)%20797-3066> <(703)%20797-3066>
>>> > Caleb.Meier@Parsons.com<ma...@Parsons.com> ♦
>>> www.parsons.com<
>>> > http://www.parsons.com/>
>>> >
>>> > --
>>> Christopher
>>>
>> --
>> Christopher
Re: third party service to poll Fluo for absence of event
Posted by Keith Turner <ke...@deenlo.com>.
On Thu, Feb 16, 2017 at 12:08 PM, Meier, Caleb <Ca...@parsons.com> wrote:
> Quick question. How timely is Fluo when it comes to processing notifications? If there are enough workers, will a notification be processed in a timely manner after writing to the observed column? Earlier we had a discussion about a periodic query service.
> If I write a notification to issue a periodic query to Fluo, can I expect that my Observer will process that query fairly quickly (provided there are enough workers)?
Fluo keeps track of the last time it saw a notification in a tablet
and exponentially increases the scan period for that tablet when it
keeps seeing nothing. The increase is up to a configurable maximum.
The following code does the backoff.
https://github.com/apache/incubator-fluo/blob/rel/fluo-1.0.0-incubating/modules/core/src/main/java/org/apache/fluo/core/worker/finder/hash/TabletData.java
The following is where it gets the max sleep time.
https://github.com/apache/incubator-fluo/blob/rel/fluo-1.0.0-incubating/modules/core/src/main/java/org/apache/fluo/core/worker/finder/hash/ScanTask.java#L72
Looking at the following, the default max sleep time for a tablet is 5
minutes. If I expanded the constant correctly, this can be changed
by setting fluo.impl.ScanTask.maxSleep. Note, impl properties are not
part of the public API and are subject to change when the
implementation changes.
https://github.com/apache/incubator-fluo/blob/rel/fluo-1.0.0-incubating/modules/core/src/main/java/org/apache/fluo/core/impl/FluoConfigurationImpl.java#L37
Also, when Fluo has notifications queued, it will wait till the queue
size halves before scanning again for notifications. So if 10,000
notifications were queued, then scanning for notifications would not
happen until the queue size was 5,000 or less. The following code
shows where that happens. I noticed while looking for the max scan
sleep code.
https://github.com/apache/incubator-fluo/blob/rel/fluo-1.0.0-incubating/modules/core/src/main/java/org/apache/fluo/core/worker/finder/hash/ScanTask.java#L85
>
> Caleb A. Meier, Ph.D.
> Software Engineer II ♦ Analyst
> Parsons Corporation
> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
> Office: (703)797-3066
> Caleb.Meier@Parsons.com ♦ www.parsons.com
>
> -----Original Message-----
> From: Keith Turner [mailto:keith@deenlo.com]
> Sent: Wednesday, February 01, 2017 11:03 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: third party service to poll Fluo for absence of event
>
> On Wed, Feb 1, 2017 at 9:54 PM, Christopher <ct...@apache.org> wrote:
>> On Wed, Feb 1, 2017 at 10:04 AM Meier, Caleb <Ca...@parsons.com>
>> wrote:
>>
>>> Yeah, this seems pretty reasonable to me. I guess it then boils down
>>> to the nitty gritty of do I store results in Fluo and have my service
>>> query Fluo (I think you guys actually advise against that in your
>>> documentation), or export results and then have the service query
>>> some external index that I am exporting to.
>>>
>>>
>> I'm not sure we advise against it, so much as recognize that it may
>> not be suitable for certain use cases and may not meet query
>> performance expectations (
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__fluo.apache.org_docs_fluo-2Drecipes_1.0.0-2Dincubating_export-2Dqueue_&d=CwIFaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=zqJSJTFo90FyUVCiF79uq3P0FHnxr0MLFKbsPsHGgyk&s=spmwJN_FBTO6TBBT2dne8sbE7MRMrlhz8lLPpfPZBbs&e= ).
>>
>
> I would advise against querying Fluo for low latency queries.
> However, this external service thats checking a few stats within Fluo and injecting new notifications probably does not care about latency.
>
> The reason Fluo is not geared towards low latency is that it does lazy
> recovery of failed transactions. Failed transactions are not cleaned
> up until something tries to read the data, which could significantly delay reads.
>
>> In any case, your observer need not write the final "last occurrence"
>> entries into a Fluo table. It could write them anywhere.
>>
>>
>>> Regarding timestamps, does the oracle server provide actual
>>> timestamps or just logical timestamps? That is, could I use the
>>> timestamps that the server provides to define some sort of now()
>>> function to obtain the current time to compare with the times of incoming events?
>>>
>>
>> Just logical time, and it delivers batches to limit locking, so it can
>> appear to jump ahead spontaneously. I'm not sure the OracleServer is
>> suitable for this purpose. What level of precision are you going for?
>> It might be enough to just run NTP, if you don't need more precision
>> than "within seconds".
>>
>>
>>> ________________________________________
>>> From: Christopher <ct...@apache.org>
>>> Sent: Tuesday, January 31, 2017 5:08 PM
>>> To: dev@fluo.incubator.apache.org
>>> Subject: Re: third party service to poll Fluo for absence of event
>>>
>>> You could write an observer which rolls up timestamps from all the
>>> events you are concerned about, and puts the most recent event
>>> timestamp into a centralized place, which you could poll. If there is
>>> no ingest of these events, then the last timestamp in this central
>>> place will exceed some threshold and the poller could detect that and trigger additional actions.
>>>
>>> On Tue, Jan 31, 2017 at 3:51 PM Meier, Caleb
>>> <Ca...@parsons.com>
>>> wrote:
>>>
>>> > Hello,
>>> >
>>> > I’m looking into using Fluo to develop an event based notification
>>> > system that incrementally generates events of increasing
>>> > complexity. The one issue that I’m running into is how to handle
>>> > the non-event event. That
>>> is,
>>> > Fluo (as I understand it) is not well-suited to handle the
>>> > following
>>> > request: “generate a notification if no events of a given type have
>>> > occurred within the last 24 hours”. This is because it is a push
>>> > based notification framework that only generates notifications when
>>> > things actually happen. So the question is, has anyone looked into
>>> > developing a service for generating notifications at regular
>>> > intervals (even if something doesn’t happen) that works with Fluo?
>>> > I’m toying with the idea of creating some sort of Twill application
>>> > that tells Fluo to wake up at regular intervals to generate a
>>> > notification about the set of events falling within the given time
>>> > window. Before doing this I just wanted to make sure that something
>>> > like this does not already exist, and I also
>>> want
>>> > to get a sense of how bad an idea it is to delegate some of the logic of
>>> > this periodic notification service to Fluo. Would it be better to
>>> > separate out the temporal portion of my notification request to be
>>> > processed entirely outside of Fluo to avoid transactional overhead?
>>> >
>>> > Caleb A. Meier, Ph.D.
>>> > Software Engineer II ♦ Analyst
>>> > Parsons Corporation
>>> > 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
>>> > Office: (703)797-3066 <(703)%20797-3066> <(703)%20797-3066>
>>> > Caleb.Meier@Parsons.com<ma...@Parsons.com> ♦
>>> www.parsons.com<
>>> > http://www.parsons.com/>
>>> >
>>> > --
>>> Christopher
>>>
>> --
>> Christopher
RE: third party service to poll Fluo for absence of event
Posted by "Meier, Caleb" <Ca...@parsons.com>.
Quick question. How timely is Fluo when it comes to processing notifications? If there are enough workers, will a notification be processed in a timely manner after writing to the observed column? Earlier we had a discussion about a periodic query service.
If I write a notification to issue a periodic query to Fluo, can I expect that my Observer will process that query fairly quickly (provided there are enough workers)?
Caleb A. Meier, Ph.D.
Software Engineer II ♦ Analyst
Parsons Corporation
1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
Office: (703)797-3066
Caleb.Meier@Parsons.com ♦ www.parsons.com
-----Original Message-----
From: Keith Turner [mailto:keith@deenlo.com]
Sent: Wednesday, February 01, 2017 11:03 PM
To: dev@fluo.incubator.apache.org
Subject: Re: third party service to poll Fluo for absence of event
On Wed, Feb 1, 2017 at 9:54 PM, Christopher <ct...@apache.org> wrote:
> On Wed, Feb 1, 2017 at 10:04 AM Meier, Caleb <Ca...@parsons.com>
> wrote:
>
>> Yeah, this seems pretty reasonable to me. I guess it then boils down
>> to the nitty gritty of do I store results in Fluo and have my service
>> query Fluo (I think you guys actually advise against that in your
>> documentation), or export results and then have the service query
>> some external index that I am exporting to.
>>
>>
> I'm not sure we advise against it, so much as recognize that it may
> not be suitable for certain use cases and may not meet query
> performance expectations (
> https://urldefense.proofpoint.com/v2/url?u=http-3A__fluo.apache.org_docs_fluo-2Drecipes_1.0.0-2Dincubating_export-2Dqueue_&d=CwIFaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=zqJSJTFo90FyUVCiF79uq3P0FHnxr0MLFKbsPsHGgyk&s=spmwJN_FBTO6TBBT2dne8sbE7MRMrlhz8lLPpfPZBbs&e= ).
>
I would advise against querying Fluo for low latency queries.
However, this external service thats checking a few stats within Fluo and injecting new notifications probably does not care about latency.
The reason Fluo is not geared towards low latency is that it does lazy
recovery of failed transactions. Failed transactions are not cleaned
up until something tries to read the data, which could significantly delay reads.
> In any case, your observer need not write the final "last occurrence"
> entries into a Fluo table. It could write them anywhere.
>
>
>> Regarding timestamps, does the oracle server provide actual
>> timestamps or just logical timestamps? That is, could I use the
>> timestamps that the server provides to define some sort of now()
>> function to obtain the current time to compare with the times of incoming events?
>>
>
> Just logical time, and it delivers batches to limit locking, so it can
> appear to jump ahead spontaneously. I'm not sure the OracleServer is
> suitable for this purpose. What level of precision are you going for?
> It might be enough to just run NTP, if you don't need more precision
> than "within seconds".
>
>
>> ________________________________________
>> From: Christopher <ct...@apache.org>
>> Sent: Tuesday, January 31, 2017 5:08 PM
>> To: dev@fluo.incubator.apache.org
>> Subject: Re: third party service to poll Fluo for absence of event
>>
>> You could write an observer which rolls up timestamps from all the
>> events you are concerned about, and puts the most recent event
>> timestamp into a centralized place, which you could poll. If there is
>> no ingest of these events, then the last timestamp in this central
>> place will exceed some threshold and the poller could detect that and trigger additional actions.
>>
>> On Tue, Jan 31, 2017 at 3:51 PM Meier, Caleb
>> <Ca...@parsons.com>
>> wrote:
>>
>> > Hello,
>> >
>> > I’m looking into using Fluo to develop an event based notification
>> > system that incrementally generates events of increasing
>> > complexity. The one issue that I’m running into is how to handle
>> > the non-event event. That
>> is,
>> > Fluo (as I understand it) is not well-suited to handle the
>> > following
>> > request: “generate a notification if no events of a given type have
>> > occurred within the last 24 hours”. This is because it is a push
>> > based notification framework that only generates notifications when
>> > things actually happen. So the question is, has anyone looked into
>> > developing a service for generating notifications at regular
>> > intervals (even if something doesn’t happen) that works with Fluo?
>> > I’m toying with the idea of creating some sort of Twill application
>> > that tells Fluo to wake up at regular intervals to generate a
>> > notification about the set of events falling within the given time
>> > window. Before doing this I just wanted to make sure that something
>> > like this does not already exist, and I also
>> want
>> > to get a sense of how bad an idea it is to delegate some of the logic of
>> > this periodic notification service to Fluo. Would it be better to
>> > separate out the temporal portion of my notification request to be
>> > processed entirely outside of Fluo to avoid transactional overhead?
>> >
>> > Caleb A. Meier, Ph.D.
>> > Software Engineer II ♦ Analyst
>> > Parsons Corporation
>> > 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
>> > Office: (703)797-3066 <(703)%20797-3066> <(703)%20797-3066>
>> > Caleb.Meier@Parsons.com<ma...@Parsons.com> ♦
>> www.parsons.com<
>> > http://www.parsons.com/>
>> >
>> > --
>> Christopher
>>
> --
> Christopher
Re: third party service to poll Fluo for absence of event
Posted by Mike Walch <mw...@apache.org>.
Hi Caleb,
I wouldn't try to follow our Twill code too closely. We have a lot of
abstraction in the code that I don't think you need. We do use TwillRunner
and TwillController. One major tip is that you should consider whether or
not you want to pull in all of Twill dependencies (which may cause
dependency conflicts). You can avoid pulling in Twill's dependencies by
creating a bundled jar and using Twill's BundleJarRunnable. We didn't do
this in Fluo right now but I would like to in the future. If you want to
see an example, take a look at the new accumulo-testing repo[1] which uses
Twill to launch tests using bundled jars. In particular, look at the class
YarnAccumuloTestRunner[2].
Best,
Mike
[1]: https://github.com/apache/accumulo-testing
[2]:
https://github.com/apache/accumulo-testing/blob/master/yarn/src/main/java/org/apache/accumulo/testing/yarn/YarnAccumuloTestRunner.java
On Thu, Feb 2, 2017 at 5:15 PM Meier, Caleb <Ca...@parsons.com> wrote:
> Thanks for the input. I'm currently looking at creating some sort of
> coordinator (which wraps a ScheduledExecutorService to generate periodic
> notifications) and a collection of workers (to process the periodic queries
> as they are issued). Most of the interaction between the workers and
> coordinator will be via Kafka (develop some sort of protocol to ensure that
> more than one worker isn't getting assigned the same query). At any rate,
> I was thinking of implementing these components as TwillRunnables.
> However, it seems like the Twill documentation is a bit sparse. Given that
> you guys implemented Fluo as a TwillApplication, do you have any
> insight/advice for writing TwillApplications? In particular, how is your
> FluoTwillApp being run? All of the examples I've seen create a client with
> a TwillRunner and TwillController. It seems like you 've created your own
> version of a YarnAppRunner -- what role is that playing in running the
> FluoTwillApp? Moreover, it is also unclear to me whether the
> TwillRunnables are bound to the client -- if the client terminates do the
> runnables terminate as well? So essentially, it is unclear to me how
> create a long running application in Twill that is not bound to a
> particular client. Sorry that this is a little off topic, but any help,
> references to documentation/examples would be very appreciated.
>
> Caleb A. Meier, Ph.D.
> Software Engineer II ♦ Analyst
> Parsons Corporation
> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
> Office: (703)797-3066 <(703)%20797-3066>
> Caleb.Meier@Parsons.com ♦ www.parsons.com
>
> -----Original Message-----
> From: Keith Turner [mailto:keith@deenlo.com]
> Sent: Wednesday, February 01, 2017 11:03 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: third party service to poll Fluo for absence of event
>
> On Wed, Feb 1, 2017 at 9:54 PM, Christopher <ct...@apache.org> wrote:
> > On Wed, Feb 1, 2017 at 10:04 AM Meier, Caleb <Ca...@parsons.com>
> > wrote:
> >
> >> Yeah, this seems pretty reasonable to me. I guess it then boils down
> >> to the nitty gritty of do I store results in Fluo and have my service
> >> query Fluo (I think you guys actually advise against that in your
> >> documentation), or export results and then have the service query
> >> some external index that I am exporting to.
> >>
> >>
> > I'm not sure we advise against it, so much as recognize that it may
> > not be suitable for certain use cases and may not meet query
> > performance expectations (
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__fluo.apache.org_docs_fluo-2Drecipes_1.0.0-2Dincubating_export-2Dqueue_&d=CwIFaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=zqJSJTFo90FyUVCiF79uq3P0FHnxr0MLFKbsPsHGgyk&s=spmwJN_FBTO6TBBT2dne8sbE7MRMrlhz8lLPpfPZBbs&e=
> ).
> >
>
> I would advise against querying Fluo for low latency queries.
> However, this external service thats checking a few stats within Fluo and
> injecting new notifications probably does not care about latency.
>
> The reason Fluo is not geared towards low latency is that it does lazy
> recovery of failed transactions. Failed transactions are not cleaned
> up until something tries to read the data, which could significantly delay
> reads.
>
> > In any case, your observer need not write the final "last occurrence"
> > entries into a Fluo table. It could write them anywhere.
> >
> >
> >> Regarding timestamps, does the oracle server provide actual
> >> timestamps or just logical timestamps? That is, could I use the
> >> timestamps that the server provides to define some sort of now()
> >> function to obtain the current time to compare with the times of
> incoming events?
> >>
> >
> > Just logical time, and it delivers batches to limit locking, so it can
> > appear to jump ahead spontaneously. I'm not sure the OracleServer is
> > suitable for this purpose. What level of precision are you going for?
> > It might be enough to just run NTP, if you don't need more precision
> > than "within seconds".
> >
> >
> >> ________________________________________
> >> From: Christopher <ct...@apache.org>
> >> Sent: Tuesday, January 31, 2017 5:08 PM
> >> To: dev@fluo.incubator.apache.org
> >> Subject: Re: third party service to poll Fluo for absence of event
> >>
> >> You could write an observer which rolls up timestamps from all the
> >> events you are concerned about, and puts the most recent event
> >> timestamp into a centralized place, which you could poll. If there is
> >> no ingest of these events, then the last timestamp in this central
> >> place will exceed some threshold and the poller could detect that and
> trigger additional actions.
> >>
> >> On Tue, Jan 31, 2017 at 3:51 PM Meier, Caleb
> >> <Ca...@parsons.com>
> >> wrote:
> >>
> >> > Hello,
> >> >
> >> > I’m looking into using Fluo to develop an event based notification
> >> > system that incrementally generates events of increasing
> >> > complexity. The one issue that I’m running into is how to handle
> >> > the non-event event. That
> >> is,
> >> > Fluo (as I understand it) is not well-suited to handle the
> >> > following
> >> > request: “generate a notification if no events of a given type have
> >> > occurred within the last 24 hours”. This is because it is a push
> >> > based notification framework that only generates notifications when
> >> > things actually happen. So the question is, has anyone looked into
> >> > developing a service for generating notifications at regular
> >> > intervals (even if something doesn’t happen) that works with Fluo?
> >> > I’m toying with the idea of creating some sort of Twill application
> >> > that tells Fluo to wake up at regular intervals to generate a
> >> > notification about the set of events falling within the given time
> >> > window. Before doing this I just wanted to make sure that something
> >> > like this does not already exist, and I also
> >> want
> >> > to get a sense of how bad an idea it is to delegate some of the logic
> of
> >> > this periodic notification service to Fluo. Would it be better to
> >> > separate out the temporal portion of my notification request to be
> >> > processed entirely outside of Fluo to avoid transactional overhead?
> >> >
> >> > Caleb A. Meier, Ph.D.
> >> > Software Engineer II ♦ Analyst
> >> > Parsons Corporation
> >> > 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
> >> > Office: (703)797-3066 <(703)%20797-3066> <(703)%20797-3066>
> <(703)%20797-3066>
> >> > Caleb.Meier@Parsons.com<ma...@Parsons.com> ♦
> >> www.parsons.com<
> >> > http://www.parsons.com/>
> >> >
> >> > --
> >> Christopher
> >>
> > --
> > Christopher
>
RE: third party service to poll Fluo for absence of event
Posted by "Meier, Caleb" <Ca...@parsons.com>.
Thanks for the feedback everyone.
Keith, we are currently exporting to Kafka, but I don't foresee needing to ingest from Kafka into Fluo for this project. I will certainly keep you updated if that changes.
Thanks,
Caleb A. Meier, Ph.D.
Software Engineer II ♦ Analyst
Parsons Corporation
1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
Office: (703)797-3066
Caleb.Meier@Parsons.com ♦ www.parsons.com
-----Original Message-----
From: Keith Turner [mailto:keith@deenlo.com]
Sent: Friday, February 03, 2017 10:05 AM
To: dev@fluo.incubator.apache.org
Subject: Re: third party service to poll Fluo for absence of event
If you have time to share as you move forward, I am interested in learning from your experiences with using Kafka and Fluo together. I have wanted to experiment with this in order to see if Fluo needed any
changes to support better interoperation. However, I have not had
the time. I opened #795[1], but its based on speculation not experience.
[1]: https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dfluo_issues_795&d=CwIFaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=5UoPZQJ9JMRl_7xZ5RY_Yw7cEUFMTQQLpbr341mlILw&s=fcHGc-fgm-SIG8XLgsQyvHOlpRVexhcGQWSlCIQETtA&e=
On Thu, Feb 2, 2017 at 5:15 PM, Meier, Caleb <Ca...@parsons.com> wrote:
> Thanks for the input. I'm currently looking at creating some sort of coordinator (which wraps a ScheduledExecutorService to generate periodic notifications) and a collection of workers (to process the periodic queries as they are issued). Most of the interaction between the workers and coordinator will be via Kafka (develop some sort of protocol to ensure that more than one worker isn't getting assigned the same query). At any rate, I was thinking of implementing these components as TwillRunnables. However, it seems like the Twill documentation is a bit sparse. Given that you guys implemented Fluo as a TwillApplication, do you have any insight/advice for writing TwillApplications? In particular, how is your FluoTwillApp being run? All of the examples I've seen create a client with a TwillRunner and TwillController. It seems like you 've created your own version of a YarnAppRunner -- what role is that playing in running the FluoTwillApp? Moreover, it is also unclear to me whether the TwillRunnables are bound to the client -- if the client terminates do the runnables terminate as well? So essentially, it is unclear to me how create a long running application in Twill that is not bound to a particular client. Sorry that this is a little off topic, but any help, references to documentation/examples would be very appreciated.
>
> Caleb A. Meier, Ph.D.
> Software Engineer II ♦ Analyst
> Parsons Corporation
> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
> Office: (703)797-3066
> Caleb.Meier@Parsons.com ♦ www.parsons.com
>
> -----Original Message-----
> From: Keith Turner [mailto:keith@deenlo.com]
> Sent: Wednesday, February 01, 2017 11:03 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: third party service to poll Fluo for absence of event
>
> On Wed, Feb 1, 2017 at 9:54 PM, Christopher <ct...@apache.org> wrote:
>> On Wed, Feb 1, 2017 at 10:04 AM Meier, Caleb
>> <Ca...@parsons.com>
>> wrote:
>>
>>> Yeah, this seems pretty reasonable to me. I guess it then boils
>>> down to the nitty gritty of do I store results in Fluo and have my
>>> service query Fluo (I think you guys actually advise against that in
>>> your documentation), or export results and then have the service
>>> query some external index that I am exporting to.
>>>
>>>
>> I'm not sure we advise against it, so much as recognize that it may
>> not be suitable for certain use cases and may not meet query
>> performance expectations (
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__fluo.apache.org_docs_fluo-2Drecipes_1.0.0-2Dincubating_export-2Dqueue_&d=CwIFaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=zqJSJTFo90FyUVCiF79uq3P0FHnxr0MLFKbsPsHGgyk&s=spmwJN_FBTO6TBBT2dne8sbE7MRMrlhz8lLPpfPZBbs&e= ).
>>
>
> I would advise against querying Fluo for low latency queries.
> However, this external service thats checking a few stats within Fluo and injecting new notifications probably does not care about latency.
>
> The reason Fluo is not geared towards low latency is that it does lazy
> recovery of failed transactions. Failed transactions are not cleaned
> up until something tries to read the data, which could significantly delay reads.
>
>> In any case, your observer need not write the final "last occurrence"
>> entries into a Fluo table. It could write them anywhere.
>>
>>
>>> Regarding timestamps, does the oracle server provide actual
>>> timestamps or just logical timestamps? That is, could I use the
>>> timestamps that the server provides to define some sort of now()
>>> function to obtain the current time to compare with the times of incoming events?
>>>
>>
>> Just logical time, and it delivers batches to limit locking, so it
>> can appear to jump ahead spontaneously. I'm not sure the OracleServer
>> is suitable for this purpose. What level of precision are you going for?
>> It might be enough to just run NTP, if you don't need more precision
>> than "within seconds".
>>
>>
>>> ________________________________________
>>> From: Christopher <ct...@apache.org>
>>> Sent: Tuesday, January 31, 2017 5:08 PM
>>> To: dev@fluo.incubator.apache.org
>>> Subject: Re: third party service to poll Fluo for absence of event
>>>
>>> You could write an observer which rolls up timestamps from all the
>>> events you are concerned about, and puts the most recent event
>>> timestamp into a centralized place, which you could poll. If there
>>> is no ingest of these events, then the last timestamp in this
>>> central place will exceed some threshold and the poller could detect that and trigger additional actions.
>>>
>>> On Tue, Jan 31, 2017 at 3:51 PM Meier, Caleb
>>> <Ca...@parsons.com>
>>> wrote:
>>>
>>> > Hello,
>>> >
>>> > I’m looking into using Fluo to develop an event based notification
>>> > system that incrementally generates events of increasing
>>> > complexity. The one issue that I’m running into is how to handle
>>> > the non-event event. That
>>> is,
>>> > Fluo (as I understand it) is not well-suited to handle the
>>> > following
>>> > request: “generate a notification if no events of a given type
>>> > have occurred within the last 24 hours”. This is because it is a
>>> > push based notification framework that only generates
>>> > notifications when things actually happen. So the question is,
>>> > has anyone looked into developing a service for generating
>>> > notifications at regular intervals (even if something doesn’t happen) that works with Fluo?
>>> > I’m toying with the idea of creating some sort of Twill
>>> > application that tells Fluo to wake up at regular intervals to
>>> > generate a notification about the set of events falling within the
>>> > given time window. Before doing this I just wanted to make sure
>>> > that something like this does not already exist, and I also
>>> want
>>> > to get a sense of how bad an idea it is to delegate some of the logic of
>>> > this periodic notification service to Fluo. Would it be better to
>>> > separate out the temporal portion of my notification request to be
>>> > processed entirely outside of Fluo to avoid transactional overhead?
>>> >
>>> > Caleb A. Meier, Ph.D.
>>> > Software Engineer II ♦ Analyst
>>> > Parsons Corporation
>>> > 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
>>> > Office: (703)797-3066 <(703)%20797-3066> <(703)%20797-3066>
>>> > Caleb.Meier@Parsons.com<ma...@Parsons.com> ♦
>>> www.parsons.com<
>>> > http://www.parsons.com/>
>>> >
>>> > --
>>> Christopher
>>>
>> --
>> Christopher
Re: third party service to poll Fluo for absence of event
Posted by Keith Turner <ke...@deenlo.com>.
If you have time to share as you move forward, I am interested in
learning from your experiences with using Kafka and Fluo together. I
have wanted to experiment with this in order to see if Fluo needed any
changes to support better interoperation. However, I have not had
the time. I opened #795[1], but its based on speculation not
experience.
[1]: https://github.com/apache/incubator-fluo/issues/795
On Thu, Feb 2, 2017 at 5:15 PM, Meier, Caleb <Ca...@parsons.com> wrote:
> Thanks for the input. I'm currently looking at creating some sort of coordinator (which wraps a ScheduledExecutorService to generate periodic notifications) and a collection of workers (to process the periodic queries as they are issued). Most of the interaction between the workers and coordinator will be via Kafka (develop some sort of protocol to ensure that more than one worker isn't getting assigned the same query). At any rate, I was thinking of implementing these components as TwillRunnables. However, it seems like the Twill documentation is a bit sparse. Given that you guys implemented Fluo as a TwillApplication, do you have any insight/advice for writing TwillApplications? In particular, how is your FluoTwillApp being run? All of the examples I've seen create a client with a TwillRunner and TwillController. It seems like you 've created your own version of a YarnAppRunner -- what role is that playing in running the FluoTwillApp? Moreover, it is also unclear to me whether the TwillRunnables are bound to the client -- if the client terminates do the runnables terminate as well? So essentially, it is unclear to me how create a long running application in Twill that is not bound to a particular client. Sorry that this is a little off topic, but any help, references to documentation/examples would be very appreciated.
>
> Caleb A. Meier, Ph.D.
> Software Engineer II ♦ Analyst
> Parsons Corporation
> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
> Office: (703)797-3066
> Caleb.Meier@Parsons.com ♦ www.parsons.com
>
> -----Original Message-----
> From: Keith Turner [mailto:keith@deenlo.com]
> Sent: Wednesday, February 01, 2017 11:03 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: third party service to poll Fluo for absence of event
>
> On Wed, Feb 1, 2017 at 9:54 PM, Christopher <ct...@apache.org> wrote:
>> On Wed, Feb 1, 2017 at 10:04 AM Meier, Caleb <Ca...@parsons.com>
>> wrote:
>>
>>> Yeah, this seems pretty reasonable to me. I guess it then boils down
>>> to the nitty gritty of do I store results in Fluo and have my service
>>> query Fluo (I think you guys actually advise against that in your
>>> documentation), or export results and then have the service query
>>> some external index that I am exporting to.
>>>
>>>
>> I'm not sure we advise against it, so much as recognize that it may
>> not be suitable for certain use cases and may not meet query
>> performance expectations (
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__fluo.apache.org_docs_fluo-2Drecipes_1.0.0-2Dincubating_export-2Dqueue_&d=CwIFaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=zqJSJTFo90FyUVCiF79uq3P0FHnxr0MLFKbsPsHGgyk&s=spmwJN_FBTO6TBBT2dne8sbE7MRMrlhz8lLPpfPZBbs&e= ).
>>
>
> I would advise against querying Fluo for low latency queries.
> However, this external service thats checking a few stats within Fluo and injecting new notifications probably does not care about latency.
>
> The reason Fluo is not geared towards low latency is that it does lazy
> recovery of failed transactions. Failed transactions are not cleaned
> up until something tries to read the data, which could significantly delay reads.
>
>> In any case, your observer need not write the final "last occurrence"
>> entries into a Fluo table. It could write them anywhere.
>>
>>
>>> Regarding timestamps, does the oracle server provide actual
>>> timestamps or just logical timestamps? That is, could I use the
>>> timestamps that the server provides to define some sort of now()
>>> function to obtain the current time to compare with the times of incoming events?
>>>
>>
>> Just logical time, and it delivers batches to limit locking, so it can
>> appear to jump ahead spontaneously. I'm not sure the OracleServer is
>> suitable for this purpose. What level of precision are you going for?
>> It might be enough to just run NTP, if you don't need more precision
>> than "within seconds".
>>
>>
>>> ________________________________________
>>> From: Christopher <ct...@apache.org>
>>> Sent: Tuesday, January 31, 2017 5:08 PM
>>> To: dev@fluo.incubator.apache.org
>>> Subject: Re: third party service to poll Fluo for absence of event
>>>
>>> You could write an observer which rolls up timestamps from all the
>>> events you are concerned about, and puts the most recent event
>>> timestamp into a centralized place, which you could poll. If there is
>>> no ingest of these events, then the last timestamp in this central
>>> place will exceed some threshold and the poller could detect that and trigger additional actions.
>>>
>>> On Tue, Jan 31, 2017 at 3:51 PM Meier, Caleb
>>> <Ca...@parsons.com>
>>> wrote:
>>>
>>> > Hello,
>>> >
>>> > I’m looking into using Fluo to develop an event based notification
>>> > system that incrementally generates events of increasing
>>> > complexity. The one issue that I’m running into is how to handle
>>> > the non-event event. That
>>> is,
>>> > Fluo (as I understand it) is not well-suited to handle the
>>> > following
>>> > request: “generate a notification if no events of a given type have
>>> > occurred within the last 24 hours”. This is because it is a push
>>> > based notification framework that only generates notifications when
>>> > things actually happen. So the question is, has anyone looked into
>>> > developing a service for generating notifications at regular
>>> > intervals (even if something doesn’t happen) that works with Fluo?
>>> > I’m toying with the idea of creating some sort of Twill application
>>> > that tells Fluo to wake up at regular intervals to generate a
>>> > notification about the set of events falling within the given time
>>> > window. Before doing this I just wanted to make sure that something
>>> > like this does not already exist, and I also
>>> want
>>> > to get a sense of how bad an idea it is to delegate some of the logic of
>>> > this periodic notification service to Fluo. Would it be better to
>>> > separate out the temporal portion of my notification request to be
>>> > processed entirely outside of Fluo to avoid transactional overhead?
>>> >
>>> > Caleb A. Meier, Ph.D.
>>> > Software Engineer II ♦ Analyst
>>> > Parsons Corporation
>>> > 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
>>> > Office: (703)797-3066 <(703)%20797-3066> <(703)%20797-3066>
>>> > Caleb.Meier@Parsons.com<ma...@Parsons.com> ♦
>>> www.parsons.com<
>>> > http://www.parsons.com/>
>>> >
>>> > --
>>> Christopher
>>>
>> --
>> Christopher
RE: third party service to poll Fluo for absence of event
Posted by "Meier, Caleb" <Ca...@parsons.com>.
Thanks for the input. I'm currently looking at creating some sort of coordinator (which wraps a ScheduledExecutorService to generate periodic notifications) and a collection of workers (to process the periodic queries as they are issued). Most of the interaction between the workers and coordinator will be via Kafka (develop some sort of protocol to ensure that more than one worker isn't getting assigned the same query). At any rate, I was thinking of implementing these components as TwillRunnables. However, it seems like the Twill documentation is a bit sparse. Given that you guys implemented Fluo as a TwillApplication, do you have any insight/advice for writing TwillApplications? In particular, how is your FluoTwillApp being run? All of the examples I've seen create a client with a TwillRunner and TwillController. It seems like you 've created your own version of a YarnAppRunner -- what role is that playing in running the FluoTwillApp? Moreover, it is also unclear to me whether the TwillRunnables are bound to the client -- if the client terminates do the runnables terminate as well? So essentially, it is unclear to me how create a long running application in Twill that is not bound to a particular client. Sorry that this is a little off topic, but any help, references to documentation/examples would be very appreciated.
Caleb A. Meier, Ph.D.
Software Engineer II ♦ Analyst
Parsons Corporation
1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
Office: (703)797-3066
Caleb.Meier@Parsons.com ♦ www.parsons.com
-----Original Message-----
From: Keith Turner [mailto:keith@deenlo.com]
Sent: Wednesday, February 01, 2017 11:03 PM
To: dev@fluo.incubator.apache.org
Subject: Re: third party service to poll Fluo for absence of event
On Wed, Feb 1, 2017 at 9:54 PM, Christopher <ct...@apache.org> wrote:
> On Wed, Feb 1, 2017 at 10:04 AM Meier, Caleb <Ca...@parsons.com>
> wrote:
>
>> Yeah, this seems pretty reasonable to me. I guess it then boils down
>> to the nitty gritty of do I store results in Fluo and have my service
>> query Fluo (I think you guys actually advise against that in your
>> documentation), or export results and then have the service query
>> some external index that I am exporting to.
>>
>>
> I'm not sure we advise against it, so much as recognize that it may
> not be suitable for certain use cases and may not meet query
> performance expectations (
> https://urldefense.proofpoint.com/v2/url?u=http-3A__fluo.apache.org_docs_fluo-2Drecipes_1.0.0-2Dincubating_export-2Dqueue_&d=CwIFaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=zqJSJTFo90FyUVCiF79uq3P0FHnxr0MLFKbsPsHGgyk&s=spmwJN_FBTO6TBBT2dne8sbE7MRMrlhz8lLPpfPZBbs&e= ).
>
I would advise against querying Fluo for low latency queries.
However, this external service thats checking a few stats within Fluo and injecting new notifications probably does not care about latency.
The reason Fluo is not geared towards low latency is that it does lazy
recovery of failed transactions. Failed transactions are not cleaned
up until something tries to read the data, which could significantly delay reads.
> In any case, your observer need not write the final "last occurrence"
> entries into a Fluo table. It could write them anywhere.
>
>
>> Regarding timestamps, does the oracle server provide actual
>> timestamps or just logical timestamps? That is, could I use the
>> timestamps that the server provides to define some sort of now()
>> function to obtain the current time to compare with the times of incoming events?
>>
>
> Just logical time, and it delivers batches to limit locking, so it can
> appear to jump ahead spontaneously. I'm not sure the OracleServer is
> suitable for this purpose. What level of precision are you going for?
> It might be enough to just run NTP, if you don't need more precision
> than "within seconds".
>
>
>> ________________________________________
>> From: Christopher <ct...@apache.org>
>> Sent: Tuesday, January 31, 2017 5:08 PM
>> To: dev@fluo.incubator.apache.org
>> Subject: Re: third party service to poll Fluo for absence of event
>>
>> You could write an observer which rolls up timestamps from all the
>> events you are concerned about, and puts the most recent event
>> timestamp into a centralized place, which you could poll. If there is
>> no ingest of these events, then the last timestamp in this central
>> place will exceed some threshold and the poller could detect that and trigger additional actions.
>>
>> On Tue, Jan 31, 2017 at 3:51 PM Meier, Caleb
>> <Ca...@parsons.com>
>> wrote:
>>
>> > Hello,
>> >
>> > I’m looking into using Fluo to develop an event based notification
>> > system that incrementally generates events of increasing
>> > complexity. The one issue that I’m running into is how to handle
>> > the non-event event. That
>> is,
>> > Fluo (as I understand it) is not well-suited to handle the
>> > following
>> > request: “generate a notification if no events of a given type have
>> > occurred within the last 24 hours”. This is because it is a push
>> > based notification framework that only generates notifications when
>> > things actually happen. So the question is, has anyone looked into
>> > developing a service for generating notifications at regular
>> > intervals (even if something doesn’t happen) that works with Fluo?
>> > I’m toying with the idea of creating some sort of Twill application
>> > that tells Fluo to wake up at regular intervals to generate a
>> > notification about the set of events falling within the given time
>> > window. Before doing this I just wanted to make sure that something
>> > like this does not already exist, and I also
>> want
>> > to get a sense of how bad an idea it is to delegate some of the logic of
>> > this periodic notification service to Fluo. Would it be better to
>> > separate out the temporal portion of my notification request to be
>> > processed entirely outside of Fluo to avoid transactional overhead?
>> >
>> > Caleb A. Meier, Ph.D.
>> > Software Engineer II ♦ Analyst
>> > Parsons Corporation
>> > 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
>> > Office: (703)797-3066 <(703)%20797-3066> <(703)%20797-3066>
>> > Caleb.Meier@Parsons.com<ma...@Parsons.com> ♦
>> www.parsons.com<
>> > http://www.parsons.com/>
>> >
>> > --
>> Christopher
>>
> --
> Christopher
Re: third party service to poll Fluo for absence of event
Posted by Keith Turner <ke...@deenlo.com>.
On Wed, Feb 1, 2017 at 9:54 PM, Christopher <ct...@apache.org> wrote:
> On Wed, Feb 1, 2017 at 10:04 AM Meier, Caleb <Ca...@parsons.com>
> wrote:
>
>> Yeah, this seems pretty reasonable to me. I guess it then boils down to
>> the nitty gritty of do I store results in Fluo and have my service query
>> Fluo (I think you guys actually advise against that in your documentation),
>> or export results and then have the service query some external index that
>> I am exporting to.
>>
>>
> I'm not sure we advise against it, so much as recognize that it may not be
> suitable for certain use cases and may not meet query performance
> expectations (
> http://fluo.apache.org/docs/fluo-recipes/1.0.0-incubating/export-queue/).
>
I would advise against querying Fluo for low latency queries.
However, this external service thats checking a few stats within Fluo
and injecting new notifications probably does not care about latency.
The reason Fluo is not geared towards low latency is that it does lazy
recovery of failed transactions. Failed transactions are not cleaned
up until something tries to read the data, which could significantly
delay reads.
> In any case, your observer need not write the final "last occurrence"
> entries into a Fluo table. It could write them anywhere.
>
>
>> Regarding timestamps, does the oracle server provide actual timestamps or
>> just logical timestamps? That is, could I use the timestamps that the
>> server provides to define some sort of now() function to obtain the current
>> time to compare with the times of incoming events?
>>
>
> Just logical time, and it delivers batches to limit locking, so it can
> appear to jump ahead spontaneously. I'm not sure the OracleServer is
> suitable for this purpose. What level of precision are you going for? It
> might be enough to just run NTP, if you don't need more precision than
> "within seconds".
>
>
>> ________________________________________
>> From: Christopher <ct...@apache.org>
>> Sent: Tuesday, January 31, 2017 5:08 PM
>> To: dev@fluo.incubator.apache.org
>> Subject: Re: third party service to poll Fluo for absence of event
>>
>> You could write an observer which rolls up timestamps from all the events
>> you are concerned about, and puts the most recent event timestamp into a
>> centralized place, which you could poll. If there is no ingest of these
>> events, then the last timestamp in this central place will exceed some
>> threshold and the poller could detect that and trigger additional actions.
>>
>> On Tue, Jan 31, 2017 at 3:51 PM Meier, Caleb <Ca...@parsons.com>
>> wrote:
>>
>> > Hello,
>> >
>> > I’m looking into using Fluo to develop an event based notification system
>> > that incrementally generates events of increasing complexity. The one
>> > issue that I’m running into is how to handle the non-event event. That
>> is,
>> > Fluo (as I understand it) is not well-suited to handle the following
>> > request: “generate a notification if no events of a given type have
>> > occurred within the last 24 hours”. This is because it is a push based
>> > notification framework that only generates notifications when things
>> > actually happen. So the question is, has anyone looked into developing a
>> > service for generating notifications at regular intervals (even if
>> > something doesn’t happen) that works with Fluo? I’m toying with the idea
>> > of creating some sort of Twill application that tells Fluo to wake up at
>> > regular intervals to generate a notification about the set of events
>> > falling within the given time window. Before doing this I just wanted to
>> > make sure that something like this does not already exist, and I also
>> want
>> > to get a sense of how bad an idea it is to delegate some of the logic of
>> > this periodic notification service to Fluo. Would it be better to
>> > separate out the temporal portion of my notification request to be
>> > processed entirely outside of Fluo to avoid transactional overhead?
>> >
>> > Caleb A. Meier, Ph.D.
>> > Software Engineer II ♦ Analyst
>> > Parsons Corporation
>> > 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
>> > Office: (703)797-3066 <(703)%20797-3066> <(703)%20797-3066>
>> > Caleb.Meier@Parsons.com<ma...@Parsons.com> ♦
>> www.parsons.com<
>> > http://www.parsons.com/>
>> >
>> > --
>> Christopher
>>
> --
> Christopher
Re: third party service to poll Fluo for absence of event
Posted by Christopher <ct...@apache.org>.
On Wed, Feb 1, 2017 at 10:04 AM Meier, Caleb <Ca...@parsons.com>
wrote:
> Yeah, this seems pretty reasonable to me. I guess it then boils down to
> the nitty gritty of do I store results in Fluo and have my service query
> Fluo (I think you guys actually advise against that in your documentation),
> or export results and then have the service query some external index that
> I am exporting to.
>
>
I'm not sure we advise against it, so much as recognize that it may not be
suitable for certain use cases and may not meet query performance
expectations (
http://fluo.apache.org/docs/fluo-recipes/1.0.0-incubating/export-queue/).
In any case, your observer need not write the final "last occurrence"
entries into a Fluo table. It could write them anywhere.
> Regarding timestamps, does the oracle server provide actual timestamps or
> just logical timestamps? That is, could I use the timestamps that the
> server provides to define some sort of now() function to obtain the current
> time to compare with the times of incoming events?
>
Just logical time, and it delivers batches to limit locking, so it can
appear to jump ahead spontaneously. I'm not sure the OracleServer is
suitable for this purpose. What level of precision are you going for? It
might be enough to just run NTP, if you don't need more precision than
"within seconds".
> ________________________________________
> From: Christopher <ct...@apache.org>
> Sent: Tuesday, January 31, 2017 5:08 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: third party service to poll Fluo for absence of event
>
> You could write an observer which rolls up timestamps from all the events
> you are concerned about, and puts the most recent event timestamp into a
> centralized place, which you could poll. If there is no ingest of these
> events, then the last timestamp in this central place will exceed some
> threshold and the poller could detect that and trigger additional actions.
>
> On Tue, Jan 31, 2017 at 3:51 PM Meier, Caleb <Ca...@parsons.com>
> wrote:
>
> > Hello,
> >
> > I’m looking into using Fluo to develop an event based notification system
> > that incrementally generates events of increasing complexity. The one
> > issue that I’m running into is how to handle the non-event event. That
> is,
> > Fluo (as I understand it) is not well-suited to handle the following
> > request: “generate a notification if no events of a given type have
> > occurred within the last 24 hours”. This is because it is a push based
> > notification framework that only generates notifications when things
> > actually happen. So the question is, has anyone looked into developing a
> > service for generating notifications at regular intervals (even if
> > something doesn’t happen) that works with Fluo? I’m toying with the idea
> > of creating some sort of Twill application that tells Fluo to wake up at
> > regular intervals to generate a notification about the set of events
> > falling within the given time window. Before doing this I just wanted to
> > make sure that something like this does not already exist, and I also
> want
> > to get a sense of how bad an idea it is to delegate some of the logic of
> > this periodic notification service to Fluo. Would it be better to
> > separate out the temporal portion of my notification request to be
> > processed entirely outside of Fluo to avoid transactional overhead?
> >
> > Caleb A. Meier, Ph.D.
> > Software Engineer II ♦ Analyst
> > Parsons Corporation
> > 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
> > Office: (703)797-3066 <(703)%20797-3066> <(703)%20797-3066>
> > Caleb.Meier@Parsons.com<ma...@Parsons.com> ♦
> www.parsons.com<
> > http://www.parsons.com/>
> >
> > --
> Christopher
>
--
Christopher
Re: third party service to poll Fluo for absence of event
Posted by "Meier, Caleb" <Ca...@parsons.com>.
Yeah, this seems pretty reasonable to me. I guess it then boils down to the nitty gritty of do I store results in Fluo and have my service query Fluo (I think you guys actually advise against that in your documentation), or export results and then have the service query some external index that I am exporting to.
Regarding timestamps, does the oracle server provide actual timestamps or just logical timestamps? That is, could I use the timestamps that the server provides to define some sort of now() function to obtain the current time to compare with the times of incoming events?
________________________________________
From: Christopher <ct...@apache.org>
Sent: Tuesday, January 31, 2017 5:08 PM
To: dev@fluo.incubator.apache.org
Subject: Re: third party service to poll Fluo for absence of event
You could write an observer which rolls up timestamps from all the events
you are concerned about, and puts the most recent event timestamp into a
centralized place, which you could poll. If there is no ingest of these
events, then the last timestamp in this central place will exceed some
threshold and the poller could detect that and trigger additional actions.
On Tue, Jan 31, 2017 at 3:51 PM Meier, Caleb <Ca...@parsons.com>
wrote:
> Hello,
>
> I’m looking into using Fluo to develop an event based notification system
> that incrementally generates events of increasing complexity. The one
> issue that I’m running into is how to handle the non-event event. That is,
> Fluo (as I understand it) is not well-suited to handle the following
> request: “generate a notification if no events of a given type have
> occurred within the last 24 hours”. This is because it is a push based
> notification framework that only generates notifications when things
> actually happen. So the question is, has anyone looked into developing a
> service for generating notifications at regular intervals (even if
> something doesn’t happen) that works with Fluo? I’m toying with the idea
> of creating some sort of Twill application that tells Fluo to wake up at
> regular intervals to generate a notification about the set of events
> falling within the given time window. Before doing this I just wanted to
> make sure that something like this does not already exist, and I also want
> to get a sense of how bad an idea it is to delegate some of the logic of
> this periodic notification service to Fluo. Would it be better to
> separate out the temporal portion of my notification request to be
> processed entirely outside of Fluo to avoid transactional overhead?
>
> Caleb A. Meier, Ph.D.
> Software Engineer II ♦ Analyst
> Parsons Corporation
> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
> Office: (703)797-3066 <(703)%20797-3066>
> Caleb.Meier@Parsons.com<ma...@Parsons.com> ♦ www.parsons.com<
> http://www.parsons.com/>
>
> --
Christopher
Re: third party service to poll Fluo for absence of event
Posted by Christopher <ct...@apache.org>.
You could write an observer which rolls up timestamps from all the events
you are concerned about, and puts the most recent event timestamp into a
centralized place, which you could poll. If there is no ingest of these
events, then the last timestamp in this central place will exceed some
threshold and the poller could detect that and trigger additional actions.
On Tue, Jan 31, 2017 at 3:51 PM Meier, Caleb <Ca...@parsons.com>
wrote:
> Hello,
>
> I’m looking into using Fluo to develop an event based notification system
> that incrementally generates events of increasing complexity. The one
> issue that I’m running into is how to handle the non-event event. That is,
> Fluo (as I understand it) is not well-suited to handle the following
> request: “generate a notification if no events of a given type have
> occurred within the last 24 hours”. This is because it is a push based
> notification framework that only generates notifications when things
> actually happen. So the question is, has anyone looked into developing a
> service for generating notifications at regular intervals (even if
> something doesn’t happen) that works with Fluo? I’m toying with the idea
> of creating some sort of Twill application that tells Fluo to wake up at
> regular intervals to generate a notification about the set of events
> falling within the given time window. Before doing this I just wanted to
> make sure that something like this does not already exist, and I also want
> to get a sense of how bad an idea it is to delegate some of the logic of
> this periodic notification service to Fluo. Would it be better to
> separate out the temporal portion of my notification request to be
> processed entirely outside of Fluo to avoid transactional overhead?
>
> Caleb A. Meier, Ph.D.
> Software Engineer II ♦ Analyst
> Parsons Corporation
> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
> Office: (703)797-3066 <(703)%20797-3066>
> Caleb.Meier@Parsons.com<ma...@Parsons.com> ♦ www.parsons.com<
> https://webportal.parsons.com/,DanaInfo=www.parsons.com+>
>
> --
Christopher