order guarantees on event processing

["Meier, Caleb" <Ca...@parsons.com>]

Hello,

We are currently using Fluo to incrementally update precomputed queries for the RDF triplestore Rya, which is another incubating Apache project.  We have a strategy in place for updating our queries as new triples are ingested, and we are now
working on delete support.

I'm wondering if Fluo provides any guarantees on the order in which it processes notifications.  Are notifications for each observer added to a queue?  For example, if Column C has an observer O and event e1 is written before event e2, is there a guarantee that O will process e1 before e2?  The reason I'm curious about these guarantees is that we want to address the case of one user adding a triple and another user deleting the same triple.  We would like the changes to percolate through our observers in a manner that is consistent with the order in which the triples are written/deleted from the triples column.

If there are no guarantees, do you have any suggestions as to how to enforce notification processing order?  Our initial thought is to require our add/delete observer to only process the event with the lowest add/delete timestamp.  That is, if O attempts to process e2, it checks for the event with the lowest timestamp (e1) and ignores e2 until it encounters and processes e1.  This seems like it could create a huge bottleneck given that it may take a large number of cycles before it processes e1.  Please let me know if you need any additional info for added context.

Thanks,
Caleb Meier

Re: order guarantees on event processing

[Keith Turner <ke...@deenlo.com>]

Caleb,

There is no order guarantee with notification processing.

Can you share a little more info about the portion of your schema related
to updates?

One possible way to solve this problem is to construct a queue for each
triple in Fluo.  As updates come in they are added to the triples queue and
a notification is set.  Eventually an observer processes the entire queue
in order.  There are multiple strategies for implementing a queue, but I
will delay going into that for now.  First want to know if this strategy
even makes sense.

The fluo-jaccard example I wrote recently uses queues per node in a graph.
Each node in the graph has a multiple queues for different types of
updates.  An observers reads all of the queue an computes the nodes new
states, while queuing updates for other nodes.

Keith

On Tue, Jul 5, 2016 at 3:22 PM, Meier, Caleb <Ca...@parsons.com>
wrote:

> Hello,
>
> We are currently using Fluo to incrementally update precomputed queries
> for the RDF triplestore Rya, which is another incubating Apache project.
> We have a strategy in place for updating our queries as new triples are
> ingested, and we are now
> working on delete support.
>
> I'm wondering if Fluo provides any guarantees on the order in which it
> processes notifications.  Are notifications for each observer added to a
> queue?  For example, if Column C has an observer O and event e1 is written
> before event e2, is there a guarantee that O will process e1 before e2?
> The reason I'm curious about these guarantees is that we want to address
> the case of one user adding a triple and another user deleting the same
> triple.  We would like the changes to percolate through our observers in a
> manner that is consistent with the order in which the triples are
> written/deleted from the triples column.
>
> If there are no guarantees, do you have any suggestions as to how to
> enforce notification processing order?  Our initial thought is to require
> our add/delete observer to only process the event with the lowest
> add/delete timestamp.  That is, if O attempts to process e2, it checks for
> the event with the lowest timestamp (e1) and ignores e2 until it encounters
> and processes e1.  This seems like it could create a huge bottleneck given
> that it may take a large number of cycles before it processes e1.  Please
> let me know if you need any additional info for added context.
>
> Thanks,
> Caleb Meier
>