[jira] [Commented] (RYA-280) PeriodicQuery Support for Fluo

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/RYA-280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087969#comment-16087969 ] 

ASF GitHub Bot commented on RYA-280:
------------------------------------

GitHub user meiercaleb opened a pull request:

    https://github.com/apache/incubator-rya/pull/177

    RYA-280-Periodic Query Service

    <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
    
      http://www.apache.org/licenses/LICENSE-2.0
    
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
    This pull request contains four main components.  The first component
    is the PeriodicQueryMetadata and observer framework for Fluo.  This capability
    adds a periodic bin id to BindingSets as they percolate through Fluo.  The second component is a
    PeriodicQueryResultStorage API, which essentially wraps existing PCJTables class and allows results to be queried based on their periodic bin id.  The third capability is the external PeriodicQueryService that generates periodic notifications.  When a notification is generated, it uses the time stamp of that notification to poll the PeriodicQueryResultStorage table for results whose periodic bin matches the notification's time stamp.  It then exports the matching results to Kafka.  The fourth and final component is a Batch Processing framework that allows Fluo to batch any process or task that could lead to an out of memory exception.  This can occur when Fluo issues a scan to its underlying table in the middle of 
    a transaction.  Using the Batch framework, a BatchInformation object is written to the BatchObserver column (the BatchInformation objects consist of a task (add, delete, update), a batch size, the column that the task will be applied to, and the query node id that it will be applied to).   Additional information is included as well, such as the Span (range of data that task is applied to).  When the BatchObserver processes a BatchInformation object, it performs the indicated task until it reaches the batch size, at which point it creates a new BatchInformation object with info detailing the stopping point of the last processing stage.  That BatchInformation object is then fed back to the BatchObserver, and this process continues until the task can be completed before reaching the batch size.
    
    ### Tests
    Integration tests were written to test that the PeriodicQueryMetadata
    was being inserted correctly into Fluo, that BindingSets were being binned
    correctly, and that all components of the external PeriodicQueryService were
    operating as expected.  Integration tests can be found in rya.periodic.service.integration.tests
    and rya.pcj.fluo.integration.
    
    ### Links
    [Jira](https://issues.apache.org/jira/browse/RYA-280)
    
    ### Checklist
    - [ ] Code Review
    - [X] Squash Commits
    
    #### People To Reivew
    Andrew Smith, Aaron Mihalik


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/meiercaleb/incubator-rya RYA-280

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-rya/pull/177.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #177
    
----
commit 4bffff21c1aa3541172905a2d0415d07cae1ae3a
Author: Caleb Meier <ca...@parsons.com>
Date:   2017-04-15T02:20:25Z

    RYA-280-Periodic Query Service

----


> PeriodicQuery Support for Fluo
> ------------------------------
>
>                 Key: RYA-280
>                 URL: https://issues.apache.org/jira/browse/RYA-280
>             Project: Rya
>          Issue Type: New Feature
>          Components: clients
>    Affects Versions: 3.2.10
>            Reporter: Caleb Meier
>            Assignee: Caleb Meier
>
> Add the capability to Rya-Fluo App to provide periodic updates for queries registered with Fluo.  That is, provide the application with the ability to satisfy the standing query "tell me every 12 hours about all of the events of a particular type that occurred within the last 24 hours".  Given that Fluo operates using a push based notification system, some external service needs to be implemented to periodically notify Fluo to generate a result notification (this helps handle the non-event event that occurs when nothing happens in a given period of time).  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)