[jira] [Updated] (USERGRID-324) [SPIKE] Prototype a few distributed realtime parallel processing systems

["George Reyes (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/USERGRID-324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

George Reyes updated USERGRID-324:
----------------------------------
    Sprint:   (was: Usergrid 34)

> [SPIKE] Prototype a few distributed realtime parallel processing systems
> ------------------------------------------------------------------------
>
>                 Key: USERGRID-324
>                 URL: https://issues.apache.org/jira/browse/USERGRID-324
>             Project: Usergrid
>          Issue Type: Story
>            Reporter: Todd Nine
>            Priority: Minor
>
> We need a system to allow us to build distributed/parallel data process flows.  Ultimately, this system must have the following requirements at a component level.
> h1. Software Requirements
> # Work well within Reactive/Streams logic
> # Allow easy operationalization.  No  additional external systems should be required
> # Allow failover during node failure, and automatic recovery. Processing should not stop/restart because of a failure, it should resume.
> # Not additional state administration should need to be built.  Distributed futures should be an inherent part of the framework.
>  Some examples are the following.
> # Migrations
> # Import/Export
> # Distributed Indexing on heavily connected entities
> # Post processing deletes
> # Collection deletes
> # Application deletes
> I have the following requirements.
> # You can define a deployment topology and limit the number of sub processes/workers in the workflow
> # Ability to reject requests when there is no capacity
> # Preferably, do not introduce another dependency (like Zookeeper) and deploy it in the stack war file
> # An easy intuitive interface for programming flows which will work in a single node, or clustered environment
> # Administrative status, giving the ability to poll operational status via REST for automated monitoring
> h2. Examples
> h3. Guaranteed post processing
> # A write occurs.  It is written to Cassandra and a post processing message is triggered. 
> ## This message must be persistent, and allow for recovery during complete failure. 
> ## If there is no capacity for post processing in the system, we should reject this request via back pressure
> ## The post processing should return a future (or similar) that could be blocked on if the caller chooses to.
> # The HTTP call is returned to the user
> # Post process indexing occurs asynchronously
> h3. Reindex all entities in the system.
> # Launch a root process.  This process emits all application ids within the system.
> # Child processes receive the application id.  For each app, create the index in elastic search, then emit all collections.
> # Child processes receive the collections and app ids. For each collection, emit the entity id
> # Child process receives the app, collection, and id.  For each entity, get it's edges, and re-index the documents within elastic search. 
> h3. Delete a collection
> Realtime -> Update collection alias to point to a new internal collection name.  Fire delete collection task.
> Job process
> # Launch root process.  Load previous collection name and emit to 2 child tasks.
> # Child task 1: Remove every entity of the previous type from elastic search using bulk delete until empty.
> # Child task 2: Iterate every entity and remove it from Cassandra, as well as it's graph edge.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)