You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@usergrid.apache.org by gr...@apache.org on 2015/05/07 00:35:52 UTC
[01/13] incubator-usergrid git commit: Initial review of new IO
framework. Rough draft and missing lower level read pipeline.
Repository: incubator-usergrid
Updated Branches:
refs/heads/two-dot-o-dev 362e7580c -> 56534b1af
Initial review of new IO framework. Rough draft and missing lower level read pipeline.
Project: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/commit/e4016d0d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/tree/e4016d0d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/diff/e4016d0d
Branch: refs/heads/two-dot-o-dev
Commit: e4016d0d30435a44f6c7f4ddd4babef08967096c
Parents: fd6305c
Author: GERey <gr...@apigee.com>
Authored: Wed Apr 29 16:01:38 2015 -0700
Committer: GERey <gr...@apigee.com>
Committed: Wed Apr 29 16:01:38 2015 -0700
----------------------------------------------------------------------
.../pipeline/PipelineDiagram.jpg | Bin 0 -> 204778 bytes
.../usergrid/corepersistence/pipeline/README.md | 93 +++++++++++++++++++
.../pipeline/cursor/CursorDiagram.jpg | Bin 0 -> 181286 bytes
.../pipeline/read/ReadDiagram.jpg | Bin 0 -> 536513 bytes
4 files changed, 93 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/e4016d0d/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/PipelineDiagram.jpg
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/PipelineDiagram.jpg b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/PipelineDiagram.jpg
new file mode 100644
index 0000000..e38470e
Binary files /dev/null and b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/PipelineDiagram.jpg differ
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/e4016d0d/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
new file mode 100644
index 0000000..cea6b34
--- /dev/null
+++ b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
@@ -0,0 +1,93 @@
+#IO Framework
+Based on the Pipes and Filters software pattern
+
+There are three main parts that are central to the framework.
+
+
+
+Above is the uml diagram of how the seperate modules are connected. We see 7 core parts
+
+
+1. PipelineModule
+ a. This handles all of the guice configuration for the Pipeline Module that we are working in. If you need to wire something do it here.
+1. PipelineBuilderFactory
+ a. This factory is used to instantiate the ReadPipelineBuilder in CpEntityManager and CpRelationshipManager. This lets us use the pipeline outside of the pipeline module.
+1. Pipeline
+ a. The Pipeline class is where it all starts. Here we:
+ 1. Define the application scope to know what application we're working on
+ 1. What pipeline operations are going to be passed through
+ 1. What type of Collector we are going to be using.
+ 1. The request and response cursor.
+ 1. The limit on the number of entities we might want to get back from a query
+ 1. Where in the pipeline we are.
+ a. The ```execute``` method contains the execution of the pipeline. It retrieves the applications and pipes the operations from one to another until we run out of operations. At that point we aggregate the response into a collection and return an instance of PipelineResult with the results and a cursor.
+ a. Things that depend on the read module
+ 1. The PipelineOperation
+ 1. The Collector
+ a. Things that depend on the cursor module
+ 1. Request and Response Cursor
+1. PipelineContext
+ a. This is a stateful representation of the current step in the pipeline. It contains the cusor that we want to run (requestCursor) and the cursor that is returned from the filter (responseCursor).
+ a. Each filter contains a Pipeline Context due to the above reason.
+1. PipelineResult
+ a. Gets created in the Pipeline after the results are collected from the filters.
+ a. Depends on the ResponseCursor class.
+1. Cursor Module
+ a. Contains the Request and ResponseCursor classes that are directly instantiated in Pipeline Module.
+ a. Contains logic that represents the cursor logic
+***
+
+###Indepth Cursor Explanation
+ 
+
+The Cursor Module is made up of 7 classes.
+
+1. ResponseCursor
+ a. This is the cursor that gets returned in the response after the filter has run.
+ b. The flow defined by the Response cursor is as follows
+ 1. Set cursor(s) that are made up of a Integer and a CursorEntry
+ 1. Response Cursor gets initalized
+ 1. We go into the CursorEntry class that consists of the Cursor ( of a raw type ) and the serializer that we would use to parse the Cursor.
+1. RequestCursor
+ a. Contains some information on the parsedCursor
+ b. This gets populated by either the User ( using a cursor that we've given them), or by the pipeline feeding the cursor into the next stage.
+ c. Could be
+
+
+
+***
+###Indepth Read Module Explanation
+ 
+
+1. PipelineOperation
+ a. Top class in the Pipeline because it defines what every pipeline operation needs to have and extend. Mandates that every class contains a PipelineContext
+ b. Primary interface for Filtering commands.
+1. CandidateResultsFilter
+ a. Is an interface
+ a. Extends PipelineOperation
+ b. Defines the types that will be requried in the filter. While not visible in the diagram the CandidateResultsFilters will consist of a <Id, CandidateResults>.
+ c. Primary filter that will be used for interfacing with ES (Elasticsearch)
+1. Filter
+ a. Extends generic PipelineOperation
+ b. Primary used to interact with Graph and Entity modules
+1. Collector
+ a. Extends generic PipelineOperation
+ b. Primary used to interact with Entity and Elasticsearch Packages
+ a. The stage of our filters that is used to reduce our stream of results into our final output.
+1. Elasticsearch Module
+ a. Contains the functions we use to actual perform filtered commands that contain elasticsearch components.
+1. Entity Module
+ a. Contains
+
+
+##How does this all work?
+Consider the following example flow:
+
+ ```pipeline = 1. appId -> 2. filter1 -> filter2 -> 3. collector -> 4. PipelineResult```
+
+ 1. First we start with an application. You want to do a certain set of operations on this application. The application gets passed into the pipeline. We extract the application id and pass unto the first filter
+ 1. The filter represents some action being done unto the application. This could be a "getCollection" to a "deleteIndex" action. We can have as many as we want. These operations will be done in the order that they are represented in above. The results of each of the filters will be fed into the next one for processing. Thus filter 1 will be done before filter 2 can be applied.
+ a. An important note here is that a cursor may or may not be generated here. this cursor ( if it exists ) will be stored in what is known as a Pipeline Context. The context contains all approriate state cursor information in case we need resume a query after we have run it.
+ 1. After we have applied all the filters, we take the results and feed them into a collector. The collector aggreates the results returned and pushes them into a ResultSet.
+ 1. The result set ( along with a cursor is one was generated ) is fed into the PipelineResult class. The PipelineResult is returned as a observable that can be iterated on by the calling method.
+
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/e4016d0d/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/cursor/CursorDiagram.jpg
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/cursor/CursorDiagram.jpg b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/cursor/CursorDiagram.jpg
new file mode 100644
index 0000000..81442b0
Binary files /dev/null and b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/cursor/CursorDiagram.jpg differ
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/e4016d0d/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg
new file mode 100644
index 0000000..9a54e9d
Binary files /dev/null and b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg differ
[05/13] incubator-usergrid git commit: Fixing some of the formatting
errors for github.
Posted by gr...@apache.org.
Fixing some of the formatting errors for github.
Project: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/commit/c6bbfba9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/tree/c6bbfba9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/diff/c6bbfba9
Branch: refs/heads/two-dot-o-dev
Commit: c6bbfba989b806b106fe6b2e4f1744123c30c760
Parents: d557294
Author: GERey <gr...@apigee.com>
Authored: Mon May 4 10:30:37 2015 -0700
Committer: GERey <gr...@apigee.com>
Committed: Mon May 4 10:30:37 2015 -0700
----------------------------------------------------------------------
.../usergrid/corepersistence/pipeline/README.md | 128 +++++++++----------
1 file changed, 61 insertions(+), 67 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/c6bbfba9/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
index 0e12c45..ad42612 100644
--- a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
+++ b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
@@ -1,7 +1,6 @@
#IO Framework
Based on the Pipes and Filters software pattern
-
##How does this all work?
@@ -11,20 +10,21 @@ Consider the following example flow:
1. First we start with an application. You want to do a certain set of operations on this application. The application gets passed into the pipeline. We extract the application id and pass unto the first filter
1. The filter represents some action being done unto the application. This could be a "getCollection" to a "deleteIndex" action. We can have as many as we want. These operations will be done in the order that they are represented in above. The results of each of the filters will be fed into the next one for processing. Thus filter 1 will be done before filter 2 can be applied.
- a. An important note here is that a cursor may or may not be generated here. this cursor ( if it exists ) will be stored in what is known as a Pipeline Context. The context contains all approriate state cursor information in case we need resume a query after we have run it.
+ * An important note here is that a cursor may or may not be generated here. this cursor ( if it exists ) will be stored in what is known as a Pipeline Context. The context contains all approriate state cursor information in case we need resume a query after we have run it.
1. After we have applied all the filters, we take the results and feed them into a collector. The collector aggreates the results returned and pushes them into a ResultSet.
1. The result set ( along with a cursor is one was generated ) is fed into the PipelineResult class. The PipelineResult is returned as a observable that can be iterated on by the calling method.
-Above is the uml diagram of how the seperate modules are connected. We see 7 core parts
-
###Pipeline Module
+
+
1. PipelineModule
- a. This handles all of the guice configuration for the Pipeline Module that we are working in. If you need to wire something do it here.
+
+ * This handles all of the guice configuration for the Pipeline Module that we are working in. If you need to wire something, do it here.
1. PipelineBuilderFactory
- a. This factory is used to instantiate the ReadPipelineBuilder in CpEntityManager and CpRelationshipManager. This lets us use the pipeline outside of the pipeline module.
+ * This factory is used to instantiate the ReadPipelineBuilder in CpEntityManager and CpRelationshipManager.
1. Pipeline
- a. The Pipeline class is where it all starts. Here we:
+ * The Pipeline class is where it all starts. Here we:
1. Define the application scope to know what application we're working on
1. What pipeline operations are going to be passed through
1. What type of Collector we are going to be using.
@@ -32,24 +32,19 @@ Above is the uml diagram of how the seperate modules are connected. We see 7 cor
1. The limit on the number of entities we might want to get back from a query
1. Where in the pipeline we are.
a. The ```execute``` method contains the execution of the pipeline. It retrieves the applications and pipes the operations from one to another until we run out of operations. At that point we aggregate the response into a collection and return an instance of PipelineResult with the results and a cursor.
- a. Things that depend on the read module
- 1. The PipelineOperation
- 1. The Collector
- a. Things that depend on the cursor module
- 1. Request and Response Cursor
1. PipelineContext
- a. This is a stateful representation of the current step in the pipeline. It contains the cusor that we want to run (requestCursor) and the cursor that is returned from the filter (responseCursor).
- a. Each filter contains a Pipeline Context due to the above reason.
+ * This is a stateful representation of the current step in the pipeline. It contains the cusor that we want to run (requestCursor) and the cursor that is returned from the filter (responseCursor).
+ * Each filter contains a Pipeline Context due to the above reason.
1. PipelineResult
- a. Gets created in the Pipeline after the results are collected from the filters.
- a. Depends on the ResponseCursor class.
+ * Gets created in the Pipeline after the results are collected from the filters.
+ * Depends on the ResponseCursor class.
1. Cursor Module
- a. Contains the Request and ResponseCursor classes that are directly instantiated in Pipeline Module.
- a. Contains logic that represents the cursor logic
+ * Contains the Request and ResponseCursor classes that are instantiated in Pipeline Module.
+ * Contains logic that represents the cursor logic.
***
###Indepth Cursor Module Explanation
- 
+ 
The Cursor Module is made up of 7 classes.
@@ -64,95 +59,94 @@ The Cursor Module is made up of 7 classes.
b. This gets populated by either the User ( using a cursor that we've given them), or by the pipeline feeding the cursor into the next stage.
c. Could be
-
-
***
###Indepth Read Module Explanation
- 
+ 
1. PipelineOperation
- a. Top class in the Pipeline because it defines what every pipeline operation needs to have and extend. Mandates that every class contains a PipelineContext
- b. Primary interface for Filtering commands.
+ * Top class in the Pipeline because it defines what every pipeline operation needs to have and extend. Mandates that every class in the Pipeline extend this class.
+ * Primary interface for Filtering commands.
1. CandidateResultsFilter
- a. Is an interface
- a. Extends PipelineOperation
- b. Defines the types that will be requried in the filter. While not visible in the diagram the CandidateResultsFilters will consist of a <Id, CandidateResults>.
- c. Primary filter that will be used for interfacing with ES (Elasticsearch)
+ * Is an interface
+ * Extends PipelineOperation
+ * Defines the types that will be requried in the filter. While not visible in the diagram the CandidateResultsFilters will consist of a <Id, CandidateResults>.
+ * Primary filter that will be used for interfacing with ES (Elasticsearch)
1. Filter
- a. Extends generic PipelineOperation
- b. Primary used to interact with Graph and Entity modules
- b. Why do we use the filter in the ReadPipeline when we could also interchange the Canadiate Results filter? Is it just the type that differentiates it.
+ * Extends generic PipelineOperation
+ * Primary used to interact with Graph and Entity modules
+ * Why do we use the filter in the ReadPipeline when we could also interchange the Canadiate Results filter? Is it just the type that differentiates it.
1. AbstractSeekingFilter
- a. This abstract filter is meant to be extended by any filter that requires a cursor and operations on that cursor.
- b. Extends from the AbstractPipelineOperation because a filter is a pipeline operation.
- c. Is used in the Graph and Elasticsearch submodules because those both use cursors.
+ * This abstract filter is meant to be extended by any filter that requires a cursor and operations on that cursor.
+ * Extends from the AbstractPipelineOperation because a filter is a pipeline operation.
+ * Is used in the Graph and Elasticsearch submodules because those both use cursors.
1. CursorSeek
- a. Protected internal class that lives in AbstractSeekingFilter
- b. Whats the deal with only seeking values on the first call? Is this not similar to pagination?
+ * Protected internal class that lives in AbstractSeekingFilter
+ * Whats the deal with only seeking values on the first call? Is this not similar to pagination?
1. Collector
- a. Extends generic PipelineOperation
- b. Primary used to interact with Entity and Elasticsearch Packages
- a. The stage of our filters that is used to reduce our stream of results into our final output.
+ * Extends generic PipelineOperation
+ * Primary used to interact with Entity and Elasticsearch Packages
+ * Used to reduce our stream of results into our final output.
1. CollectorState
- a. The state that holds a singleton collector instance and what type of collector the Collector filter should be using.
- a. The collector state gets initialized with a CollectorFactory and then gets set with which collector it should use for the Collector object that it holds.
- b. This is a private inner class within ReadPipelineBuilderImpl
+ * The state that holds a singleton collector instance and what type of collector the Collector filter should be using.
+ * The collector state gets initialized with a CollectorFactory and then gets set with which collector it should use for the Collector object that it holds.
+ * This is a private inner class within ReadPipelineBuilderImpl
1. Elasticsearch Module
- a. Contains the functions we use to actual perform filtered commands that contain elasticsearch components.
- b. These will typically return Canadiate Result sets that will be processed by the collector.
+ * Contains the functions we use to actual perform filtered commands that contain elasticsearch components.
+ * These will typically return Canadiate Result sets that will be processed by the collector.
1. Entity Module
- a. Contains a single filter that maps id's, and the collector that processes entity id's.
+ * Contains a single filter that maps id's, and the collector that processes entity id's.
1. Graph Module
- a. Contains the filters that are used to interact with the lower level Graph Module.
+ * Contains the filters that are used to interact with the lower level Graph Module.
1. FilterFactory
- a. Defines all of the possible read filters that can be added to a pipeline.
- b. Contain comments on what each type of filter should accomplish.
+ * Defines all of the possible read filters that can be added to a pipeline.
+ * Contain comments on what each type of filter should accomplish.
1. ReadPipelineBuilder
- a. Contains the builder interface that will assemble the underlying pipe along with updating and keeping track of its state.
+ * Contains the builder interface that will assemble the underlying pipe along with updating and keeping track of its state.
1. ReadPipelineBuilderImpl
- a. Contains the builder implementation of the ReadPipelineBuilder.
- b. Adds on filters from FilterFactory depending on the type of action we take.
- c. Contains execute method when the pipeline is finished building. This pushes results as a PipelineResult back up.
+ * Contains the builder implementation of the ReadPipelineBuilder.
+ * Adds on filters from FilterFactory depending on the type of action we take.
+ * Contains execute method when the pipeline is finished building. This pushes results as a PipelineResult back up.
***
###Indepth Entity Module Explanation
The entity module only contains two classes. So I won't attach the uml diagram as they aren't related to each other in any way.
1. EntityIdFilter
- a. A stopgap filter that helps migrating from the service tier and its entities. Just makes a list of entities.
+ * A stopgap filter that helps migrating from the service tier and its entities. Just makes a list of entities.
2. EntityLoadCollector
- a. The EntityLoadCollector loops through entity id's and then converts them to our old entity model so that they can go through the service and rest tier.
+ * The EntityLoadCollector loops through entity id's and then converts them to our old entity model so that they can go through the service and rest tier.
+
***
###Indepth Graph Module Explanation
- 
+ 
1. EdgeCursorSerializer
- a. The serializer we use to decode and make sense of the graph cursor that gets returned.
+ * The serializer we use to decode and make sense of the graph cursor that gets returned.
1. AbstractReadGraph(EdgeById)Filter
- a. An abstract class that defines how we should read graph edges from name(id).
+ * An abstract class that defines how we should read graph edges from name(id).
1. ReadGraphConnection(ById)Filter
- a. Defines how to read Connections out of the graph using names(id).
+ * Defines how to read Connections out of the graph using names(id).
1. ReadGraphCollection(ById)Filter
- a. Defines how to read Collections out of the graph using names(id).
+ * Defines how to read Collections out of the graph using names(id).
+ 1. ReadGraphconnectionByTypeFilter
+ * A filter that reads graph connections by type.
- 2. ReadGraphconnectionByTypeFilter
- a. A filter that reads graph connections by type.
***
###Indepth Elasticsearch Module Explanation
- 
+ 
1. Impl Module
- a. contains all the implementations and verfiers and loaders for elasticsearch
+ * contains all the implementations and verfiers and loaders for elasticsearch
2. AbstractElasticSearchFilter
- a. This extends into the same pattern as the Graph Module where we make a abstract filter so we can extend it to easily accomodate Collection or Connection searching.
+ * This extends into the same pattern as the Graph Module where we make a abstract filter so we can extend it to easily accomodate Collection or Connection searching.
3. CandidateResultsEntityResultsCollector
- a. Collects the results from Elasticsearch then retrieves them from cassandra and converts them from 2.0 to 1.0 entities that are suitable for return.
+ * Collects the results from Elasticsearch then retrieves them from cassandra and converts them from 2.0 to 1.0 entities that are suitable for return.
4. CandidateResultsIdVerifyFilter
- a. Filter that verifies that the canaidate results id's are correct???? What else does this do ? isn't that what the collector does?
+ * Filter that verifies that the canaidate results id's are correct???? What else does this do ? isn't that what the collector does?
5. ElasticsearchCursorSerializer
- a. The serializer we use to decode and make sense of the elasticsearch cursor.
+ * The serializer we use to decode and make sense of the elasticsearch cursor.
[04/13] incubator-usergrid git commit: Merge branch 'two-dot-o-dev'
of https://git-wip-us.apache.org/repos/asf/incubator-usergrid into
USERGRID-609
Posted by gr...@apache.org.
Merge branch 'two-dot-o-dev' of https://git-wip-us.apache.org/repos/asf/incubator-usergrid into USERGRID-609
# By Shawn Feldman
# Via Shawn Feldman
* 'two-dot-o-dev' of https://git-wip-us.apache.org/repos/asf/incubator-usergrid:
fixing test consistency
Project: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/commit/d5572940
Tree: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/tree/d5572940
Diff: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/diff/d5572940
Branch: refs/heads/two-dot-o-dev
Commit: d5572940475d77d08f14bf9bf833be8efa4de05b
Parents: 54871b5 d18b8ee
Author: GERey <gr...@apigee.com>
Authored: Mon May 4 10:26:26 2015 -0700
Committer: GERey <gr...@apigee.com>
Committed: Mon May 4 10:26:26 2015 -0700
----------------------------------------------------------------------
.../persistence/index/impl/EntityIndexTest.java | 121 ++++++++++---------
1 file changed, 61 insertions(+), 60 deletions(-)
----------------------------------------------------------------------
[08/13] incubator-usergrid git commit: Updated all the diagrams for
newest refactor.
Posted by gr...@apache.org.
Updated all the diagrams for newest refactor.
Project: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/commit/5312bbc6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/tree/5312bbc6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/diff/5312bbc6
Branch: refs/heads/two-dot-o-dev
Commit: 5312bbc6b6e8c35305a5103c176385643eeb2699
Parents: c3897d3
Author: GERey <gr...@apigee.com>
Authored: Mon May 4 11:21:14 2015 -0700
Committer: GERey <gr...@apigee.com>
Committed: Mon May 4 11:21:14 2015 -0700
----------------------------------------------------------------------
.../usergrid/corepersistence/pipeline/README.md | 36 +++++++++-----------
1 file changed, 16 insertions(+), 20 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/5312bbc6/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
index ad42612..f469e30 100644
--- a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
+++ b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
@@ -1,25 +1,21 @@
#IO Framework
Based on the Pipes and Filters software pattern
-
-
##How does this all work?
Consider the following example flow:
- ```pipeline = 1. appId -> 2. filter1 -> filter2 -> 3. collector -> 4. PipelineResult```
+ ```pipeline = 1. appId -> 2. filter1 -> filter2 -> 3. collector -> 4. Observable```
1. First we start with an application. You want to do a certain set of operations on this application. The application gets passed into the pipeline. We extract the application id and pass unto the first filter
1. The filter represents some action being done unto the application. This could be a "getCollection" to a "deleteIndex" action. We can have as many as we want. These operations will be done in the order that they are represented in above. The results of each of the filters will be fed into the next one for processing. Thus filter 1 will be done before filter 2 can be applied.
* An important note here is that a cursor may or may not be generated here. this cursor ( if it exists ) will be stored in what is known as a Pipeline Context. The context contains all approriate state cursor information in case we need resume a query after we have run it.
1. After we have applied all the filters, we take the results and feed them into a collector. The collector aggreates the results returned and pushes them into a ResultSet.
- 1. The result set ( along with a cursor is one was generated ) is fed into the PipelineResult class. The PipelineResult is returned as a observable that can be iterated on by the calling method.
-
+ 1. The result set ( along with a cursor is one was generated ) is fed into a observable that can be iterated on.
###Pipeline Module
-
+
1. PipelineModule
-
* This handles all of the guice configuration for the Pipeline Module that we are working in. If you need to wire something, do it here.
1. PipelineBuilderFactory
* This factory is used to instantiate the ReadPipelineBuilder in CpEntityManager and CpRelationshipManager.
@@ -31,20 +27,23 @@ Consider the following example flow:
1. The request and response cursor.
1. The limit on the number of entities we might want to get back from a query
1. Where in the pipeline we are.
- a. The ```execute``` method contains the execution of the pipeline. It retrieves the applications and pipes the operations from one to another until we run out of operations. At that point we aggregate the response into a collection and return an instance of PipelineResult with the results and a cursor.
+ * The ```execute``` method contains the execution of the pipeline. It retrieves the applications and pipes the operations from one to another until we run out of operations. At that point we aggregate the response into a collection and return an observable with the elements.
1. PipelineContext
* This is a stateful representation of the current step in the pipeline. It contains the cusor that we want to run (requestCursor) and the cursor that is returned from the filter (responseCursor).
* Each filter contains a Pipeline Context due to the above reason.
-1. PipelineResult
- * Gets created in the Pipeline after the results are collected from the filters.
- * Depends on the ResponseCursor class.
+1. PipelineOperation
+ * Top class in the Pipeline because it defines what every pipeline operation needs to have and extend. Mandates that every class in the Pipeline extend this class.
+ * Primary interface for Filtering and Collection commands.
1. Cursor Module
* Contains the Request and ResponseCursor classes that are instantiated in Pipeline Module.
* Contains logic that represents the cursor logic.
+1. Read Module
+ * Contains the logic behind reading from graph and elasticsearch.
+
***
###Indepth Cursor Module Explanation
- 
+ 
The Cursor Module is made up of 7 classes.
@@ -61,11 +60,9 @@ The Cursor Module is made up of 7 classes.
***
###Indepth Read Module Explanation
- 
+ 
+
-1. PipelineOperation
- * Top class in the Pipeline because it defines what every pipeline operation needs to have and extend. Mandates that every class in the Pipeline extend this class.
- * Primary interface for Filtering commands.
1. CandidateResultsFilter
* Is an interface
* Extends PipelineOperation
@@ -82,7 +79,6 @@ The Cursor Module is made up of 7 classes.
1. CursorSeek
* Protected internal class that lives in AbstractSeekingFilter
* Whats the deal with only seeking values on the first call? Is this not similar to pagination?
-
1. Collector
* Extends generic PipelineOperation
* Primary used to interact with Entity and Elasticsearch Packages
@@ -106,7 +102,7 @@ The Cursor Module is made up of 7 classes.
1. ReadPipelineBuilderImpl
* Contains the builder implementation of the ReadPipelineBuilder.
* Adds on filters from FilterFactory depending on the type of action we take.
- * Contains execute method when the pipeline is finished building. This pushes results as a PipelineResult back up.
+ * Contains execute method when the pipeline is finished building. This pushes results as an observable back up.
***
###Indepth Entity Module Explanation
@@ -119,7 +115,7 @@ The entity module only contains two classes. So I won't attach the uml diagram a
***
###Indepth Graph Module Explanation
- 
+ 
1. EdgeCursorSerializer
* The serializer we use to decode and make sense of the graph cursor that gets returned.
@@ -135,7 +131,7 @@ The entity module only contains two classes. So I won't attach the uml diagram a
***
###Indepth Elasticsearch Module Explanation
- 
+ 
1. Impl Module
* contains all the implementations and verfiers and loaders for elasticsearch
[10/13] incubator-usergrid git commit: Removing the cursor module
comment on 7 classes.
Posted by gr...@apache.org.
Removing the cursor module comment on 7 classes.
Project: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/commit/fc4b48af
Tree: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/tree/fc4b48af
Diff: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/diff/fc4b48af
Branch: refs/heads/two-dot-o-dev
Commit: fc4b48af21dca6ff8509e040105466e66061920d
Parents: 8d1618f
Author: GERey <gr...@apigee.com>
Authored: Mon May 4 14:56:38 2015 -0700
Committer: GERey <gr...@apigee.com>
Committed: Mon May 4 14:56:38 2015 -0700
----------------------------------------------------------------------
.../java/org/apache/usergrid/corepersistence/pipeline/README.md | 2 --
1 file changed, 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/fc4b48af/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
index 09b3877..ff123b0 100644
--- a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
+++ b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
@@ -45,8 +45,6 @@ Consider the following example flow:
###Indepth Cursor Module Explanation
-The Cursor Module is made up of 7 classes.
-
1. ResponseCursor
* This is the cursor that gets returned in the response after the filter has run.
* The flow defined by the Response cursor is as follows
[13/13] incubator-usergrid git commit: Merge remote-tracking branch
'origin/USERGRID-609' into two-dot-o-dev
Posted by gr...@apache.org.
Merge remote-tracking branch 'origin/USERGRID-609' into two-dot-o-dev
Project: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/commit/56534b1a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/tree/56534b1a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/diff/56534b1a
Branch: refs/heads/two-dot-o-dev
Commit: 56534b1affddc2c67749ea860a1b65da579d0f01
Parents: 362e758 eae4cbd
Author: GERey <gr...@apigee.com>
Authored: Wed May 6 15:35:42 2015 -0700
Committer: GERey <gr...@apigee.com>
Committed: Wed May 6 15:35:42 2015 -0700
----------------------------------------------------------------------
.../pipeline/PipelineDiagram.jpg | Bin 0 -> 183035 bytes
.../usergrid/corepersistence/pipeline/README.md | 187 +++++++++++++++++++
.../pipeline/cursor/CursorDiagram.jpg | Bin 0 -> 167390 bytes
.../pipeline/read/ReadDiagram.jpg | Bin 0 -> 818565 bytes
.../pipeline/read/collect/CollectDiagram.jpg | Bin 0 -> 98007 bytes
.../read/elasticsearch/Elasticsearchdiagram.jpg | Bin 0 -> 316655 bytes
.../pipeline/read/graph/GraphDiagram.jpg | Bin 0 -> 347711 bytes
7 files changed, 187 insertions(+)
----------------------------------------------------------------------
[02/13] incubator-usergrid git commit: Merge branch 'two-dot-o-dev'
of https://git-wip-us.apache.org/repos/asf/incubator-usergrid into
USERGRID-609
Posted by gr...@apache.org.
Merge branch 'two-dot-o-dev' of https://git-wip-us.apache.org/repos/asf/incubator-usergrid into USERGRID-609
# By Shawn Feldman
# Via Shawn Feldman
* 'two-dot-o-dev' of https://git-wip-us.apache.org/repos/asf/incubator-usergrid:
moving back to refresh its faster
moving back to refresh its faster
refresh to observable
adding refresh back
test passing
null safety
Project: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/commit/264e67ff
Tree: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/tree/264e67ff
Diff: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/diff/264e67ff
Branch: refs/heads/two-dot-o-dev
Commit: 264e67ff9d68d5c28290e0db219e315fdddcbb26
Parents: e4016d0 faafa37
Author: GERey <gr...@apigee.com>
Authored: Fri May 1 09:48:45 2015 -0700
Committer: GERey <gr...@apigee.com>
Committed: Fri May 1 09:48:45 2015 -0700
----------------------------------------------------------------------
.../corepersistence/CpRelationManager.java | 34 ++---
.../usergrid/persistence/EntityManagerIT.java | 4 +-
.../graph/impl/SimpleSearchByEdge.java | 5 +-
.../usergrid/persistence/index/IndexFig.java | 11 +-
.../persistence/index/IndexRefreshCommand.java | 2 +-
.../index/impl/EsEntityIndexImpl.java | 2 +-
.../index/impl/IndexRefreshCommandImpl.java | 136 +++++++++++--------
.../persistence/index/impl/EntityIndexTest.java | 2 +-
8 files changed, 112 insertions(+), 84 deletions(-)
----------------------------------------------------------------------
[06/13] incubator-usergrid git commit: Merge branch 'two-dot-o-dev'
of https://git-wip-us.apache.org/repos/asf/incubator-usergrid into
USERGRID-609
Posted by gr...@apache.org.
Merge branch 'two-dot-o-dev' of https://git-wip-us.apache.org/repos/asf/incubator-usergrid into USERGRID-609
# By Todd Nine
# Via Todd Nine
* 'two-dot-o-dev' of https://git-wip-us.apache.org/repos/asf/incubator-usergrid:
Updates observable short circuit and fixes NPE on empty results
Fixes graph cursor resume state.
Fixes resume logic by loading then filtering first id
Massive refactor. Paths for cursor generation are now part of our I/O results. This allows the collector to take until satisfied, then generate a serializable path.
Updated mapping to fix missing doc_values and disable norms since we use external sorting
Project: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/commit/7b215250
Tree: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/tree/7b215250
Diff: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/diff/7b215250
Branch: refs/heads/two-dot-o-dev
Commit: 7b215250afce3303af6a26ecf4ec8918840e37e0
Parents: c6bbfba 9b939d1
Author: GERey <gr...@apigee.com>
Authored: Mon May 4 10:32:43 2015 -0700
Committer: GERey <gr...@apigee.com>
Committed: Mon May 4 10:32:43 2015 -0700
----------------------------------------------------------------------
.../corepersistence/CpRelationManager.java | 5 +-
.../corepersistence/pipeline/Pipeline.java | 12 +-
.../pipeline/PipelineContext.java | 16 +-
.../pipeline/PipelineOperation.java | 39 ++++
.../pipeline/PipelineResult.java | 57 -----
.../pipeline/cursor/ResponseCursor.java | 81 ++++---
.../pipeline/read/AbstractFilter.java | 45 ++++
.../pipeline/read/AbstractPathFilter.java | 109 +++++++++
.../read/AbstractPipelineOperation.java | 44 ----
.../pipeline/read/AbstractSeekingFilter.java | 102 --------
.../pipeline/read/CandidateResultsFilter.java | 31 ---
.../pipeline/read/Collector.java | 13 +-
.../pipeline/read/CollectorFactory.java | 12 +-
.../corepersistence/pipeline/read/EdgePath.java | 79 +++++++
.../corepersistence/pipeline/read/Filter.java | 9 +-
.../pipeline/read/FilterFactory.java | 40 +++-
.../pipeline/read/FilterResult.java | 56 +++++
.../pipeline/read/PipelineOperation.java | 38 ---
.../pipeline/read/ReadPipelineBuilder.java | 5 +-
.../pipeline/read/ReadPipelineBuilderImpl.java | 80 ++++---
.../pipeline/read/ResultsPage.java | 26 ++-
.../read/collect/AbstractCollector.java | 46 ++++
.../pipeline/read/collect/EntityFilter.java | 68 ++++++
.../read/collect/IdCursorSerializer.java | 41 ++++
.../read/collect/ResultsPageCollector.java | 80 +++++++
.../AbstractElasticSearchFilter.java | 45 ++--
.../pipeline/read/elasticsearch/Candidate.java | 55 +++++
.../elasticsearch/CandidateEntityFilter.java | 234 +++++++++++++++++++
.../read/elasticsearch/CandidateIdFilter.java | 201 ++++++++++++++++
.../CandidateResultsEntityResultsCollector.java | 217 -----------------
.../CandidateResultsIdVerifyFilter.java | 193 ---------------
.../impl/CollectionRefsVerifier.java | 44 ----
.../CollectionResultsLoaderFactoryImpl.java | 65 ------
.../impl/ConnectionRefsVerifier.java | 59 -----
.../ConnectionResultsLoaderFactoryImpl.java | 73 ------
.../impl/ElasticSearchQueryExecutor.java | 224 ------------------
.../read/elasticsearch/impl/EntityVerifier.java | 127 ----------
.../elasticsearch/impl/FilteringLoader.java | 219 -----------------
.../read/elasticsearch/impl/IdsVerifier.java | 46 ----
.../read/elasticsearch/impl/ResultsLoader.java | 43 ----
.../impl/ResultsLoaderFactory.java | 41 ----
.../elasticsearch/impl/ResultsVerifier.java | 52 -----
.../elasticsearch/impl/VersionVerifier.java | 85 -------
.../pipeline/read/entity/EntityIdFilter.java | 53 -----
.../read/entity/EntityLoadCollector.java | 94 --------
.../graph/AbstractReadGraphEdgeByIdFilter.java | 12 +-
.../read/graph/AbstractReadGraphFilter.java | 65 +++++-
.../pipeline/read/graph/EntityIdFilter.java | 54 +++++
.../pipeline/read/graph/EntityLoadFilter.java | 155 ++++++++++++
.../graph/ReadGraphConnectionByTypeFilter.java | 20 +-
.../results/ObservableQueryExecutor.java | 24 +-
.../apache/usergrid/persistence/Results.java | 2 +-
.../pipeline/cursor/CursorTest.java | 20 +-
.../persistence/index/CandidateResults.java | 11 +-
.../impl/EsApplicationEntityIndexImpl.java | 7 +-
.../persistence/index/usergrid-mappings.json | 3 +
56 files changed, 1578 insertions(+), 2099 deletions(-)
----------------------------------------------------------------------
[07/13] incubator-usergrid git commit: Committing new diagrams that
reflect the latest refactor of the io framework.
Posted by gr...@apache.org.
Committing new diagrams that reflect the latest refactor of the io framework.
Project: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/commit/c3897d3a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/tree/c3897d3a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/diff/c3897d3a
Branch: refs/heads/two-dot-o-dev
Commit: c3897d3abac7226d9a93a831c020567abd00536c
Parents: 7b21525
Author: GERey <gr...@apigee.com>
Authored: Mon May 4 11:15:25 2015 -0700
Committer: GERey <gr...@apigee.com>
Committed: Mon May 4 11:15:25 2015 -0700
----------------------------------------------------------------------
.../pipeline/PipelineDiagram.jpg | Bin 204778 -> 183035 bytes
.../pipeline/cursor/CursorDiagram.jpg | Bin 181286 -> 167390 bytes
.../pipeline/read/ReadDiagram.jpg | Bin 704305 -> 568372 bytes
.../pipeline/read/collect/CollectDiagram.jpg | Bin 0 -> 98007 bytes
.../read/elasticsearch/Elasticsearchdiagram.jpg | Bin 355271 -> 316655 bytes
.../pipeline/read/graph/GraphDiagram.jpg | Bin 224755 -> 347711 bytes
6 files changed, 0 insertions(+), 0 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/c3897d3a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/PipelineDiagram.jpg
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/PipelineDiagram.jpg b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/PipelineDiagram.jpg
index e38470e..7724697 100644
Binary files a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/PipelineDiagram.jpg and b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/PipelineDiagram.jpg differ
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/c3897d3a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/cursor/CursorDiagram.jpg
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/cursor/CursorDiagram.jpg b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/cursor/CursorDiagram.jpg
index 81442b0..7410cc3 100644
Binary files a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/cursor/CursorDiagram.jpg and b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/cursor/CursorDiagram.jpg differ
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/c3897d3a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg
index d83ff38..60e4b66 100644
Binary files a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg and b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg differ
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/c3897d3a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/collect/CollectDiagram.jpg
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/collect/CollectDiagram.jpg b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/collect/CollectDiagram.jpg
new file mode 100644
index 0000000..809c554
Binary files /dev/null and b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/collect/CollectDiagram.jpg differ
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/c3897d3a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/elasticsearch/Elasticsearchdiagram.jpg
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/elasticsearch/Elasticsearchdiagram.jpg b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/elasticsearch/Elasticsearchdiagram.jpg
index 1f6a616..08970e3 100644
Binary files a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/elasticsearch/Elasticsearchdiagram.jpg and b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/elasticsearch/Elasticsearchdiagram.jpg differ
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/c3897d3a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/graph/GraphDiagram.jpg
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/graph/GraphDiagram.jpg b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/graph/GraphDiagram.jpg
index e808060..c0308bd 100644
Binary files a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/graph/GraphDiagram.jpg and b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/graph/GraphDiagram.jpg differ
[09/13] incubator-usergrid git commit: Added dependencies in the
ReadDiagram. First complete rough draft of the README for the ioframework.
Posted by gr...@apache.org.
Added dependencies in the ReadDiagram.
First complete rough draft of the README for the ioframework.
Project: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/commit/8d1618f8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/tree/8d1618f8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/diff/8d1618f8
Branch: refs/heads/two-dot-o-dev
Commit: 8d1618f845ccd4680b27b08dbff7053a04e02b34
Parents: 5312bbc
Author: GERey <gr...@apigee.com>
Authored: Mon May 4 14:54:22 2015 -0700
Committer: GERey <gr...@apigee.com>
Committed: Mon May 4 14:54:22 2015 -0700
----------------------------------------------------------------------
.../usergrid/corepersistence/pipeline/README.md | 110 +++++++++++++------
.../pipeline/read/ReadDiagram.jpg | Bin 568372 -> 818565 bytes
2 files changed, 78 insertions(+), 32 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/8d1618f8/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
index f469e30..09b3877 100644
--- a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
+++ b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
@@ -33,7 +33,7 @@ Consider the following example flow:
* Each filter contains a Pipeline Context due to the above reason.
1. PipelineOperation
* Top class in the Pipeline because it defines what every pipeline operation needs to have and extend. Mandates that every class in the Pipeline extend this class.
- * Primary interface for Filtering and Collection commands.
+ * Primary interface for Filtering and Collection commands in the Read module.
1. Cursor Module
* Contains the Request and ResponseCursor classes that are instantiated in Pipeline Module.
* Contains logic that represents the cursor logic.
@@ -48,49 +48,43 @@ Consider the following example flow:
The Cursor Module is made up of 7 classes.
1. ResponseCursor
- a. This is the cursor that gets returned in the response after the filter has run.
- b. The flow defined by the Response cursor is as follows
+ * This is the cursor that gets returned in the response after the filter has run.
+ * The flow defined by the Response cursor is as follows
1. Set cursor(s) that are made up of a Integer and a CursorEntry
1. Response Cursor gets initalized
1. We go into the CursorEntry class that consists of the Cursor ( of a raw type ) and the serializer that we would use to parse the Cursor.
1. RequestCursor
- a. Contains some information on the parsedCursor
- b. This gets populated by either the User ( using a cursor that we've given them), or by the pipeline feeding the cursor into the next stage.
- c. Could be
+ * Contains some information on the parsedCursor
+ * This gets populated by either the User ( using a cursor that we've given them), or by the pipeline feeding the cursor into the next stage.
+1. AbstractCursorSerializer
+ * Used exclusivly in the read module and should probably be refactored there
+ * Is a CursorSerializer that implements the the base cursor methods.
+1. CursorSerializerUtil
+ * Defines the type of serialization we encode the cursors with. In this case Smile Jackson Serialization.
***
###Indepth Read Module Explanation
-
-1. CandidateResultsFilter
- * Is an interface
- * Extends PipelineOperation
- * Defines the types that will be requried in the filter. While not visible in the diagram the CandidateResultsFilters will consist of a <Id, CandidateResults>.
- * Primary filter that will be used for interfacing with ES (Elasticsearch)
1. Filter
* Extends generic PipelineOperation
- * Primary used to interact with Graph and Entity modules
- * Why do we use the filter in the ReadPipeline when we could also interchange the Canadiate Results filter? Is it just the type that differentiates it.
-1. AbstractSeekingFilter
+ * Interacts with anything that classifies itself as a filter.
+ * Defines output as a element T and a FilterResult.
+1. AbstractPathFilter
* This abstract filter is meant to be extended by any filter that requires a cursor and operations on that cursor.
* Extends from the AbstractPipelineOperation because a filter is a pipeline operation.
- * Is used in the Graph and Elasticsearch submodules because those both use cursors.
+ * Is used in all the submodules as a way to deal with cursors.
1. CursorSeek
- * Protected internal class that lives in AbstractSeekingFilter
+ * Protected internal class that lives in AbstractPathFilter
* Whats the deal with only seeking values on the first call? Is this not similar to pagination?
1. Collector
* Extends generic PipelineOperation
- * Primary used to interact with Entity and Elasticsearch Packages
+ * Primary used to interact with the collect module
* Used to reduce our stream of results into our final output.
-1. CollectorState
- * The state that holds a singleton collector instance and what type of collector the Collector filter should be using.
- * The collector state gets initialized with a CollectorFactory and then gets set with which collector it should use for the Collector object that it holds.
- * This is a private inner class within ReadPipelineBuilderImpl
1. Elasticsearch Module
* Contains the functions we use to actual perform filtered commands that contain elasticsearch components.
* These will typically return Canadiate Result sets that will be processed by the collector.
-1. Entity Module
+1. Collect Module
* Contains a single filter that maps id's, and the collector that processes entity id's.
1. Graph Module
* Contains the filters that are used to interact with the lower level Graph Module.
@@ -103,15 +97,37 @@ The Cursor Module is made up of 7 classes.
* Contains the builder implementation of the ReadPipelineBuilder.
* Adds on filters from FilterFactory depending on the type of action we take.
* Contains execute method when the pipeline is finished building. This pushes results as an observable back up.
+1. CollectorState
+ * The state that holds a singleton collector instance and what type of collector the Collector filter should be using.
+ * The collector state gets initialized with a CollectorFactory and then gets set with which collector it should use for the Collector object that it holds.
+ * This is a private inner class within ReadPipelineBuilderImpl
+1. Results Page
+ * Contains the encapsulation of entities as a group of responses.
+ * Holds the list of entities along with the limit of the entities we want for a response and the cursor that gets returned.
+ * Maybe refactor to collect module?
+1. EdgePath
+ * Represents the path from the intial entity to the emitted element.
+ * A list of these represnt a path through the graph to a specific element.
***
-###Indepth Entity Module Explanation
-The entity module only contains two classes. So I won't attach the uml diagram as they aren't related to each other in any way.
-1. EntityIdFilter
- * A stopgap filter that helps migrating from the service tier and its entities. Just makes a list of entities.
-2. EntityLoadCollector
- * The EntityLoadCollector loops through entity id's and then converts them to our old entity model so that they can go through the service and rest tier.
+###Indepth Collect Module Explanation
+
+
+
+
+1. EntityFilter
+ * A filter that is used when we can potentially serialize pages via cursor. ? (not entirely sure I know what that means.)
+1. IdCursorSerializer
+ * The serializer for Id's.
+1. AbstractCollector
+ * Abstract class that derives from Collector class
+ * Adds a pipelineContext for the collector to work with when looking at cursors.
+1. ResultsPageCollector
+ * Takes the entities and collects them into results so we can return them through the service and rest tier. Exists for 1.0 compatibility.
+1. ResultsPageWithCursorCollector
+ * This collector aggregates our results together using an arrayList.
+
***
###Indepth Graph Module Explanation
@@ -119,6 +135,9 @@ The entity module only contains two classes. So I won't attach the uml diagram a
1. EdgeCursorSerializer
* The serializer we use to decode and make sense of the graph cursor that gets returned.
+
+ The Main difference between ReadGraph and ReadGraph by id is that the Id won't ever bother itself with cursors because it doesn't need to worry about cursor generation. Hence the distinction but very similar patterns.
+
1. AbstractReadGraph(EdgeById)Filter
* An abstract class that defines how we should read graph edges from name(id).
1. ReadGraphConnection(ById)Filter
@@ -127,6 +146,15 @@ The entity module only contains two classes. So I won't attach the uml diagram a
* Defines how to read Collections out of the graph using names(id).
1. ReadGraphconnectionByTypeFilter
* A filter that reads graph connections by type.
+ 1. EntityIdFilter
+ * A stopgap filter that helps migrating from the service tier and its entities. Just makes a list of entities.
+ 1. EdgeLoadFilter
+ * Loads entities from a set of Ids.
+ 1. EntityVerifier
+ * Verifies that the id's in the filter results exist and can be added to the results.
+ * Functions as a collector.
+ 1. EdgeState
+ * A wrapper class that addresses the problem with skipping a value if a concurrent change has been made on the data set. In some cases we would be skipping a value. Now the cursor will always try to seek to the same position that we ended instead of the new position created by the change in data.
***
###Indepth Elasticsearch Module Explanation
@@ -135,15 +163,33 @@ The entity module only contains two classes. So I won't attach the uml diagram a
1. Impl Module
* contains all the implementations and verfiers and loaders for elasticsearch
- 2. AbstractElasticSearchFilter
- * This extends into the same pattern as the Graph Module where we make a abstract filter so we can extend it to easily accomodate Collection or Connection searching.
+
3. CandidateResultsEntityResultsCollector
* Collects the results from Elasticsearch then retrieves them from cassandra and converts them from 2.0 to 1.0 entities that are suitable for return.
4. CandidateResultsIdVerifyFilter
* Filter that verifies that the canaidate results id's are correct???? What else does this do ? isn't that what the collector does?
+
5. ElasticsearchCursorSerializer
* The serializer we use to decode and make sense of the elasticsearch cursor.
-
+ 1. Candidate
+ * Contains the candidate result and the search edge that was searched for that result.
+ * Is Canadidate really a good name for this class if it actually contains the CandidateResults? What does this class represent?
+ 1. CandidateIdFilter
+ * Takes in candidate results and outputs a stream of validated Ids.
+ * Uses the EntityCollector to map a fresh new cp entity to an old 1.0 version of the entity. Then we return those results to the upper tiers.
+ 1. EntityCollector
+ * I'm not entirely clear how the collector actually does the mapping. Seems like it just does the elasticsearch repair and checks entity versions. Then collects the entities into a result set
+ 1. AbstractElasticSearchFilter
+ * This extends into the same pattern as the Graph Module where we make a abstract filter so we can extend it to easily accomodate Collection or Connection searching.
+ 2. ElasticSearchConnectionFilter
+ * Creates the filter that will go and search for connections in elasticsearch.
+ 3. ElasticSearchCollectionFilter
+ * Creates the filter that will go and search for collections in elasticsearch.
+ 4. CandidateEntityFilter
+ * Searches on incoming Candidate entity and returns an entity instead of an Id like the CandidateIdFilter.
+ * Does a similar repair using the EntityVerifier.
+ 5. EntityVerifier
+ * Collects the entities emitted and returns them as a result set. Also verifies that the entities exist or if they need to be repaired in elasticsearch.
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/8d1618f8/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg
index 60e4b66..4fd18a0 100644
Binary files a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg and b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg differ
[03/13] incubator-usergrid git commit: [USERGRID-609 ]Added
additional um diagrams for the lower levels. Added tons of additional
information about how the lower levels of the io framework...work.
Posted by gr...@apache.org.
[USERGRID-609 ]Added additional um diagrams for the lower levels.
Added tons of additional information about how the lower levels of the io framework...work.
Project: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/commit/54871b5a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/tree/54871b5a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/diff/54871b5a
Branch: refs/heads/two-dot-o-dev
Commit: 54871b5a5647d45d25636f82a780d0d79acd6a2a
Parents: 264e67f
Author: GERey <gr...@apigee.com>
Authored: Fri May 1 15:20:40 2015 -0700
Committer: GERey <gr...@apigee.com>
Committed: Fri May 1 15:20:40 2015 -0700
----------------------------------------------------------------------
.../usergrid/corepersistence/pipeline/README.md | 95 ++++++++++++++++---
.../pipeline/read/ReadDiagram.jpg | Bin 536513 -> 704305 bytes
.../read/elasticsearch/Elasticsearchdiagram.jpg | Bin 0 -> 355271 bytes
.../pipeline/read/graph/GraphDiagram.jpg | Bin 0 -> 224755 bytes
4 files changed, 82 insertions(+), 13 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/54871b5a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
index cea6b34..0e12c45 100644
--- a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
+++ b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
@@ -1,12 +1,23 @@
#IO Framework
Based on the Pipes and Filters software pattern
-There are three main parts that are central to the framework.
-
+
+##How does this all work?
+Consider the following example flow:
+
+ ```pipeline = 1. appId -> 2. filter1 -> filter2 -> 3. collector -> 4. PipelineResult```
+
+ 1. First we start with an application. You want to do a certain set of operations on this application. The application gets passed into the pipeline. We extract the application id and pass unto the first filter
+ 1. The filter represents some action being done unto the application. This could be a "getCollection" to a "deleteIndex" action. We can have as many as we want. These operations will be done in the order that they are represented in above. The results of each of the filters will be fed into the next one for processing. Thus filter 1 will be done before filter 2 can be applied.
+ a. An important note here is that a cursor may or may not be generated here. this cursor ( if it exists ) will be stored in what is known as a Pipeline Context. The context contains all approriate state cursor information in case we need resume a query after we have run it.
+ 1. After we have applied all the filters, we take the results and feed them into a collector. The collector aggreates the results returned and pushes them into a ResultSet.
+ 1. The result set ( along with a cursor is one was generated ) is fed into the PipelineResult class. The PipelineResult is returned as a observable that can be iterated on by the calling method.
+
Above is the uml diagram of how the seperate modules are connected. We see 7 core parts
+###Pipeline Module
1. PipelineModule
a. This handles all of the guice configuration for the Pipeline Module that we are working in. If you need to wire something do it here.
@@ -37,7 +48,7 @@ Above is the uml diagram of how the seperate modules are connected. We see 7 cor
a. Contains logic that represents the cursor logic
***
-###Indepth Cursor Explanation
+###Indepth Cursor Module Explanation
The Cursor Module is made up of 7 classes.
@@ -57,7 +68,7 @@ The Cursor Module is made up of 7 classes.
***
###Indepth Read Module Explanation
- 
+ 
1. PipelineOperation
a. Top class in the Pipeline because it defines what every pipeline operation needs to have and extend. Mandates that every class contains a PipelineContext
@@ -70,24 +81,82 @@ The Cursor Module is made up of 7 classes.
1. Filter
a. Extends generic PipelineOperation
b. Primary used to interact with Graph and Entity modules
+ b. Why do we use the filter in the ReadPipeline when we could also interchange the Canadiate Results filter? Is it just the type that differentiates it.
+1. AbstractSeekingFilter
+ a. This abstract filter is meant to be extended by any filter that requires a cursor and operations on that cursor.
+ b. Extends from the AbstractPipelineOperation because a filter is a pipeline operation.
+ c. Is used in the Graph and Elasticsearch submodules because those both use cursors.
+1. CursorSeek
+ a. Protected internal class that lives in AbstractSeekingFilter
+ b. Whats the deal with only seeking values on the first call? Is this not similar to pagination?
+
1. Collector
a. Extends generic PipelineOperation
b. Primary used to interact with Entity and Elasticsearch Packages
a. The stage of our filters that is used to reduce our stream of results into our final output.
+1. CollectorState
+ a. The state that holds a singleton collector instance and what type of collector the Collector filter should be using.
+ a. The collector state gets initialized with a CollectorFactory and then gets set with which collector it should use for the Collector object that it holds.
+ b. This is a private inner class within ReadPipelineBuilderImpl
1. Elasticsearch Module
a. Contains the functions we use to actual perform filtered commands that contain elasticsearch components.
+ b. These will typically return Canadiate Result sets that will be processed by the collector.
1. Entity Module
- a. Contains
+ a. Contains a single filter that maps id's, and the collector that processes entity id's.
+1. Graph Module
+ a. Contains the filters that are used to interact with the lower level Graph Module.
+1. FilterFactory
+ a. Defines all of the possible read filters that can be added to a pipeline.
+ b. Contain comments on what each type of filter should accomplish.
+1. ReadPipelineBuilder
+ a. Contains the builder interface that will assemble the underlying pipe along with updating and keeping track of its state.
+1. ReadPipelineBuilderImpl
+ a. Contains the builder implementation of the ReadPipelineBuilder.
+ b. Adds on filters from FilterFactory depending on the type of action we take.
+ c. Contains execute method when the pipeline is finished building. This pushes results as a PipelineResult back up.
+
+***
+###Indepth Entity Module Explanation
+The entity module only contains two classes. So I won't attach the uml diagram as they aren't related to each other in any way.
+1. EntityIdFilter
+ a. A stopgap filter that helps migrating from the service tier and its entities. Just makes a list of entities.
+2. EntityLoadCollector
+ a. The EntityLoadCollector loops through entity id's and then converts them to our old entity model so that they can go through the service and rest tier.
+***
+###Indepth Graph Module Explanation
+ 
+
+ 1. EdgeCursorSerializer
+ a. The serializer we use to decode and make sense of the graph cursor that gets returned.
+ 1. AbstractReadGraph(EdgeById)Filter
+ a. An abstract class that defines how we should read graph edges from name(id).
+ 1. ReadGraphConnection(ById)Filter
+ a. Defines how to read Connections out of the graph using names(id).
+ 1. ReadGraphCollection(ById)Filter
+ a. Defines how to read Collections out of the graph using names(id).
-##How does this all work?
-Consider the following example flow:
+ 2. ReadGraphconnectionByTypeFilter
+ a. A filter that reads graph connections by type.
+***
+###Indepth Elasticsearch Module Explanation
- ```pipeline = 1. appId -> 2. filter1 -> filter2 -> 3. collector -> 4. PipelineResult```
+ 
- 1. First we start with an application. You want to do a certain set of operations on this application. The application gets passed into the pipeline. We extract the application id and pass unto the first filter
- 1. The filter represents some action being done unto the application. This could be a "getCollection" to a "deleteIndex" action. We can have as many as we want. These operations will be done in the order that they are represented in above. The results of each of the filters will be fed into the next one for processing. Thus filter 1 will be done before filter 2 can be applied.
- a. An important note here is that a cursor may or may not be generated here. this cursor ( if it exists ) will be stored in what is known as a Pipeline Context. The context contains all approriate state cursor information in case we need resume a query after we have run it.
- 1. After we have applied all the filters, we take the results and feed them into a collector. The collector aggreates the results returned and pushes them into a ResultSet.
- 1. The result set ( along with a cursor is one was generated ) is fed into the PipelineResult class. The PipelineResult is returned as a observable that can be iterated on by the calling method.
+ 1. Impl Module
+ a. contains all the implementations and verfiers and loaders for elasticsearch
+ 2. AbstractElasticSearchFilter
+ a. This extends into the same pattern as the Graph Module where we make a abstract filter so we can extend it to easily accomodate Collection or Connection searching.
+ 3. CandidateResultsEntityResultsCollector
+ a. Collects the results from Elasticsearch then retrieves them from cassandra and converts them from 2.0 to 1.0 entities that are suitable for return.
+ 4. CandidateResultsIdVerifyFilter
+ a. Filter that verifies that the canaidate results id's are correct???? What else does this do ? isn't that what the collector does?
+ 5. ElasticsearchCursorSerializer
+ a. The serializer we use to decode and make sense of the elasticsearch cursor.
+
+
+
+
+
+
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/54871b5a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg
index 9a54e9d..d83ff38 100644
Binary files a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg and b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/ReadDiagram.jpg differ
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/54871b5a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/elasticsearch/Elasticsearchdiagram.jpg
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/elasticsearch/Elasticsearchdiagram.jpg b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/elasticsearch/Elasticsearchdiagram.jpg
new file mode 100644
index 0000000..1f6a616
Binary files /dev/null and b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/elasticsearch/Elasticsearchdiagram.jpg differ
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/54871b5a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/graph/GraphDiagram.jpg
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/graph/GraphDiagram.jpg b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/graph/GraphDiagram.jpg
new file mode 100644
index 0000000..e808060
Binary files /dev/null and b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/read/graph/GraphDiagram.jpg differ
[12/13] incubator-usergrid git commit: Updates on questions in the IO
framework document.
Posted by gr...@apache.org.
Updates on questions in the IO framework document.
Project: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/commit/eae4cbd5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/tree/eae4cbd5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/diff/eae4cbd5
Branch: refs/heads/two-dot-o-dev
Commit: eae4cbd599772615dce01bf3e8fe6d9d093ddb32
Parents: 944253a
Author: GERey <gr...@apigee.com>
Authored: Tue May 5 10:52:04 2015 -0700
Committer: GERey <gr...@apigee.com>
Committed: Tue May 5 10:52:04 2015 -0700
----------------------------------------------------------------------
.../usergrid/corepersistence/pipeline/README.md | 23 ++++++--------------
1 file changed, 7 insertions(+), 16 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/eae4cbd5/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
index 5b98c31..780ada5 100644
--- a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
+++ b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
@@ -102,7 +102,6 @@ Consider the following example flow:
1. Results Page
* Contains the encapsulation of entities as a group of responses.
* Holds the list of entities along with the limit of the entities we want for a response and the cursor that gets returned.
- * Maybe refactor to collect module?
1. EdgePath
* Represents the path from the intial entity to the emitted element.
* A list of these represnt a path through the graph to a specific element.
@@ -115,7 +114,7 @@ Consider the following example flow:
1. EntityFilter
- * A filter that is used when we can potentially serialize pages via cursor. ? (not entirely sure I know what that means.)
+ * This filter is our intermediate resume filter. So if we're returning in consistent results around the end of the limit query then it is possible for the last result on the last page and first result on the resume page to have the same entity. This filter detects is that is the case and will filter the first result if the id's of the last entity of the last page and the first entity of the new page match.
1. IdCursorSerializer
* The serializer for Id's.
1. AbstractCollector
@@ -158,20 +157,12 @@ Consider the following example flow:
###Indepth Elasticsearch Module Explanation
-
- 1. Impl Module
- * contains all the implementations and verfiers and loaders for elasticsearch
-
- 3. CandidateResultsEntityResultsCollector
- * Collects the results from Elasticsearch then retrieves them from cassandra and converts them from 2.0 to 1.0 entities that are suitable for return.
- 4. CandidateResultsIdVerifyFilter
- * Filter that verifies that the canaidate results id's are correct???? What else does this do ? isn't that what the collector does?
- 5. ElasticsearchCursorSerializer
+ 1. ElasticsearchCursorSerializer
* The serializer we use to decode and make sense of the elasticsearch cursor.
1. Candidate
* Contains the candidate result and the search edge that was searched for that result.
- * Is Canadidate really a good name for this class if it actually contains the CandidateResults? What does this class represent?
+ * Since we needed the search edge along with the CandidateResults we packaged them together into a single class.
1. CandidateIdFilter
* Takes in candidate results and outputs a stream of validated Ids.
* Uses the EntityCollector to map a fresh new cp entity to an old 1.0 version of the entity. Then we return those results to the upper tiers.
@@ -179,14 +170,14 @@ Consider the following example flow:
* I'm not entirely clear how the collector actually does the mapping. Seems like it just does the elasticsearch repair and checks entity versions. Then collects the entities into a result set
1. AbstractElasticSearchFilter
* This extends into the same pattern as the Graph Module where we make a abstract filter so we can extend it to easily accomodate Collection or Connection searching.
- 2. ElasticSearchConnectionFilter
+ 1. ElasticSearchConnectionFilter
* Creates the filter that will go and search for connections in elasticsearch.
- 3. ElasticSearchCollectionFilter
+ 1. ElasticSearchCollectionFilter
* Creates the filter that will go and search for collections in elasticsearch.
- 4. CandidateEntityFilter
+ 1. CandidateEntityFilter
* Searches on incoming Candidate entity and returns an entity instead of an Id like the CandidateIdFilter.
* Does a similar repair using the EntityVerifier.
- 5. EntityVerifier
+ 1. EntityVerifier
* Collects the entities emitted and returns them as a result set. Also verifies that the entities exist or if they need to be repaired in elasticsearch.
[11/13] incubator-usergrid git commit: Added clarification for
cursors in the CursorSeek part of the documentation.
Posted by gr...@apache.org.
Added clarification for cursors in the CursorSeek part of the documentation.
Project: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/commit/944253a3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/tree/944253a3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-usergrid/diff/944253a3
Branch: refs/heads/two-dot-o-dev
Commit: 944253a3f9f8fbc90088b2d4b2f291daaf09a8e2
Parents: fc4b48a
Author: GERey <gr...@apigee.com>
Authored: Mon May 4 15:27:55 2015 -0700
Committer: GERey <gr...@apigee.com>
Committed: Mon May 4 15:27:55 2015 -0700
----------------------------------------------------------------------
.../java/org/apache/usergrid/corepersistence/pipeline/README.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-usergrid/blob/944253a3/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
----------------------------------------------------------------------
diff --git a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
index ff123b0..5b98c31 100644
--- a/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
+++ b/stack/core/src/main/java/org/apache/usergrid/corepersistence/pipeline/README.md
@@ -74,7 +74,7 @@ Consider the following example flow:
* Is used in all the submodules as a way to deal with cursors.
1. CursorSeek
* Protected internal class that lives in AbstractPathFilter
- * Whats the deal with only seeking values on the first call? Is this not similar to pagination?
+ * When resuming we use the RequestCursor to page for values. After use the cursor is no longer valid, and we only need to seek on the values that were returned from the cursor call. Any calls on the RequestCursor will be empty afterwards.
1. Collector
* Extends generic PipelineOperation
* Primary used to interact with the collect module