You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@usergrid.apache.org by GERey <gi...@git.apache.org> on 2015/05/26 20:15:42 UTC
[GitHub] incubator-usergrid pull request: Usergrid 608
GitHub user GERey opened a pull request:
https://github.com/apache/incubator-usergrid/pull/256
Usergrid 608
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/incubator-usergrid USERGRID-608
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-usergrid/pull/256.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #256
----
commit b52ac2f47d918a3390fd2bc65fbd543c2e78067a
Author: GERey <gr...@apigee.com>
Date: 2015-05-13T19:26:58Z
Added tests for usergrid 607 and other issues were fixed in previous releases.
Added new method that gets all edge documents out of elastic search and returns them.
Added comment to IndexEdge about the differentiation between IndexEdge and SearchEdge.
Removed the timeout in EsIndexBufferConsumerImpl.
commit 79aea6ab9716966235b01b2aa4969862003e557a
Author: GERey <gr...@apigee.com>
Date: 2015-05-13T19:27:50Z
Fixed naming mismatch in CandidateResults.
commit 50f8a56fdfa558f9204c348bec59c938347c7485
Author: GERey <gr...@apigee.com>
Date: 2015-05-14T20:59:30Z
[USERGRID-608] Copied method indexEntity to deleteEntityIndexes and made it a deindex call that requires version.
Added version to method signature.
Created Observable to get first marked entry.
commit ed317b295d96c150693d441ba7d306d0ef3ff40b
Author: GERey <gr...@apigee.com>
Date: 2015-05-15T19:23:28Z
[USERGRID-608] Added initial version of delete flow
commit 11648ab1c84be349a45020e0cc4a9cafd60d005b
Author: GERey <gr...@apigee.com>
Date: 2015-05-18T22:08:15Z
Removed inner observable and converted it into regular code that functions. Will convert back in next push.
commit cf823c47d3fc2bb92198e7d1e65ecc4be96ccfcc
Author: GERey <gr...@apigee.com>
Date: 2015-05-22T16:53:44Z
[USERGRID-677] Added a new SearchRequestBuilderStrategy that allows you to make term post filters easily without the cruft of the regular SearchRequestBuilderStrategy.
Added a version of the scrolling api and an additional parser that will let us aggregate the from the constant queries and return a single CandidateResults set.
Added a test that currently just checks to make sure the new scrolling search still functions as a regular search ( it does! Yay! ).
commit 630cb4a87e0943d03b9d515166b1529b4cd649d6
Author: GERey <gr...@apigee.com>
Date: 2015-05-22T18:21:09Z
[USERGRID-677] Added revised test for the scrolling query.
Added fixes to method so that it properly logs aggregation and searching for versions.
commit 00aa293d0c161a056c8f8e10d3f5880eeac2df5b
Author: GERey <gr...@apigee.com>
Date: 2015-05-22T19:49:01Z
[USERGRID-608] Fixed signature to make method self documenting
Added fix for UniqueCleanup.java where it was comparing things to the wrong standard.
Added fixes for IndexServiceImpl so that it can properly follow the delete async process it is called from.
commit aa8b20d9ae33cddff289aa7652ee7e4fe88c9b0e
Author: GERey <gr...@apigee.com>
Date: 2015-05-22T21:37:56Z
[USERGRID-607] Refactored getAllEdgeDocuments to use scroll api and to do a simplified edge search.
Refactored deleteIndexEdge to take advantage of the scroll api.
commit 3f8899dfa10a22a4b67f0099d86c19ed22ba6c1e
Author: GERey <gr...@apigee.com>
Date: 2015-05-26T17:55:31Z
Merge branch 'two-dot-o-dev' of https://git-wip-us.apache.org/repos/asf/incubator-usergrid into USERGRID-608
# By Todd Nine (7) and others
# Via Shawn Feldman (2) and others
* 'two-dot-o-dev' of https://git-wip-us.apache.org/repos/asf/incubator-usergrid:
[USERGRID-642] Remove password properties from user objects in portal
Addresses some of George's comments
Fixes exception on missing keyspace during bootstrap
Updates REST api calls. POST is used to create a re-index job, PUT is used to resume.
test fixes
fix serviceinvocation it
fix serviceinvocation it
Change usergrid SDK logging to be set via config.
Fixes bugs and cleans up tests
Finishes changes before tests
fix connections
rename connecting to source and connected to target
rename vars
Refactored observable methods to correct name
Update of Apis and tests
[USERGRID-572] - ignore sensitive params from QP in response params
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-usergrid pull request: Usergrid 608
Posted by shawnfeldman <gi...@git.apache.org>.
Github user shawnfeldman commented on a diff in the pull request:
https://github.com/apache/incubator-usergrid/pull/256#discussion_r31177680
--- Diff: stack/corepersistence/queryindex/src/main/java/org/apache/usergrid/persistence/index/impl/EsApplicationEntityIndexImpl.java ---
@@ -173,6 +184,159 @@ public CandidateResults search( final SearchEdge searchEdge, final SearchTypes s
}
+ @Override
+ public CandidateResults getAllEdgeDocuments( final IndexEdge edge, final Id entityId ) {
+ /**
+ * Take a list of IndexEdge, with an entityId
+ and query Es directly for matches
+
+ */
+ IndexValidationUtils.validateSearchEdge( edge );
+ Preconditions.checkNotNull( entityId, "entityId cannot be null" );
+
+ SearchResponse searchResponse;
+
+ List<CandidateResult> candidates = new ArrayList<>();
+
+ final ParsedQuery parsedQuery = ParsedQueryBuilder.build( "select *" );
+
+ final SearchRequestBuilder srb = searchRequestBuilderStrategyV2.getBuilder();
+
+ //I can't just search on the entity Id.
+
+ FilterBuilder entityEdgeFilter = FilterBuilders.termFilter( IndexingUtils.EDGE_NODE_ID_FIELDNAME,
+ IndexingUtils.idString( edge.getNodeId() ));
+
+ srb.setPostFilter(entityEdgeFilter);
+
+ if ( logger.isDebugEnabled() ) {
+ logger.debug( "Searching for marked versions in index (read alias): {}\n nodeId: {},\n query: {} ",
+ this.alias.getReadAlias(),entityId, srb );
+ }
+
+ try {
+ //Added For Graphite Metrics
+ Timer.Context timeSearch = searchTimer.time();
+
+ //set the timeout on the scroll cursor to 6 seconds and set the number of values returned per shard to 100.
+ //The settings for the scroll aren't tested and so we aren't sure what vlaues would be best in a production enviroment
+ //TODO: review this and make them not magic numbers when acking this PR.
+ searchResponse = srb.setScroll( new TimeValue( 6000 ) ).setSize( 100 ).execute().actionGet();
+
+
+ while(true){
--- End diff --
make this into observable
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-usergrid pull request: Usergrid 608
Posted by GERey <gi...@git.apache.org>.
Github user GERey commented on a diff in the pull request:
https://github.com/apache/incubator-usergrid/pull/256#discussion_r31070692
--- Diff: stack/corepersistence/queryindex/src/main/java/org/apache/usergrid/persistence/index/impl/EsApplicationEntityIndexImpl.java ---
@@ -173,6 +184,161 @@ public CandidateResults search( final SearchEdge searchEdge, final SearchTypes s
}
+ @Override
+ public CandidateResults getAllEdgeDocuments( final IndexEdge edge, final Id entityId ) {
+ /**
+ * Take a list of IndexEdge, with an entityId
+ and query Es directly for matches
+
+ */
+ IndexValidationUtils.validateSearchEdge( edge );
+ Preconditions.checkNotNull( entityId, "entityId cannot be null" );
+
+ SearchResponse searchResponse;
+
+ List<CandidateResult> candidates = new ArrayList<>();
+
+ final ParsedQuery parsedQuery = ParsedQueryBuilder.build( "select *" );
+
+ final SearchRequestBuilder srb = searchRequestBuilderStrategyV2.getBuilder();
+
+ //I can't just search on the entity Id.
+
+ FilterBuilder entityEdgeFilter = FilterBuilders.termFilter( IndexingUtils.EDGE_NODE_ID_FIELDNAME,
+ IndexingUtils.idString( edge.getNodeId() ));
+
+ srb.setPostFilter(entityEdgeFilter);
+
+ if ( logger.isDebugEnabled() ) {
+ logger.debug( "Searching for marked versions in index (read alias): {}\n nodeId: {},\n query: {} ",
+ this.alias.getReadAlias(),entityId, srb );
+ }
+
+ try {
+ //Added For Graphite Metrics
+ Timer.Context timeSearch = searchTimer.time();
+
+ //set the timeout on the scroll cursor to 6 seconds and set the number of values returned per shard to 100.
+ //The settings for the scroll aren't tested and so we aren't sure what vlaues would be best in a production enviroment
+ //TODO: review this and make them not magic numbers when acking this PR.
+ searchResponse = srb.setScroll( new TimeValue( 6000 ) ).setSize( 100 ).execute().actionGet();
--- End diff --
This some comments on. Not sure what values we want here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-usergrid pull request: Usergrid 608
Posted by GERey <gi...@git.apache.org>.
Github user GERey commented on a diff in the pull request:
https://github.com/apache/incubator-usergrid/pull/256#discussion_r31176723
--- Diff: stack/core/src/main/java/org/apache/usergrid/corepersistence/asyncevents/EventBuilderImpl.java ---
@@ -125,12 +128,30 @@ public EntityDeleteResults queueEntityDelete( final ApplicationScope application
final GraphManager gm = graphManagerFactory.createEdgeManager( applicationScope );
+ //needs get versions here.
+
+
+ //TODO: change this to be an observable
+ //so we get these versions and loop through them until we find the MvccLogEntry that is marked as delete.
+ //TODO: evauluate this to possibly be an observable to pass to the nextmethod.
+ MvccLogEntry mostRecentlyMarked = ecm.getVersions( entityId ).toBlocking()
+ .firstOrDefault( null,
--- End diff --
Should be combined with the observable below. There isn't any reasons we should go through entity versions twice.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-usergrid pull request: Usergrid 608
Posted by shawnfeldman <gi...@git.apache.org>.
Github user shawnfeldman commented on a diff in the pull request:
https://github.com/apache/incubator-usergrid/pull/256#discussion_r31177697
--- Diff: stack/corepersistence/queryindex/src/main/java/org/apache/usergrid/persistence/index/impl/EsApplicationEntityIndexImpl.java ---
@@ -173,6 +184,159 @@ public CandidateResults search( final SearchEdge searchEdge, final SearchTypes s
}
+ @Override
+ public CandidateResults getAllEdgeDocuments( final IndexEdge edge, final Id entityId ) {
+ /**
+ * Take a list of IndexEdge, with an entityId
+ and query Es directly for matches
+
+ */
+ IndexValidationUtils.validateSearchEdge( edge );
+ Preconditions.checkNotNull( entityId, "entityId cannot be null" );
+
+ SearchResponse searchResponse;
+
+ List<CandidateResult> candidates = new ArrayList<>();
+
+ final ParsedQuery parsedQuery = ParsedQueryBuilder.build( "select *" );
+
+ final SearchRequestBuilder srb = searchRequestBuilderStrategyV2.getBuilder();
+
+ //I can't just search on the entity Id.
+
+ FilterBuilder entityEdgeFilter = FilterBuilders.termFilter( IndexingUtils.EDGE_NODE_ID_FIELDNAME,
+ IndexingUtils.idString( edge.getNodeId() ));
+
+ srb.setPostFilter(entityEdgeFilter);
+
+ if ( logger.isDebugEnabled() ) {
+ logger.debug( "Searching for marked versions in index (read alias): {}\n nodeId: {},\n query: {} ",
+ this.alias.getReadAlias(),entityId, srb );
+ }
+
+ try {
+ //Added For Graphite Metrics
+ Timer.Context timeSearch = searchTimer.time();
+
+ //set the timeout on the scroll cursor to 6 seconds and set the number of values returned per shard to 100.
+ //The settings for the scroll aren't tested and so we aren't sure what vlaues would be best in a production enviroment
+ //TODO: review this and make them not magic numbers when acking this PR.
+ searchResponse = srb.setScroll( new TimeValue( 6000 ) ).setSize( 100 ).execute().actionGet();
+
+
+ while(true){
--- End diff --
needs to be like IndexBufferConsumer
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-usergrid pull request: Usergrid 608
Posted by shawnfeldman <gi...@git.apache.org>.
Github user shawnfeldman commented on a diff in the pull request:
https://github.com/apache/incubator-usergrid/pull/256#discussion_r31176980
--- Diff: stack/core/src/main/java/org/apache/usergrid/corepersistence/asyncevents/EventBuilderImpl.java ---
@@ -125,12 +128,30 @@ public EntityDeleteResults queueEntityDelete( final ApplicationScope application
final GraphManager gm = graphManagerFactory.createEdgeManager( applicationScope );
+ //needs get versions here.
+
+
+ //TODO: change this to be an observable
+ //so we get these versions and loop through them until we find the MvccLogEntry that is marked as delete.
+ //TODO: evauluate this to possibly be an observable to pass to the nextmethod.
+ MvccLogEntry mostRecentlyMarked = ecm.getVersions( entityId ).toBlocking()
+ .firstOrDefault( null,
+ mvccLogEntry -> mvccLogEntry.getState() == MvccLogEntry.State.DELETED );
+
+ //If there is nothing marked then we shouldn't return any results.
+ //TODO: evaluate if we want to return null or return empty observable when we don't have any results marked as deleted.
+ if(mostRecentlyMarked == null)
+ return null;
//observable of index operation messages
+ //this method will need the most recent version.
+ //When we go to compact the graph make sure you turn on the debugging mode for the deleted nodes so
+ //we can verify that we mark them. That said that part seems kinda done. as we also delete the mvcc buffers.
final Observable<IndexOperationMessage> edgeObservable =
--- End diff --
probably needs to be flatmap
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-usergrid pull request: Usergrid 608
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-usergrid/pull/256
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-usergrid pull request: Usergrid 608
Posted by GERey <gi...@git.apache.org>.
Github user GERey commented on a diff in the pull request:
https://github.com/apache/incubator-usergrid/pull/256#discussion_r31070708
--- Diff: stack/corepersistence/queryindex/src/main/java/org/apache/usergrid/persistence/index/impl/EsApplicationEntityIndexImpl.java ---
@@ -173,6 +184,161 @@ public CandidateResults search( final SearchEdge searchEdge, final SearchTypes s
}
+ @Override
+ public CandidateResults getAllEdgeDocuments( final IndexEdge edge, final Id entityId ) {
+ /**
+ * Take a list of IndexEdge, with an entityId
+ and query Es directly for matches
+
+ */
+ IndexValidationUtils.validateSearchEdge( edge );
+ Preconditions.checkNotNull( entityId, "entityId cannot be null" );
+
+ SearchResponse searchResponse;
+
+ List<CandidateResult> candidates = new ArrayList<>();
+
+ final ParsedQuery parsedQuery = ParsedQueryBuilder.build( "select *" );
+
+ final SearchRequestBuilder srb = searchRequestBuilderStrategyV2.getBuilder();
+
+ //I can't just search on the entity Id.
+
+ FilterBuilder entityEdgeFilter = FilterBuilders.termFilter( IndexingUtils.EDGE_NODE_ID_FIELDNAME,
+ IndexingUtils.idString( edge.getNodeId() ));
+
+ srb.setPostFilter(entityEdgeFilter);
+
+ if ( logger.isDebugEnabled() ) {
+ logger.debug( "Searching for marked versions in index (read alias): {}\n nodeId: {},\n query: {} ",
+ this.alias.getReadAlias(),entityId, srb );
+ }
+
+ try {
+ //Added For Graphite Metrics
+ Timer.Context timeSearch = searchTimer.time();
+
+ //set the timeout on the scroll cursor to 6 seconds and set the number of values returned per shard to 100.
+ //The settings for the scroll aren't tested and so we aren't sure what vlaues would be best in a production enviroment
+ //TODO: review this and make them not magic numbers when acking this PR.
+ searchResponse = srb.setScroll( new TimeValue( 6000 ) ).setSize( 100 ).execute().actionGet();
+
+
+ while(true){
+ //add search result hits to some sort of running tally of hits.
+ candidates = aggregateScrollResults( candidates, searchResponse );
+
+ SearchScrollRequestBuilder ssrb = searchRequestBuilderStrategyV2
+ .getScrollBuilder( searchResponse.getScrollId() )
+ .setScroll( new TimeValue( 6000 ) );
--- End diff --
same as above
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---