You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@usergrid.apache.org by snoopdave <gi...@git.apache.org> on 2014/12/10 16:52:19 UTC
[GitHub] incubator-usergrid pull request: USERGRID-254: Use event system to...
GitHub user snoopdave opened a pull request:
https://github.com/apache/incubator-usergrid/pull/120
USERGRID-254: Use event system to clean-up stale indexes
DO NOT MERGE. FOR REVIEW ONLY.
This branch implements Events https://issues.apache.org/jira/browse/USERGRID-254
When the Core module EntityManager deletes or updates Entities in the Collections module, the Query Index records for the old versions of that Entity need to be deleted. To enable this, we added an eventing system to the Collections module so that callers can hook into entity create, update and delete events.
And we added event handlers to clean up the Query Index records for those events.
See the new package in Collections module:
org.apache.usergrid.persistence.collection.event
And in Core:
org.apache.usergrid.corepersistence.events
One issue with this PR is the AbstractIteratingQueryIT, which now requires at least a 5ms write-delay or one of the tests will fail and then cause subsequent tests in other classes to fail too.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/snoopdave/incubator-usergrid two-dot-o-events
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-usergrid/pull/120.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #120
----
commit 6446cd64d086608013006c2694d0b35c61e0ebd4
Author: Shawn Feldman <sf...@apache.org>
Date: 2014-10-07T19:25:47Z
Adding EntityDeletedImpl
commit 406f32d2ee4333bc9c7fe879b659ae1809531f73
Author: Shawn Feldman <sf...@apache.org>
Date: 2014-10-07T19:26:46Z
Merge branch 'two-dot-o' of https://git-wip-us.apache.org/repos/asf/incubator-usergrid into two-dot-o-events
commit f87c33a1e0b63be24db4c95e8a10821300cf0e15
Author: Shawn Feldman <sf...@apache.org>
Date: 2014-10-07T19:34:09Z
Merge branch 'collection_multiget' of https://git-wip-us.apache.org/repos/asf/incubator-usergrid into two-dot-o-events
commit abd401447756ccb08f2b961f75465b04368aa08a
Author: Shawn Feldman <sf...@apache.org>
Date: 2014-10-07T20:10:12Z
refactor tasks
commit c0c108f78c7fcc60cd2f199843573d5841a545ee
Author: Shawn Feldman <sf...@apache.org>
Date: 2014-10-08T15:40:21Z
Merge branch 'two-dot-o' into two-dot-o-events
commit a09bdda6859d55f14be5c283339846737bcf89b5
Author: Shawn Feldman <sf...@apache.org>
Date: 2014-10-08T15:50:17Z
fix test
commit 9bd46b5a543c88b177f514587882c9e4836d7bb6
Author: Shawn Feldman <sf...@apache.org>
Date: 2014-10-08T16:34:00Z
delete current version
commit 23ff82e05aa38ca56d16016d65ef8d7316618299
Author: Shawn Feldman <sf...@apache.org>
Date: 2014-10-09T00:07:06Z
latest
commit 5a7aa5b28cdd516e516d26cd453ca2bc74ab1797
Author: Shawn Feldman <sf...@apache.org>
Date: 2014-10-09T15:22:11Z
Adding factory classes
commit d9ace9e0916e69e6b4d9d11a9c062219a5ed73c0
Author: Shawn Feldman <sf...@apache.org>
Date: 2014-10-10T15:49:13Z
merge from two-o
commit 38f659822989926a6f52aee2556906e433f704f5
Author: grey <gr...@apigee.com>
Date: 2014-10-16T20:14:21Z
Merge remote-tracking branch 'origin/two-dot-o' into two-dot-o-events
* origin/two-dot-o: (32 commits)
Fixes bug with lifecycle manager cache lazy loading
taking extra line out
fix bug if you have other devices
Fixes issue with setup not creating indexes correctly
notifications test fix
changing to uuid
fixing query to all
fix service
real tests for devices, BA
register devices
adding collections test
fix device removal, remove devices
adding device deletion
change queue name
fixing queue prefix name
change queuename
Fixed UUID lookup bug
Moved index create to be on new application entity manager
Removed obsolete tests.
Fixes problem with deleting connections
...
Conflicts:
stack/corepersistence/collection/src/main/java/org/apache/usergrid/persistence/collection/impl/EntityCollectionManagerImpl.java
commit 3551f3c6331e7edafc871727d0f1bdeeb2c7cb39
Author: Dave Johnson <dm...@apigee.com>
Date: 2014-10-20T13:46:03Z
Merge branch 'two-dot-o' into two-dot-o-events
Conflicts:
stack/corepersistence/collection/src/main/java/org/apache/usergrid/persistence/collection/impl/EntityCollectionManagerImpl.java
commit cff9ee8c537e769ef93b12a11a9342a950e96f18
Author: Dave Johnson <dm...@apigee.com>
Date: 2014-10-20T13:49:40Z
Move event handlers under corepersistence package
commit 6ca862fa56573061560d8bfdad273cf75f0bdc99
Author: Dave Johnson <dm...@apigee.com>
Date: 2014-10-20T17:10:17Z
More complete implementation of EntityDeletedImpl, see the TODO item in there for next steps.
commit fa2662788e062003d1653e05a30872978a61cd6f
Author: grey <gr...@apigee.com>
Date: 2014-10-20T18:02:46Z
Merge branch 'two-dot-o-events' of https://git-wip-us.apache.org/repos/asf/incubator-usergrid into two-dot-o-events
# By Todd Nine (30) and others
# Via amuramoto (12) and others
* 'two-dot-o-events' of https://git-wip-us.apache.org/repos/asf/incubator-usergrid: (53 commits)
More complete implementation of EntityDeletedImpl, see the TODO item in there for next steps.
Move event handlers under corepersistence package
Fixes cassandra heap
Fixes cassandra heap
Removed tokens and using vnodes
Changed health check to 10 seconds
Fixes issue with AWS props location
change access key logic
Fixed source execution order
add graphite
Fixes raid configuration to mount ephemeral drives in autoscaling instances
add comments for all devices query
removing all devices hack
Fix syntax for installation and configure to use smtp
Fixes graphite installation
Merge branch 'two-dot-o' of https://git-wip-us.apache.org/repos/asf/incubator-usergrid into two-dot-o
Fix some issues found during testing.
fixing dependency tree for httpcomponents
Added kopf and bigdesk to elasticsearch
Fixes missing env variable for graphite server
...
Conflicts:
stack/corepersistence/collection/src/main/java/org/apache/usergrid/persistence/collection/impl/EntityCollectionManagerImpl.java
commit 31e2061add13c157251b8a4d401bab6bc2652c8b
Author: grey <gr...@apigee.com>
Date: 2014-10-20T20:10:55Z
Fixed issues surrounding EntityDeletedImpl and EntityVersionDeletedImpl.
commit 4474ca91dace7c14d780986d02ea8c8ef0a90620
Author: grey <gr...@apigee.com>
Date: 2014-10-22T21:04:34Z
Fixed spelling.
Changed factory to take in collectionScope.
Fixed and added in wiring for EntityEvents.
Added list support in the module.
commit 4e2e41117190feba3029238b463025b53b024450
Author: grey <gr...@apigee.com>
Date: 2014-10-22T21:06:02Z
Merge branch 'two-dot-o' of https://git-wip-us.apache.org/repos/asf/incubator-usergrid into two-dot-o-events
# By Shawn Feldman (13) and others
# Via Shawn Feldman (4) and others
* 'two-dot-o' of https://git-wip-us.apache.org/repos/asf/incubator-usergrid: (28 commits)
checking additional threads
added error message to script
Changed configuration to disable system administrator approval
fix xml for gatling
cache queue names
change method to get queue url
license headers
Fixes missing license headers
add expected count to history
add gatling script
changing con path
portal should send debug flag
fix send to all parameter
fix send to all parameter
fixing query to send to all
Fixes bucket name
change default property value to turn off graphite
updated loadtest README
updated loadtest simulations
updated load tests
...
commit daebd252b14f57c8ce091ece5b2104310c94b188
Author: Shawn Feldman <sf...@apache.org>
Date: 2014-10-23T20:30:15Z
add queuescope factory
commit c5195a4dff1d8223b04e7d9f4ac64c3fe276917e
Author: Shawn Feldman <sf...@apache.org>
Date: 2014-10-23T21:42:35Z
Merge branch 'two-dot-o' of https://git-wip-us.apache.org/repos/asf/incubator-usergrid into two-dot-o-events
commit 9fc4f0afa1b6a2d9b5fa3fac6e17ae0aab74ed8b
Author: Shawn Feldman <sf...@apache.org>
Date: 2014-10-23T22:32:58Z
wire up deleted task
commit 5f28208f5fc018646fd4376216c172c792fc6d6d
Author: Dave Johnson <dm...@apigee.com>
Date: 2014-10-24T14:47:55Z
Implemented deleteEntity() from all index scopes methods, using ES delete by query, next up: tests.
commit 6e5b185e9d1760b474f4cf8d746ebec5a47a5768
Author: grey <gr...@apigee.com>
Date: 2014-10-24T16:23:06Z
Wired in EntityDeletedFactory and changed implementations to support using the factory.
commit 300353f02afc9156525120fcffa38a886136ec40
Author: Dave Johnson <dm...@apigee.com>
Date: 2014-10-24T21:22:19Z
Fix two of the EntityVersionCleanupTest tests, there are still 4 error/failures in that test.
commit 63bcf8de0a9e149d060fb6cafa42659acca83e78
Author: Dave Johnson <dm...@apigee.com>
Date: 2014-10-27T14:21:13Z
Fix four tests, ignore two that appear to be works in progress.
commit 57c28ba3342d53537a883fbab1851fd8d612059a
Author: Dave Johnson <dm...@apigee.com>
Date: 2014-10-27T14:29:19Z
Clean up no longer used classes and tests.
commit 05429c518cfbdf79f1ae4579909a882fdc9c50a2
Author: Dave Johnson <dm...@apigee.com>
Date: 2014-10-27T14:31:24Z
Minor formatting changes only.
commit 91314e2a00635a7b3b063a01dc79eee227226b2b
Author: Dave Johnson <dm...@apigee.com>
Date: 2014-10-27T14:35:07Z
Import cleanup only.
commit 50728fee145018e79d00c345c18c0cbc1f0e50a3
Author: Dave Johnson <dm...@apigee.com>
Date: 2014-10-27T20:00:31Z
Fix Multi-bindings setup, which requires a Set and not a List for injected multi-bindings.
commit bf347dabfff03a7f81bdb2045fd8ef61bf7a812e
Author: Dave Johnson <dm...@apigee.com>
Date: 2014-10-27T20:02:25Z
Give two Guice-created objects EntityDeleted and EntityVersionDeletedImpl, access to the Spring-created EntityManagerFactory, also: test fixes.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-usergrid pull request: USERGRID-254: Use event system to...
Posted by snoopdave <gi...@git.apache.org>.
Github user snoopdave commented on a diff in the pull request:
https://github.com/apache/incubator-usergrid/pull/120#discussion_r21743937
--- Diff: stack/core/src/main/java/org/apache/usergrid/corepersistence/CpSetup.java ---
@@ -82,11 +78,9 @@ public CpSetup( EntityManagerFactory emf, CassandraService cass ) {
}
-
-
public static Injector getInjector() {
if ( injector == null ) {
- injector = Guice.createInjector( new GuiceModule() );
+ injector = Guice.createInjector( new GuiceModule( emf ) );
--- End diff --
The reason for that is that the EntityDeletedHandler and other handers need access to the EMF and those handlers are created by Guice, so we need the EMF to be injectable -- BUT the one and only EMF instance is created by Spring.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-usergrid pull request: USERGRID-254: Use event system to...
Posted by snoopdave <gi...@git.apache.org>.
Github user snoopdave commented on a diff in the pull request:
https://github.com/apache/incubator-usergrid/pull/120#discussion_r21743340
--- Diff: stack/core/src/main/java/org/apache/usergrid/corepersistence/CpSetup.java ---
@@ -82,11 +78,9 @@ public CpSetup( EntityManagerFactory emf, CassandraService cass ) {
}
-
-
public static Injector getInjector() {
if ( injector == null ) {
- injector = Guice.createInjector( new GuiceModule() );
+ injector = Guice.createInjector( new GuiceModule( emf ) );
--- End diff --
That's not true. EMF does not depend on Guice.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-usergrid pull request: USERGRID-254: Use event system to...
Posted by tnine <gi...@git.apache.org>.
Github user tnine commented on a diff in the pull request:
https://github.com/apache/incubator-usergrid/pull/120#discussion_r21706204
--- Diff: stack/core/src/main/java/org/apache/usergrid/corepersistence/CpSetup.java ---
@@ -82,11 +78,9 @@ public CpSetup( EntityManagerFactory emf, CassandraService cass ) {
}
-
-
public static Injector getInjector() {
if ( injector == null ) {
- injector = Guice.createInjector( new GuiceModule() );
+ injector = Guice.createInjector( new GuiceModule( emf ) );
--- End diff --
Do we need to do this here? This seems like it's creating a circular dependency of the EMF into Guice, but the EMF depends on Guice.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-usergrid pull request: USERGRID-254: Use event system to...
Posted by tnine <gi...@git.apache.org>.
Github user tnine commented on the pull request:
https://github.com/apache/incubator-usergrid/pull/120#issuecomment-67041058
Looks good. Acking
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-usergrid pull request: USERGRID-254: Use event system to...
Posted by snoopdave <gi...@git.apache.org>.
Github user snoopdave commented on the pull request:
https://github.com/apache/incubator-usergrid/pull/120#issuecomment-66812656
@tnine I was able to remove the need for the EMF in the event handler classes. All they needed was the ManagerCache. So, no more circular dependency here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-usergrid pull request: USERGRID-254: Use event system to...
Posted by snoopdave <gi...@git.apache.org>.
Github user snoopdave closed the pull request at:
https://github.com/apache/incubator-usergrid/pull/120
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---