You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@allura.apache.org by "Joy P. Ku" <jo...@stanford.edu> on 2014/07/22 23:05:51 UTC
Migrating to MongoDB?
We're currently running our software forge off an old version of GForge
that uses postgres. If we switched to Allura, we'd have to migrate that
over to MongoDB. Are there any scripts or tools out there for doing so?
Secondly, we'd read that there can be challenges with using MongoDB for
certain types of queries. For instance, we tried probing MongoDB to pull
out the administrators for a certain project and had to perform several
find() steps to get that info, as opposed to issuing a single command when
working with postgres. (Specific commands below). Is there a better way
to do this?
Thanks!
Joy
(1) Find project id that contains the name "simtkproject2" from the
Collection "project". Result: ObjectId("53b39cd1574cff4112728520")
db.project.findOne({"shortname":/simtkproject2/},{"shortname":1,"_id":1})
{
"_id" : ObjectId("53b39cd1574cff4112728520"),
"shortname" : "simtkproject2"
}
(2) Find the ObjectId of the "Admin" role for that project from the
Collection "project_role". Result: ObjectId("53b39cd1574cff4112728521")
db.project_role.find({"project_id":ObjectId("53b39cd1574cff4112728520"),"n
ame":"Admin"})
{ "_id" : ObjectId("53b39cd1574cff4112728521"), "project_id" :
ObjectId("53b39cd1574cff4112728520"), "user_id" : null, "name" : "Admin",
"roles" : [ ObjectId("53b39cd1574cff4112728522") ] }
(3) Find the ObjectId of user(s) with "Admin" role ObjectId for that
project from the Collection "project_role". Result:
ObjectId("53b39b78574cff4112728492")
db.projecdb.project_role.find({"project_id":ObjectId("53b39cd1574cff411272
8520"),"roles":[ObjectId("53b39cd1574cff4112728521")]})
{ "_id" : ObjectId("53b39cd1574cff4112728526"), "project_id" :
ObjectId("53b39cd1574cff4112728520"), "user_id" :
ObjectId("53b39b78574cff4112728492"), "name" : null, "roles" : [
ObjectId("53b39cd1574cff4112728521") ] }
(4) Look up information on the "Admin" user from the Collection "user".
Result: "testsimtkuser1"
db.user.find({"_id": ObjectId("53b39b78574cff4112728492")})
{ "_id" : ObjectId("53b39b78574cff4112728492"), "username" :
"testsimtkuser1", "socialnetworks" : [ ], "localization" : { "city" :
null, "country" : null }, "last_password_updated" :
ISODate("2014-07-02T05:41:12.445Z"), "sex" : "Unknown", "disabled" :
false, "sent_user_message_times" : [ ], "webpages" : [ ], "timezone" :
null, "password" :
"sha256Lg\u00131\u0016hr|YOVDOh9bMkplxs4zg/xRdPOtsrYM6Qh1BQEoX0+xVPQ=",
"availability" : [ ], "skypeaccount" : null, "tool_preferences" : { },
"display_name" : "Test Simtk User 1", "preferences" : { "email_address" :
null, "email_format" : null, "disable_user_messages" : null,
"results_per_page" : 25 }, "skills" : [ ], "birthdate" : null, "tool_data"
: { }, "inactiveperiod" : [ ], "telnumbers" : [ ], "stats_id" :
ObjectId("53b39b78574cff4112728493"), "email_addresses" : [ ] }
---
Joy P. Ku, PhD
Director, <http://simbios.stanford.edu/> Simbios
Director of Communications & Training, <http://opensim.stanford.edu/>
NCSRR
Stanford University
(w) 650.736.8434, (f) 650.723.7461
Email: <ma...@stanford.edu> joyku@stanford.edu
Re: Migrating to MongoDB?
Posted by Dave Brondsema <da...@brondsema.net>.
On 7/22/14 5:05 PM, Joy P. Ku wrote:
> We're currently running our software forge off an old version of GForge
> that uses postgres. If we switched to Allura, we'd have to migrate that
> over to MongoDB. Are there any scripts or tools out there for doing so?
>
We at SourceForge migrated thousands of projects from our old system, which has
some common ancestry with GForge, so our custom migration script might be able
to be modified for re-use with GForge. I don't know how much of a difference
there is between GForge and the old SourceForge platform -- it probably would be
some amount of work to adapt the script.
If/when you know you want to use Allura, I can work on removing the
SourceForge-specific parts of our migration script and open sourcing it. It
would still need further development from there to update for any GForge needs.
>
> Secondly, we'd read that there can be challenges with using MongoDB for
> certain types of queries. For instance, we tried probing MongoDB to pull
> out the administrators for a certain project and had to perform several
> find() steps to get that info, as opposed to issuing a single command when
> working with postgres. (Specific commands below). Is there a better way
> to do this?
Yes MongoDB isn't always as suited for adhoc querying & reporting since you
can't just join tables together. And the permissions system in Allura is quite
flexible which also means its a bit complex for those types of reports. I would
generally recommend to work with the Allura code to get the data I need, either
through existing Allura logic, Allura APIs, or a custom python script that uses
Allura models & functions. Rather than querying mongo directly (although
sometimes I've done that too). A few out-of-the-box options for getting admin
info are:
* web page: /p/simtkproject2/_members/
* json API: /rest/p/simtkproject2/ (see developers field. A good enhancement
would be to denote admins separately)
* list for all projects in the 'p' neighborhood: /p/_admin/stats/adminlist
Or a little python code using the Allura models is as easy as:
from allura import model as M
print [u.username for u in M.Project.query.get(shortname='simtkproject2').admins()]
To run code like that, go into the Allura directory and run 'paster shell
development.ini' and then enter the code. Or save it as a file and run 'paster
script development.ini myscript.py'
>
>
>
> Thanks!
>
> Joy
>
>
>
> (1) Find project id that contains the name "simtkproject2" from the
> Collection "project". Result: ObjectId("53b39cd1574cff4112728520")
>
>
>
> db.project.findOne({"shortname":/simtkproject2/},{"shortname":1,"_id":1})
> {
> "_id" : ObjectId("53b39cd1574cff4112728520"),
> "shortname" : "simtkproject2"
> }
>
>
>
> (2) Find the ObjectId of the "Admin" role for that project from the
> Collection "project_role". Result: ObjectId("53b39cd1574cff4112728521")
>
>
> db.project_role.find({"project_id":ObjectId("53b39cd1574cff4112728520"),"n
> ame":"Admin"})
> { "_id" : ObjectId("53b39cd1574cff4112728521"), "project_id" :
> ObjectId("53b39cd1574cff4112728520"), "user_id" : null, "name" : "Admin",
> "roles" : [ ObjectId("53b39cd1574cff4112728522") ] }
>
>
>
>
>
> (3) Find the ObjectId of user(s) with "Admin" role ObjectId for that
> project from the Collection "project_role". Result:
> ObjectId("53b39b78574cff4112728492")
>
>
>
> db.projecdb.project_role.find({"project_id":ObjectId("53b39cd1574cff411272
> 8520"),"roles":[ObjectId("53b39cd1574cff4112728521")]})
> { "_id" : ObjectId("53b39cd1574cff4112728526"), "project_id" :
> ObjectId("53b39cd1574cff4112728520"), "user_id" :
> ObjectId("53b39b78574cff4112728492"), "name" : null, "roles" : [
> ObjectId("53b39cd1574cff4112728521") ] }
>
>
>
> (4) Look up information on the "Admin" user from the Collection "user".
> Result: "testsimtkuser1"
>
>
>
> db.user.find({"_id": ObjectId("53b39b78574cff4112728492")})
> { "_id" : ObjectId("53b39b78574cff4112728492"), "username" :
> "testsimtkuser1", "socialnetworks" : [ ], "localization" : { "city" :
> null, "country" : null }, "last_password_updated" :
> ISODate("2014-07-02T05:41:12.445Z"), "sex" : "Unknown", "disabled" :
> false, "sent_user_message_times" : [ ], "webpages" : [ ], "timezone" :
> null, "password" :
> "sha256Lg\u00131\u0016hr|YOVDOh9bMkplxs4zg/xRdPOtsrYM6Qh1BQEoX0+xVPQ=",
> "availability" : [ ], "skypeaccount" : null, "tool_preferences" : { },
> "display_name" : "Test Simtk User 1", "preferences" : { "email_address" :
> null, "email_format" : null, "disable_user_messages" : null,
> "results_per_page" : 25 }, "skills" : [ ], "birthdate" : null, "tool_data"
> : { }, "inactiveperiod" : [ ], "telnumbers" : [ ], "stats_id" :
> ObjectId("53b39b78574cff4112728493"), "email_addresses" : [ ] }
>
>
>
>
>
> ---
>
> Joy P. Ku, PhD
>
> Director, <http://simbios.stanford.edu/> Simbios
>
> Director of Communications & Training, <http://opensim.stanford.edu/>
> NCSRR
>
> Stanford University
>
>
>
> (w) 650.736.8434, (f) 650.723.7461
>
> Email: <ma...@stanford.edu> joyku@stanford.edu
>
>
>
>
--
Dave Brondsema : dave@brondsema.net
http://www.brondsema.net : personal
http://www.splike.com : programming
<><