You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Zitan Broth <zi...@mediasculpt.net> on 2003/04/28 06:36:13 UTC
Blue Sky: Scripting Language / SQL database ?
Greetings All,
Been having a look at Subversion and it is an excellent open source project alright, far more flexible than CVS, and particularly awesome to support DAV and DeltaV. And my interest in Subversion comes *especially* from the DeltaV support. To be able integrate Subversion with into a web service-based architecture it would require some extra functionality mentioned in the "Future" section of the Subversion Design doc. My serving side scripting language of choice is PHP but I'm sure you could achieve a bridge to PHP from Perl also.
With a scripting language glue layer in between the Subversion "business logic" (eg how to implement DeltaV, DAV) and the current "persistance" layer (currently Berkely DB) it would be possible to take advantage of the logic that handles DAV and DeltaV but extend the logic and change how things are persisted in powerful ways (ie write another backend), say for example persisting part of the data in an SQL database (metadata, text docs, collections even and the other part (larger media files) in a file repository, and so on. It would also mean that organisations would likely extend the mod_dav backend via svn also, so that their custom backends could support DeltaV where required. A language like PHP supports almost every SQL database under the sun as well as being able to call Perl, Pthyon, Java and even .NET allowing integration into any platform or backend desired.
I'm installing a windows version locally of Subversion now and will look over the API documentation (svn_fs.h), however I'm (not yet) a C developer. So I'm interested to know what the interest in this project extension? How hard would it be to implement? Is work currently underway? I guess if the architecture is well abstracted (and Subversion looks to be an excellently managed project ;)) it would be a matter of editing the current API between the logic and the persistance to call a series of classes which are currently blank but would be filled in by the developers as required. Have I got the "right end of the stick" here or is the work far more complicated (eg is part of the logic embedded with the persistance layer interface)?
Any comments a greatly appreciated.
Awesome,
Z.
Re: Blue Sky: Scripting Language / SQL database ?
Posted by Zitan Broth <zi...@mediasculpt.net>.
It is almost like the hooks, triggered by repository events, were created for this very purpose. I didn't expect it to be that easy ;-)
Also looks like the SQL DB backend is something that is being worked on, which is also true.
Well looks like svn is good to go ... Z.
----- Original Message -----
From: Zitan Broth
To: dev@subversion.tigris.org
Sent: Monday, April 28, 2003 3:28 AM
Subject: Re: Blue Sky: Scripting Language / SQL database ?
Greetings Again :-)
I've had a slight rethink .... what I am really wanting is to leave subversion entirely as it is and interact with it in two ways.
Firstly to communicate with SVN via its API (1) to do things like check things out and in etc... which I can do by simply using the command line interface or with SWIG allowing intergration with whatever applications you want.
The other thing I'm considering is keeping a syncronised SQL database (2) going with Subversion which could be used to do things like allocate custom permissions to resources, more complicated searching and the storage of custom "resource types". This is very similar to the mod_dav_dbms architecture that allows searching (DASL) to DAV resources. All that is required is that when subversion commits a change, (eg a document's locations is changed) that it spawns a call to a wrapper class that logs this to the database.
I know (1) is possible what about (2)? I will continue to read the documentation on the subject and Subversion in general :-)
Awesome, Z.
----- Original Message -----
From: Zitan Broth
To: dev@subversion.tigris.org
Sent: Sunday, April 27, 2003 11:36 PM
Subject: Blue Sky: Scripting Language / SQL database ?
Greetings All,
Been having a look at Subversion and it is an excellent open source project alright, far more flexible than CVS, and particularly awesome to support DAV and DeltaV. And my interest in Subversion comes *especially* from the DeltaV support. To be able integrate Subversion with into a web service-based architecture it would require some extra functionality mentioned in the "Future" section of the Subversion Design doc. My serving side scripting language of choice is PHP but I'm sure you could achieve a bridge to PHP from Perl also.
With a scripting language glue layer in between the Subversion "business logic" (eg how to implement DeltaV, DAV) and the current "persistance" layer (currently Berkely DB) it would be possible to take advantage of the logic that handles DAV and DeltaV but extend the logic and change how things are persisted in powerful ways (ie write another backend), say for example persisting part of the data in an SQL database (metadata, text docs, collections even and the other part (larger media files) in a file repository, and so on. It would also mean that organisations would likely extend the mod_dav backend via svn also, so that their custom backends could support DeltaV where required. A language like PHP supports almost every SQL database under the sun as well as being able to call Perl, Pthyon, Java and even .NET allowing integration into any platform or backend desired.
I'm installing a windows version locally of Subversion now and will look over the API documentation (svn_fs.h), however I'm (not yet) a C developer. So I'm interested to know what the interest in this project extension? How hard would it be to implement? Is work currently underway? I guess if the architecture is well abstracted (and Subversion looks to be an excellently managed project ;)) it would be a matter of editing the current API between the logic and the persistance to call a series of classes which are currently blank but would be filled in by the developers as required. Have I got the "right end of the stick" here or is the work far more complicated (eg is part of the logic embedded with the persistance layer interface)?
Any comments a greatly appreciated.
Awesome,
Z.
Re: Blue Sky: Scripting Language / SQL database ?
Posted by "Glenn A. Thompson" <gt...@cdr.net>.
Hey,
Zitan Broth wrote:
>Btw, not sure how svn currently handles it, but when we were looking at
>virtual directories we decided only to store text files *in* the database,
>and leave everything else on the server. The DB effectively *knows* where
>everything is (holding all the live props), but the app layer puts the dead
>props resource requests together. Or perhaps you were thinking of just
>using an sql db for the repository library .... I can wait for the doc of
>course :-)
>
>
Well, the doc that is coming has nothing to do with SQL other than I
need to make the pluggable changes first so I can co-exist with the BDB
FS. The idea is that other folks can have a go at doing it *their* way.
I can tell you I'm not a huge fan of storing *anything* outside the DB.
Performance issues associated with raw and/or LOBs can usually be dealt
with in other ways.
I did write a quick SQL design sorta thingy a while back:
http://www.cdrguys.com/subversion/sql_fs_docs/svn_fs_sql.htm. It was
based on the http://www.cdrguys.com/subversion/fs_refactoring/index.htm
work. The refactoring document is what I'm re-documenting. The SQL
document will be improved later.
If you haven't read "subversion/libsvn_fs/structure", you need to. It's
required reading for anyone interested in the FS. Also read the
comments in "subversion/include/svn_fs.h". For starters;-) It should
give you an idea of what has to be implemented. The first SQL
implementation will leverage as much of the existing logic as possible.
gat
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Blue Sky: Scripting Language / SQL database ?
Posted by Zitan Broth <zi...@mediasculpt.net>.
Btw, not sure how svn currently handles it, but when we were looking at
virtual directories we decided only to store text files *in* the database,
and leave everything else on the server. The DB effectively *knows* where
everything is (holding all the live props), but the app layer puts the dead
props resource requests together. Or perhaps you were thinking of just
using an sql db for the repository library .... I can wait for the doc of
course :-)
Z.
----- Original Message -----
From: "Zitan Broth" <zi...@mediasculpt.net>
To: <gt...@cdr.net>
Cc: "Subversion dev" <de...@subversion.tigris.org>
Sent: Monday, April 28, 2003 8:43 PM
Subject: Re: Blue Sky: Scripting Language / SQL database ?
> Greetings Glenn,
>
> Thanks for your message :-)
>
> > Before a SQL effort can be cranked out, the FS backend needs to be made
> > "pluggable". Watch for threads with "pluggable-db" in the subject.
> Will do ;-)
>
> > The SQL work I've done (I can't speak for anyone else) is focused on
> > providing a SQL DB backend as an alternative to BDB. The FS API will
> > not change early on. I'm sure any query extensions to the FS API will
> > require much debate. Also don't assume that once we have a SQL backend
> > that it will be as simple as whipping up few fancy queries behind the
> > scenes. The Subversion FS schema will make for some tricky queries. I
> > plan to *grow* into the more sophisticated query capabilities over
time:-)
> >
> > I got behind do to illness but I'm back on track now. Watch for a new
> > pluggable-db design document soon.
>
> Sounds primo, and I will enjoy reading it. I wasn't sure how much logic
> would be tied up with BDB, but I certainly wasn't under the illusion it
was
> a simple project ;-) We had much more simple problem to solve with a
> virtual FS (very little of that time dimension) and the queries to that
look
> to be complicated enough .... however in the meantime I am happy to
call
> the FS API from the command line/swigs to make things happen from our
> system, and to use the repository hooks to fire up PHP scripts which load
> the data into another SQL database I can use for extension.
>
> I will certainly look forward to having a pluggable DB and it is
reassuring
> to know that someone is working on it. However I didn't want to let that
> stop me having a play with svn :-)
>
> Awesome, Z.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Blue Sky: Scripting Language / SQL database ?
Posted by Zitan Broth <zi...@mediasculpt.net>.
Greetings Glenn,
Thanks for your message :-)
> Before a SQL effort can be cranked out, the FS backend needs to be made
> "pluggable". Watch for threads with "pluggable-db" in the subject.
Will do ;-)
> The SQL work I've done (I can't speak for anyone else) is focused on
> providing a SQL DB backend as an alternative to BDB. The FS API will
> not change early on. I'm sure any query extensions to the FS API will
> require much debate. Also don't assume that once we have a SQL backend
> that it will be as simple as whipping up few fancy queries behind the
> scenes. The Subversion FS schema will make for some tricky queries. I
> plan to *grow* into the more sophisticated query capabilities over time:-)
>
> I got behind do to illness but I'm back on track now. Watch for a new
> pluggable-db design document soon.
Sounds primo, and I will enjoy reading it. I wasn't sure how much logic
would be tied up with BDB, but I certainly wasn't under the illusion it was
a simple project ;-) We had much more simple problem to solve with a
virtual FS (very little of that time dimension) and the queries to that look
to be complicated enough .... however in the meantime I am happy to call
the FS API from the command line/swigs to make things happen from our
system, and to use the repository hooks to fire up PHP scripts which load
the data into another SQL database I can use for extension.
I will certainly look forward to having a pluggable DB and it is reassuring
to know that someone is working on it. However I didn't want to let that
stop me having a play with svn :-)
Awesome, Z.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Blue Sky: Scripting Language / SQL database ?
Posted by "Glenn A. Thompson" <gt...@cdr.net>.
Hey,
> The other thing I'm considering is keeping a syncronised SQL database
> (2) going with Subversion which could be used to do things like
> allocate custom permissions to resources, more complicated searching
> and the storage of custom "resource types". This is very similar to
> the mod_dav_dbms architecture that allows searching (DASL) to DAV
> resources. All that is required is that when subversion commits a
> change, (eg a document's locations is changed) that it spawns a call
> to a wrapper class that logs this to the database.
>
Before a SQL effort can be cranked out, the FS backend needs to be made
"pluggable". Watch for threads with "pluggable-db" in the subject.
The SQL work I've done (I can't speak for anyone else) is focused on
providing a SQL DB backend as an alternative to BDB. The FS API will
not change early on. I'm sure any query extensions to the FS API will
require much debate. Also don't assume that once we have a SQL backend
that it will be as simple as whipping up few fancy queries behind the
scenes. The Subversion FS schema will make for some tricky queries. I
plan to *grow* into the more sophisticated query capabilities over time:-)
I got behind do to illness but I'm back on track now. Watch for a new
pluggable-db design document soon.
gat
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Blue Sky: Scripting Language / SQL database ?
Posted by Zitan Broth <zi...@mediasculpt.net>.
Greetings Again :-)
I've had a slight rethink .... what I am really wanting is to leave subversion entirely as it is and interact with it in two ways.
Firstly to communicate with SVN via its API (1) to do things like check things out and in etc... which I can do by simply using the command line interface or with SWIG allowing intergration with whatever applications you want.
The other thing I'm considering is keeping a syncronised SQL database (2) going with Subversion which could be used to do things like allocate custom permissions to resources, more complicated searching and the storage of custom "resource types". This is very similar to the mod_dav_dbms architecture that allows searching (DASL) to DAV resources. All that is required is that when subversion commits a change, (eg a document's locations is changed) that it spawns a call to a wrapper class that logs this to the database.
I know (1) is possible what about (2)? I will continue to read the documentation on the subject and Subversion in general :-)
Awesome, Z.
----- Original Message -----
From: Zitan Broth
To: dev@subversion.tigris.org
Sent: Sunday, April 27, 2003 11:36 PM
Subject: Blue Sky: Scripting Language / SQL database ?
Greetings All,
Been having a look at Subversion and it is an excellent open source project alright, far more flexible than CVS, and particularly awesome to support DAV and DeltaV. And my interest in Subversion comes *especially* from the DeltaV support. To be able integrate Subversion with into a web service-based architecture it would require some extra functionality mentioned in the "Future" section of the Subversion Design doc. My serving side scripting language of choice is PHP but I'm sure you could achieve a bridge to PHP from Perl also.
With a scripting language glue layer in between the Subversion "business logic" (eg how to implement DeltaV, DAV) and the current "persistance" layer (currently Berkely DB) it would be possible to take advantage of the logic that handles DAV and DeltaV but extend the logic and change how things are persisted in powerful ways (ie write another backend), say for example persisting part of the data in an SQL database (metadata, text docs, collections even and the other part (larger media files) in a file repository, and so on. It would also mean that organisations would likely extend the mod_dav backend via svn also, so that their custom backends could support DeltaV where required. A language like PHP supports almost every SQL database under the sun as well as being able to call Perl, Pthyon, Java and even .NET allowing integration into any platform or backend desired.
I'm installing a windows version locally of Subversion now and will look over the API documentation (svn_fs.h), however I'm (not yet) a C developer. So I'm interested to know what the interest in this project extension? How hard would it be to implement? Is work currently underway? I guess if the architecture is well abstracted (and Subversion looks to be an excellently managed project ;)) it would be a matter of editing the current API between the logic and the persistance to call a series of classes which are currently blank but would be filled in by the developers as required. Have I got the "right end of the stick" here or is the work far more complicated (eg is part of the logic embedded with the persistance layer interface)?
Any comments a greatly appreciated.
Awesome,
Z.