You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Gregory Tappero <co...@gmail.com> on 2010/08/15 23:32:52 UTC
File system data representation
Hello couchers,
I was looking in a way to build S3 like with couchdb.
- buckets and files
- sharing between users
- update/move folders in buckets
Using a separate doc for Each File with the pathj of the file in json
seemed like a good plan. We Can easily lits files and folders
children of a given path when emiting on each /foo/ part of the path
parameter.
Sharing is also solved by creating sharing doc for a given path and
user using pattern matching we can find out if a file is accessible to
a user. /foo/bar matches /foo/ so Bob is allowed to see bar.
For thé move folders action however i see.no other solution t han
performing an update of path for all docs touched by the move
operation. / foo/bar and foo/bar/a plus sharings will need to be
changed if foo become 'bizz', quite inneficient.
Do you see some other solutions ?
--
Greg Tappero
CTO co founder Edoboard
http://www.edoboard.com
+33 0645764425
Re: File system data representation
Posted by J Chris Anderson <jc...@apache.org>.
On Aug 15, 2010, at 2:32 PM, Gregory Tappero wrote:
> Hello couchers,
>
> I was looking in a way to build S3 like with couchdb.
>
> - buckets and files
> - sharing between users
> - update/move folders in buckets
>
> Using a separate doc for Each File with the pathj of the file in json
> seemed like a good plan. We Can easily lits files and folders
> children of a given path when emiting on each /foo/ part of the path
> parameter.
> Sharing is also solved by creating sharing doc for a given path and
> user using pattern matching we can find out if a file is accessible to
> a user. /foo/bar matches /foo/ so Bob is allowed to see bar.
>
> For thé move folders action however i see.no other solution t han
> performing an update of path for all docs touched by the move
> operation. / foo/bar and foo/bar/a plus sharings will need to be
> changed if foo become 'bizz', quite inneficient.
>
> Do you see some other solutions ?
>
This looks like the way to do it. Not much getting around the need to bulk update docs in a path when renaming.
Although renames are rare, and they won't involve rewriting the binary data, so it's not as bad as it seems.
Chris
>
> --
> Greg Tappero
> CTO co founder Edoboard
> http://www.edoboard.com
> +33 0645764425
Re: File system data representation
Posted by Mirko Kiefer <mi...@arcor.de>.
Hi Gregory,
there is an early-stage implementation of a virtual filesystem on top of CouchDB using FUSE:
http://github.com/narkisr/couch-fuse
Ronen Narkis who is responsible for couch-fuse also wrote about his project on this mailing-list some time back. I'm not sure on the current status of his project - back in June he was heavily working on it.
Best,
Mirko
On Aug 17, 2010, at 8:41 AM, Gregory Tappero wrote:
> That's very interesting i look foward to testing it.
> How would an S3 datastructure mimic using '/' in keys will look on couchdb ?
> I am no sure i get the difference with a path attribute '/my/fake/folders'.
>
> Thanks
>
> On Tue, Aug 17, 2010 at 2:41 AM, sgoto <sa...@gmail.com> wrote:
>
>> I am working on something related to that
>>
>> couchdb backend
>>
>> http://kumbaya.bounceme.net:9094/_utils/database.html?testdb/_design/kumbaya/_view/folders
>>
>> and use webdav to mount this
>>
>> webdav frontend for my couchdb db
>> konqueror http://kumbaya.bounceme.net:9095
>>
>> i'll be open sourcing this soon
>>
>> On Sun, Aug 15, 2010 at 10:21 PM, Alan Mock <al...@gmail.com> wrote:
>>
>>> On Aug 15, 2010, at 2:32 PM, Gregory Tappero wrote:
>>>
>>>> Hello couchers,
>>>>
>>>> I was looking in a way to build S3 like with couchdb.
>>>>
>>>> - buckets and files
>>>> - sharing between users
>>>
>>
>> this is where i'm stuck at the moment. i'm not sure how couchdb handles
>> authentication while replicating data (eg can a malicious couchdb node
>> write
>> data to my local couchdb node on behalf of a user?). have you figured how
>> authentication and authorization is going to work on your system with
>> untrusted couchdb nodes ?
>>
>>
>>>> - update/move folders in buckets
>>>>
>>>> Using a separate doc for Each File with the pathj of the file in json
>>>> seemed like a good plan. We Can easily lits files and folders
>>>> children of a given path when emiting on each /foo/ part of the path
>>>> parameter.
>>>> Sharing is also solved by creating sharing doc for a given path and
>>>> user using pattern matching we can find out if a file is accessible to
>>>> a user. /foo/bar matches /foo/ so Bob is allowed to see bar.
>>>>
>>>> For thé move folders action however i see.no other solution t han
>>>> performing an update of path for all docs touched by the move
>>>> operation. / foo/bar and foo/bar/a plus sharings will need to be
>>>> changed if foo become 'bizz', quite inneficient.
>>>>
>>>> Do you see some other solutions ?
>>>>
>>>
>>> I don't know how closely you want to mimic S3 or if you're just using it
>> as
>>> an example, but S3 is a key-value store which means it has no folders.
>> It
>>> allows / as part of a key so that you can use it's prefix function to
>> allow
>>> listing of keys like they were in a folder, but that's as much folder
>>> support as it provides. If you're using S3 and it works then maybe
>> copying
>>> it's functionality more closely would be easier to implement.
>>>
>>> Alan
>>>
>>>>
>>>> --
>>>> Greg Tappero
>>>> CTO co founder Edoboard
>>>> http://www.edoboard.com
>>>> +33 0645764425
>>>
>>>
>>
>>
>> --
>> f u cn rd ths u cn b a gd prgmr !
>>
>
>
>
> --
> Greg Tappero
> CTO co founder Edoboard
> http://www.edoboard.com
> +33 0645764425
Re: File system data representation
Posted by Gregory Tappero <co...@gmail.com>.
That's very interesting i look foward to testing it.
How would an S3 datastructure mimic using '/' in keys will look on couchdb ?
I am no sure i get the difference with a path attribute '/my/fake/folders'.
Thanks
On Tue, Aug 17, 2010 at 2:41 AM, sgoto <sa...@gmail.com> wrote:
> I am working on something related to that
>
> couchdb backend
>
> http://kumbaya.bounceme.net:9094/_utils/database.html?testdb/_design/kumbaya/_view/folders
>
> and use webdav to mount this
>
> webdav frontend for my couchdb db
> konqueror http://kumbaya.bounceme.net:9095
>
> i'll be open sourcing this soon
>
> On Sun, Aug 15, 2010 at 10:21 PM, Alan Mock <al...@gmail.com> wrote:
>
> > On Aug 15, 2010, at 2:32 PM, Gregory Tappero wrote:
> >
> > > Hello couchers,
> > >
> > > I was looking in a way to build S3 like with couchdb.
> > >
> > > - buckets and files
> > > - sharing between users
> >
>
> this is where i'm stuck at the moment. i'm not sure how couchdb handles
> authentication while replicating data (eg can a malicious couchdb node
> write
> data to my local couchdb node on behalf of a user?). have you figured how
> authentication and authorization is going to work on your system with
> untrusted couchdb nodes ?
>
>
> > > - update/move folders in buckets
> > >
> > > Using a separate doc for Each File with the pathj of the file in json
> > > seemed like a good plan. We Can easily lits files and folders
> > > children of a given path when emiting on each /foo/ part of the path
> > > parameter.
> > > Sharing is also solved by creating sharing doc for a given path and
> > > user using pattern matching we can find out if a file is accessible to
> > > a user. /foo/bar matches /foo/ so Bob is allowed to see bar.
> > >
> > > For thé move folders action however i see.no other solution t han
> > > performing an update of path for all docs touched by the move
> > > operation. / foo/bar and foo/bar/a plus sharings will need to be
> > > changed if foo become 'bizz', quite inneficient.
> > >
> > > Do you see some other solutions ?
> > >
> >
> > I don't know how closely you want to mimic S3 or if you're just using it
> as
> > an example, but S3 is a key-value store which means it has no folders.
> It
> > allows / as part of a key so that you can use it's prefix function to
> allow
> > listing of keys like they were in a folder, but that's as much folder
> > support as it provides. If you're using S3 and it works then maybe
> copying
> > it's functionality more closely would be easier to implement.
> >
> > Alan
> >
> > >
> > > --
> > > Greg Tappero
> > > CTO co founder Edoboard
> > > http://www.edoboard.com
> > > +33 0645764425
> >
> >
>
>
> --
> f u cn rd ths u cn b a gd prgmr !
>
--
Greg Tappero
CTO co founder Edoboard
http://www.edoboard.com
+33 0645764425
Re: File system data representation
Posted by sgoto <sa...@gmail.com>.
I am working on something related to that
couchdb backend
http://kumbaya.bounceme.net:9094/_utils/database.html?testdb/_design/kumbaya/_view/folders
and use webdav to mount this
webdav frontend for my couchdb db
konqueror http://kumbaya.bounceme.net:9095
i'll be open sourcing this soon
On Sun, Aug 15, 2010 at 10:21 PM, Alan Mock <al...@gmail.com> wrote:
> On Aug 15, 2010, at 2:32 PM, Gregory Tappero wrote:
>
> > Hello couchers,
> >
> > I was looking in a way to build S3 like with couchdb.
> >
> > - buckets and files
> > - sharing between users
>
this is where i'm stuck at the moment. i'm not sure how couchdb handles
authentication while replicating data (eg can a malicious couchdb node write
data to my local couchdb node on behalf of a user?). have you figured how
authentication and authorization is going to work on your system with
untrusted couchdb nodes ?
> > - update/move folders in buckets
> >
> > Using a separate doc for Each File with the pathj of the file in json
> > seemed like a good plan. We Can easily lits files and folders
> > children of a given path when emiting on each /foo/ part of the path
> > parameter.
> > Sharing is also solved by creating sharing doc for a given path and
> > user using pattern matching we can find out if a file is accessible to
> > a user. /foo/bar matches /foo/ so Bob is allowed to see bar.
> >
> > For thé move folders action however i see.no other solution t han
> > performing an update of path for all docs touched by the move
> > operation. / foo/bar and foo/bar/a plus sharings will need to be
> > changed if foo become 'bizz', quite inneficient.
> >
> > Do you see some other solutions ?
> >
>
> I don't know how closely you want to mimic S3 or if you're just using it as
> an example, but S3 is a key-value store which means it has no folders. It
> allows / as part of a key so that you can use it's prefix function to allow
> listing of keys like they were in a folder, but that's as much folder
> support as it provides. If you're using S3 and it works then maybe copying
> it's functionality more closely would be easier to implement.
>
> Alan
>
> >
> > --
> > Greg Tappero
> > CTO co founder Edoboard
> > http://www.edoboard.com
> > +33 0645764425
>
>
--
f u cn rd ths u cn b a gd prgmr !
Re: File system data representation
Posted by Alan Mock <al...@gmail.com>.
On Aug 15, 2010, at 2:32 PM, Gregory Tappero wrote:
> Hello couchers,
>
> I was looking in a way to build S3 like with couchdb.
>
> - buckets and files
> - sharing between users
> - update/move folders in buckets
>
> Using a separate doc for Each File with the pathj of the file in json
> seemed like a good plan. We Can easily lits files and folders
> children of a given path when emiting on each /foo/ part of the path
> parameter.
> Sharing is also solved by creating sharing doc for a given path and
> user using pattern matching we can find out if a file is accessible to
> a user. /foo/bar matches /foo/ so Bob is allowed to see bar.
>
> For thé move folders action however i see.no other solution t han
> performing an update of path for all docs touched by the move
> operation. / foo/bar and foo/bar/a plus sharings will need to be
> changed if foo become 'bizz', quite inneficient.
>
> Do you see some other solutions ?
>
I don't know how closely you want to mimic S3 or if you're just using it as an example, but S3 is a key-value store which means it has no folders. It allows / as part of a key so that you can use it's prefix function to allow listing of keys like they were in a folder, but that's as much folder support as it provides. If you're using S3 and it works then maybe copying it's functionality more closely would be easier to implement.
Alan
>
> --
> Greg Tappero
> CTO co founder Edoboard
> http://www.edoboard.com
> +33 0645764425