You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Chris Anderson <jc...@apache.org> on 2010/01/04 19:58:06 UTC
Re: Dealing with Atomic Operations
On Mon, Jan 4, 2010 at 7:48 PM, Andreas Pavlogiannis
<pa...@gmail.com> wrote:
> Hello,
>
> I 've been experimenting with Couchdb and I have reached a point in which I
> need to do some transactions in an atomic way. I know that Couchdb lacks
> locking, so is there a uniform way to deal with this?
> In particular, I have created a filesystem in which each file and folder is
> represented by a single document, while its pathname is a list. For example:
>
> ['root', 'home']
> ['root', 'home', pictures']
> ['root', 'home', 'pictures', 'picture.jpg']
>
> Now, when creating the document with pathname ['root', 'home', pictures',
> 'my_dog.jpg'], I need to make sure that its parent, ['root', 'home',
> pictures'] exists in the db. Moreover, I need to ensure that the user is not
> exceeding his quota.
Why do you need the parent to exist? You can use views to create
parents "virtually." This way each file would have a path, and you
could use a view to sort all files so that those in paths and subpaths
together are listed together. There is no need for a document to
denote the existence of an empty directory.
If you need to model directory permissions, then it makes sense to
have a document for that directory, but this also doesn't seem like it
needs to be transactionally checked / modified with the creation of
new files.
The quota stuff seems like it doesn't need to be transactional. If a
user is over quota, they can't create new files. If they delete some
files to get below quota again, it seems ok if this is reflected
"eventually" instead of transactionally.
> The bulk update way would be ideal if a possible conflict would create the
> appropriate error messages but as far as I know, that's not the case.
> Moreover, there are cases in which a bulk update seems inevitable (for
> example, when deleting a folder, so all its contents must be deleted too),
> but its poor behavior in a bad scenario (with document conflicts) makes it a
> bit difficult to use.
>
> Is there any way to circumvent all these problems?
> Is the behavior of the bulk updates going to change in the near future?
There are no plans to add transactional bulk updates as they are
incompatible with large partitioned clusters.
--
Chris Anderson
http://jchrisa.net
http://couch.io
Re: Dealing with Atomic Operations
Posted by Chris Anderson <jc...@apache.org>.
On Mon, Jan 4, 2010 at 10:04 PM, Andreas Pavlogiannis
<pa...@gmail.com> wrote:
> On 1/4/2010 10:58 AM, Chris Anderson wrote:
>>
>> On Mon, Jan 4, 2010 at 7:48 PM, Andreas Pavlogiannis
>> <pa...@gmail.com> wrote:
>>
>>>
>>> Hello,
>>>
>>> I 've been experimenting with Couchdb and I have reached a point in which
>>> I
>>> need to do some transactions in an atomic way. I know that Couchdb lacks
>>> locking, so is there a uniform way to deal with this?
>>> In particular, I have created a filesystem in which each file and folder
>>> is
>>> represented by a single document, while its pathname is a list. For
>>> example:
>>>
>>> ['root', 'home']
>>> ['root', 'home', pictures']
>>> ['root', 'home', 'pictures', 'picture.jpg']
>>>
>>> Now, when creating the document with pathname ['root', 'home', pictures',
>>> 'my_dog.jpg'], I need to make sure that its parent, ['root', 'home',
>>> pictures'] exists in the db. Moreover, I need to ensure that the user is
>>> not
>>> exceeding his quota.
>>>
>>
>> Why do you need the parent to exist? You can use views to create
>> parents "virtually." This way each file would have a path, and you
>> could use a view to sort all files so that those in paths and subpaths
>> together are listed together. There is no need for a document to
>> denote the existence of an empty directory.
>>
>> If you need to model directory permissions, then it makes sense to
>> have a document for that directory, but this also doesn't seem like it
>> needs to be transactionally checked / modified with the creation of
>> new files.
>>
>
> I need to ensure that when creating a new
> document, its parent exists in the db.
If you drop this constraint, you should still be able to handle the
business case, while doing away with a bunch of complexity.
>
> Hm, what about the case in which a user has enough space to upload just
> one file, but several upload requests arrive concurrently. As these requests
> are handled locally, each one will allow for the document to be submitted.
> But in this way the user can exceed his quota unboundedly (extreme scenario,
> but possible).
>>
You can always handle uploads by setting a custom piece of data in the
validation function. if you run a view query to say what the users
free space is, you can use this to reject oversize attachment uploads.
Still, I'd rather accept all uploads, and initiate a billing system
action or something when they cross the threshold.
--
Chris Anderson
http://jchrisa.net
http://couch.io
Re: Dealing with Atomic Operations
Posted by Anh <7z...@gmail.com>.
On Jan 4, 2010, at 10:04 PM, Andreas Pavlogiannis <paulogiann.couchdb@gmail.com
> wrote:
>
>>>
>>>
>>>
>>
> Hm, what about the case in which a user has enough space to upload
> just one file, but several upload requests arrive concurrently. As
> these requests are handled locally, each one will allow for the
> document to be submitted. But in this way the user can exceed his
> quota unboundedly (extreme scenario, but possible).
>>
>
Why is this a transactional issue?
Assuming your middle-ware is enforcing the quota,
seems you would have the same issue even if you were using an RDBMS
with transaction support
Re: Dealing with Atomic Operations
Posted by Andreas Pavlogiannis <pa...@gmail.com>.
On 1/4/2010 10:58 AM, Chris Anderson wrote:
> On Mon, Jan 4, 2010 at 7:48 PM, Andreas Pavlogiannis
> <pa...@gmail.com> wrote:
>
>> Hello,
>>
>> I 've been experimenting with Couchdb and I have reached a point in which I
>> need to do some transactions in an atomic way. I know that Couchdb lacks
>> locking, so is there a uniform way to deal with this?
>> In particular, I have created a filesystem in which each file and folder is
>> represented by a single document, while its pathname is a list. For example:
>>
>> ['root', 'home']
>> ['root', 'home', pictures']
>> ['root', 'home', 'pictures', 'picture.jpg']
>>
>> Now, when creating the document with pathname ['root', 'home', pictures',
>> 'my_dog.jpg'], I need to make sure that its parent, ['root', 'home',
>> pictures'] exists in the db. Moreover, I need to ensure that the user is not
>> exceeding his quota.
>>
> Why do you need the parent to exist? You can use views to create
> parents "virtually." This way each file would have a path, and you
> could use a view to sort all files so that those in paths and subpaths
> together are listed together. There is no need for a document to
> denote the existence of an empty directory.
>
> If you need to model directory permissions, then it makes sense to
> have a document for that directory, but this also doesn't seem like it
> needs to be transactionally checked / modified with the creation of
> new files.
>
Yes, this is the case. The idea is to store the full pathname in each
document, so as to retrieve it in a single step. So when the user asks
for the document 'root/home/foo/bar', I just return the document with
that pathname, instead of following the conventional pathname
translation (first obtain root, then home etc..). The existence of
folder documents is needed, as a user must have the ability to create
folders, and possibly share them with other users. Given the above, I
need to ensure that when creating a new document, its parent exists in
the db. If not, the new document exists (and can be accessed) while, the
branch it belongs to has been deleted.
> The quota stuff seems like it doesn't need to be transactional. If a
> user is over quota, they can't create new files. If they delete some
> files to get below quota again, it seems ok if this is reflected
> "eventually" instead of transactionally.
>
Hm, what about the case in which a user has enough space to upload
just one file, but several upload requests arrive concurrently. As these
requests are handled locally, each one will allow for the document to be
submitted. But in this way the user can exceed his quota unboundedly
(extreme scenario, but possible).
>
>
>> The bulk update way would be ideal if a possible conflict would create the
>> appropriate error messages but as far as I know, that's not the case.
>> Moreover, there are cases in which a bulk update seems inevitable (for
>> example, when deleting a folder, so all its contents must be deleted too),
>> but its poor behavior in a bad scenario (with document conflicts) makes it a
>> bit difficult to use.
>>
>> Is there any way to circumvent all these problems?
>> Is the behavior of the bulk updates going to change in the near future?
>>
> There are no plans to add transactional bulk updates as they are
> incompatible with large partitioned clusters.
>
>
>
>