You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Marijn Stollenga <m....@gmail.com> on 2009/10/05 15:58:47 UTC
slow deleting
Hello,
I am new to this mailinglist. I am using CouchDB as backend to an
online game. I notice that deleting documents is really slow (about 4
deletes per second). Is there any way to improve this speed?
I use several views, do these views prohibit fast deletion? I added a
view is use in the database.
Thanks in advance,
Marijn
{
"all": {
"map": "function(doc) {if (doc.type == 'area'){emit(doc.name,
doc._id);}}"
},
"to_character": {
"map": "function(doc) {if (doc.type == 'character'){emit
(doc.location, doc.name);}}"
},
"to_paths": {
"map": "function(doc) {if (doc.type == 'path'){emit(doc.from,
doc.to);}}"
},
"path_to_area": {
"map": "function(doc) {if (doc.type == 'path'){emit(doc.from + doc.name
, doc.to);}}"
}
}
Re: slow deleting
Posted by Robert Newson <ro...@gmail.com>.
without that setting you are sync'ing each delete to disk (along with
updated database headers, etc). with it, your deletes are not yet
guaranteed to be durable. Someone else can state the magnitude of the
delay with the setting, but I think it's small. A call to
_ensure_full_commit periodically will put delete durability back under
your control, at least.
B.
On Mon, Oct 5, 2009 at 4:47 PM, Marijn Stollenga <m....@gmail.com> wrote:
> delayed_commits did the trick! Why does that help and more imporantly what
> are the side-effects. Does it take more time before the documents are up to
> date?
>
> On 5 okt 2009, at 17:35, Chris Anderson wrote:
>
>> On Mon, Oct 5, 2009 at 8:29 AM, Marijn Stollenga <m....@gmail.com>
>> wrote:
>>>
>>> When using a python script that calls curl to delete the objects, it
>>> remains
>>> equally slow. Curl does not have that header problem right? Other ideas?
>>>
>>
>> How many documents do you have? Does the deletion size vary with # of
>> docs?
>>
>> Try setting delayed_commits=true in your config and see if that helps.
>>
>> Chris
>>
>>>
>>> On 5 okt 2009, at 17:14, Robert Newson wrote:
>>>
>>>> I haven't checked but it's conceivable that couchdb4j (since it uses
>>>> httpclient) is issuing an "Expect: continue" header, and couchdb,
>>>> until very recently, treated that (incorrectly) case-sensitively,
>>>> causing an unnecessary wait for a timeout.
>>>>
>>>> A trace of http request/response headers for a single delete would
>>>> confirm or deny that hypothesis.
>>>>
>>>> B.
>>>>
>>>> On Mon, Oct 5, 2009 at 4:04 PM, Marijn Stollenga <m....@gmail.com>
>>>> wrote:
>>>>>
>>>>> By the way, I use couchdb4j for interfacing with couchdb. I will try to
>>>>> find
>>>>> out if that is the problem.
>>>>>
>>>>> On 5 okt 2009, at 16:36, Paul Davis wrote:
>>>>>
>>>>>> Views won't affect deletion speeds at all. What version of CouchDB on
>>>>>> what operating system are you using? Also, if possible you'll want to
>>>>>> group as many deletes as possible into a single _bulk_docs call.
>>>>>>
>>>>>> Regardless of _bulk_docs, 4rps seems slow if you're on one of the
>>>>>> releases. If you're on an older version of trunk, try updating to the
>>>>>> current version to see if that helps out.
>>>>>>
>>>>>> Paul Davis
>>>>>>
>>>>>> On Mon, Oct 5, 2009 at 9:58 AM, Marijn Stollenga
>>>>>> <m....@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>> I am new to this mailinglist. I am using CouchDB as backend to an
>>>>>>> online
>>>>>>> game. I notice that deleting documents is really slow (about 4
>>>>>>> deletes
>>>>>>> per
>>>>>>> second). Is there any way to improve this speed?
>>>>>>> I use several views, do these views prohibit fast deletion? I added a
>>>>>>> view
>>>>>>> is use in the database.
>>>>>>>
>>>>>>> Thanks in advance,
>>>>>>> Marijn
>>>>>>>
>>>>>>> {
>>>>>>> "all": {
>>>>>>> "map": "function(doc) {if (doc.type == 'area'){emit(doc.name,
>>>>>>> doc._id);}}"
>>>>>>> },
>>>>>>> "to_character": {
>>>>>>> "map": "function(doc) {if (doc.type ==
>>>>>>> 'character'){emit(doc.location,
>>>>>>> doc.name);}}"
>>>>>>> },
>>>>>>> "to_paths": {
>>>>>>> "map": "function(doc) {if (doc.type == 'path'){emit(doc.from,
>>>>>>> doc.to);}}"
>>>>>>> },
>>>>>>> "path_to_area": {
>>>>>>> "map": "function(doc) {if (doc.type == 'path'){emit(doc.from +
>>>>>>> doc.name, doc.to);}}"
>>>>>>> }
>>>>>>> }
>>>>>
>>>>>
>>>
>>>
>>
>>
>>
>> --
>> Chris Anderson
>> http://jchrisa.net
>> http://couch.io
>
>
Re: slow deleting
Posted by Marijn Stollenga <m....@gmail.com>.
delayed_commits did the trick! Why does that help and more imporantly
what are the side-effects. Does it take more time before the documents
are up to date?
On 5 okt 2009, at 17:35, Chris Anderson wrote:
> On Mon, Oct 5, 2009 at 8:29 AM, Marijn Stollenga <m.stollenga@gmail.com
> > wrote:
>> When using a python script that calls curl to delete the objects,
>> it remains
>> equally slow. Curl does not have that header problem right? Other
>> ideas?
>>
>
> How many documents do you have? Does the deletion size vary with #
> of docs?
>
> Try setting delayed_commits=true in your config and see if that helps.
>
> Chris
>
>>
>> On 5 okt 2009, at 17:14, Robert Newson wrote:
>>
>>> I haven't checked but it's conceivable that couchdb4j (since it uses
>>> httpclient) is issuing an "Expect: continue" header, and couchdb,
>>> until very recently, treated that (incorrectly) case-sensitively,
>>> causing an unnecessary wait for a timeout.
>>>
>>> A trace of http request/response headers for a single delete would
>>> confirm or deny that hypothesis.
>>>
>>> B.
>>>
>>> On Mon, Oct 5, 2009 at 4:04 PM, Marijn Stollenga <m.stollenga@gmail.com
>>> >
>>> wrote:
>>>>
>>>> By the way, I use couchdb4j for interfacing with couchdb. I will
>>>> try to
>>>> find
>>>> out if that is the problem.
>>>>
>>>> On 5 okt 2009, at 16:36, Paul Davis wrote:
>>>>
>>>>> Views won't affect deletion speeds at all. What version of
>>>>> CouchDB on
>>>>> what operating system are you using? Also, if possible you'll
>>>>> want to
>>>>> group as many deletes as possible into a single _bulk_docs call.
>>>>>
>>>>> Regardless of _bulk_docs, 4rps seems slow if you're on one of the
>>>>> releases. If you're on an older version of trunk, try updating
>>>>> to the
>>>>> current version to see if that helps out.
>>>>>
>>>>> Paul Davis
>>>>>
>>>>> On Mon, Oct 5, 2009 at 9:58 AM, Marijn Stollenga <m.stollenga@gmail.com
>>>>> >
>>>>> wrote:
>>>>>>
>>>>>> Hello,
>>>>>> I am new to this mailinglist. I am using CouchDB as backend to an
>>>>>> online
>>>>>> game. I notice that deleting documents is really slow (about 4
>>>>>> deletes
>>>>>> per
>>>>>> second). Is there any way to improve this speed?
>>>>>> I use several views, do these views prohibit fast deletion? I
>>>>>> added a
>>>>>> view
>>>>>> is use in the database.
>>>>>>
>>>>>> Thanks in advance,
>>>>>> Marijn
>>>>>>
>>>>>> {
>>>>>> "all": {
>>>>>> "map": "function(doc) {if (doc.type == 'area'){emit(doc.name,
>>>>>> doc._id);}}"
>>>>>> },
>>>>>> "to_character": {
>>>>>> "map": "function(doc) {if (doc.type ==
>>>>>> 'character'){emit(doc.location,
>>>>>> doc.name);}}"
>>>>>> },
>>>>>> "to_paths": {
>>>>>> "map": "function(doc) {if (doc.type == 'path'){emit(doc.from,
>>>>>> doc.to);}}"
>>>>>> },
>>>>>> "path_to_area": {
>>>>>> "map": "function(doc) {if (doc.type == 'path'){emit
>>>>>> (doc.from +
>>>>>> doc.name, doc.to);}}"
>>>>>> }
>>>>>> }
>>>>
>>>>
>>
>>
>
>
>
> --
> Chris Anderson
> http://jchrisa.net
> http://couch.io
Re: slow deleting
Posted by Chris Anderson <jc...@apache.org>.
On Mon, Oct 5, 2009 at 8:29 AM, Marijn Stollenga <m....@gmail.com> wrote:
> When using a python script that calls curl to delete the objects, it remains
> equally slow. Curl does not have that header problem right? Other ideas?
>
How many documents do you have? Does the deletion size vary with # of docs?
Try setting delayed_commits=true in your config and see if that helps.
Chris
>
> On 5 okt 2009, at 17:14, Robert Newson wrote:
>
>> I haven't checked but it's conceivable that couchdb4j (since it uses
>> httpclient) is issuing an "Expect: continue" header, and couchdb,
>> until very recently, treated that (incorrectly) case-sensitively,
>> causing an unnecessary wait for a timeout.
>>
>> A trace of http request/response headers for a single delete would
>> confirm or deny that hypothesis.
>>
>> B.
>>
>> On Mon, Oct 5, 2009 at 4:04 PM, Marijn Stollenga <m....@gmail.com>
>> wrote:
>>>
>>> By the way, I use couchdb4j for interfacing with couchdb. I will try to
>>> find
>>> out if that is the problem.
>>>
>>> On 5 okt 2009, at 16:36, Paul Davis wrote:
>>>
>>>> Views won't affect deletion speeds at all. What version of CouchDB on
>>>> what operating system are you using? Also, if possible you'll want to
>>>> group as many deletes as possible into a single _bulk_docs call.
>>>>
>>>> Regardless of _bulk_docs, 4rps seems slow if you're on one of the
>>>> releases. If you're on an older version of trunk, try updating to the
>>>> current version to see if that helps out.
>>>>
>>>> Paul Davis
>>>>
>>>> On Mon, Oct 5, 2009 at 9:58 AM, Marijn Stollenga <m....@gmail.com>
>>>> wrote:
>>>>>
>>>>> Hello,
>>>>> I am new to this mailinglist. I am using CouchDB as backend to an
>>>>> online
>>>>> game. I notice that deleting documents is really slow (about 4 deletes
>>>>> per
>>>>> second). Is there any way to improve this speed?
>>>>> I use several views, do these views prohibit fast deletion? I added a
>>>>> view
>>>>> is use in the database.
>>>>>
>>>>> Thanks in advance,
>>>>> Marijn
>>>>>
>>>>> {
>>>>> "all": {
>>>>> "map": "function(doc) {if (doc.type == 'area'){emit(doc.name,
>>>>> doc._id);}}"
>>>>> },
>>>>> "to_character": {
>>>>> "map": "function(doc) {if (doc.type ==
>>>>> 'character'){emit(doc.location,
>>>>> doc.name);}}"
>>>>> },
>>>>> "to_paths": {
>>>>> "map": "function(doc) {if (doc.type == 'path'){emit(doc.from,
>>>>> doc.to);}}"
>>>>> },
>>>>> "path_to_area": {
>>>>> "map": "function(doc) {if (doc.type == 'path'){emit(doc.from +
>>>>> doc.name, doc.to);}}"
>>>>> }
>>>>> }
>>>
>>>
>
>
--
Chris Anderson
http://jchrisa.net
http://couch.io
Re: slow deleting
Posted by Marijn Stollenga <m....@gmail.com>.
When using a python script that calls curl to delete the objects, it
remains equally slow. Curl does not have that header problem right?
Other ideas?
On 5 okt 2009, at 17:14, Robert Newson wrote:
> I haven't checked but it's conceivable that couchdb4j (since it uses
> httpclient) is issuing an "Expect: continue" header, and couchdb,
> until very recently, treated that (incorrectly) case-sensitively,
> causing an unnecessary wait for a timeout.
>
> A trace of http request/response headers for a single delete would
> confirm or deny that hypothesis.
>
> B.
>
> On Mon, Oct 5, 2009 at 4:04 PM, Marijn Stollenga <m.stollenga@gmail.com
> > wrote:
>> By the way, I use couchdb4j for interfacing with couchdb. I will
>> try to find
>> out if that is the problem.
>>
>> On 5 okt 2009, at 16:36, Paul Davis wrote:
>>
>>> Views won't affect deletion speeds at all. What version of CouchDB
>>> on
>>> what operating system are you using? Also, if possible you'll want
>>> to
>>> group as many deletes as possible into a single _bulk_docs call.
>>>
>>> Regardless of _bulk_docs, 4rps seems slow if you're on one of the
>>> releases. If you're on an older version of trunk, try updating to
>>> the
>>> current version to see if that helps out.
>>>
>>> Paul Davis
>>>
>>> On Mon, Oct 5, 2009 at 9:58 AM, Marijn Stollenga <m.stollenga@gmail.com
>>> >
>>> wrote:
>>>>
>>>> Hello,
>>>> I am new to this mailinglist. I am using CouchDB as backend to an
>>>> online
>>>> game. I notice that deleting documents is really slow (about 4
>>>> deletes
>>>> per
>>>> second). Is there any way to improve this speed?
>>>> I use several views, do these views prohibit fast deletion? I
>>>> added a
>>>> view
>>>> is use in the database.
>>>>
>>>> Thanks in advance,
>>>> Marijn
>>>>
>>>> {
>>>> "all": {
>>>> "map": "function(doc) {if (doc.type == 'area'){emit(doc.name,
>>>> doc._id);}}"
>>>> },
>>>> "to_character": {
>>>> "map": "function(doc) {if (doc.type ==
>>>> 'character'){emit(doc.location,
>>>> doc.name);}}"
>>>> },
>>>> "to_paths": {
>>>> "map": "function(doc) {if (doc.type == 'path'){emit(doc.from,
>>>> doc.to);}}"
>>>> },
>>>> "path_to_area": {
>>>> "map": "function(doc) {if (doc.type == 'path'){emit(doc.from +
>>>> doc.name, doc.to);}}"
>>>> }
>>>> }
>>
>>
Re: slow deleting
Posted by Marijn Stollenga <m....@gmail.com>.
For extra information, the put command takes equally long. I tried two
runs of my python script, which directly uses curl:
done in 12.9758610725 n_docs: 74 time per doc: 0.175349473953
done in 13.5452620983 n_docs: 74 time per doc: 0.18304408241
So on average it takes about 0.18 sec per doc.
On 5 okt 2009, at 17:14, Robert Newson wrote:
> I haven't checked but it's conceivable that couchdb4j (since it uses
> httpclient) is issuing an "Expect: continue" header, and couchdb,
> until very recently, treated that (incorrectly) case-sensitively,
> causing an unnecessary wait for a timeout.
>
> A trace of http request/response headers for a single delete would
> confirm or deny that hypothesis.
>
> B.
>
> On Mon, Oct 5, 2009 at 4:04 PM, Marijn Stollenga <m.stollenga@gmail.com
> > wrote:
>> By the way, I use couchdb4j for interfacing with couchdb. I will
>> try to find
>> out if that is the problem.
>>
>> On 5 okt 2009, at 16:36, Paul Davis wrote:
>>
>>> Views won't affect deletion speeds at all. What version of CouchDB
>>> on
>>> what operating system are you using? Also, if possible you'll want
>>> to
>>> group as many deletes as possible into a single _bulk_docs call.
>>>
>>> Regardless of _bulk_docs, 4rps seems slow if you're on one of the
>>> releases. If you're on an older version of trunk, try updating to
>>> the
>>> current version to see if that helps out.
>>>
>>> Paul Davis
>>>
>>> On Mon, Oct 5, 2009 at 9:58 AM, Marijn Stollenga <m.stollenga@gmail.com
>>> >
>>> wrote:
>>>>
>>>> Hello,
>>>> I am new to this mailinglist. I am using CouchDB as backend to an
>>>> online
>>>> game. I notice that deleting documents is really slow (about 4
>>>> deletes
>>>> per
>>>> second). Is there any way to improve this speed?
>>>> I use several views, do these views prohibit fast deletion? I
>>>> added a
>>>> view
>>>> is use in the database.
>>>>
>>>> Thanks in advance,
>>>> Marijn
>>>>
>>>> {
>>>> "all": {
>>>> "map": "function(doc) {if (doc.type == 'area'){emit(doc.name,
>>>> doc._id);}}"
>>>> },
>>>> "to_character": {
>>>> "map": "function(doc) {if (doc.type ==
>>>> 'character'){emit(doc.location,
>>>> doc.name);}}"
>>>> },
>>>> "to_paths": {
>>>> "map": "function(doc) {if (doc.type == 'path'){emit(doc.from,
>>>> doc.to);}}"
>>>> },
>>>> "path_to_area": {
>>>> "map": "function(doc) {if (doc.type == 'path'){emit(doc.from +
>>>> doc.name, doc.to);}}"
>>>> }
>>>> }
>>
>>
Re: slow deleting
Posted by Robert Newson <ro...@gmail.com>.
I haven't checked but it's conceivable that couchdb4j (since it uses
httpclient) is issuing an "Expect: continue" header, and couchdb,
until very recently, treated that (incorrectly) case-sensitively,
causing an unnecessary wait for a timeout.
A trace of http request/response headers for a single delete would
confirm or deny that hypothesis.
B.
On Mon, Oct 5, 2009 at 4:04 PM, Marijn Stollenga <m....@gmail.com> wrote:
> By the way, I use couchdb4j for interfacing with couchdb. I will try to find
> out if that is the problem.
>
> On 5 okt 2009, at 16:36, Paul Davis wrote:
>
>> Views won't affect deletion speeds at all. What version of CouchDB on
>> what operating system are you using? Also, if possible you'll want to
>> group as many deletes as possible into a single _bulk_docs call.
>>
>> Regardless of _bulk_docs, 4rps seems slow if you're on one of the
>> releases. If you're on an older version of trunk, try updating to the
>> current version to see if that helps out.
>>
>> Paul Davis
>>
>> On Mon, Oct 5, 2009 at 9:58 AM, Marijn Stollenga <m....@gmail.com>
>> wrote:
>>>
>>> Hello,
>>> I am new to this mailinglist. I am using CouchDB as backend to an online
>>> game. I notice that deleting documents is really slow (about 4 deletes
>>> per
>>> second). Is there any way to improve this speed?
>>> I use several views, do these views prohibit fast deletion? I added a
>>> view
>>> is use in the database.
>>>
>>> Thanks in advance,
>>> Marijn
>>>
>>> {
>>> "all": {
>>> "map": "function(doc) {if (doc.type == 'area'){emit(doc.name,
>>> doc._id);}}"
>>> },
>>> "to_character": {
>>> "map": "function(doc) {if (doc.type ==
>>> 'character'){emit(doc.location,
>>> doc.name);}}"
>>> },
>>> "to_paths": {
>>> "map": "function(doc) {if (doc.type == 'path'){emit(doc.from,
>>> doc.to);}}"
>>> },
>>> "path_to_area": {
>>> "map": "function(doc) {if (doc.type == 'path'){emit(doc.from +
>>> doc.name, doc.to);}}"
>>> }
>>> }
>
>
Re: slow deleting
Posted by Marijn Stollenga <m....@gmail.com>.
By the way, I use couchdb4j for interfacing with couchdb. I will try
to find out if that is the problem.
On 5 okt 2009, at 16:36, Paul Davis wrote:
> Views won't affect deletion speeds at all. What version of CouchDB on
> what operating system are you using? Also, if possible you'll want to
> group as many deletes as possible into a single _bulk_docs call.
>
> Regardless of _bulk_docs, 4rps seems slow if you're on one of the
> releases. If you're on an older version of trunk, try updating to the
> current version to see if that helps out.
>
> Paul Davis
>
> On Mon, Oct 5, 2009 at 9:58 AM, Marijn Stollenga <m.stollenga@gmail.com
> > wrote:
>> Hello,
>> I am new to this mailinglist. I am using CouchDB as backend to an
>> online
>> game. I notice that deleting documents is really slow (about 4
>> deletes per
>> second). Is there any way to improve this speed?
>> I use several views, do these views prohibit fast deletion? I added
>> a view
>> is use in the database.
>>
>> Thanks in advance,
>> Marijn
>>
>> {
>> "all": {
>> "map": "function(doc) {if (doc.type == 'area'){emit(doc.name,
>> doc._id);}}"
>> },
>> "to_character": {
>> "map": "function(doc) {if (doc.type == 'character'){emit
>> (doc.location,
>> doc.name);}}"
>> },
>> "to_paths": {
>> "map": "function(doc) {if (doc.type == 'path'){emit(doc.from,
>> doc.to);}}"
>> },
>> "path_to_area": {
>> "map": "function(doc) {if (doc.type == 'path'){emit(doc.from +
>> doc.name, doc.to);}}"
>> }
>> }
Re: slow deleting
Posted by Marijn Stollenga <m....@gmail.com>.
I am running MacOSX with Apache CouchDB 0.11.0b821321. Is that an old
version?
On 5 okt 2009, at 16:36, Paul Davis wrote:
> Views won't affect deletion speeds at all. What version of CouchDB on
> what operating system are you using? Also, if possible you'll want to
> group as many deletes as possible into a single _bulk_docs call.
>
> Regardless of _bulk_docs, 4rps seems slow if you're on one of the
> releases. If you're on an older version of trunk, try updating to the
> current version to see if that helps out.
>
> Paul Davis
>
> On Mon, Oct 5, 2009 at 9:58 AM, Marijn Stollenga <m.stollenga@gmail.com
> > wrote:
>> Hello,
>> I am new to this mailinglist. I am using CouchDB as backend to an
>> online
>> game. I notice that deleting documents is really slow (about 4
>> deletes per
>> second). Is there any way to improve this speed?
>> I use several views, do these views prohibit fast deletion? I added
>> a view
>> is use in the database.
>>
>> Thanks in advance,
>> Marijn
>>
>> {
>> "all": {
>> "map": "function(doc) {if (doc.type == 'area'){emit(doc.name,
>> doc._id);}}"
>> },
>> "to_character": {
>> "map": "function(doc) {if (doc.type == 'character'){emit
>> (doc.location,
>> doc.name);}}"
>> },
>> "to_paths": {
>> "map": "function(doc) {if (doc.type == 'path'){emit(doc.from,
>> doc.to);}}"
>> },
>> "path_to_area": {
>> "map": "function(doc) {if (doc.type == 'path'){emit(doc.from +
>> doc.name, doc.to);}}"
>> }
>> }
Re: slow deleting
Posted by Paul Davis <pa...@gmail.com>.
Views won't affect deletion speeds at all. What version of CouchDB on
what operating system are you using? Also, if possible you'll want to
group as many deletes as possible into a single _bulk_docs call.
Regardless of _bulk_docs, 4rps seems slow if you're on one of the
releases. If you're on an older version of trunk, try updating to the
current version to see if that helps out.
Paul Davis
On Mon, Oct 5, 2009 at 9:58 AM, Marijn Stollenga <m....@gmail.com> wrote:
> Hello,
> I am new to this mailinglist. I am using CouchDB as backend to an online
> game. I notice that deleting documents is really slow (about 4 deletes per
> second). Is there any way to improve this speed?
> I use several views, do these views prohibit fast deletion? I added a view
> is use in the database.
>
> Thanks in advance,
> Marijn
>
> {
> "all": {
> "map": "function(doc) {if (doc.type == 'area'){emit(doc.name,
> doc._id);}}"
> },
> "to_character": {
> "map": "function(doc) {if (doc.type == 'character'){emit(doc.location,
> doc.name);}}"
> },
> "to_paths": {
> "map": "function(doc) {if (doc.type == 'path'){emit(doc.from,
> doc.to);}}"
> },
> "path_to_area": {
> "map": "function(doc) {if (doc.type == 'path'){emit(doc.from +
> doc.name, doc.to);}}"
> }
> }