You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Vladimir Ralev <vl...@gmail.com> on 2013/12/27 01:51:06 UTC
Burst PUT updates
Hi all,
I have a script that makes a burst of around 100 very fast PUT requests
mixed with another 300 or so GET requests against a 4GB database. 40
seconds into the script (30-40 PUTs and the CouchDB starts freezing, taking
longer and longer to respond until it reaches y timeouts). Is this a known
problem? Is there something I can do about it.
The PUTs update existing documents in the DB and there some autocompaction.
The documents are less than 1KB. Now I run the script with 30 second pauses
between the writes and that works fine, but it's too slow.
Cheers.
Re: Burst PUT updates
Posted by Vladimir Ralev <vl...@gmail.com>.
The client is not a problem. I observed the traffic with wireshark and the
requests make it while the the responses are getting slower and slower.
Today I observed that the server is busy with indexer tasks (10s of these),
but they only show up in http://HOST/_active_tasks not from futon. I will
think how can I confirm that this is the problem. This changes_done 385 and
432 don't make much sense to me since I it happens only after 50 or so PUTs.
- {
- pid: "<0.3769.9>",
- changes_done: 385,
- database: "account1",
- design_document: "_design/account",
- progress: 32,
- started_on: 1388178498,
- total_changes: 1194,
- type: "indexer",
- updated_on: 1388178500
},
-
{
- pid: "<0.3770.9>",
- changes_done: 432,
- database: "account2",
- design_document: "_design/account",
- progress: 36,
- started_on: 1388178498,
- total_changes: 1178,
- type: "indexer",
- updated_on: 1388178500
},
On Fri, Dec 27, 2013 at 10:49 AM, Benoit Chesneau <bc...@gmail.com>wrote:
> On Fri, Dec 27, 2013 at 8:06 AM, Stanley Iriele <siriele2x3@gmail.com
> >wrote:
>
> > Even without bulk docs...that doesn't sound right..it sounds to me like
> the
> > library you're using is having a problem with its http client and the
> time
> > gap in closing a connection and opening a new one. Try looking at how
> many
> > open connections your app has when it starts to slow down.
> >
> > I'm guessing in the loop it tries to reuse the same TCP connection but it
> > can't so it just creates a new one..? Then slowly but surely your
> > application begins to die...I would override the TCP dial in the function
> > to count the number of created connections or use lsof.
> >
> > This is often why I stay away from SDK s for couchdb... They often limit
> > you more than they enable you... Your driver should be the http client
> > itself not some library that wraps making the calls for you.. In my
> opinion
> > Anyways 😀
> >
>
> The problem could also be be that no framework here is used but a client.
> Couchdbkit for example is reusing connections. These frameworks aren't only
> "wrapping" the API call, they do much. At least for some.
>
> Anyway the problem can indeed be that either the client or couchdb run out
> of free fds. Then the system is waiting. Do you have any log in couchdb
> that can help? Or from your client?
>
> What is the limit of fds on your system ? (ulimit -n could help you on
> POSIX systems) ?
>
> - benoit
>
Re: Burst PUT updates
Posted by Benoit Chesneau <bc...@gmail.com>.
On Fri, Dec 27, 2013 at 8:06 AM, Stanley Iriele <si...@gmail.com>wrote:
> Even without bulk docs...that doesn't sound right..it sounds to me like the
> library you're using is having a problem with its http client and the time
> gap in closing a connection and opening a new one. Try looking at how many
> open connections your app has when it starts to slow down.
>
> I'm guessing in the loop it tries to reuse the same TCP connection but it
> can't so it just creates a new one..? Then slowly but surely your
> application begins to die...I would override the TCP dial in the function
> to count the number of created connections or use lsof.
>
> This is often why I stay away from SDK s for couchdb... They often limit
> you more than they enable you... Your driver should be the http client
> itself not some library that wraps making the calls for you.. In my opinion
> Anyways 😀
>
The problem could also be be that no framework here is used but a client.
Couchdbkit for example is reusing connections. These frameworks aren't only
"wrapping" the API call, they do much. At least for some.
Anyway the problem can indeed be that either the client or couchdb run out
of free fds. Then the system is waiting. Do you have any log in couchdb
that can help? Or from your client?
What is the limit of fds on your system ? (ulimit -n could help you on
POSIX systems) ?
- benoit
Re: Burst PUT updates
Posted by Stanley Iriele <si...@gmail.com>.
Even without bulk docs...that doesn't sound right..it sounds to me like the
library you're using is having a problem with its http client and the time
gap in closing a connection and opening a new one. Try looking at how many
open connections your app has when it starts to slow down.
I'm guessing in the loop it tries to reuse the same TCP connection but it
can't so it just creates a new one..? Then slowly but surely your
application begins to die...I would override the TCP dial in the function
to count the number of created connections or use lsof.
This is often why I stay away from SDK s for couchdb... They often limit
you more than they enable you... Your driver should be the http client
itself not some library that wraps making the calls for you.. In my opinion
Anyways 😀
On Dec 26, 2013 7:11 PM, "Jens Alfke" <je...@couchbase.com> wrote:
>
> On Dec 26, 2013, at 4:51 PM, Vladimir Ralev <vl...@gmail.com>
> wrote:
>
> > I have a script that makes a burst of around 100 very fast PUT requests
> > mixed with another 300 or so GET requests against a 4GB database.
>
> Try using _bulk_docs instead, to update multiple docs at once. It should
> be more efficient. (It’s what the replicator uses to push revisions.)
>
> —Jens
>
>
Re: Burst PUT updates
Posted by Jens Alfke <je...@couchbase.com>.
On Dec 26, 2013, at 4:51 PM, Vladimir Ralev <vl...@gmail.com> wrote:
> I have a script that makes a burst of around 100 very fast PUT requests
> mixed with another 300 or so GET requests against a 4GB database.
Try using _bulk_docs instead, to update multiple docs at once. It should be more efficient. (It’s what the replicator uses to push revisions.)
—Jens
Re: Burst PUT updates
Posted by Alexander Shorin <kx...@gmail.com>.
Hi Vladimir,
Which CouchDB/Erlang version you're using?
Do you have delay_commits enabled and how much loaded your disk I/O?
How often autocompaction runs and does freezes happened during
compaction or just eventually?
Can you share your script to let us reproduce the behaviour? It would
be awesome, if you'll be able to provide some usage statistics to see
when anomaly freezing starts and what's tasks/requests CouchDB serves
in that moment.
Thanks!
--
,,,^..^,,,
On Fri, Dec 27, 2013 at 4:51 AM, Vladimir Ralev
<vl...@gmail.com> wrote:
> Hi all,
>
> I have a script that makes a burst of around 100 very fast PUT requests
> mixed with another 300 or so GET requests against a 4GB database. 40
> seconds into the script (30-40 PUTs and the CouchDB starts freezing, taking
> longer and longer to respond until it reaches y timeouts). Is this a known
> problem? Is there something I can do about it.
>
> The PUTs update existing documents in the DB and there some autocompaction.
> The documents are less than 1KB. Now I run the script with 30 second pauses
> between the writes and that works fine, but it's too slow.
>
> Cheers.
Re: Burst PUT updates
Posted by Vladimir Ralev <vl...@gmail.com>.
It's CouchDB 1.1.1, erlang R14B03 (erts-5.8.4)
The script takes all docs from a view in several DBs, changes one field and
then puts it back. I've tried several versions of the script, one that
maintains the TCP socket and all kinds of reconnect logic, more or less the
same effect. I will review my settings, but there shouldn't be anything too
crazy there. The machine is solid - xeon 4 core server with 16G and there
is very little CPU and IO utilisation, Debian. It's unknown right now if
compaction is correlated. I will have to get back to you on that. This DB
was migrated from a bigcouch, so I guess I will have to rule out some
problem with that too.
The python3 script looks something very close to the minimal script that
demonstrates the issue for me.
def update(db, conn):
conn.request("GET", "/" + db + "/_design/account/_view/by_name")
response = conn.getresponse().read().decode()
rj = json.loads(response)
for account in rj["rows"]:
conn.request("GET", "/" + db + "/" + account["id"])
docresp = conn.getresponse().read().decode()
jdocresp = json.loads(docresp)
jdocresp["url"] = "http://myurl0"
conn.request("PUT","/" + db + "/" + account["id"],
json.dumps(jdocresp))
conn.getresponse().read()
print ('Done ' + json.dumps(jdocresp))
def main():
conn = http.client.HTTPConnection('host',5984, True, 20)
conn.request("GET", "/_all_dbs")
response = conn.getresponse()
body = response.read().decode()
jdbs = json.loads(body)
for db in jdbs:
updatenotify(db, conn)
conn.close()
time.sleep(20) # HERE IS THE SLEEP I mentioned
conn = http.client.HTTPConnection('host', 5984, True, 20)
On Fri, Dec 27, 2013 at 3:27 AM, Stanley Iriele <si...@gmail.com>wrote:
> What's the hardware... Cocudb version...operating system of your
> project...also that doesn't sound like a lot..sounds like there's something
> funky going on...you're not spinning up new processes for each request are
> you?
> On Dec 26, 2013 7:51 PM, "Vladimir Ralev" <vl...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I have a script that makes a burst of around 100 very fast PUT requests
> > mixed with another 300 or so GET requests against a 4GB database. 40
> > seconds into the script (30-40 PUTs and the CouchDB starts freezing,
> taking
> > longer and longer to respond until it reaches y timeouts). Is this a
> known
> > problem? Is there something I can do about it.
> >
> > The PUTs update existing documents in the DB and there some
> autocompaction.
> > The documents are less than 1KB. Now I run the script with 30 second
> pauses
> > between the writes and that works fine, but it's too slow.
> >
> > Cheers.
> >
>
Re: Burst PUT updates
Posted by Stanley Iriele <si...@gmail.com>.
What's the hardware... Cocudb version...operating system of your
project...also that doesn't sound like a lot..sounds like there's something
funky going on...you're not spinning up new processes for each request are
you?
On Dec 26, 2013 7:51 PM, "Vladimir Ralev" <vl...@gmail.com> wrote:
> Hi all,
>
> I have a script that makes a burst of around 100 very fast PUT requests
> mixed with another 300 or so GET requests against a 4GB database. 40
> seconds into the script (30-40 PUTs and the CouchDB starts freezing, taking
> longer and longer to respond until it reaches y timeouts). Is this a known
> problem? Is there something I can do about it.
>
> The PUTs update existing documents in the DB and there some autocompaction.
> The documents are less than 1KB. Now I run the script with 30 second pauses
> between the writes and that works fine, but it's too slow.
>
> Cheers.
>