You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "robanstha (via GitHub)" <gi...@apache.org> on 2024/01/24 22:23:36 UTC
[I] Memory spike after running _bulk_docs endpoint with around 2700 docs (~500kb) [couchdb-docker]
robanstha opened a new issue, #252:
URL: https://github.com/apache/couchdb-docker/issues/252
Using couchdb 3.3.3 (Also tested on older versions like 3.0.1, 2.x) in docker container and posting documents to database in using _bulk_docs endpoint and the memory usage in the docker container spikes up from ~1GB to ~6GB. I'm posting ~2700 documents which totals up ~500kb for each 5 databases.
Also, after deleting the database, the memory consumption goes down.
OTP 25 is supposed to have memory leak issue but couchdb 3.3.3 is using OTP 24:
```
> User-Agent: curl/8.4.0
> Accept: */*
> Content-Type: application/json
> Connection: close
> Content-Length: 260959
>
* We are completely uploaded and fine
< HTTP/1.1 201 Created
< Cache-Control: must-revalidate
< Connection: close
< Content-Length: 233056
< Content-Type: application/json
< Date: Wed, 24 Jan 2024 21:09:09 GMT
< Server: CouchDB/3.3.3 (Erlang OTP/24)
< X-Couch-Request-ID: 2b52ff2219
< X-CouchDB-Body-Time: 2
```
Example of all databases and its size created using curl:
Spike in memory consumption after running _bulk_docs of ~500kb (2700 docs) for 5 databases:
## Expected Behavior
500kb of docs should not spike up memory by 1Gigs?
## Current Behavior
Memory spikes up by about 1GB for 500kb of documents posted via _bulk_docs
## Steps to Reproduce (for bugs)
1. Create mock_documents.json with ~2700 documents
2. Create database using curl.
3. POST documents using _bulk_docs.
4. Check memory consumption of the container.
Ex:
```
curl -X PUT http://test:test@localhost:5984/testbuild1
curl -X POST -H "Content-Type: application/json" http://test:test@localhost:5984/testbuild1/_bulk_docs -d @mock_documents.json
```
## Context
We have a process that creates 2700 docs (~500kb total) with new database every day. The high memory consumption increases every day and the container stops and restarts once it reaches the peak.
## Your Environment
Local docker container and kubernetes pods.
* Version used: 3.3.3, 3.2.2, 3.0.1, 2.x
* Browser Name and version: any
* Operating System and version (desktop or mobile): Desktop MacOS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Gradual memory consumption increment (in GBs) after running _bulk_docs endpoint with around 2700 docs (~500kb) with _views for per day in three node cluster [couchdb-docker]
Posted by "robanstha (via GitHub)" <gi...@apache.org>.
robanstha commented on issue #252:
URL: https://github.com/apache/couchdb-docker/issues/252#issuecomment-1919292669
Found a workaround for the issue:
1. Delete the views before deleting the database.
2. Run database compact to remove the deleted views permanently.
3. Delete the database.
This has free up all the memory that were consumed by views in the above case:
From docker stats, you can see that the memory usage was reduced by 1.xGB to 1xxMB. If views were not deleted and db wasn't compacted, it would have stuck in 1.xGB even after deleting database.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Gradual memory consumption increment (in GBs) after running _bulk_docs endpoint with around 2700 docs (~500kb) with _views for per day in three node cluster [couchdb-docker]
Posted by "robanstha (via GitHub)" <gi...@apache.org>.
robanstha closed issue #252: Gradual memory consumption increment (in GBs) after running _bulk_docs endpoint with around 2700 docs (~500kb) with _views for per day in three node cluster
URL: https://github.com/apache/couchdb-docker/issues/252
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org