You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2020/01/10 17:42:02 UTC
[GitHub] [couchdb] arifcse019 opened a new issue #2441: Memory Usage
Increases Gradually
arifcse019 opened a new issue #2441: Memory Usage Increases Gradually
URL: https://github.com/apache/couchdb/issues/2441
[NOTE]: # ( ^^ Provide a general summary of the issue in the title above. ^^ )
## Description
[NOTE]: # ( Describe the problem you're encountering. )
[TIP]: # ( Do NOT give us access or passwords to your actual CouchDB! )
Memory usage of our CouchDB Cluster keeps increasing week by week until we have to restart it to release the memory. There is no attachment or conflict (verified programmatically using views) or big document (our biggest document size is ~27 KB)
## Steps to Reproduce
- Running the cluster for a few weeks, you would see the memory usage increasing week by week.
## Expected Behaviour
Memory usage remains relatively stable/flat
## Your Environment
[TIP]: # ( Include as many relevant details about your environment as possible. )
[TIP]: # ( You can paste the output of curl http://YOUR-COUCHDB:5984/ here. )
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"4ebb08d132299ae512bd001e673fc7fa","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
* CouchDB version used: 2.3.1
* 3 Nodes Cluster, n=3, q=9
* ~15 thousand databases, total data size in the disk is ~11G in each of the three nodes.
* Browser name and version: Not Applicable
* Operating system and version: Ubuntu 14.04
## Additional Context
- Among the CouchDB memory components, it is the "processes" which is gradually increasing as can be seen in the diagram below (the dip is when we restarted it last time):
- I tried to find individual processes' memory usage, but the total Heap+Stack from `i().` =~ 40MB does not match up with the processes component of `erlang:memory().` =~ 2.7GB
Following are the commands I tried
`root@ip-10-0-3-10:~# erl -name test -remsh couchdb@first.couchcluster.internal -setcookie redacted`
> Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:2:2] [async-threads:10] [kernel-poll:false]
> Eshell V8.3.5 (abort with ^G)
`(couchdb@first.couchcluster.internal)1> erlang:memory().`
> [{total,2834080280},
> {processes,2709523496},
> {processes_used,2709484152},
> {system,124556784},
> {atom,512625},
> {atom_used,505104},
> {binary,1510536},
> {code,11770466},
> {ets,103666808}]
`(couchdb@first.couchcluster.internal)2> i().`
> Pid Initial Call Heap Reds Msgs Registered Current Function Stack
> <0.0.0> otp_ring0:start/2 1598 4032 0 init init:loop/1 2
>
> .
> .
> .
>
> Total 40590405 31290642 0 10320
- We upgraded the cluster in-place from 2.1.2 to 2.3.1 a few months ago. We did not have this memory usage issue/trend on 2.1.2. Not sure if it is related or not since we upgraded, we are seeing this error regularly: https://github.com/apache/couchdb/issues/1600#issuecomment-523982028
- Not sure about its relevance, but the following are two design documents we have and use in all of our 15 thousand databases:
`{"_id":"_design/permissions","_rev":"3-e99851a9513d6e9c973a598d69523645","filters":{"user_documents_to_account":"function(doc, req) {\n if (doc._id.indexOf('_') === 0) return false;\n if (!req.query.account_id || !doc.accountId) return false;\n if (req.query.account_id != doc.accountId) return false;\n if (req.query.roles.indexOf('account_writer') !== -1) return true;\n if (doc._id.indexOf('account' === 0) && req.query.roles.indexOf('account_info_writer') !== -1) return true;\n return false;\n }"}}`
`{"_id":"_design/logDocs","_rev":"1-e64774d95fb1fc9e9c20842eb38edffd","validate_doc_update":"function (newDoc, oldDoc, userCtx){\n var logData = JSON.stringify({newDoc: newDoc, oldDoc: oldDoc, userCtx: userCtx})\n log('[ATVENU] ' + logData);\n }"}`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] noahbailey commented on issue #2441: Memory Usage Keeps
Increasing Gradually
Posted by GitBox <gi...@apache.org>.
noahbailey commented on issue #2441: Memory Usage Keeps Increasing Gradually
URL: https://github.com/apache/couchdb/issues/2441#issuecomment-597722282
@pseudooriginal My team hasn't migrated to v3.x yet, so it could be some time before we get good data on that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] pseudooriginal commented on issue #2441: Memory Usage
Keeps Increasing Gradually
Posted by GitBox <gi...@apache.org>.
pseudooriginal commented on issue #2441: Memory Usage Keeps Increasing Gradually
URL: https://github.com/apache/couchdb/issues/2441#issuecomment-597665350
Same problem here on a single node installation. The memory usage is increasing and the server crashes. CouchDB 2.3.1. Does v3.0.0 resolve this issue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] toumbask commented on issue #2441: Memory Usage Keeps
Increasing Gradually
Posted by GitBox <gi...@apache.org>.
toumbask commented on issue #2441: Memory Usage Keeps Increasing Gradually
URL: https://github.com/apache/couchdb/issues/2441#issuecomment-575093353
Also facing the same issue, on Single Node installation , though the increase span is much faster (within 2-4 days, RAM usage jumps to over 8GB and eventually the server crashes. DB usage is minimal, with only 2 DBs and minimal data (<10MB overall). Using latest docker built with version 2.3.1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] noahbailey commented on issue #2441: Memory Usage Keeps
Increasing Gradually
Posted by GitBox <gi...@apache.org>.
noahbailey commented on issue #2441: Memory Usage Keeps Increasing Gradually
URL: https://github.com/apache/couchdb/issues/2441#issuecomment-579463531
I have a very similar issue, also on 2.3.1 but on Ubuntu 16.04 and 18.04 virtual machines.
The memory usage grows for several weeks before it crashes for about 3 seconds and restarts the service. It's completely recoverable (at least for us) and doesn't cause any sort of outage but it is concerning.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [couchdb] sklassen commented on issue #2441: Memory Usage Keeps
Increasing Gradually
Posted by GitBox <gi...@apache.org>.
sklassen commented on issue #2441: Memory Usage Keeps Increasing Gradually
URL: https://github.com/apache/couchdb/issues/2441#issuecomment-606720618
I had recurring memory problems that I fixed by upgrading the version of jiffy (see https://github.com/davisp/jiffy/commit/0ba322e42171bb48ffdd0c053cef28a42eb35fc9)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services