You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "sergey-safarov (via GitHub)" <gi...@apache.org> on 2024/02/20 22:34:15 UTC
[I] couchspawnkillable: Cannot allocate memory [couchdb]
sergey-safarov opened a new issue, #4988:
URL: https://github.com/apache/couchdb/issues/4988
## Description
I use `apache/couchdb:3.3.2` docker image on my server.
Today couchdb process crashed because no free memory on the server.
CouchDB logs contians
> -- Logs begin at Tue 2024-01-09 16:32:03 UTC, end at Tue 2024-02-20 22:09:12 UTC. --
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: sh: 1: exec: /opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memory
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: sh: 1: exec: sh: 1: exec: /opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memory/opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memorysh: 1: exec: sh: 1: exec: /opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memory
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: /opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memory
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: ^M
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: sh: 1: exec: sh: 1: exec: /opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memory
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: sh: 1: exec:
Feb 20 20:49:59 ippbx-0a.example.com docker[999209]: /opt/couchdb/bin/../lib/couch-3.3.2/priv/couchspawnkillable: Cannot allocate memory
Feb 20 20:50:07 ippbx-0a.example.com docker[999209]: out of memory
Feb 20 21:06:23 ippbx-0a.example.com docker[999209]: out of memory
Feb 20 21:10:08 ippbx-0a.example.com docker[999209]: eheap_alloc: Cannot allocate 71302344 bytes of memory (of type "heap").
Feb 20 21:10:08 ippbx-0a.example.com docker[999209]: ^M^M
Feb 20 21:19:07 ippbx-0a.example.com docker[999209]: Crash dump is being written to: erl_crash.dump...
Feb 20 21:19:07 ippbx-0a.example.com systemd[1]: couchdb-docker.service: Main process exited, code=exited, status=137/n/a
Feb 20 21:19:07 ippbx-0a.example.com systemd[1]: couchdb-docker.service: Failed with result 'exit-code'.
Feb 20 21:19:07 ippbx-0a.example.com systemd[1]: couchdb-docker.service: Service RestartSec=100ms expired, scheduling restart.
Feb 20 21:19:07 ippbx-0a.example.com systemd[1]: couchdb-docker.service: Scheduled restart job, restart counter is at 2.
Feb 20 21:19:07 ippbx-0a.example.com systemd[1]: Stopped CouchDB container.
## Steps to Reproduce
Do not know
## Expected Behaviour
All external processes should not lead to all server memory usage.
## Your Environment
* CouchDB version used: 3.3.2
* Browser name and version: do not used
* Operating system and version: CentOS 8 Stream, aarch64
## Additional Context
Here is graph for swap memory usage on the server. Swap usage start grow about one week ago
And the disk cache started shrinking about 8 days ago
A related ticket found at https://issues.apache.org/jira/browse/COUCHDB-371 and looks like issue is not solved and closed with comment
> If this issue remains in CouchDB 2.x, please file a new issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] couchspawnkillable: Cannot allocate memory [couchdb]
Posted by "sergey-safarov (via GitHub)" <gi...@apache.org>.
sergey-safarov commented on issue #4988:
URL: https://github.com/apache/couchdb/issues/4988#issuecomment-1955248793
Before the issue was used 170Mb of swap memory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] couchspawnkillable: Cannot allocate memory [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #4988:
URL: https://github.com/apache/couchdb/issues/4988#issuecomment-1955460577
I don't typically run CouchDB in docker container so this all speculative advice.
A few things to try perhaps:
* Try adjusting couchjs process numbers their number with https://docs.couchdb.org/en/stable/config/query-servers.html#query_server_config/os_process_limit. Make the hard limit say 75 and soft limit 25. The default is 100 I think.
* Try lowering [max_dbs_open](https://docs.couchdb.org/en/stable/maintenance/performance.html#max-dbs-open). The default is 500 so maybe try making it 400 or 300.
* When running in docker, Erlang VM doesn't always do a good job auto-detecting the CPU quotas so start a scheduler thread for each core on the host machine. So try lowering the number of schedulers artificially in `vm.args` with `+S 4:4`, for example to use only 4 schedulers, that could help with memory usage a bit as well.
* Erlang VM has more advanced memory management settings. I haven't tried most of them. There is one to use a max amount of heap, so if you have a way to test it out on a test cluster first, try something like:
vm.args
```
+MMscs 4096
+Musac false
+MMsco true
```
That sets the super-carrier to 4GB max. Again, I haven't tried running with those settings but someone reported some success with that: https://elixirforum.com/t/running-elixir-with-limited-heap-size/23633/13
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org