You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2020/07/30 07:29:12 UTC
[GitHub] [couchdb] ngspinu opened a new issue #3047: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os
ngspinu opened a new issue #3047:
URL: https://github.com/apache/couchdb/issues/3047
[NOTE]: # ( ^^ Provide a general summary of the issue in the title above. ^^ )
## Description
Hi there We have recently upgraded one of our production environments to 3.1.0. Prior to that we've had the same issue while running couchdb 3.1.0 on a Mac-os machine.
Basically what happens is: as soon you start couch and access some data it starts indexing(see attached fauxton screenshot)
 and never stops consuming 100% of the CPU.
While doing some reading out there, we found out that deleting old(2.*) .compact files would make things chill down - but actually this was not the case.
We ended up by setting `autoupdate: false` to all our _design/docs. However, we are looking for a real fix to this one. Anybody can give a hand?
## Steps to Reproduce
1. Upgrade couch from 2.3.1 to 3.1.0
2. Start couch
3. Access some view
## Your Environment
Mac-os, and Cent-os7
[TIP]: # ( Include as many relevant details about your environment as possible. )
[TIP]: # ( You can paste the output of curl http://YOUR-COUCHDB:5984/ here. )
* CouchDB version used: 3.1.0
* Browser name and version:
* Operating system and version:
## Additional Context
[TIP]: # ( Add any other context about the problem here. )
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] ngspinu commented on issue #3047: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os
Posted by GitBox <gi...@apache.org>.
ngspinu commented on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-667868676
@wohali We have tried your suggestion IE: Increasing the 16MB to way more as our dbs are pretty big: ~1GB. Ratio too.
Is there anything else we can try? We have around 35 environments that we need to upgrade to 3.* and we can't do it until we understand what is causing the continuous indexing/unorthodox CPU consumption.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] ngspinu edited a comment on issue #3047: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os
Posted by GitBox <gi...@apache.org>.
ngspinu edited a comment on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-667868676
@wohali thank you for the hints. We have tried your suggestion IE: Increasing the 16MB to way more as our dbs are pretty big: ~1GB. Ratio too. Sadly - it still keeps indexing :(.
Is there anything else we can try? We have around 35 environments that we need to upgrade to 3.* and we can't do it until we understand what is causing the continuous indexing/unorthodox CPU consumption.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] wohali edited a comment on issue #3047: Increase slack_db min priority (was: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os)
Posted by GitBox <gi...@apache.org>.
wohali edited a comment on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-854230247
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] ngspinu edited a comment on issue #3047: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os
Posted by GitBox <gi...@apache.org>.
ngspinu edited a comment on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-665916311
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] molecular edited a comment on issue #3047: Increase slack_db min priority (was: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os)
Posted by GitBox <gi...@apache.org>.
molecular edited a comment on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-803536567
I ran into this (100% cpu 1-2 minutes after couchdb start). It was a view compaction being constantly triggered on a ~5 million document db that gets new document(s) every 10 seconds.
I copied the defaults from https://docs.couchdb.org/en/latest/maintenance/compaction.html#channel-configuration and that resulted in much less frequent view compaction being triggered.
So those "defaults2 are likely not the defaults on my system (archlinux, couchdb 3.1.1-2)
I also learned how to interpret db metadata to evaluate what's a good treshold priority
```sh
curl -s -u admin:<pw> -X GET http://blackbox:5984/<db> | jq '(.sizes.file - .sizes.active) / 1E6'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] molecular edited a comment on issue #3047: Increase slack_db min priority (was: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os)
Posted by GitBox <gi...@apache.org>.
molecular edited a comment on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-803536567
I ran into this (100% cpu 1-2 minutes after couchdb start, never stopped).
Reason was a view compaction being constantly triggered on a ~5 million document db that gets new document(s) every 10 seconds.
I copied the defaults from https://docs.couchdb.org/en/latest/maintenance/compaction.html#channel-configuration and that resulted in much less frequent view compaction being triggered.
So those "defaults2 are likely not the defaults on my system (archlinux, couchdb 3.1.1-2)
I also learned how to interpret db metadata to evaluate what's a good treshold priority
```sh
#> curl -s -u admin:<pw> -X GET http://blackbox:5984/<db> | jq '(.sizes.file - .sizes.active) / 1E6'
62.852902
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] wohali commented on issue #3047: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os
Posted by GitBox <gi...@apache.org>.
wohali commented on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-666029699
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] ngspinu edited a comment on issue #3047: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os
Posted by GitBox <gi...@apache.org>.
ngspinu edited a comment on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-667868676
@wohali thank you for the hints. We have tried your suggestion IE: Increasing the 16MB to way more as our dbs are pretty big: ~1GB. Ratio too.
Is there anything else we can try? We have around 35 environments that we need to upgrade to 3.* and we can't do it until we understand what is causing the continuous indexing/unorthodox CPU consumption.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] molecular edited a comment on issue #3047: Increase slack_db min priority (was: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os)
Posted by GitBox <gi...@apache.org>.
molecular edited a comment on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-803536567
I ran into this (100% cpu 1-2 minutes after couchdb start, never stopped).
Reason was a view compaction being triggered (and repeatedly re-triggered right after it finished) on a ~5 million document db that gets new document(s) every 10 seconds.
I copied the defaults from https://docs.couchdb.org/en/latest/maintenance/compaction.html#channel-configuration and that resulted is much less frequent view compaction being triggered.
So those "defaults2 are likely not the defaults on my system (archlinux, couchdb 3.1.1-2)
I also learned how to interpret db metadata to evaluate what's a good treshold priority
```sh
#> curl -s -u admin:<pw> -X GET http://blackbox:5984/<db> | jq '(.sizes.file - .sizes.active) / 1E6'
62.852902
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] wohali edited a comment on issue #3047: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os
Posted by GitBox <gi...@apache.org>.
wohali edited a comment on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-666030056
@rnewson This has come up here and in #3019. I think we should change the default. What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] molecular edited a comment on issue #3047: Increase slack_db min priority (was: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os)
Posted by GitBox <gi...@apache.org>.
molecular edited a comment on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-803536567
I ran into this (100% cpu 1-2 minutes after couchdb start). It was a view compaction being constantly triggered on a ~5 million document db that gets new document(s) every 10 seconds.
I copied the defaults from https://docs.couchdb.org/en/latest/maintenance/compaction.html#channel-configuration and that resulted in much less frequent view compaction being triggered.
So those "defaults2 are likely not the defaults on my system (archlinux, couchdb 3.1.1-2)
I also learned how to interpret db metadata to evaluate what's a good treshold priority
```sh
#> curl -s -u admin:<pw> -X GET http://blackbox:5984/<db> | jq '(.sizes.file - .sizes.active) / 1E6'
62.852902
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] rjharmon commented on issue #3047: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os
Posted by GitBox <gi...@apache.org>.
rjharmon commented on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-665908783
I'm guessing you may have many design docs that aren't being actively used by your application, and the autoupdate traffic on those is causing the undesired load? Maybe some of them are simply obsolete?
Did you consider setting autoupdate: true on one design-doc at a time, for those design docs whose indexes you KNOW are going to be taking traffic? The intent of the autoupdate behavior, as I understand it, is to shift indexing workload away from time-of-need and to execute it earlier - so it's ready when your application does a query.
Administratively, setting autoupdate: true gradually for all your less-used indexes, and removing any obsolete design docs, could get you caught up and cleaned up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] seigel commented on issue #3047: Increase slack_db min priority (was: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os)
Posted by GitBox <gi...@apache.org>.
seigel commented on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-854096423
Any updates on this topic, just wondering if for @molecular or the author @ngspinu , if this ever resolved?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] wohali edited a comment on issue #3047: Increase slack_db min priority (was: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os)
Posted by GitBox <gi...@apache.org>.
wohali edited a comment on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-854230247
Based on info from the above, we're going to bump the `slack_dbs` and `slack_views` defaults to 512MB, but leave the ratio at 2.0. If people have evidence to suggest other appropriate defaults, let us know.
Related PRs:
*
* https://github.com/apache/couchdb-documentation/pull/663
The problem is that any default setting here will be too aggressive for some users, and too lenient for others.
A potential future smoosh 2.0 could autotune in response to remaining disk space and current foreground load.
A potential future smooth 3.0 would simply calculate the future state of the universe and then change these values to avoid any bad outcomes.
Towards either improvement... "pull requests welcome."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] wohali commented on issue #3047: Increase slack_db min priority (was: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os)
Posted by GitBox <gi...@apache.org>.
wohali commented on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-854230247
We're going to bump the slack_dbs/slack_views values to 512MB, but leave the ratio at 2.0. If people have evidence to suggest other appropriate defaults, let us know. The problem is that any setting here will be too aggressive for some users, and too lenient for others.
A potential future smoosh 2.0 could autotune in response to remaining disk space and current foreground load.
A potential future smooth 3.0 would simply calculate the future state of the universe and then change these values to avoid any bad outcomes.
Towards either improvement... "pull requests welcome."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] bdoyle0182 commented on issue #3047: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os
Posted by GitBox <gi...@apache.org>.
bdoyle0182 commented on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-737589956
I'm a little confused on what gets shipped by default. So you have the ratio channel and the slack channel. Are they both used by default or is one prioritized? If the slack default is 16mb, when would the ratio channel ever even get used? I think this should change it's pretty scary for people upgrading from 2.x and most people have much larger db's than this so they're treated to constant compaction
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] wohali commented on issue #3047: Increase slack_db min priority (was: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os)
Posted by GitBox <gi...@apache.org>.
wohali commented on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-854230247
We're going to bump the slack_dbs/slack_views values to 512MB, but leave the ratio at 2.0. If people have evidence to suggest other appropriate defaults, let us know. The problem is that any setting here will be too aggressive for some users, and too lenient for others.
A potential future smoosh 2.0 could autotune in response to remaining disk space and current foreground load.
A potential future smooth 3.0 would simply calculate the future state of the universe and then change these values to avoid any bad outcomes.
Towards either improvement... "pull requests welcome."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] molecular edited a comment on issue #3047: Increase slack_db min priority (was: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os)
Posted by GitBox <gi...@apache.org>.
molecular edited a comment on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-803536567
I ran into this (100% cpu 1-2 minutes after couchdb start, never stopped).
Reason was a view compaction being triggered (and repeatedly re-triggered right after it finished) on a ~5 million document db that gets new document(s) every 10 seconds.
I copied the defaults from https://docs.couchdb.org/en/latest/maintenance/compaction.html#channel-configuration and that resulted in much less frequent view compaction being triggered.
So those "defaults2 are likely not the defaults on my system (archlinux, couchdb 3.1.1-2)
I also learned how to interpret db metadata to evaluate what's a good treshold priority
```sh
#> curl -s -u admin:<pw> -X GET http://blackbox:5984/<db> | jq '(.sizes.file - .sizes.active) / 1E6'
62.852902
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] molecular commented on issue #3047: Increase slack_db min priority (was: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os)
Posted by GitBox <gi...@apache.org>.
molecular commented on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-803536567
I ran into this (100% cpu 1-2 minutes after couchdb start). It was a view compaction being constantly triggered on a ~5 million document db that gets new document(s) every 10 seconds.
I copied the defaults from https://docs.couchdb.org/en/latest/maintenance/compaction.html#channel-configuration and that resulted in much less frequent view compaction being triggered.
So those "defaults2 are likely not the defaults on my system (archlinux, couchdb 3.1.1-2)
I also learned how to interpret db metadata to evaluate what's a good treshold priority
```sh
curl -s -u admin:<pw> -X GET http://blackbox:5984/solar | jq '(.sizes.file - .sizes.active) / 1E6'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] wohali commented on issue #3047: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os
Posted by GitBox <gi...@apache.org>.
wohali commented on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-667958679
Increase them more. If all you're seeing is forever running db and view compactions, smoosh is still being triggered too fast. Use the output of `GET /{db}` to determine the appropriate ratio for your needs. Or you can just disable certain DBs if they cause you trouble.
If what you're seeing is index _builds_ (not compactions) then `ken` is your issue, and you'll need to use configs to disable ken from building all your views. That's also explained in the documentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] seigel commented on issue #3047: Increase slack_db min priority (was: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os)
Posted by GitBox <gi...@apache.org>.
seigel commented on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-854096423
Any updates on this topic, just wondering if for @molecular or the author @ngspinu , if this ever resolved?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] ngspinu commented on issue #3047: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os
Posted by GitBox <gi...@apache.org>.
ngspinu commented on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-665916311
@rjharmon thanks for hints. We have indeed removed all outdated/obsolete index and also compacted the views. We have quite a few views that are required so that the app can work, so whatever we would be doing - we still endup with a bunch of non-stop indexing views, that will eventually take our whole CPU.
As for `autoupdate: true` - that is indeed used to keep the indexes warm, but I reckon.. there must be a middle way to keep things in line: IE: No 100%
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] wohali edited a comment on issue #3047: Increase slack_db min priority (was: Upgrade from 2.3.1 to 3.1.0 - leads to 100% CPU Usage on both CentOs 7 and Mac-os)
Posted by GitBox <gi...@apache.org>.
wohali edited a comment on issue #3047:
URL: https://github.com/apache/couchdb/issues/3047#issuecomment-854230247
Based on info from the above, we're going to bump the `slack_dbs` and `slack_views` defaults to 512MB, but leave the ratio at 2.0. If people have evidence to suggest other appropriate defaults, let us know.
Related PRs:
* https://github.com/apache/couchdb/pull/3600
* https://github.com/apache/couchdb-documentation/pull/663
The problem is that any default setting here will be too aggressive for some users, and too lenient for others.
A potential future smoosh 2.0 could autotune in response to remaining disk space and current foreground load.
A potential future smooth 3.0 would simply calculate the future state of the universe and then change these values to avoid any bad outcomes.
Towards either improvement... "pull requests welcome."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org