You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by GitBox <gi...@apache.org> on 2020/04/11 12:06:59 UTC
[GitHub] [openwhisk] ShubhamGupta9582 opened a new issue #4877: couchdb is
restarting itself very frequently
ShubhamGupta9582 opened a new issue #4877: couchdb is restarting itself very frequently
URL: https://github.com/apache/openwhisk/issues/4877
Hi guys,
I'm invoking 18-20 actions in every minute using some cron. After 12-14 hours the openwhisk_db (couchdb) will start restarting itself very frequently and it uses its 100% memory at that time and its cpu will also go to 200% or 300% and its 5984 port (couchdb port) will also become unavailable due to which invoker gives error.
can anyone explain me why it happens? The openwhisk_db container's mem_limit is already 6 GB.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] ShubhamGupta9582 edited a comment on issue #4877:
couchdb is restarting itself very frequently
Posted by GitBox <gi...@apache.org>.
ShubhamGupta9582 edited a comment on issue #4877: couchdb is restarting itself very frequently
URL: https://github.com/apache/openwhisk/issues/4877#issuecomment-612763382
First time when i noticed that couchdb restarted itself. I had stopped my cron which was invoking several actions in a minute. and after few minutes i ran cleanUpActivations.py file to clean the activations and after that ow restarted itself very frequently.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] ShubhamGupta9582 commented on issue #4877: couchdb is
restarting itself very frequently
Posted by GitBox <gi...@apache.org>.
ShubhamGupta9582 commented on issue #4877: couchdb is restarting itself very frequently
URL: https://github.com/apache/openwhisk/issues/4877#issuecomment-614459348
Do i need to change the design docs ? and what does this error means
**[os_mon] cpu supervisor port (cpu_sup): Erlang has closed**
**[os_mon] memory supervisor port (memsup): Erlang has closed**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] style95 commented on issue #4877: couchdb is restarting
itself very frequently
Posted by GitBox <gi...@apache.org>.
style95 commented on issue #4877: couchdb is restarting itself very frequently
URL: https://github.com/apache/openwhisk/issues/4877#issuecomment-614525589
That means the internal components of CouchDB are closed.
Since CouchDB is running on ErlangVM, I think this is because CouchDB is crashed.
What I can infer from the logs is your view request is timed out.
So it responded with the 500 status code.
I think you invoked `wsk activation list` and it triggered the view updates.
Since there are too many documents to update the view, CouchDB might be crashed.
I think what you try is you can periodically trigger view updates before too many documents are piled up. Please see if it happens again with the approach.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] ShubhamGupta9582 removed a comment on issue #4877:
couchdb is restarting itself very frequently
Posted by GitBox <gi...@apache.org>.
ShubhamGupta9582 removed a comment on issue #4877: couchdb is restarting itself very frequently
URL: https://github.com/apache/openwhisk/issues/4877#issuecomment-612763382
First time when i noticed that couchdb restarted itself. I had stopped my cron which was invoking several actions in a minute. and after few minutes i ran cleanUpActivations.py file to clean the activations and after that ow restarted itself very frequently.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] ShubhamGupta9582 commented on issue #4877: couchdb is
restarting itself very frequently
Posted by GitBox <gi...@apache.org>.
ShubhamGupta9582 commented on issue #4877: couchdb is restarting itself very frequently
URL: https://github.com/apache/openwhisk/issues/4877#issuecomment-614459994
Do i need to change the design docs ? and what do these errors mean
**[os_mon] cpu supervisor port (cpu_sup): Erlang has closed**
**[os_mon] memory supervisor port (memsup): Erlang has closed**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] style95 commented on issue #4877: couchdb is restarting
itself very frequently
Posted by GitBox <gi...@apache.org>.
style95 commented on issue #4877: couchdb is restarting itself very frequently
URL: https://github.com/apache/openwhisk/issues/4877#issuecomment-612580728
Please share the logs you found. It would be also great if you share the size of databases in CouchDB.
Did CouchDB crash without any user intervention? or did you run anything such as `wsk activation list`?
It seems CouchDB crashed due to OOM.
I observed a similar symptom when I try to refresh a view results with a high volume of data.
It happens when you've not refreshed your view for a while and some data are piled up in the database and finally, you trigger the view refresh.
Then CouchDB tries to load data and build a view in memory and be crashed eventually.
Since I could observe this situation even with just 10GB data(actually not huge), we are internally using ElasticSearch as an alternative.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] ShubhamGupta9582 removed a comment on issue #4877:
couchdb is restarting itself very frequently
Posted by GitBox <gi...@apache.org>.
ShubhamGupta9582 removed a comment on issue #4877: couchdb is restarting itself very frequently
URL: https://github.com/apache/openwhisk/issues/4877#issuecomment-614459348
Do i need to change the design docs ? and what does this error means
**[os_mon] cpu supervisor port (cpu_sup): Erlang has closed**
**[os_mon] memory supervisor port (memsup): Erlang has closed**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] ShubhamGupta9582 commented on issue #4877: couchdb is
restarting itself very frequently
Posted by GitBox <gi...@apache.org>.
ShubhamGupta9582 commented on issue #4877: couchdb is restarting itself very frequently
URL: https://github.com/apache/openwhisk/issues/4877#issuecomment-613318598
I didn't make any changes for design docs. So, it should be as it is as it was earlier.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] ShubhamGupta9582 commented on issue #4877: couchdb is
restarting itself very frequently
Posted by GitBox <gi...@apache.org>.
ShubhamGupta9582 commented on issue #4877: couchdb is restarting itself very frequently
URL: https://github.com/apache/openwhisk/issues/4877#issuecomment-612791942
First time when i noticed that couchdb restarted itself. I had stopped my cron which was invoking several actions in a minute. and after few minutes i ran cleanUpActivations.py file to clean the activations and after that ow restarted itself very frequently.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] style95 commented on issue #4877: couchdb is restarting
itself very frequently
Posted by GitBox <gi...@apache.org>.
style95 commented on issue #4877: couchdb is restarting itself very frequently
URL: https://github.com/apache/openwhisk/issues/4877#issuecomment-612826061
It seems it is looking for the design docs.
Do all DB have proper design docs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] ShubhamGupta9582 edited a comment on issue #4877:
couchdb is restarting itself very frequently
Posted by GitBox <gi...@apache.org>.
ShubhamGupta9582 edited a comment on issue #4877: couchdb is restarting itself very frequently
URL: https://github.com/apache/openwhisk/issues/4877#issuecomment-612763382
I ran cleanUpActivations.py file to clean the activations and after that ow restarted itself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] ShubhamGupta9582 commented on issue #4877: couchdb is
restarting itself very frequently
Posted by GitBox <gi...@apache.org>.
ShubhamGupta9582 commented on issue #4877: couchdb is restarting itself very frequently
URL: https://github.com/apache/openwhisk/issues/4877#issuecomment-612763382
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] style95 edited a comment on issue #4877: couchdb is
restarting itself very frequently
Posted by GitBox <gi...@apache.org>.
style95 edited a comment on issue #4877: couchdb is restarting itself very frequently
URL: https://github.com/apache/openwhisk/issues/4877#issuecomment-614526102
And for the record, if it is because of view updates, this is a CouchDB issue rather than OpenWhisk's one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] style95 commented on issue #4877: couchdb is restarting
itself very frequently
Posted by GitBox <gi...@apache.org>.
style95 commented on issue #4877: couchdb is restarting itself very frequently
URL: https://github.com/apache/openwhisk/issues/4877#issuecomment-614526102
And for the record, If it is because of view updates, this is a CouchDB issue rather than OpenWhisk's one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services