You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2022/12/10 07:34:33 UTC
[GitHub] [couchdb] zdravko123 opened a new issue, #4300: Couch 3.1 restarts on all nodes but only 2 come back online sportically
zdravko123 opened a new issue, #4300:
URL: https://github.com/apache/couchdb/issues/4300
[NOTE]: # ( ^^ Provide a general summary of the issue in the title above. ^^ )
## Description
We are having issues with all our Production CloudDB clusters running Ubuntu 18.04
we have about 14 clusters of 3 nodes each so a total of 52 hosted in Azure.
Since last Friday we saw that the nodes were falling in traffic & CPU (suspect losing connections or something was restarting) then they would ramp up but only 2 out of the 3 nodes would rejoin the cluster.
Although we saw sporadic issues in the days leading up to it but also a few months prior but they were recovering faster without outages.
We suspect something has changed recently but are unsure what exactly, we did note a new kernel was installed a few weeks prior.
Needless to say this is causing great hardship to the team with a person monitoring the servers 24/7 and manually restarting them.
Problem start time id say 7:30Pm on last Friday the 2nd as no one was around to restart them, it lasted a good hour.
[NOTE]: # ( Describe the problem you're encountering. )
[TIP]: # ( Do NOT give us access or passwords to your actual CouchDB! )
## Steps to Reproduce
[NOTE]: # ( Include commands to reproduce, if possible. curl is preferred. )
## Expected Behaviour
[NOTE]: # ( Tell us what you expected to happen. )
## Your Environment
[TIP]: # ( Include as many relevant details about your environment as possible. )
[TIP]: # ( You can paste the output of curl http://YOUR-COUCHDB:5984/ here. )
* CouchDB version used:
* Browser name and version:
* Operating system and version:
## Additional Context
[TIP]: # ( Add any other context about the problem here. )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] big-r81 commented on issue #4300: Couch 3.1 restarts on all nodes but only 2 come back online sportically
Posted by GitBox <gi...@apache.org>.
big-r81 commented on issue #4300:
URL: https://github.com/apache/couchdb/issues/4300#issuecomment-1346163617
Closing this for now, if happen again, please re-open this and put a paste from your logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] zdravko123 commented on issue #4300: Couch 3.1 restarts on all nodes but only 2 come back online sportically
Posted by GitBox <gi...@apache.org>.
zdravko123 commented on issue #4300:
URL: https://github.com/apache/couchdb/issues/4300#issuecomment-1345185105
image
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] zdravko123 commented on issue #4300: Couch 3.1 restarts on all nodes but only 2 come back online sportically
Posted by GitBox <gi...@apache.org>.
zdravko123 commented on issue #4300:
URL: https://github.com/apache/couchdb/issues/4300#issuecomment-1345689769
Question 1: What was the cause of instability?
Answer: Bad Ubuntu kernel (Version: 5.4.0-1095.101~18.04.1)
Question 2: Why it all started Friday, 2nd Dec?
Answer: This is when we started upsizing VMs in anticipation of holidays load
Question 3: How did the instability crawl in?
Answer: The bad kernel was downloaded around 18th of Nov. However Ubuntu requires VM to reboot for the new kernel to kick in. That reboot happened on 2nd, 3rd and 4th of Dec (Fri, Sat, Sun)
Question 3: What was the fix?
Answer: Ubuntu issued a fixed kernel 9th Dec (Friday). VMs were patched manually (with exception of NZ and MISC clusters ) between 3 am and 4:30 am today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] zdravko123 commented on issue #4300: Couch 3.1 restarts on all nodes but only 2 come back online sportically
Posted by GitBox <gi...@apache.org>.
zdravko123 commented on issue #4300:
URL: https://github.com/apache/couchdb/issues/4300#issuecomment-1345185023
here is a screen shot of some more metrics, notice the network sockets increase until
![Uploading 2022-12-10 19_29_34-Node Exporter (AU-WEST-02) - Dashboards - Grafana.png…]()
it maxes out at 5000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] zdravko123 commented on issue #4300: Couch 3.1 restarts on all nodes but only 2 come back online sportically
Posted by GitBox <gi...@apache.org>.
zdravko123 commented on issue #4300:
URL: https://github.com/apache/couchdb/issues/4300#issuecomment-1345690031
https://www.ubuntuupdates.org/package/core/bionic/main/updates/linux-azure-5.4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] big-r81 commented on issue #4300: Couch 3.1 restarts on all nodes but only 2 come back online sportically
Posted by GitBox <gi...@apache.org>.
big-r81 commented on issue #4300:
URL: https://github.com/apache/couchdb/issues/4300#issuecomment-1345185700
Any interesting in your log files, when the issues are rising?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] big-r81 closed issue #4300: Couch 3.1 restarts on all nodes but only 2 come back online sportically
Posted by GitBox <gi...@apache.org>.
big-r81 closed issue #4300: Couch 3.1 restarts on all nodes but only 2 come back online sportically
URL: https://github.com/apache/couchdb/issues/4300
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] zdravko123 commented on issue #4300: Couch 3.1 restarts on all nodes but only 2 come back online sportically
Posted by GitBox <gi...@apache.org>.
zdravko123 commented on issue #4300:
URL: https://github.com/apache/couchdb/issues/4300#issuecomment-1345689610
we suspect that the new kernel was buggy, we doubled the size of the machines a few days prior or when it started happening, we believe this kicked in the new kernal install and then we had issues. There was a new one that fixed some socket/tcp race conditions Friday we installed this on all the clusters and it looks like the issue has gone away.. luckily.... So i put it down to OS Kernal update for now, or Santa.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org