You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2020/03/31 15:39:35 UTC
[GitHub] [couchdb] wolfman1956 opened a new issue #2736: Couchdb -
intermittent fabric timeout errors
wolfman1956 opened a new issue #2736: Couchdb - intermittent fabric timeout errors
URL: https://github.com/apache/couchdb/issues/2736
**_## Description_**
When running a specific string using the following command string:
sup crossbar_maintenance init_apps /var/www/html/monster-ui/apps http://our_1st_master01.kazoo.our_domain.ca:8000/v2
..the script will load flawlessly on average, 3 out of 5 times with no errors, but will have errors the 2 other times (timeouts) but will also eventually finish with an "OK" status, taking longer than the usual 100% successful load of between 5-6 seconds. Depending on the # of timeouts encountered, it can take up to 1m30s, but as mentioned will eventually load.
We have looked into most if not all timeout related parameters in the default.ini file, tested and still get same results with intermittent fabric_timeout errors in the couchdb log.
Sample of timeout errors (they will vary from "init app" initialization:
sup crossbar_maintenance init_apps /var/www/html/monster-ui/apps http://mainserverkazookapps01.kazoo.ourprodbox.ca:8000/v2
trying to init app from /var/www/html/monster-ui/apps/pbxs
app pbxs already loaded in system
not updating api_url, it is unchanged
no metadata changes for app 6f12f101f866c972d04a43f960ff41b7
failed to save PBXconnector_app.png to 6f12f101f866c972d04a43f960ff41b7: timeout
saved pbxconnector1.png to 6f12f101f866c972d04a43f960ff41b7
saved pbxconnector2.png to 6f12f101f866c972d04a43f960ff41b7
trying to init app from /var/www/html/monster-ui/apps/webhooks
app webhooks already loaded in system
not updating api_url, it is unchanged
no metadata changes for app e6e5774256edd2201f838865baa25ef3
saved WebHooks_app.png to e6e5774256edd2201f838865baa25ef3
saved webhooks1.png to e6e5774256edd2201f838865baa25ef3
saved webhooks2.png to e6e5774256edd2201f838865baa25ef3
trying to init app from /var/www/html/monster-ui/apps/fax
app fax already loaded in system
not updating api_url, it is unchanged
no metadata changes for app e21e143c33312aa3495558d7936242ec
failed to save Fax_app.png to e21e143c33312aa3495558d7936242ec: timeout
saved OutboundFaxes.png to e21e143c33312aa3495558d7936242ec
trying to init app from /var/www/html/monster-ui/apps/csv-onboarding
app csv-onboarding already loaded in system
not updating api_url, it is unchanged
no metadata changes for app bdca179bf1754c18f5d739dc91c859b6
failed to find icon in bdca179bf1754c18f5d739dc91c859b6
failed to find screenshots in bdca179bf1754c18f5d739dc91c859b6
trying to init app from /var/www/html/monster-ui/apps/accounts
app accounts already loaded in system
not updating api_url, it is unchanged
no metadata changes for app 1b7093aa6cbf7b50708027d03a106a12
saved Accounts_app.png to 1b7093aa6cbf7b50708027d03a106a12
saved Account-AvailableApps.png to 1b7093aa6cbf7b50708027d03a106a12
saved Account-Limits.png to 1b7093aa6cbf7b50708027d03a106a12
saved AccountOverview.png to 1b7093aa6cbf7b50708027d03a106a12
trying to init app from /var/www/html/monster-ui/apps/callflows
app callflows already loaded in system
not updating api_url, it is unchanged
no metadata changes for app 184e7c891e2b9c64e8f4e8eec5b00a7b
saved Callflows_app.png to 184e7c891e2b9c64e8f4e8eec5b00a7b
saved callflows_1.png to 184e7c891e2b9c64e8f4e8eec5b00a7b
saved callflows_2.png to 184e7c891e2b9c64e8f4e8eec5b00a7b
saved callflows_3.png to 184e7c891e2b9c64e8f4e8eec5b00a7b
trying to init app from /var/www/html/monster-ui/apps/voip
app voip already loaded in system
not updating api_url, it is unchanged
no metadata changes for app e9cc159ebcb93062e51c5f38604ec77f
saved SmartPBX_app.png to e9cc159ebcb93062e51c5f38604ec77f
saved smartpbx1.png to e9cc159ebcb93062e51c5f38604ec77f
saved smartpbx2.png to e9cc159ebcb93062e51c5f38604ec77f
saved smartpbx3.png to e9cc159ebcb93062e51c5f38604ec77f
saved smartpbx4.png to e9cc159ebcb93062e51c5f38604ec77f
saved smartpbx5.png to e9cc159ebcb93062e51c5f38604ec77f
trying to init app from /var/www/html/monster-ui/apps/numbers
app numbers already loaded in system
not updating api_url, it is unchanged
no metadata changes for app 8637570ad7b41bc7f3906882507094db
saved Numbers_app.png to 8637570ad7b41bc7f3906882507094db
saved numbers1.png to 8637570ad7b41bc7f3906882507094db
saved numbers2.png to 8637570ad7b41bc7f3906882507094db
trying to init app from /var/www/html/monster-ui/apps/voicemails
app voicemails already loaded in system
not updating api_url, it is unchanged
no metadata changes for app 48630c8173c977fdc460ae93e96010ec
saved Voicemail_app.png to 48630c8173c977fdc460ae93e96010ec
saved PlayVoicemail.png to 48630c8173c977fdc460ae93e96010ec
saved SelectedVoicemails.png to 48630c8173c977fdc460ae93e96010ec
ok
_**## Steps to Reproduce**_
Reproduction is achieved by running the command line string;
sup crossbar_maintenance init_apps /var/www/html/monster-ui/apps http://our_1st_master01.kazoo.our_domain.ca:8000/v2
**_## Expected Behaviour_**
This should be the ouput on every run of the script, without any timeout messages:
trying to init app from /var/www/html/monster-ui/apps/pbxs
app pbxs already loaded in system
not updating api_url, it is unchanged
no metadata changes for app 6f12f101f866c972d04a43f960ff41b7
saved PBXconnector_app.png to 6f12f101f866c972d04a43f960ff41b7
saved pbxconnector1.png to 6f12f101f866c972d04a43f960ff41b7
saved pbxconnector2.png to 6f12f101f866c972d04a43f960ff41b7
trying to init app from /var/www/html/monster-ui/apps/webhooks
app webhooks already loaded in system
not updating api_url, it is unchanged
no metadata changes for app e6e5774256edd2201f838865baa25ef3
saved WebHooks_app.png to e6e5774256edd2201f838865baa25ef3
saved webhooks1.png to e6e5774256edd2201f838865baa25ef3
saved webhooks2.png to e6e5774256edd2201f838865baa25ef3
trying to init app from /var/www/html/monster-ui/apps/fax
app fax already loaded in system
not updating api_url, it is unchanged
no metadata changes for app e21e143c33312aa3495558d7936242ec
saved Fax_app.png to e21e143c33312aa3495558d7936242ec
saved OutboundFaxes.png to e21e143c33312aa3495558d7936242ec
trying to init app from /var/www/html/monster-ui/apps/csv-onboarding
app csv-onboarding already loaded in system
not updating api_url, it is unchanged
no metadata changes for app bdca179bf1754c18f5d739dc91c859b6
failed to find icon in bdca179bf1754c18f5d739dc91c859b6
failed to find screenshots in bdca179bf1754c18f5d739dc91c859b6
trying to init app from /var/www/html/monster-ui/apps/accounts
app accounts already loaded in system
not updating api_url, it is unchanged
no metadata changes for app 1b7093aa6cbf7b50708027d03a106a12
saved Accounts_app.png to 1b7093aa6cbf7b50708027d03a106a12
saved Account-AvailableApps.png to 1b7093aa6cbf7b50708027d03a106a12
saved Account-Limits.png to 1b7093aa6cbf7b50708027d03a106a12
saved AccountOverview.png to 1b7093aa6cbf7b50708027d03a106a12
trying to init app from /var/www/html/monster-ui/apps/callflows
app callflows already loaded in system
not updating api_url, it is unchanged
no metadata changes for app 184e7c891e2b9c64e8f4e8eec5b00a7b
saved Callflows_app.png to 184e7c891e2b9c64e8f4e8eec5b00a7b
saved callflows_1.png to 184e7c891e2b9c64e8f4e8eec5b00a7b
saved callflows_2.png to 184e7c891e2b9c64e8f4e8eec5b00a7b
saved callflows_3.png to 184e7c891e2b9c64e8f4e8eec5b00a7b
trying to init app from /var/www/html/monster-ui/apps/voip
app voip already loaded in system
not updating api_url, it is unchanged
no metadata changes for app e9cc159ebcb93062e51c5f38604ec77f
saved SmartPBX_app.png to e9cc159ebcb93062e51c5f38604ec77f
saved smartpbx1.png to e9cc159ebcb93062e51c5f38604ec77f
saved smartpbx2.png to e9cc159ebcb93062e51c5f38604ec77f
saved smartpbx3.png to e9cc159ebcb93062e51c5f38604ec77f
saved smartpbx4.png to e9cc159ebcb93062e51c5f38604ec77f
saved smartpbx5.png to e9cc159ebcb93062e51c5f38604ec77f
trying to init app from /var/www/html/monster-ui/apps/numbers
app numbers already loaded in system
not updating api_url, it is unchanged
no metadata changes for app 8637570ad7b41bc7f3906882507094db
saved Numbers_app.png to 8637570ad7b41bc7f3906882507094db
saved numbers1.png to 8637570ad7b41bc7f3906882507094db
saved numbers2.png to 8637570ad7b41bc7f3906882507094db
trying to init app from /var/www/html/monster-ui/apps/voicemails
app voicemails already loaded in system
not updating api_url, it is unchanged
no metadata changes for app 48630c8173c977fdc460ae93e96010ec
saved Voicemail_app.png to 48630c8173c977fdc460ae93e96010ec
saved PlayVoicemail.png to 48630c8173c977fdc460ae93e96010ec
saved SelectedVoicemails.png to 48630c8173c977fdc460ae93e96010ec
ok
This normally runs under 6 seconds. When timeouts are encountered, execution can take up to 1m30s to complete.
**_## Our Environment_**
Details about our environment:
• Nutanix cluster; 2 masters and 2 x 3-node clusters (1 x primary and 1 x secondary). We test using the primary only.
• Kazoo 4.3-91.el7.centos
• CentOS Linux release 7.7.1908 (Core)
• Couchdb.x86_64 2.3.1-1.el7
Output of curl http://OUR-COUCHDB:5984/ :
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"238a9bf0d5831a4702bbfd52b5000dca","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Contents and details of local.ini, default.ini, vm.args and prlimit output.
See attached file)
[GitHub_params_file_for_prob_submission.docx](https://github.com/apache/couchdb/files/4410044/GitHub_params_file_for_prob_submission.docx)
Extract from /var/log/couchdb/couchdb.log after a run that displayed timeout error:
[error] 2020-03-19T20:09:01.224839Z couchdb@ourprodbox.kazoo.xxxxitelecom.ca <0.510.125> 93969851c9 rexi_server: from: couchdb@ourprodbox.kazoo.xxxxitelecom.ca(<14056.18571.128>) mfa: fabric_rpc:update_docs/3 exit:timeout [{fabric_doc_atts,'-receiver_callback/2-fun-1-',1,[{file,"src/fabric_doc_atts.erl"},{line,56}]},{couch_att,write_streamed_attachment,3,[{file,"src/couch_att.erl"},{line,611}]},{couch_db,with_stream,3,[{file,"src/couch_db.erl"},{line,1413}]},{couch_db,'-doc_flush_atts/2-lc$^0/1-0-',2,[{file,"src/couch_db.erl"},{line,1365}]},{couch_db,doc_flush_atts,2,[{file,"src/couch_db.erl"},{line,1365}]},{couch_db,'-update_docs/4-lc$^4/1-4-',2,[{file,"src/couch_db.erl"},{line,1184}]},{couch_db,'-update_docs/4-lc$^3/1-3-',2,[{file,"src/couch_db.erl"},{line,1183}]},{couch_db,update_docs,4,[{file,"src/couch_db.erl"},{line,1183}]}]
**_- QUESTIONS:_**
1- how can pinpoint the source of the fabric timeouts ?
2- which parameter(s) in the config file(s) needs to be adjusted ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services