You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2022/03/14 15:58:02 UTC

[GitHub] [couchdb] ronag opened a new issue #3959: uknown internal server error

ronag opened a new issue #3959:
URL: https://github.com/apache/couchdb/issues/3959


   We seem to be getting a lot of 500 errors with "function_clause" and "unknown_error"?
   
   e.g.
   
   ```
   [2022-03-14 15:19:50.597 +0000] WARN  (deepstream-record-worker-12): bulkDocs error
       err: {
         "type": "InternalServerError",
         "message": "Internal Server Error",
         "stack":
             InternalServerError: Internal Server Error
                 at makeError (/usr/src/app/node_modules/@nxtedition/lib/couch.js:101:12)
                 at Object.bulkDocs (/usr/src/app/node_modules/@nxtedition/lib/couch.js:502:13)
                 at runMicrotasks (<anonymous>)
                 at processTicksAndRejections (node:internal/process/task_queues:96:5)
                 at async _write (file:///usr/src/app/src/record/record-storage.js:103:9)
         "data": "{\n  \"req\": {\n    \"path\": \"/nxt/_bulk_docs\",\n    \"method\": \"POST\",\n    \"body\": \"\\\"{\\\\\\\"new_edits\\\\\\\":false,\\\\\\\"docs\\\\\\\":[{\\\\n      \\\\\\\"_deleted\\\\\\\": true,\\\\n      \\\\\\\"_id\\\\\\\": \\\\\\\"749bfdadb7:general._asset\\\\\\\",\\\\n      \\\\\\\"_rev\\\\\\\": \\\\\\\"3-T04LakhRFb9a4J-asset\\\\\\\",\\\\n      \\\\\\\"_revisions\\\\\\\": {\\\\n        \\\\\\\"ids\\\\\\\": [\\\\\\\"T04LakhRFb9a4J-asset\\\\\\\",\\\\\\\"NIE8HiDHRKbFD3-_asset\\\\\\\"],\\\\n        \\\\\\\"start\\\\\\\": 3\\\\n      }\\\\n    },{\\\\n      \\\\\\\"asset\\\\\\\":\\\\\\\"6aaffc6debbcea77229146de2824b8f6de14a7c1\\\\\\\",\\\\n      \\\\\\\"_id\\\\\\\": \\\\\\\"749bfdadb7:general._asset\\\\\\\",\\\\n      \\\\\\\"_rev\\\\\\\": \\\\\\\"4-T04Lb0rRDMqxBR-asset\\\\\\\",\\\\n      \\\\\\\"_revisions\\\\\\\": {\\\\n        \\\\\\\"ids\\\\\\\": [\\\\\\\"T04Lb0rRDMqxBR-asset\\\\\\\",\\\\\\\"T04LakhRFb9a4J-asset\\\\\\\"],\\\\n        \\\\\\\"start\\\\\\\": 4\
 \\\n      }\\\\n    },{\\\\n      \\\\\\\"mimeType\\\\\\\":\\\\\\\"video/MP2T\\\\\\\",\\\\\\\"size\\\\\\\":2102216,\\\\\\\"seekable\\\\\\\":true,\\\\\\\"atime\\\\\\\":1647216000000,\\\\\\\"refs\\\\\\\":[\\\\\\\"93ae0721c5127f0e.mp4\\\\\\\"],\\\\n      \\\\\\\"_id\\\\\\\": \\\\\\\"2bd5b22f4d1b0255e1b5.ts:file\\\\\\\",\\\\n      \\\\\\\"_rev\\\\\\\": \\\\\\\"8-T04LbmaQHdkIu2-MoNlFct5_PI7HL-storage-2\\\\\\\",\\\\n      \\\\\\\"_revisions\\\\\\\": {\\\\n        \\\\\\\"ids\\\\\\\": [\\\\\\\"T04LbmaQHdkIu2-MoNlFct5_PI7HL-storage-2\\\\\\\",\\\\\\\"SyElZEy9XpJTil-MoNlSCy59YTcOu-storage-3\\\\\\\"],\\\\n        \\\\\\\"start\\\\\\\": 8\\\\n      }\\\\n    },{\\\\n      \\\\\\\"asset\\\\\\\":\\\\\\\"ccb4e5163175782ff359740b9f36cc2226107130\\\\\\\",\\\\n      \\\\\\\"_id\\\\\\\": \\\\\\\"98d310dfcb:general._asset\\\\\\\",\\\\n      \\\\\\\"_rev\\\\\\\": \\\\\\\"3-T04LbnMRIlM4zp-asset\\\\\\\",\\\\n      \\\\\\\"_revisions\\\\\\\": {\\\\n        \\\\\\\"ids\\\\\\\": [\\\\\\\"T04LbnMRIlM4zp-asset
 \\\\\\\",\\\\\\\"NIE8I1I5GgGyN~-_asset\\\\\\\"],\\\\n        \\\\\\\"start\\\\\\\": 3\\\\n      }\\\\n    },{\\\\n      \\\\\\\"_deleted\\\\\\\": true,\\\\n      \\\\\\\"_id\\\\\\\": \\\\\\\"072256f9c7:general._asset\\\\\\\",\\\\n      \\\\\\\"_rev\\\\\\\": \\\\\\\"3-T04LbnNRF5rAql-asset\\\\\\\",\\\\n      \\\\\\\"_revisions\\\\\\\": {\\\\n        \\\\\\\"ids\\\\\\\": [\\\\\\\"T04LbnNRF5rAql-asset\\\\\\\",\\\\\\\"NIE8I7p5HOETXF-_asset\\\\\\\"],\\\\n        \\\\\\\"start\\\\\\\": 3\\\\n      }\\\\n    },{\\\\n      \\\\\\\"asset\\\\\\\":\\\\\\\"ccb4e5163175782ff359740b9f36cc2226107130\\\\\\\",\\\\n      \\\\\\\"_id\\\\\\\": \\\\\\\"072256f9c7:general._asset\\\\\\\",\\\\n      \\\\\\\"_rev\\\\\\\": \\\\\\\"4-T04Lc3cRDYIHTX-asset\\\\\\\",\\\\n      \\\\\\\"_revisions\\\\\\\": {\\\\n        \\\\\\\"ids\\\\\\\": [\\\\\\\"T04Lc3cRDYIHTX-asset\\\\\\\",\\\\\\\"T04LbnNRF5rAql-asset\\\\\\\"],\\\\n        \\\\\\\"start\\\\\\\": 4\\\\n      }\\\\n    },{\\\\n      \\\\\\\"asset\\\\\\\":\\\\\\\"
 af2cfd5623a56efd27cb80c47adad06a6b2e8d88\\\\\\\",\\\\\\\"storage\\\\\\\":\\\\\\\"2d8356a8099e3f9deb00b2fe3c16b57d8841cf72\\\\\\\",\\\\\\\"media\\\\\\\":\\\\\\\"51a51052e4bfa94901cde1f4ff5956109be63662\\\\\\\",\\\\n      \\\\\\\"_id\\\\\\\": \\\\\\\"MxTjuda5pC08gJ:general._asset\\\\\\\",\\\\n      \\\\\\\"_rev\\\\\\\": \\\\\\\"24-T04Lcq2RDXl9lo-asset\\\\\\\",\\\\n      \\\\\\\"_revisions\\\\\\\": {\\\\n        \\\\\\\"ids\\\\\\\": [\\\\\\\"T04Lcq2RDXl9lo-asset\\\\\\\",\\\\\\\"SxXantebPobiSL-asset-asset\\\\\\\"],\\\\n        \\\\\\\"start\\\\\\\": 24\\\\n      }\\\\n    },{\\\\n      \\\\\\\"asset\\\\\\\":\\\\\\\"76b5485774761f626c3df34d9cfde3913f4d6753\\\\\\\",\\\\n      \\\\\\\"_id\\\\\\\": \\\\\\\"2aamu3s:general._asset\\\\\\\",\\\\n      \\\\\\\"_rev\\\\\\\": \\\\\\\"4-T04Lcq3RI9SPze-asset\\\\\\\",\\\\n      \\\\\\\"_revisions\\\\\\\": {\\\\n        \\\\\\\"ids\\\\\\\": [\\\\\\\"T04Lcq3RI9SPze-asset\\\\\\\",\\\\\\\"NmG~jnA8BdwJHf-_asset\\\\\\\"],\\\\n        \\\\\\\"start\\\\\\\":
  4\\\\n      }\\\\n    },{\\\\n      \\\\\\\"_deleted\\\\\\\": true,\\\\n      \\\\\\\"_id\\\\\\\": \\\\\\\"MxTqjB~40inVhC:general._asset\\\\\\\",\\\\n      \\\\\\\"_rev\\\\\\\": \\\\\\\"3-T04LeOvRGxXdRE-asset\\\\\\\",\\\\n      \\\\\\\"_revisions\\\\\\\": {\\\\n        \\\\\\\"ids\\\\\\\": [\\\\\\\"T04LeOvRGxXdRE-asset\\\\\\\",\\\\\\\"NIE8Is15LDoG5t-_asset\\\\\\\"],\\\\n        \\\\\\\"start\\\\\\\": 3\\\\n      }\\\\n    },{\\\\n      \\\\\\\"asset\\\\\\\":\\\\\\\"3d28ed928c23dcb61a9525772c592a71a8a80160\\\\\\\",\\\\n      \\\\\\\"_id\\\\\\\": \\\\\\\"MxTqjB~40inVhC:general._asset\\\\\\\",\\\\n      \\\\\\\"_rev\\\\\\\": \\\\\\\"4-T04LevYREMROXZ-asset\\\\\\\",\\\\n      \\\\\\\"_revisions\\\\\\\": {\\\\n        \\\\\\\"ids\\\\\\\": [\\\\\\\"T04LevYREMROXZ-asset\\\\\\\",\\\\\\\"T04LeOvRGxXdRE-asset\\\\\\\"],\\\\n        \\\\\\\"start\\\\\\\": 4\\\\n      }\\\\n    },{\\\\n      \\\\\\\"_deleted\\\\\\\": true,\\\\n      \\\\\\\"_id\\\\\\\": \\\\\\\"9376ae59ed:general._asset\\\\\\\",
 \\\\n      \\\\\\\"_rev\\\\\\\": \\\\\\\"3-T04LfhwRFehyzz-asset\\\\\\\",\\\\n      \\\\\\\"_revisions\\\\\\\": {\\\\n        \\\\\\\"ids\\\\\\\": [\\\\\\\"T04LfhwRFehyzz-asset\\\\\\\",\\\\\\\"NIE8JKYHX5kSno-_asset\\\\\\\"],\\\\n        \\\\\\\"start\\\\\\\": 3\\\\n      }\\\\n    },{\\\\n      \\\\\\\"asset\\\\\\\":\\\\\\\"ccb4e5163175782ff359740b9f36cc2226107130\\\\\\\",\\\\n      \\\\\\\"_id\\\\\\\": \\\\\\\"9376ae59ed:general._asset\\\\\\\",\\\\n      \\\\\\\"_rev\\\\\\\": \\\\\\\"4-T04Lfy4RFsI8TH-asset\\\\\\\",\\\\n      \\\\\\\"_revisions\\\\\\\": {\\\\n        \\\\\\\"ids\\\\\\\": [\\\\\\\"T04Lfy4RFsI8TH-asset\\\\\\\",\\\\\\\"T04LfhwRFehyzz-asset\\\\\\\"],\\\\n        \\\\\\\"start\\\\\\\": 4\\\\n      }\\\\n    },{\\\\n      \\\\\\\"_deleted\\\\\\\": true,\\\\n      \\\\\\\"_id\\\\\\\": \\\\\\\"MxUKo8LC8Ns641:general._asset\\\\\\\",\\\\n      \\\\\\\"_rev\\\\\\\": \\\\\\\"4-T04LgkURFHQsc6-asset\\\\\\\",\\\\n      \\\\\\\"_revisions\\\\\\\": {\\\\n        \\\\\\\"ids\\\\\\\": 
 [\\\\\\\"T04LgkURFHQsc6-asset\\\\\\\",\\\\\\\"NIE8JcO7A97Sy8-_asset\\\\\\\"],\\\\n        \\\\\\\"start\\\\\\\": 4\\\\n      }\\\\n    },{\\\\n      \\\\\\\"_deleted\\\\\\\": true,\\\\n      \\\\\\\"_id\\\\\\\": \\\\\\\"68ab2b9b83:general._asset\\\\\\\",\\\\n      \\\\\\\"_rev\\\\\\\": \\\\\\\"3-T04LgkURGx9Uz2-asset\\\\\\\",\\\\n      \\\\\\\"_revisions\\\\\\\": {\\\\n        \\\\\\\"ids\\\\\\\": [\\\\\\\"T04LgkURGx9Uz2-asset\\\\\\\",\\\\\\\"N\"\n  },\n  \"res\": {\n    \"status\": 500,\n    \"headers\": {\n      \"cache-control\": \"must-revalidate\",\n      \"content-length\": \"68\",\n      \"content-type\": \"application/json\",\n      \"date\": \"Mon, 14 Mar 2022 15:19:49 GMT\",\n      \"server\": \"CouchDB/3.2.1 (Erlang OTP/20)\",\n      \"x-couch-request-id\": \"84771c9a5d\",\n      \"x-couch-stack-hash\": \"27933023\",\n      \"x-couchdb-body-time\": \"0\"\n    },\n    \"body\": {\n      \"error\": \"unknown_error\",\n      \"reason\": \"function_clause\",\n      \"ref\": 27
 933023\n    }\n  }\n}",
         "reason": "function_clause",
         "error": "unknown_error",
         "status": 500,
         "statusCode": 500,
         "expose": false
       }
   ```
   
   Any ideas what this could be and/or how we could provide better debug information?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] ronag commented on issue #3959: uknown internal server error

Posted by GitBox <gi...@apache.org>.
ronag commented on issue #3959:
URL: https://github.com/apache/couchdb/issues/3959#issuecomment-1076461705


   It seems related to heavy/extreme load. Everything works fine usually but when on heavy load we get various kind of "unknown" internal server errors.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] nickva commented on issue #3959: uknown internal server error

Posted by GitBox <gi...@apache.org>.
nickva commented on issue #3959:
URL: https://github.com/apache/couchdb/issues/3959#issuecomment-1076460204


   @ronag I think the _revisions field provided might not be valid CouchDB revisions such as: `NIE8HiDHRKbFD3-_asset` (just an initial guess).
   
   Would the logs from CouchDB be available?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] nickva commented on issue #3959: uknown internal server error

Posted by GitBox <gi...@apache.org>.
nickva commented on issue #3959:
URL: https://github.com/apache/couchdb/issues/3959#issuecomment-1076489068


   @ronag that makes sense. In that case keep an eye out for a stacktrace in the couchdb logs. If it happens again see if you look for the `"x-couch-stack-hash": "27933023"` value (`27933023` in that case) in the logs. That ID should help pin-point the stack trace associated with that request.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] ronag commented on issue #3959: uknown internal server error

Posted by GitBox <gi...@apache.org>.
ronag commented on issue #3959:
URL: https://github.com/apache/couchdb/issues/3959#issuecomment-1076462919


   >  I think the _revisions field provided might not be valid CouchDB revisions such
   
   They work and should work. I believe it was discussed in couch 4 to deprecate the possibility of custom revisions. If I retry the exact same request now everything works fine.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] ronag edited a comment on issue #3959: uknown internal server error

Posted by GitBox <gi...@apache.org>.
ronag edited a comment on issue #3959:
URL: https://github.com/apache/couchdb/issues/3959#issuecomment-1066991256


   ```
   \"server\": \"CouchDB/3.2.1 (Erlang OTP/20)\",
   \"x-couch-request-id\": \"84771c9a5d\",
   \"x-couch-stack-hash\": \"27933023\",\n
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] ronag commented on issue #3959: uknown internal server error

Posted by GitBox <gi...@apache.org>.
ronag commented on issue #3959:
URL: https://github.com/apache/couchdb/issues/3959#issuecomment-1066991256


   \"server\": \"CouchDB/3.2.1 (Erlang OTP/20)\",\n      \"x-couch-request-id\": \"84771c9a5d\",\n      \"x-couch-stack-hash\": \"27933023\",\n


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] ronag commented on issue #3959: uknown internal server error

Posted by GitBox <gi...@apache.org>.
ronag commented on issue #3959:
URL: https://github.com/apache/couchdb/issues/3959#issuecomment-1076492696


   I'll re-open this issue once/if it occurs again and I can grab the stack trace.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] ronag closed issue #3959: uknown internal server error

Posted by GitBox <gi...@apache.org>.
ronag closed issue #3959:
URL: https://github.com/apache/couchdb/issues/3959


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] ronag commented on issue #3959: uknown internal server error

Posted by GitBox <gi...@apache.org>.
ronag commented on issue #3959:
URL: https://github.com/apache/couchdb/issues/3959#issuecomment-1076462080


   I don't have more info on this atm as the logs have rotated since then.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org