You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "nickva (via GitHub)" <gi...@apache.org> on 2023/03/29 16:21:24 UTC

[GitHub] [couchdb] nickva opened a new issue, #4504: Out-of-memory InternalError exceptions in couchjs

nickva opened a new issue, #4504:
URL: https://github.com/apache/couchdb/issues/4504

   Spidermonkey sometimes throws an `{<<"InternalError">>,{[{<<"message">>,<<"allocation size overflow">>}` exception when exceeding memory limits when normally we'd expect it to crash or exit with a non-0 exit code.
   
   At first it seems like a nice feature, on the other hand, because we trap exceptions and sometimes continue emitting rows, it seems possible for users views to randomly miss indexed rows based on whether GC had run or not, other internal runtime state which may have been consuming more or less memory until that time.
   
   So far I have observed this on spidermonkey 1.8.5 and only in cases when the rows are already emitted are being serialized in https://github.com/apache/couchdb/blob/75ee0f0fbd3fe2d759d13d1eda1968597f4e3aec/share/server/views.js#L134.
   
   However, it seems this error could also occur inside a view functions, if users created a large object, for instance, before emitting anything. In that case our try ... catch handler would catch `InternalError` and then skip the rows (return `[]` to couch indicated success) from https://github.com/apache/couchdb/blob/75ee0f0fbd3fe2d759d13d1eda1968597f4e3aec/share/server/views.js#L58-L80
   
   To avoid non-determinism and apparent random data loss in the view results perhaps it's better to treat `InternalError` as fatal in [handleViewError](https://github.com/apache/couchdb/blob/main/share/server/views.js#L58-L80) ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [couchdb] nickva closed issue #4504: Out-of-memory InternalError exceptions in couchjs

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva closed issue #4504: Out-of-memory InternalError exceptions in couchjs
URL: https://github.com/apache/couchdb/issues/4504


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [couchdb] nickva commented on issue #4504: Out-of-memory InternalError exceptions in couchjs

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #4504:
URL: https://github.com/apache/couchdb/issues/4504#issuecomment-1489064857

   Quick and dirty proof of concept:
   
   `% cat ~/view_oom_small_emit.json`
   
   ```json
   ["reset"]
   ["add_fun", "function(doc) {function mkstr(l) {var r = 'r'; while (r.length < l) {r=r+r;} return r;} var s = mkstr(96*1024*1024); var a = [s,s,s,s,s,s,s,s]; var j=JSON.stringify(a); emit(42,j.length);}"]
   ["map_doc", {"_id":"foo"}]
   ```
   
   ```
   cat ~/view_oom_small_emit.json | ./src/couch/priv/couchjs ./share/server/main.js
   true
   true
   ["log","function raised exception (new InternalError(\"allocation size overflow\", \"<unknown>\", 20)) with doc._id foo"]
   [[]]
   ```
   
   Seems to happen on both 91 and 1.8.5, so pretty much the whole range of supported JS engines


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [couchdb] davisp commented on issue #4504: Out-of-memory InternalError exceptions in couchjs

Posted by "davisp (via GitHub)" <gi...@apache.org>.
davisp commented on issue #4504:
URL: https://github.com/apache/couchdb/issues/4504#issuecomment-1488968527

   Seems reasonable to die on anything labeled InternalError.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [couchdb] nickva commented on issue #4504: Out-of-memory InternalError exceptions in couchjs

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #4504:
URL: https://github.com/apache/couchdb/issues/4504#issuecomment-1489192566

   A PR with a above proof of concept code as a test https://github.com/apache/couchdb/pull/4505


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org