Ugliness in header processing, mochiweb, and Erlang

[Joan Touzet <wo...@apache.org>]

Arrgh, I had an entire email typed and lost it, so I'm going to do this briefly:

This morning my time, this bug report came in against 2.3.0:

  https://github.com/apache/couchdb/issues/1810

Robert Newson, Nick Vatamaniuc and I have been trying to unravel this on IRC. It turns out to be quite complex.

Here are some links to information that will help explain how we got here and what the root cause is:

  https://github.com/apache/couchdb/issues/1409 - why it changed and why we can't just revert the change
  https://gist.github.com/wohali/908dc3dfe5126a32b64383fda17e4c95 - 2.2.0 vs. 2.3.0 defaults on a Linux system interrogating recbuf and buffer sizes
  http://erlang.org/pipermail/erlang-questions/2011-June/059571.html - why Erlang isn't doing the right thing, read the whole thread for more context
  https://github.com/ninenines/cowboy/issues/3 - how cowboy encountered this (we use mochiweb, not cowboy)
  https://github.com/ninenines/cowboy/commit/29e71cf4daec684c13047952a95ec0dc9540aad5 - how cowboy fixed this (raw packets)
  https://github.com/apache/couchdb/commit/d23025ebd7176f6c307ddf49902cf20b33bd55c4 - a side effect of this http parsing bug
   
Nick is writing up a PR against mochiweb that will set [{buffer, 8192}] if it's not explicitly set in server_options (right now mochiweb doesn't honour this option even if we pass it), which should fix things, but it's still not great. For example, the maximum document ID length hack listed above is another side effect of this failure.

The ideal solution is what cowboy has done - use a raw packet, keep recv()ing and passing to erlang:decode_packet until it doesn't say "i need more data", with safeguards to prevent endless headers from effectively DoSsing the server - but to put that in mochiweb is a huge change, and no one is signing up to do the work.

Any other ideas on what to do from the assembled brains on this list would be welcome. As are PRs. :)

-Joan