[GitHub] [couchdb] nickva commented on issue #4400: Add zstd support as a compression method

["nickva (via GitHub)" <gi...@apache.org>]

nickva commented on issue #4400:
URL: https://github.com/apache/couchdb/issues/4400#issuecomment-1399405229

   That's a good idea. We've even discussed it 7 years or so ago :-)
   
   https://lists.apache.org/thread/kvvjodld2ly0t9rrllfd4d27pwf43hy5
   
   My comment from 2016 was:
   
   > There has been a surprising resurgence of compression research lately with things like Brotli from Google and zstd from Facebook (http://facebook.github.io/zstd/). zstd has an interesting "training" mode where it can do a pass over small documents and learn a common dictionary, and CouchDB already passed over data during compaction, would that be a good time to train a compression dictionary?
   
   
   At the time it wasn't sure how zstd would do, but it seems to have survived pretty well and even made its way into the Linux kernel.
   
   That being said, if we just do the naive compression of doc bodies like we do now, it might not be worth it as there is a real future backward compatibility cost having to support another compression scheme essentially forever. It's kind of like that with snappy already. Now, one motivating factor about zstd would be if we could apply per-btree or per db compression using a dictionary. CouchDB's compaction provides a nice place to learn the dictionary and update during each compaction cycle.
   
   > CouchDB already supports multiple compression codecs (by the way, which exactly?
   
   The current compression methods include snappy and the built-in Erlang term compression. See the [docs](https://docs.couchdb.org/en/latest/config/couchdb.html#couchdb/file_compression) for more details. The built-in Erlang compression is essentially deflate with 10 configurable levels.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org