[PR] Basic zstd compression [couchdb]

["nickva (via GitHub)" <gi...@apache.org>]

nickva opened a new pull request, #6028:
URL: https://github.com/apache/couchdb/pull/6028

   This is using the built-in zstd API from Erlang/OTP 28+. For 26 and 27 it falls back to `deflate`. Otherwise it's not that different than deflate and snappy which we already support.
   
   An interesting feature of zstd is to use dictionaries. This PR doesn't do it, but it sets up a shim `couch_zstd` module where that could plug-in eventually. The idea is to train a dictionary it based on document bodies for each database before compaction and then saving the dictionary. zstd as is in Erlang/OTP doesn't expose a trainign API but we can still use a concatenative/raw dictionary:
   
   ```erlang
   
   > Dict = ~"{\"type\":\"foo\",\"bar\":1,\"baz\":\"a\"}".
   <<"{\"type\":\"foo\",\"bar\":1,\"baz\":\"a\"}">>
   
   > {ok, C} = zstd:dict(compress, Dict).
   {ok,{cdict,#Ref<0.4075923125.4131520536.31325>}}
   
   > Doc = ~"{\"type\":\"foo\", \"bar\":2, \"baz\":\"b\"}".
   <<"{\"type\":\"foo\", \"bar\":2, \"baz\":\"b\"}">>
   
   > byte_size(iolist_to_binary(zstd:compress(Doc, #{dictionary=>C}))).
   29
   
   > byte_size(iolist_to_binary(zstd:compress(Doc))).
   43
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Basic zstd compression [couchdb]

["nickva (via GitHub)" <gi...@apache.org>]

nickva commented on PR #6028:
URL: https://github.com/apache/couchdb/pull/6028#issuecomment-4662703394

   > Don't know why the trixie packaging failed, seems unrelated.
   
   I had figured it was something with the ubuntu fsn worker. I had updated all the workers and restarted them and now it seems to work again. Not really sure the exact reason of what happened. It could have been I hard forgotten to restart it last time after an `apt upgrade`...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Basic zstd compression [couchdb]

["nickva (via GitHub)" <gi...@apache.org>]

nickva merged PR #6028:
URL: https://github.com/apache/couchdb/pull/6028


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org