[jira] [Commented] (NIFI-2827) CompressContent should support brotli (MIT) and zstd (BSD)

["Matthew Hawkins (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17577413#comment-17577413 ] 

Matthew Hawkins commented on NIFI-2827:
---------------------------------------

Attached is a quick & dirty zstd implementation; it compiled with JDK11 and passes the test.

Key thing is that Apache Commons has zstandard support since 1.16 and 1.21 (latest) is included in NiFi 1.17.0 pom.

The same technique could be used to add brotli and others supported by org.apache.commons.compress and in fact you could probably refactor the whole thing to use Apache Commons instead of a gazillion other libraries. That was out of scope for me, I just wanted zstd support and I'm glad I did it because the test file compresses to 80 bytes, which is the winner of all the compression algorithms in the test. Can't wait to use it on real data!

Of note, I do NOT add support for compression levels or using the dictionary (read zstandard docs); this is beyond my Java hacking skills tonight.

 

> CompressContent should support brotli (MIT) and zstd (BSD)
> ----------------------------------------------------------
>
>                 Key: NIFI-2827
>                 URL: https://issues.apache.org/jira/browse/NIFI-2827
>             Project: Apache NiFi
>          Issue Type: New Feature
>            Reporter: Andre F de Miranda
>            Priority: Major
>         Attachments: NIFI-2827-zstd.patch
>
>
> brotli and zstd are gaining lots of momentum across the IoT sphere and I suspect it would be a nice to have in NiFi to be able to support those two emerging compression algorithms.
> Both brotli and zstd JAVA implementations are JNIs 
> https://github.com/MeteoGroup/jbrotli (ASL 2)
> https://github.com/luben/zstd-jni (3 clause BSD)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)