You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@pekko.apache.org by "mdedetrich (via GitHub)" <gi...@apache.org> on 2025/10/28 12:50:55 UTC
[PR] Add zstd compression and decompression for pekko streams [pekko]
mdedetrich opened a new pull request, #2409:
URL: https://github.com/apache/pekko/pull/2409
Resolves: https://github.com/apache/pekko/issues/2404
This PR adds a zstd compression/decompression stream/flow using [zstd-jni](https://github.com/luben/zstd-jni), this project was chosen because its the only one that has high performance (it uses the [reference zstd implementation](https://github.com/facebook/zstd) via JNI) and also supports at least JDK 17 (its published with JDK 11).
The PR still needs to be completed (documentation needs to be added along with MiMa exclusions) but I am creating a PR now with the necessary barebones earlier so that people can comment on whether the PR is on the right track, tests have been added (there is already a base testing framework for pekko-streams compression flows).
The implementation of `ZstdCompressor`/`ZstdDecompressor` uses `ZstdDirectBufferCompressingStreamNoFinalizer`/`ZstdDirectBufferDecompressingStreamNoFinalizer` as these are the abstractions provided by zstd-jni to do streaming compression, note that the versions with `NoFinalizer` just mean that you need to explicitly shutdown the resource (which is what we want since Pekko Streams handles resource cleaning). These compression abstractions use direct `ByteBuffer` to handle the shuffling of data between the JNI boundary so that the C implementation can do its work directly in memory, the [zstd-jni tests](https://github.com/luben/zstd-jni/blob/9c3386d306086078155f58116a4d905e07239db4/src/test/scala/Zstd.scala) was the basis used to write the implementation.
Some extra notes
- `CoderSpec` had to be modified as the test which catches the exception to be thrown on corrupt input was hardcoded to `DataFormatException` where as zstd throws its own bespoke exception on corrupt input
- The `ZstdCompressor` implements the `Compressor` abstraction which does a lot of heavy lifting (especially when it comes to tests) however the `ZstdDecompressor` intentionally does not implement `DeflateDecompressorBase` as the design is heavily tied to Java's deflate/compression API's, instead we use `SimpleLinearGraphStage[ByteString]` backed by `ZstdDirectBufferDecompressingStreamNoFinalizer`
- The current API also allows you to specify a [dictionary](https://github.com/facebook/zstd?tab=readme-ov-file#dictionary-compression-how-to) when doing compression. Note that to do this, you need to pass a `com.github.luben.zstd.ZstdDictCompress` datastructure which is tied to the implementation of zstd-jni. There is an argument to create our own pekko equivalent of `ZstdDictCompress` which will internally map to a `com.github.luben.zstd.ZstdDictCompress`, doing so would allow us to swap to a different implementation of zstd without breaking the API.
- This is the only part of the API that is tied to the zstd-jni implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "raboof (via GitHub)" <gi...@apache.org>.
raboof commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2556136982
##########
actor/src/main/java/org/apache/pekko/io/ByteBufferCleaner.java:
##########
@@ -36,7 +37,8 @@
*
* <p>See <a href=https://bugs.openjdk.java.net/browse/JDK-4724038>JDK-4724038</a>
*/
-final class ByteBufferCleaner {
+@InternalStableApi
Review Comment:
This annotation is for APIs that need to remain binary compatible because other, independently-released Pekko components depend on them.
As you're using this API within Pekko Core, and we expect all Pekko Core artifact versions to be consistent, the annotation is not needed for this use.
Are you planning to use this API in other Pekko modules? If not then we can remove this annotation here. If so then we should add a note to the scaladoc pointing to where it's (going to be) used.
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.io.ByteBufferCleaner
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArrayUnsafe())
+ inputBB.flip()
+ compressingStream.compress(inputBB)
+ val result = ByteString.fromByteBuffer(targetBuffer)
Review Comment:
Are you still planning on this?
##########
.scala-steward.conf:
##########
@@ -5,6 +5,8 @@ updates.pin = [
{ groupId = "org.scala-lang", artifactId = "scala3-library", version = "3.3." }
# sbt-assembly 2.3 causes build issues (https://github.com/apache/pekko/pull/1744)
{ groupId = "com.eed3si9n", artifactId = "sbt-assembly", version = "2.2." }
+ # zstd-jni upgrades are fine as long as they are backwards compatible
+ { groupId = "com.github.luben", artifactId = "zstd-jni", version = "1."}
Review Comment:
Shouldn't we even allow major version updates? Then we can decide on a case-by-case basis whether the incompatibility that caused them to bump the major version is a reason for us to avoid the update.
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.io.ByteBufferCleaner
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArrayUnsafe())
+ inputBB.flip()
+ compressingStream.compress(inputBB)
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.flip()
+ if (ByteBufferCleaner.isSupported)
+ ByteBufferCleaner.clean(inputBB)
+ result
+ }
+
+ override def flush(): ByteString = {
+ targetBuffer.flip()
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.clear()
+ compressingStream.flush()
+ result
+ }
+
+ override def finish(): ByteString = {
+ compressingStream.close()
+ targetBuffer.flip()
+ val arr = Array.ofDim[Byte](targetBuffer.limit())
+ targetBuffer.get(arr)
+ val result = ByteString.fromArrayUnsafe(arr)
+ if (ByteBufferCleaner.isSupported)
+ ByteBufferCleaner.clean(targetBuffer)
+ result
+ }
+
+ override def compressAndFlush(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArray)
Review Comment:
the recoding is now complete, right? This line still seems to be here, should it still be changed to `Unsafe`?
##########
docs/src/main/paradox/stream/operators/Compression/zstd.md:
##########
@@ -0,0 +1,30 @@
+# Compression.zstd
+
+Creates a flow that zstd-compresses a stream of ByteStrings.
+
+@ref[Compression operators](../index.md#compression-operators)
+
+## Signature
+
+@apidoc[Compression.zstd](stream.*.Compression$) { scala="#zstd:org.apache.pekko.stream.scaladsl.Flow[org.apache.pekko.util.ByteString,org.apache.pekko.util.ByteString,org.apache.pekko.NotUsed]" java="#zstd()" }
+
+## Description
+
+Creates a flow that zstd-compresses a stream of ByteStrings. Note that the compressor
+will SYNC_FLUSH after every @apidoc[util.ByteString] so that it is guaranteed that every @apidoc[util.ByteString]
Review Comment:
This also means that if the way the incoming data is cut up into `ByteString`s is not deterministic, the compressed stream will not be deterministic, right? I think that is acceptable, but would be worth calling out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2573523286
##########
.scala-steward.conf:
##########
@@ -5,6 +5,8 @@ updates.pin = [
{ groupId = "org.scala-lang", artifactId = "scala3-library", version = "3.3." }
# sbt-assembly 2.3 causes build issues (https://github.com/apache/pekko/pull/1744)
{ groupId = "com.eed3si9n", artifactId = "sbt-assembly", version = "2.2." }
+ # zstd-jni upgrades are fine as long as they are backwards compatible
+ { groupId = "com.github.luben", artifactId = "zstd-jni", version = "1."}
Review Comment:
Okay Ill remove the entry then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "jrudolph (via GitHub)" <gi...@apache.org>.
jrudolph commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3461319698
> on the `compress` method in `ZstdCompressor` as this will trigger a cleanup on every `onPush` of the stream. I don't know how expensive this cleanup is, there can be an argument that for this specific case we should defer it for GC as GC cleaning up many ByteBuffers as a batch would be faster, @jrudolph can you comment?
There used to be a reason to use buffer pools especially for direct buffers, probably not least because you might fragment the native heap. Would probably make sense to not churn through them in quick succession. Why not use the buffer pool from the zstd-jni jar?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3508172320
I have just updated the PR by adding our own `ZstdDictionary` datastructure so that the public API of the zstd compress/decompress doesn't depend on any internals from zstd-jni.
Aside from making it possible to swap out zstd-jni in the future for any alternative implementations it also makes the API cleaner for both scala and java users.
@He-Pin @pjfanning @raboof Let me know if the design `ZstdDictionary` works (I made it idiomatic for both scala and javadsl) and also that its location (i.e. `org.apache.pekko.stream.javadsl.compression.ZstdDictionary`/`org.apache.pekko.stream.scaladsl.compression.ZstdDictionary`) is appropriate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2472100795
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArrayUnsafe())
+ inputBB.flip()
+ compressingStream.compress(input.asByteBuffer)
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.flip()
+ result
+ }
+
+ override def flush(): ByteString = {
+ targetBuffer.flip()
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.clear()
+ compressingStream.flush()
Review Comment:
The flush operation calls `compressingStream.flush()` after extracting the result from the buffer. The flush result is discarded. The correct order should be: flip buffer, flush stream (which may write more data to buffer), extract result, then clear buffer.
```suggestion
compressingStream.flush()
val result = ByteString.fromByteBuffer(targetBuffer)
targetBuffer.clear()
```
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArrayUnsafe())
+ inputBB.flip()
Review Comment:
Unused variable `inputBB` created but never used. The compress operation uses `input.asByteBuffer` instead. Either remove lines 39-41 or use `inputBB` in line 42 instead of `input.asByteBuffer`.
```suggestion
```
##########
stream/src/main/scala/org/apache/pekko/stream/javadsl/Compression.scala:
##########
@@ -81,4 +87,25 @@ object Compression {
def deflate(level: Int, nowrap: Boolean): Flow[ByteString, ByteString, NotUsed] =
scaladsl.Compression.deflate(level, nowrap).asJava
+ /**
+ * @since 2.0.0
+ */
+ def zstd: Flow[ByteString, ByteString, NotUsed] =
+ scaladsl.Compression.zstd.asJava
+
+ /**
+ * Same as [[zstd]] with a custom level and an optional dictionary.
+ * @param level The compression level, must be greater or equal to [[Zstd.minCompressionLevel]] and less than or equal
+ * to [[Zstd.maxCompressionLevel]]
+ * @param dictionary An optional dictionary that can be used for compression
+ * @since 2.0.0
+ */
+ def zstd(level: Int, dictionary: Optional[ZstdDictCompress]): Flow[ByteString, ByteString, NotUsed] =
+ scaladsl.Compression.zstd(level, dictionary.toScala).asJava
+
+ /**
Review Comment:
Missing scaladoc description and parameter documentation. This should include a description and document the `maxBytesPerChunk` parameter, consistent with other decompression methods.
```suggestion
/**
* Creates a Flow that decompresses Zstandard-compressed stream of data.
*
* @param maxBytesPerChunk Maximum length of the output [[pekko.util.ByteString]] chunk.
```
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdDecompressor.scala:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.stream.Attributes
+import pekko.stream.impl.fusing.GraphStages.SimpleLinearGraphStage
+import pekko.stream.stage.{ GraphStageLogic, InHandler, OutHandler }
+import pekko.util.ByteString
+
+import com.github.luben.zstd.ZstdDirectBufferDecompressingStreamNoFinalizer
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdDecompressor(maxBytesPerChunk: Int) extends SimpleLinearGraphStage[ByteString] {
+
+ override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = {
+ new GraphStageLogic(shape) with InHandler with OutHandler {
+
+ private val sourceBuffer = ByteBuffer.allocateDirect(maxBytesPerChunk)
+
+ // This is initialized here to avoid an allocation per onPush, remember to clear before using
+ private val outputBuffer = ByteBuffer.allocateDirect(maxBytesPerChunk)
+ private val decompressingStream = new ZstdDirectBufferDecompressingStreamNoFinalizer(sourceBuffer)
+
+ override def onPush(): Unit = {
+ sourceBuffer.clear()
+ val inputArray = grab(in).toArrayUnsafe()
+ sourceBuffer.put(inputArray)
+ sourceBuffer.flip()
+ if (sourceBuffer.hasRemaining) {
+ var result = ByteString.empty
+ while (sourceBuffer.hasRemaining) {
+ outputBuffer.clear()
+ decompressingStream.read(outputBuffer)
+ outputBuffer.flip()
+ val outputArray = new Array[Byte](outputBuffer.limit())
+ outputBuffer.get(outputArray)
+ result = result.concat(ByteString.fromArrayUnsafe(outputArray))
+ }
+
+ // Fencepost case, it's possible to still have sourceBuffer.hasRemaining and yet compression result
Review Comment:
Comment refers to 'compression result' but this is decompression code. Should be 'decompression result'.
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdDecompressor.scala:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.stream.Attributes
+import pekko.stream.impl.fusing.GraphStages.SimpleLinearGraphStage
+import pekko.stream.stage.{ GraphStageLogic, InHandler, OutHandler }
+import pekko.util.ByteString
+
+import com.github.luben.zstd.ZstdDirectBufferDecompressingStreamNoFinalizer
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdDecompressor(maxBytesPerChunk: Int) extends SimpleLinearGraphStage[ByteString] {
+
+ override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = {
+ new GraphStageLogic(shape) with InHandler with OutHandler {
+
+ private val sourceBuffer = ByteBuffer.allocateDirect(maxBytesPerChunk)
+
+ // This is initialized here to avoid an allocation per onPush, remember to clear before using
+ private val outputBuffer = ByteBuffer.allocateDirect(maxBytesPerChunk)
+ private val decompressingStream = new ZstdDirectBufferDecompressingStreamNoFinalizer(sourceBuffer)
+
+ override def onPush(): Unit = {
+ sourceBuffer.clear()
+ val inputArray = grab(in).toArrayUnsafe()
+ sourceBuffer.put(inputArray)
+ sourceBuffer.flip()
+ if (sourceBuffer.hasRemaining) {
+ var result = ByteString.empty
+ while (sourceBuffer.hasRemaining) {
+ outputBuffer.clear()
+ decompressingStream.read(outputBuffer)
+ outputBuffer.flip()
+ val outputArray = new Array[Byte](outputBuffer.limit())
+ outputBuffer.get(outputArray)
+ result = result.concat(ByteString.fromArrayUnsafe(outputArray))
+ }
+
+ // Fencepost case, it's possible to still have sourceBuffer.hasRemaining and yet compression result
+ // not output anything
+ if (result.nonEmpty)
+ push(out, result)
+ else
+ pull(in)
+ } else pull(in)
+ sourceBuffer.flip()
Review Comment:
The sourceBuffer is flipped after the while loop completes, but at this point sourceBuffer.hasRemaining is already false (the loop just exited). This flip operation serves no purpose and leaves the buffer in an incorrect state. It should be removed.
```suggestion
```
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArrayUnsafe())
+ inputBB.flip()
+ compressingStream.compress(input.asByteBuffer)
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.flip()
Review Comment:
The targetBuffer should be flipped before reading from it with `fromByteBuffer`, not after. The current code reads from an unflipped buffer (in write mode) and then flips it, leaving the buffer in an incorrect state for subsequent operations.
```suggestion
targetBuffer.flip()
val result = ByteString.fromByteBuffer(targetBuffer)
```
##########
stream/src/main/scala/org/apache/pekko/stream/scaladsl/Compression.scala:
##########
@@ -85,4 +87,28 @@ object Compression {
*/
def inflate(maxBytesPerChunk: Int, nowrap: Boolean): Flow[ByteString, ByteString, NotUsed] =
Flow[ByteString].via(new DeflateDecompressor(maxBytesPerChunk, nowrap)).named("inflate")
+
+ /**
+ * @since 2.0.0
+ */
+ def zstd: Flow[ByteString, ByteString, NotUsed] = zstd(Zstd.defaultCompressionLevel())
+
+ /**
+ * Same as [[zstd]] with a custom level and an optional dictionary.
+ * @param level The compression level, must be greater or equal to [[Zstd.minCompressionLevel]] and less than or equal
+ * to [[Zstd.maxCompressionLevel]]
+ * @param dictionary An optional dictionary that can be used for compression
+ * @since 2.0.0
+ */
+ def zstd(level: Int, dictionary: Option[ZstdDictCompress] = None): Flow[ByteString, ByteString, NotUsed] = {
+ require(level <= Zstd.maxCompressionLevel() && level >= Zstd.minCompressionLevel())
+ CompressionUtils.compressorFlow(() => new ZstdCompressor(level, dictionary))
+ }
+
+ /**
+ * @since 2.0.0
+ */
Review Comment:
Missing scaladoc description and parameter documentation. Following the pattern of `gzipDecompress` and `inflate`, this should include a description such as 'Creates a Flow that decompresses a zstd-compressed stream of data.' and document the `maxBytesPerChunk` parameter.
```suggestion
*/
/**
* Creates a Flow that decompresses a zstd-compressed stream of data.
*
* @param maxBytesPerChunk Maximum length of an output [[pekko.util.ByteString]] chunk.
* @since 2.0.0
*/
```
##########
stream/src/main/scala/org/apache/pekko/stream/javadsl/Compression.scala:
##########
@@ -81,4 +87,25 @@ object Compression {
def deflate(level: Int, nowrap: Boolean): Flow[ByteString, ByteString, NotUsed] =
scaladsl.Compression.deflate(level, nowrap).asJava
+ /**
Review Comment:
Missing scaladoc description. This should include a description consistent with the Scala version, such as 'Creates a flow that zstd-compresses a stream of ByteStrings.'
```suggestion
/**
* Creates a flow that zstd-compresses a stream of ByteStrings.
*
```
##########
stream/src/main/scala/org/apache/pekko/stream/scaladsl/Compression.scala:
##########
@@ -85,4 +87,28 @@ object Compression {
*/
def inflate(maxBytesPerChunk: Int, nowrap: Boolean): Flow[ByteString, ByteString, NotUsed] =
Flow[ByteString].via(new DeflateDecompressor(maxBytesPerChunk, nowrap)).named("inflate")
+
+ /**
+ * @since 2.0.0
+ */
+ def zstd: Flow[ByteString, ByteString, NotUsed] = zstd(Zstd.defaultCompressionLevel())
+
+ /**
+ * Same as [[zstd]] with a custom level and an optional dictionary.
+ * @param level The compression level, must be greater or equal to [[Zstd.minCompressionLevel]] and less than or equal
+ * to [[Zstd.maxCompressionLevel]]
+ * @param dictionary An optional dictionary that can be used for compression
+ * @since 2.0.0
+ */
+ def zstd(level: Int, dictionary: Option[ZstdDictCompress] = None): Flow[ByteString, ByteString, NotUsed] = {
+ require(level <= Zstd.maxCompressionLevel() && level >= Zstd.minCompressionLevel())
Review Comment:
The require statement lacks a descriptive error message. Add a message explaining the valid compression level range, e.g., `require(level <= Zstd.maxCompressionLevel() && level >= Zstd.minCompressionLevel(), s\"Compression level must be between \${Zstd.minCompressionLevel()} and \${Zstd.maxCompressionLevel()}, but was $level\")`.
```suggestion
require(
level <= Zstd.maxCompressionLevel() && level >= Zstd.minCompressionLevel(),
s"Compression level must be between ${Zstd.minCompressionLevel()} and ${Zstd.maxCompressionLevel()}, but was $level"
)
```
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdDecompressor.scala:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.stream.Attributes
+import pekko.stream.impl.fusing.GraphStages.SimpleLinearGraphStage
+import pekko.stream.stage.{ GraphStageLogic, InHandler, OutHandler }
+import pekko.util.ByteString
+
+import com.github.luben.zstd.ZstdDirectBufferDecompressingStreamNoFinalizer
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdDecompressor(maxBytesPerChunk: Int) extends SimpleLinearGraphStage[ByteString] {
+
+ override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = {
+ new GraphStageLogic(shape) with InHandler with OutHandler {
+
+ private val sourceBuffer = ByteBuffer.allocateDirect(maxBytesPerChunk)
+
+ // This is initialized here to avoid an allocation per onPush, remember to clear before using
+ private val outputBuffer = ByteBuffer.allocateDirect(maxBytesPerChunk)
+ private val decompressingStream = new ZstdDirectBufferDecompressingStreamNoFinalizer(sourceBuffer)
+
+ override def onPush(): Unit = {
+ sourceBuffer.clear()
+ val inputArray = grab(in).toArrayUnsafe()
+ sourceBuffer.put(inputArray)
+ sourceBuffer.flip()
+ if (sourceBuffer.hasRemaining) {
+ var result = ByteString.empty
+ while (sourceBuffer.hasRemaining) {
+ outputBuffer.clear()
+ decompressingStream.read(outputBuffer)
+ outputBuffer.flip()
+ val outputArray = new Array[Byte](outputBuffer.limit())
+ outputBuffer.get(outputArray)
+ result = result.concat(ByteString.fromArrayUnsafe(outputArray))
+ }
+
+ // Fencepost case, it's possible to still have sourceBuffer.hasRemaining and yet compression result
+ // not output anything
+ if (result.nonEmpty)
+ push(out, result)
+ else
+ pull(in)
+ } else pull(in)
+ sourceBuffer.flip()
+ }
+
+ override def onPull(): Unit = pull(in)
+
+ override def onUpstreamFinish(): Unit = {
+ sourceBuffer.flip()
+ if (sourceBuffer.hasRemaining) {
+ var result = ByteString.empty
+ while (sourceBuffer.hasRemaining) {
+ outputBuffer.clear()
+ decompressingStream.read(outputBuffer)
+ outputBuffer.flip()
+ val outputArray = new Array[Byte](outputBuffer.limit())
+ outputBuffer.get(outputArray)
+ result = result.concat(ByteString.fromArrayUnsafe(outputArray))
+ }
+ decompressingStream.close()
+
+ // Fencepost case, it's possible to still have sourceBuffer.hasRemaining and yet compression result
Review Comment:
Comment refers to 'compression result' but this is decompression code. Should be 'decompression result'.
##########
stream/src/main/scala/org/apache/pekko/stream/scaladsl/Compression.scala:
##########
@@ -85,4 +87,28 @@ object Compression {
*/
def inflate(maxBytesPerChunk: Int, nowrap: Boolean): Flow[ByteString, ByteString, NotUsed] =
Flow[ByteString].via(new DeflateDecompressor(maxBytesPerChunk, nowrap)).named("inflate")
+
+ /**
Review Comment:
Missing scaladoc description. Following the pattern of other methods in this file (like `gzip` and `deflate`), this should include a description such as 'Creates a flow that zstd-compresses a stream of ByteStrings.'
```suggestion
/**
* Creates a flow that zstd-compresses a stream of ByteStrings using the default compression level.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3461351929
> > on the `compress` method in `ZstdCompressor` as this will trigger a cleanup on every `onPush` of the stream. I don't know how expensive this cleanup is, there can be an argument that for this specific case we should defer it for GC as GC cleaning up many ByteBuffers as a batch would be faster, @jrudolph can you comment?
>
> There used to be a reason to use buffer pools especially for direct buffers, probably not least because you might fragment the native heap. Would probably make sense to not churn through them in quick succession. Why not use the buffer pool from the zstd-jni jar?
So for `ZstdDecompressor` this is easy as all of the `DirectBuffers` are the same size. With `ZstdCompressor` its a bit more complicated as one `ByteBuffer` is statically sized (this is the `targetBuffer`) where as the temporary direct input buffers that we create whenever we get an element from a stream has a dynamic size so I am not sure what is the best way to handle this (ideally I would have just done `compressingStream.compress(input.asByteByffer)` where `input` is the `ByteString` from the stream but `ZstdDirectBufferCompressingStreamNoFinalizer.compress` only supports direct byte buffers).
I guess in this case using a pool of `ByteBuffers` would be better anyways, even with a static size? In which case I would have to redo the logic a bit but it is what it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2469518733
##########
project/Dependencies.scala:
##########
@@ -351,7 +351,8 @@ object Dependencies {
// pekko stream
- lazy val stream = l ++= Seq[sbt.ModuleID](reactiveStreams, TestDependencies.scalatest)
+ lazy val stream =
+ l ++= Seq[sbt.ModuleID](reactiveStreams, "com.github.luben" % "zstd-jni" % "1.5.7-6", TestDependencies.scalatest)
Review Comment:
> How about having a separate pekko-streams-ztsd or pekko-streams-compression so that pekko-streams core lib doesn't need this dependency? ZIO have done something similar where a small number of compression operators are in zio-streams but others have their own libs, eg zio-streams-compress-zstd.
You could make the same argument as to why lz4Java isn't its own artifact, or any of the other compression algorithms. Also, due to the way the current API is designed, all of the compression algorithms are contained within the pekko-streams `Compression` object (which we could monkey patch to add more methods as a separate artifact, but thats starting to look ugly already). Ontop of that, when adding zstd compression to pekko http request/response headers (see https://github.com/apache/pekko-http/issues/860), we would likewise have to create yet another artifact for pekko-http.
I understand the sentiment, but honestly this is creating a huge amount of ceremony for no real benefit and the only reason we are having this discussion is because of how behind Java is in adding what are de facto standard compression algorithms to the stdlib (of which zstd has been for at least half a decade now).
Put differently, the entire point of `Compression` object for scaladsl/javadsl is to contain the standard compression/decompression algorithms, and zstd is one of those. While having to use zstd-jni is not ideal, to the end user there is no difference as it already supports almost every architecture that would be used in the wild.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2472123454
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArrayUnsafe())
+ inputBB.flip()
+ compressingStream.compress(input.asByteBuffer)
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.flip()
Review Comment:
The ordering here doesn't really matter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2476327751
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.io.ByteBufferCleaner
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArrayUnsafe())
+ inputBB.flip()
+ compressingStream.compress(inputBB)
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.flip()
+ if (ByteBufferCleaner.isSupported)
+ ByteBufferCleaner.clean(inputBB)
+ result
+ }
+
+ override def flush(): ByteString = {
+ targetBuffer.flip()
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.clear()
+ compressingStream.flush()
+ result
+ }
+
+ override def finish(): ByteString = {
+ compressingStream.close()
+ targetBuffer.flip()
+ val arr = Array.ofDim[Byte](targetBuffer.limit())
+ targetBuffer.get(arr)
+ val result = ByteString.fromArrayUnsafe(arr)
+ if (ByteBufferCleaner.isSupported)
+ ByteBufferCleaner.clean(targetBuffer)
+ result
+ }
+
+ override def compressAndFlush(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArray)
Review Comment:
Should we mark it `@ApiMayChange` too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3501121627
> @mdedetrich, in Netty, we have `ZstdOptions` too. I think we can take a look at Netty's one for some inspiration before we merge this.
So I have been waiting for a response to https://github.com/apache/pekko/pull/2409#issuecomment-3469014680 but haven't gotten one yet.
In thinking about it, I am increasingly getting to the view of implementing the more complicated but smart solution, i.e. buffering data up until upper limit or if downstream signals demand. Reason being I think this is a fairly novel approach that takes advantage of pekko-streams bi-directional nature and it can even makes good advertisement for pekko streams (can make a blog post about it)
> I think we should have this in 1.3.0 release.
This would have to be a separate artifact and that would open a whole can of worms, not sure I could even make it for 1.3.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3600633105
https://github.com/netty/netty/pull/15861
@mdedetrich There is a related pr in netty too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3595831135
> @mdedetrich are you still planning further changes? I see you pushed some changes yesterday but AFAICT there's still some open review comments
Yes, I have a couple of tests to fix to pass and then I will push changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "raboof (via GitHub)" <gi...@apache.org>.
raboof commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3468221534
> > This is a trade-off between latency and compression performance: in a naive implementation, if we for instance 'keep asking elements to buffer' on the boundary of 2 elements of a gRPC stream, it would delay sending+processing the first element while waiting for the next element to show up - which could take a long time on 'event streams', for example. It's possible to avoid this problem: we could 'keep asking for elements to buffer' only when there's currently no demand from downstream, but compress and send 'everything we have' as soon as demand is signaled. That might be significantly more complex, though.
>
> Good point, I did miss that. I am currently doing an initial implementation that follows the logic I described, but it can be improved on later to bypass the accumulation of data if downstream is demanding data right now. I assume that you would use `hasBeenPulled(port)` to figure out if there is downstream demand?
From https://pekko.apache.org/docs/pekko/current/stream/stream-customize.html it looks like `hasBeenPulled` is for input ports, and `isAvailable` would be the thing to check. You'd also need to make sure that your flow signals demand even when downstream doesn't (until the buffer is 'full').
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2469518733
##########
project/Dependencies.scala:
##########
@@ -351,7 +351,8 @@ object Dependencies {
// pekko stream
- lazy val stream = l ++= Seq[sbt.ModuleID](reactiveStreams, TestDependencies.scalatest)
+ lazy val stream =
+ l ++= Seq[sbt.ModuleID](reactiveStreams, "com.github.luben" % "zstd-jni" % "1.5.7-6", TestDependencies.scalatest)
Review Comment:
> How about having a separate pekko-streams-ztsd or pekko-streams-compression so that pekko-streams core lib doesn't need this dependency? ZIO have done something similar where a small number of compression operators are in zio-streams but others have their own libs, eg zio-streams-compress-zstd.
You could make the same argument as to why lz4Java isn't its own artifact, or any of the other compression algorithms. Also, due to the way the current API is designed, all of the compression algorithms are contained within the `Compression` object (which we could monkey patch to add more methods as a separate artifact, but thats starting to look ugly already). Ontop of that, when adding zstd compression to pekko http request/response headers, we would likewise have to create yet another artifact in pekko-http.
I understand the sentiment, but honestly this is creating a huge amount of ceremony for no real benefit and the only reason we are having this discussion is because of how behind Java is in adding what are de facto standard compression algorithms to the stdlib (of which zstd has been for at least half a decade now).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2469507179
##########
project/Dependencies.scala:
##########
@@ -351,7 +351,8 @@ object Dependencies {
// pekko stream
- lazy val stream = l ++= Seq[sbt.ModuleID](reactiveStreams, TestDependencies.scalatest)
+ lazy val stream =
+ l ++= Seq[sbt.ModuleID](reactiveStreams, "com.github.luben" % "zstd-jni" % "1.5.7-6", TestDependencies.scalatest)
Review Comment:
On another note there is a silver lining here, since zstd-jni uses JNI, the C shared objects which are platform dependent are contained within the jar artifact. The `"com.github.luben" % "zstd-jni" % "1.5.7-6"` artifact contains the C shared objects for **all** of the platforms (see https://github.com/luben/zstd-jni?tab=readme-ov-file#binary-releases for which ones are supported) but there are platform dependent jars if you only care about a single platform.
This does give some credence to using provided, as a user can provide their own zstd-jni artifact however there are still some things to note here. If we end up using `provided` then we force all pekko end users to add a zstd-jni artifact to their dependencies (with `"com.github.luben" % "zstd-jni" % "1.5.7-6"` being the documented default) as its not going to be including by in the dependency resolution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2469483179
##########
project/Dependencies.scala:
##########
@@ -351,7 +351,8 @@ object Dependencies {
// pekko stream
- lazy val stream = l ++= Seq[sbt.ModuleID](reactiveStreams, TestDependencies.scalatest)
+ lazy val stream =
+ l ++= Seq[sbt.ModuleID](reactiveStreams, "com.github.luben" % "zstd-jni" % "1.5.7-6", TestDependencies.scalatest)
Review Comment:
Making it optional/provided is not going to give as any benefits as we would need to force users to redundantly provide a zstd implementation (which is going to be `"com.github.luben" % "zstd-jni" % "1.5.7-6"` anyways).
To do this properly, we would need to make a zstd agnostic API and then provided concrete implementations in a different artifact and considering that there is only a single candidate for a proper implementation (as discussed at https://github.com/apache/pekko/issues/2404#issuecomment-3450106736) we wouldn't gain any benefit out of it.
Hence zstd is being treated the same way as lz4Java is (for the same reasons), where we have a [simple direct dependency](https://github.com/apache/pekko/blob/1b54c97914e91af287af8681875bee317e968c58/project/Dependencies.scala#L106)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2474062079
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.io.ByteBufferCleaner
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArrayUnsafe())
+ inputBB.flip()
+ compressingStream.compress(inputBB)
+ val result = ByteString.fromByteBuffer(targetBuffer)
Review Comment:
I will rewrite the implementation to use the pekko `DirectByteBufferPool`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3461371718
> > on the `compress` method in `ZstdCompressor` as this will trigger a cleanup on every `onPush` of the stream. I don't know how expensive this cleanup is, there can be an argument that for this specific case we should defer it for GC as GC cleaning up many ByteBuffers as a batch would be faster, @jrudolph can you comment?
>
> There used to be a reason to use buffer pools especially for direct buffers, probably not least because you might fragment the native heap. Would probably make sense to not churn through them in quick succession. Why not use the buffer pool from the zstd-jni jar?
Ah I just noticed the buffer pool in zstd-jni, it wasn't being used in the sample code with `ZstdDirectBufferCompressingStream`/ZstdDirectBufferDecompressingStream`, ill look into it, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-4194327901
> @mdedetrich what's the status of this pr?
It requires upstream changes to zstd in order for the PR to make sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2473844334
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.io.ByteBufferCleaner
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
Review Comment:
And should we provide a helper utility to train the dictionary, too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2473841943
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.io.ByteBufferCleaner
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
Review Comment:
dictionary can be a `OptionVal`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3460360966
> @mdedetrich Not sure if it's better to be a dedicated module, I will review it this weekend.
There is a discussion on this topic at https://github.com/apache/pekko/pull/2409#discussion_r2469468261, I personally think making this an extra module adds a lot of ceremony (as we would also have to add it another module in pekko-http) for no real benefit to end user but if a lot of people feel strongly about it I can reconsider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3614314634
So upstream has commented at https://github.com/luben/zstd-jni/issues/324#issuecomment-3614203657, it may make more sense to add this functionality (specifically `ZstdCompressCtx.compressByteBufferStream()` and `ZstdDecompressCtx.decompressByteBufferStream()`) into zstd-jni proper as it would help others and make the pekko-streams implementation also far more trivial.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2560536479
##########
stream/src/main/scala/org/apache/pekko/stream/javadsl/compression/ZstdDictionary.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.javadsl.compression;
+
+import org.apache.pekko.annotation.InternalApi;
+import org.apache.pekko.stream.impl.io.compression.ZstdDictionaryImpl;
+import org.apache.pekko.util.ByteString;
+import org.apache.pekko.util.OptionalUtil;
+import scala.Option;
+
+import java.util.OptionalInt;
+
+/**
+ * @since 2.0.0
+ */
+public class ZstdDictionary {
+ byte[] dictionary;
Review Comment:
> Is this somehow a standardized "zstd thing" or something still tied to this particular implementation
Yes its entirely standerdized as part of the official C API (see https://github.com/facebook/zstd?tab=readme-ov-file#dictionary-compression-how-to and https://facebook.github.io/zstd/zstd_manual.html#Chapter10.
> think that needs to be documented. We could consider keeping the dictionary-related things `@ApiMayChange` to expedite the mergability of this PR while still giving us room to iterate later.
So the core premise of `ZstdDictionary` is for it to be a stable API that is decoupled from zstd-jni (incase other future better implementations come along). I can still put in `@ApiMayChange` but it is in contention with that core point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3517329970
I have updated the PR, I have made the various zstd constants (i.e. min level, max level, default block size etc etc) be part of the `Compression` object which makes the Pekko API fully separate from the underlying zstd-jni implementation.
I have also rebased against main to use the latest configurable `autoFlush` changes add as part of https://github.com/apache/pekko/pull/2467
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3462868253
* avoiding unnecessary intermediate transformation to InputStreams/ByteBuffers seems good
* worth prototyping anyway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2576984917
##########
actor/src/main/java/org/apache/pekko/io/ByteBufferCleaner.java:
##########
@@ -36,7 +37,8 @@
*
* <p>See <a href=https://bugs.openjdk.java.net/browse/JDK-4724038>JDK-4724038</a>
*/
-final class ByteBufferCleaner {
+@InternalStableApi
Review Comment:
> Are you planning to use this API in other Pekko modules? If not then we can remove this annotation here. If so then we should add a note to the scaladoc pointing to where it's (going to be) used.
As you allude to, the reason why I added `@InternalStableApi` is that I can imagine cases where other Pekko libraries (i.e. pekko-http or even pekko-connectors) might want to use `ByteBufferCleaner` and obviously this would need to be available in the Pekko 2.0.0 release to fullfil this.
Also this API is quite stable, hence why I added `@InternalStableApi`, but yes in general it was a forward thinking rationality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "raboof (via GitHub)" <gi...@apache.org>.
raboof commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2577097461
##########
actor/src/main/java/org/apache/pekko/io/ByteBufferCleaner.java:
##########
@@ -36,7 +37,8 @@
*
* <p>See <a href=https://bugs.openjdk.java.net/browse/JDK-4724038>JDK-4724038</a>
*/
-final class ByteBufferCleaner {
+@InternalStableApi
Review Comment:
OK. I'm not super fond of committing ourselves to `@InternalStableApi` "just in case" without a specific plan on where to use it. Also we should seriously consider using the implementation from `commons-io` - definitely in satellite projects, but also here. The use of `sun.misc.Unsafe` is icky as well.
I'm OK with this not blocking the merge though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2474051229
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.io.ByteBufferCleaner
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
Review Comment:
This won't give as anything, as we would have to convert from an `Optional`/`Optional` to an `OptionVal` since the scaladsl/javadsl uses `Option`/`Optional`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2469518733
##########
project/Dependencies.scala:
##########
@@ -351,7 +351,8 @@ object Dependencies {
// pekko stream
- lazy val stream = l ++= Seq[sbt.ModuleID](reactiveStreams, TestDependencies.scalatest)
+ lazy val stream =
+ l ++= Seq[sbt.ModuleID](reactiveStreams, "com.github.luben" % "zstd-jni" % "1.5.7-6", TestDependencies.scalatest)
Review Comment:
> How about having a separate pekko-streams-ztsd or pekko-streams-compression so that pekko-streams core lib doesn't need this dependency? ZIO have done something similar where a small number of compression operators are in zio-streams but others have their own libs, eg zio-streams-compress-zstd.
You could make the same argument as to why lz4Java isn't its own artifact, or any of the other compression algorithms. Also, due to the way the current API is designed, all of the compression algorithms are contained within the pekko-streams `Compression` object (which we could monkey patch to add more methods as a separate artifact, but thats starting to look ugly already). Ontop of that, when adding zstd compression to pekko http request/response headers (see https://github.com/apache/pekko-http/issues/860), we would likewise have to create yet another artifact in pekko-http.
I understand the sentiment, but honestly this is creating a huge amount of ceremony for no real benefit and the only reason we are having this discussion is because of how behind Java is in adding what are de facto standard compression algorithms to the stdlib (of which zstd has been for at least half a decade now).
Put differently, the entire point of `Compression` object for scaladsl/javadsl is to contain the standard compression/decompression algorithms, and zstd is one of those. While having to use zstd-jni is not ideal, to the end user there is no difference as it already supports almost every architecture that would be used in the wild.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2472114664
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdDecompressor.scala:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.stream.Attributes
+import pekko.stream.impl.fusing.GraphStages.SimpleLinearGraphStage
+import pekko.stream.stage.{ GraphStageLogic, InHandler, OutHandler }
+import pekko.util.ByteString
+
+import com.github.luben.zstd.ZstdDirectBufferDecompressingStreamNoFinalizer
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdDecompressor(maxBytesPerChunk: Int) extends SimpleLinearGraphStage[ByteString] {
+
+ override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = {
+ new GraphStageLogic(shape) with InHandler with OutHandler {
+
+ private val sourceBuffer = ByteBuffer.allocateDirect(maxBytesPerChunk)
+
+ // This is initialized here to avoid an allocation per onPush, remember to clear before using
+ private val outputBuffer = ByteBuffer.allocateDirect(maxBytesPerChunk)
+ private val decompressingStream = new ZstdDirectBufferDecompressingStreamNoFinalizer(sourceBuffer)
+
+ override def onPush(): Unit = {
+ sourceBuffer.clear()
+ val inputArray = grab(in).toArrayUnsafe()
+ sourceBuffer.put(inputArray)
+ sourceBuffer.flip()
+ if (sourceBuffer.hasRemaining) {
+ var result = ByteString.empty
+ while (sourceBuffer.hasRemaining) {
+ outputBuffer.clear()
+ decompressingStream.read(outputBuffer)
+ outputBuffer.flip()
+ val outputArray = new Array[Byte](outputBuffer.limit())
+ outputBuffer.get(outputArray)
+ result = result.concat(ByteString.fromArrayUnsafe(outputArray))
+ }
+
+ // Fencepost case, it's possible to still have sourceBuffer.hasRemaining and yet compression result
+ // not output anything
+ if (result.nonEmpty)
+ push(out, result)
+ else
+ pull(in)
+ } else pull(in)
+ sourceBuffer.flip()
+ }
+
+ override def onPull(): Unit = pull(in)
+
+ override def onUpstreamFinish(): Unit = {
+ sourceBuffer.flip()
+ if (sourceBuffer.hasRemaining) {
+ var result = ByteString.empty
+ while (sourceBuffer.hasRemaining) {
+ outputBuffer.clear()
+ decompressingStream.read(outputBuffer)
+ outputBuffer.flip()
+ val outputArray = new Array[Byte](outputBuffer.limit())
+ outputBuffer.get(outputArray)
+ result = result.concat(ByteString.fromArrayUnsafe(outputArray))
+ }
+ decompressingStream.close()
+
+ // Fencepost case, it's possible to still have sourceBuffer.hasRemaining and yet compression result
Review Comment:
Done
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdDecompressor.scala:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.stream.Attributes
+import pekko.stream.impl.fusing.GraphStages.SimpleLinearGraphStage
+import pekko.stream.stage.{ GraphStageLogic, InHandler, OutHandler }
+import pekko.util.ByteString
+
+import com.github.luben.zstd.ZstdDirectBufferDecompressingStreamNoFinalizer
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdDecompressor(maxBytesPerChunk: Int) extends SimpleLinearGraphStage[ByteString] {
+
+ override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = {
+ new GraphStageLogic(shape) with InHandler with OutHandler {
+
+ private val sourceBuffer = ByteBuffer.allocateDirect(maxBytesPerChunk)
+
+ // This is initialized here to avoid an allocation per onPush, remember to clear before using
+ private val outputBuffer = ByteBuffer.allocateDirect(maxBytesPerChunk)
+ private val decompressingStream = new ZstdDirectBufferDecompressingStreamNoFinalizer(sourceBuffer)
+
+ override def onPush(): Unit = {
+ sourceBuffer.clear()
+ val inputArray = grab(in).toArrayUnsafe()
+ sourceBuffer.put(inputArray)
+ sourceBuffer.flip()
+ if (sourceBuffer.hasRemaining) {
+ var result = ByteString.empty
+ while (sourceBuffer.hasRemaining) {
+ outputBuffer.clear()
+ decompressingStream.read(outputBuffer)
+ outputBuffer.flip()
+ val outputArray = new Array[Byte](outputBuffer.limit())
+ outputBuffer.get(outputArray)
+ result = result.concat(ByteString.fromArrayUnsafe(outputArray))
+ }
+
+ // Fencepost case, it's possible to still have sourceBuffer.hasRemaining and yet compression result
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2472754494
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
Review Comment:
Possibly? I just read the implementation now and it appears that the main use case of `ByteBufferPool ` is if you need a pool of buffers where as in our case we only have a static set amount of direct byte buffers and they have different sizes.
However from reading the `ByteBufferPool` code I did found another improvement, we can use `ByteBufferCleaner.clean(byteBuffer)` to clean up the byte buffers in a timely manner (will implement this).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "raboof (via GitHub)" <gi...@apache.org>.
raboof commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3467244878
> So I have been thinking about this, and I am coming to the conclusion that I should just reimplement the zstd compressor from scratch using the low level zstd-jni API (i.e. `public int compress(byte[] dst, byte[] src)` within the `ZstdCompressCtx` class. The reasoning is as follows
>
> * `ZstdDirectBufferCompressingStreamNoFinalizer` and related classes were designed to work with sources that provide direct `ByteBuffer`'s i.e. Netty/Java NIO however in our case we are dealing with `ByteString` where we don't have direct `ByteBuffer`'s (`ByteString.asByteBuffer` is not a direct byte buffer) so we always need up having to copy into an array that then gets pushed into a direct `ByteBuffer` before going into the compress function. Using `ZstdCompressCtx` we can just directly feed/get the byte arrays.
This seems like a convincing argument
> * Now lets get onto buffering
>
> * When it comes to compression, whether we want to buffer or not actually depends on the size of the `ByteString` element thats incoming onto the stream.
> * If the element happens to be really "big", its actually ideal to compress that element as is without feeding it into any buffer as the larger the element, the better the compression ratio is. In this case we can just use `input.toByteArrayUnsafe` and feed it directly into `ZstdCompressCtx.compress`, even better we don't need to worry about any copying in happy path case for small `ByteString`.
This sounds like the resulting bytes will be different depending on how the input was split, which could be nondeterministic (e.g. depending on network conditions). I guess it's fine since this will most likely be used for 'ephemeral' streams, we can 'blame' the nondeterminism on the component that does the splitting, and someone who cares about determinism could insert a component to convert the input into fixed-size chunks.
> * If the input `ByteStrings` is "small", then we can keep on asking for elements to buffer and once it hits a constant size limit we can then feed it into `ZstdCompressCtx.compress`
This is a trade-off between latency and compression performance: in a naive implementation, if we for instance 'keep asking elements to buffer' on the boundary of 2 elements of a gRPC stream, it would delay sending+processing the first element while waiting for the next element to show up - which could take a long time on 'event streams', for example. It's possible to avoid this problem: we could 'keep asking for elements to buffer' only when there's currently no demand from downstream, but compress and send 'everything we have' as soon as demand is signaled. That might be significantly more complex, though.
> * When it comes to decompression, the logic can remain as is. We don't need to use a `DirectByteBufferPool` because we only create 2 buffers per stream instantiation (**not** per element of stream) and those `ByteBuffer`'s are reused for each incoming element, albeit it should be updated so that those `ByteBuffer`'s get cleaned up with `if (ByteBufferCleaner.isSupported) ByteBufferCleaner.clean(byteBuffer)`. Using `ZstdDecompressCtx` isn't going to gain us much net benefit as we need to use the native streaming capabilities of zstd-jni to accept incoming data until we can get to a point where we can decompress.
Sounds good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3468007365
> This sounds like the resulting bytes will be different depending on how the input was split, which could be nondeterministic (e.g. depending on network conditions). I guess it's fine since this will most likely be used for 'ephemeral' streams, we can 'blame' the nondeterminism on the component that does the splitting, and someone who cares about determinism could insert a component to convert the input into fixed-size chunks.
Yes this is precisely my thinking as well. Hypothetically speaking someone could just feed 1gbyte sized chunks as individual stream elements but if you are doing that then your already in a troublesome spot, regardless of compression or not.
> This is a trade-off between latency and compression performance: in a naive implementation, if we for instance 'keep asking elements to buffer' on the boundary of 2 elements of a gRPC stream, it would delay sending+processing the first element while waiting for the next element to show up - which could take a long time on 'event streams', for example. It's possible to avoid this problem: we could 'keep asking for elements to buffer' only when there's currently no demand from downstream, but compress and send 'everything we have' as soon as demand is signaled. That might be significantly more complex, though.
Good point, I did miss that. I am currently doing an initial implementation that follows this logic, but it can be improved on later to bypass the accumulation of data if downstream is demanding data right now. I assume that you would use `hasBeenPulled(port)` to figure out if there is downstream demand?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "jrudolph (via GitHub)" <gi...@apache.org>.
jrudolph commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3468248622
> `ZstdDirectBufferCompressingStreamNoFinalizer` and related classes were designed to work with sources that provide direct `ByteBuffer`'s i.e. Netty/Java NIO however in our case we are dealing with `ByteString` where we don't have direct `ByteBuffer`'s (`ByteString.asByteBuffer` is not a direct byte buffer) so we always need up having to copy into an array that then gets pushed into a direct `ByteBuffer` before going into the compress function. Using `ZstdCompressCtx` we can just directly feed/get the byte arrays.
You can ignore the copy cost, compression is not cheap anyway. Also, the copying is likely happening anyways as you otherwise have to globally lock the heap array and interfere with GC...
> Now lets get onto buffering
It's a complex topic how and when a (streaming) compressor should outputs compressed data. Roughly speaking, for best compression, you leave the decision completely to the compressor. For more interactive usage you want to send any data as early as possible (kind of a nagle situation but for compression). The compressor cannot know what you want and we cannot know what the user wants. Also note, that streaming does not equal interactive usage, you might want to stream data to run with less memory, so for the (native) compressor API you don't want to make the size of the chunks part of the decision of when to flush.
For our streaming API, the problem is that we cannot easily provide the signal when to flush using a raw ByteString based stream. For that reason, in the deflate compressor we allow to set a flush mode and flush after each ByteString. That way the user still has full control over when the flushing does happen (by buffering data in the stream if required). We should not add any inner logic on top of that. Let's not overthink the interface between our compression component and the native component. In general, if we can provide all of incoming data in one go to the compressor that's ok because it's already on the heap anyway and owned by us. But we should never hold anything back.
> When it comes to decompression, the logic can reman as is. We don't need to use a `DirectByteBufferPool` because we only create 2 buffers per stream instantiation (**not** per element of stream) and those `ByteBuffer`'s are reused for each incoming element, albeit it should be updated so that those `ByteBuffer`'s get cleaned up with `if (ByteBufferCleaner.isSupported) ByteBufferCleaner.clean(byteBuffer)`. Using `ZstdDecompressCtx` isn't going to gain us much net benefit as we need to use the native streaming capabilities of zstd-jni to accept incoming data until we can get to a point where we can decompress.
In any case, we should avoid cycling through direct buffers. E.g. using one for the lifetime of the whole stream could be acceptable.
Btw. zstd is somewhat different from deflate in that regard, you can only ever decompress full blocks while deflate can at least in theory start streaming while still receiving a block (see https://github.com/facebook/zstd/issues/1359 and https://fastcompression.blogspot.com/2016/04/working-with-streaming.html, the tldr; is that a compressed block is max 128kb anyway and that usually dwarfs any LZ77 context buffers you might have to keep around anyways).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "raboof (via GitHub)" <gi...@apache.org>.
raboof commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2556773317
##########
stream/src/main/scala/org/apache/pekko/stream/javadsl/compression/ZstdDictionary.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.javadsl.compression;
+
+import org.apache.pekko.annotation.InternalApi;
+import org.apache.pekko.stream.impl.io.compression.ZstdDictionaryImpl;
+import org.apache.pekko.util.ByteString;
+import org.apache.pekko.util.OptionalUtil;
+import scala.Option;
+
+import java.util.OptionalInt;
+
+/**
+ * @since 2.0.0
+ */
+public class ZstdDictionary {
+ byte[] dictionary;
Review Comment:
Is this somehow a standardized "zstd thing" or something still tied to this particular implementation? I think that needs to be documented. We could consider keeping the dictionary-related things `@ApiMayChange` to expedite the mergability of this PR while still giving us room to iterate later.
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+import com.github.luben.zstd.{ ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+import org.apache.pekko
+import org.apache.pekko.stream.scaladsl.Compression
+import pekko.annotation.InternalApi
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Compression.ZstdDefaultCompressionLevel, dictionary: Option[ZstdDictionaryImpl] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach { dict =>
Review Comment:
(nit: I find `match` almost always easier to read compared to `foreach` on `Option`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2573522834
##########
stream/src/main/scala/org/apache/pekko/stream/javadsl/compression/ZstdDictionary.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.javadsl.compression;
+
+import org.apache.pekko.annotation.InternalApi;
+import org.apache.pekko.stream.impl.io.compression.ZstdDictionaryImpl;
+import org.apache.pekko.util.ByteString;
+import org.apache.pekko.util.OptionalUtil;
+import scala.Option;
+
+import java.util.OptionalInt;
+
+/**
+ * @since 2.0.0
+ */
+public class ZstdDictionary {
+ byte[] dictionary;
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "raboof (via GitHub)" <gi...@apache.org>.
raboof commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2576985405
##########
docs/src/main/paradox/stream/operators/Compression/zstd.md:
##########
@@ -0,0 +1,30 @@
+# Compression.zstd
+
+Creates a flow that zstd-compresses a stream of ByteStrings.
+
+@ref[Compression operators](../index.md#compression-operators)
+
+## Signature
+
+@apidoc[Compression.zstd](stream.*.Compression$) { scala="#zstd:org.apache.pekko.stream.scaladsl.Flow[org.apache.pekko.util.ByteString,org.apache.pekko.util.ByteString,org.apache.pekko.NotUsed]" java="#zstd()" }
+
+## Description
+
+Creates a flow that zstd-compresses a stream of ByteStrings. Note that the compressor
+will SYNC_FLUSH after every @apidoc[util.ByteString] so that it is guaranteed that every @apidoc[util.ByteString]
Review Comment:
That is correct. I wasn't trying to say the original comment is wrong (it isn't), I meant to say that it would be valuable to highlight that the way the incoming data is split up influences the result of the compression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3606898178
@raboof @pjfanning Apologies for missing this for the Pekko 2.0.0-M1 release train, I am still working on this but its taking longer as expected. I am actually currently implementing this work as a test in zstd-jni to make my life easier and also with the hope of getting feedback from the author that it is on the right track.
Essentially the problem is that there isn't any real prior art for this, all of the current zstd streaming tests in zstd-jni either involve creating a direct `ByteBuffer` with the same size of the input data (which we obviously don't want to do in pekko-streams as these allocations are extremely expensive) or the input source is already a direct `ByteBuffer` (i.e. java nio).
I am slightly annoying that I didn't make the 2.0.0-M1 release, but as long as we are happy with using a nightly in pekko-http to implement the http request/response zstd work in the end it shouldn't be that problematic. You could also argue its better this way, as pekko 2.0.0 milestone was primarily about updating all of the core libraries/jvm versions + removing deprecated code and bookkeeping and we can have 2.0.0-M2 be about major features.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2576560301
##########
docs/src/main/paradox/stream/operators/Compression/zstd.md:
##########
@@ -0,0 +1,30 @@
+# Compression.zstd
+
+Creates a flow that zstd-compresses a stream of ByteStrings.
+
+@ref[Compression operators](../index.md#compression-operators)
+
+## Signature
+
+@apidoc[Compression.zstd](stream.*.Compression$) { scala="#zstd:org.apache.pekko.stream.scaladsl.Flow[org.apache.pekko.util.ByteString,org.apache.pekko.util.ByteString,org.apache.pekko.NotUsed]" java="#zstd()" }
+
+## Description
+
+Creates a flow that zstd-compresses a stream of ByteStrings. Note that the compressor
+will SYNC_FLUSH after every @apidoc[util.ByteString] so that it is guaranteed that every @apidoc[util.ByteString]
Review Comment:
Actually its the opposite, since a sync flush happens on every stream element it means that there is a direct one to one relationship of every element having the exact same data compressed (there is actually a test that validates this)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "raboof (via GitHub)" <gi...@apache.org>.
raboof commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3570656232
Needs a rebase for the merge conflict and the fix for the #2476 failure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3506519953
zstd is indeed quite complex. In our work, it actually includes a complete platform and toolchain for dictionary training, end-to-end deployment, version synchronization, etc.
Therefore, could we consider releasing a separate zstd toolkit, such as pekko-zstd or pekko-http-zstd, etc.? This would allow for a more complete design and implementation, and also enable independent evolution.
Without a dictionary, zstd is simply a slightly faster gzip implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2478901764
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.io.ByteBufferCleaner
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArrayUnsafe())
+ inputBB.flip()
+ compressingStream.compress(inputBB)
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.flip()
+ if (ByteBufferCleaner.isSupported)
+ ByteBufferCleaner.clean(inputBB)
+ result
+ }
+
+ override def flush(): ByteString = {
+ targetBuffer.flip()
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.clear()
+ compressingStream.flush()
+ result
+ }
+
+ override def finish(): ByteString = {
+ compressingStream.close()
+ targetBuffer.flip()
+ val arr = Array.ofDim[Byte](targetBuffer.limit())
+ targetBuffer.get(arr)
+ val result = ByteString.fromArrayUnsafe(arr)
+ if (ByteBufferCleaner.isSupported)
+ ByteBufferCleaner.clean(targetBuffer)
+ result
+ }
+
+ override def compressAndFlush(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArray)
Review Comment:
> Should we mark it `@ApiMayChange` too?
This is internal api, `@ApiMayChange` is only for public API's
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2478898981
##########
actor/src/main/java/org/apache/pekko/io/ByteBufferCleaner.java:
##########
Review Comment:
I had to make both this class and specific methods public because currently its package private to `org.apache.pekko.io` and the pekko stream implementations are not in that package.
I marked the class to be `InternalStableApi`, let me know if should be something else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2469483179
##########
project/Dependencies.scala:
##########
@@ -351,7 +351,8 @@ object Dependencies {
// pekko stream
- lazy val stream = l ++= Seq[sbt.ModuleID](reactiveStreams, TestDependencies.scalatest)
+ lazy val stream =
+ l ++= Seq[sbt.ModuleID](reactiveStreams, "com.github.luben" % "zstd-jni" % "1.5.7-6", TestDependencies.scalatest)
Review Comment:
Making it optional/provided is not going to give as any benefits as we would need to force users to redundantly provide a zstd implementation (which is going to be `"com.github.luben" % "zstd-jni" % "1.5.7-6"` anyways).
To do this properly, we would need to make a zstd agnostic API and then provided concrete implementations in a different artifact and considering that there is only a single candidate for a proper implementation (as discussed at https://github.com/apache/pekko/issues/2404#issuecomment-3450106736) we wouldn't gain any benefit out of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2469518733
##########
project/Dependencies.scala:
##########
@@ -351,7 +351,8 @@ object Dependencies {
// pekko stream
- lazy val stream = l ++= Seq[sbt.ModuleID](reactiveStreams, TestDependencies.scalatest)
+ lazy val stream =
+ l ++= Seq[sbt.ModuleID](reactiveStreams, "com.github.luben" % "zstd-jni" % "1.5.7-6", TestDependencies.scalatest)
Review Comment:
> How about having a separate pekko-streams-ztsd or pekko-streams-compression so that pekko-streams core lib doesn't need this dependency? ZIO have done something similar where a small number of compression operators are in zio-streams but others have their own libs, eg zio-streams-compress-zstd.
You could make the same argument as to why lz4Java isn't its own artifact, or any of the other compression algorithms. Also, due to the way the current API is designed, all of the compression algorithms are contained within the pekko-streams `Compression` object (which we could monkey patch to add more methods as a separate artifact, but thats starting to look ugly already). Ontop of that, when adding zstd compression to pekko http request/response headers (see https://github.com/apache/pekko-http/issues/860), we would likewise have to create yet another artifact for pekko-http.
`zio-streams-compress` is also not the best to compare to, since if you look at https://zio.dev/zio-streams-compress/ you will see they also have separate artifacts for `gzip` and `lz4` something is contained within pekko-streams core artifact for us. zio appears to have a different design ethos here, where they err on the side of hyper modularity and put everything in its own artifact where as pekko is on the other side.
I understand the sentiment, but honestly this is creating a huge amount of ceremony for no real benefit and the only reason we are having this discussion is because of how behind Java is in adding what are de facto standard compression algorithms to the stdlib (of which zstd has been for at least half a decade now). The only reason I can see to use separate artifacts would be to support Pekko 1.x series, but zstd-jni only supports JDK 11 or higher and this would then need to be advertised to end users.
`zio-streams-compress` is also not the best comparison, since if you look at https://zio.dev/zio-streams-compress/ you will see they also have separate artifacts for `gzip` and `lz4`, where as for us these compression algorithms are contained within the core pekko-streams artifact. zio appears to have a different design ethos here, where they err on the side of hyper modularity and put everything in its own artifact where as pekko is on the other side.
Put differently, the entire point of `Compression` object for scaladsl/javadsl is to contain the standard compression/decompression algorithms, and zstd is one of those. While having to use zstd-jni is not ideal, to the end user there is no difference as it already supports almost every architecture that would be used in the wild.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2469518733
##########
project/Dependencies.scala:
##########
@@ -351,7 +351,8 @@ object Dependencies {
// pekko stream
- lazy val stream = l ++= Seq[sbt.ModuleID](reactiveStreams, TestDependencies.scalatest)
+ lazy val stream =
+ l ++= Seq[sbt.ModuleID](reactiveStreams, "com.github.luben" % "zstd-jni" % "1.5.7-6", TestDependencies.scalatest)
Review Comment:
> How about having a separate pekko-streams-ztsd or pekko-streams-compression so that pekko-streams core lib doesn't need this dependency? ZIO have done something similar where a small number of compression operators are in zio-streams but others have their own libs, eg zio-streams-compress-zstd.
You could make the same argument as to why lz4Java isn't its own artifact, or any of the other compression algorithms. Also, due to the way the current API is designed, all of the compression algorithms are contained within the pekko-streams `Compression` object (which we could monkey patch to add more methods as a separate artifact, but thats starting to look ugly already). Ontop of that, when adding zstd compression to pekko http request/response headers (see https://github.com/apache/pekko-http/issues/860), we would likewise have to create yet another artifact for pekko-http which would have its own issues (i.e. `application/zstd` is a standard RFC header which should be contained within pekko-http core but we cannot add it there because the implementation would be in its own artifact).
I understand the sentiment, but honestly this is creating a huge amount of ceremony for no real benefit and the only reason we are having this discussion is because of how behind Java is in adding what are de facto standard compression algorithms to the stdlib (of which zstd has been for at least half a decade now). The only reason I can see to use separate artifacts would be to support Pekko 1.x series, but zstd-jni only supports JDK 11 or higher and this would then need to be advertised to end users.
Put differently, the entire point of `Compression` object for scaladsl/javadsl is to contain the standard compression/decompression algorithms, and zstd is one of those. While having to use zstd-jni is not ideal, to the end user there is no difference as it already supports almost every architecture that would be used in the wild.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2469505671
##########
project/Dependencies.scala:
##########
@@ -351,7 +351,8 @@ object Dependencies {
// pekko stream
- lazy val stream = l ++= Seq[sbt.ModuleID](reactiveStreams, TestDependencies.scalatest)
+ lazy val stream =
+ l ++= Seq[sbt.ModuleID](reactiveStreams, "com.github.luben" % "zstd-jni" % "1.5.7-6", TestDependencies.scalatest)
Review Comment:
How about having a separate pekko-streams-ztsd or pekko-streams-compression so that pekko-streams core lib doesn't need this dependency? ZIO have done something similar where a small number of compression operators are in zio-streams but others have their own libs, eg zio-streams-compress-zstd.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2472119101
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArrayUnsafe())
+ inputBB.flip()
Review Comment:
Fixed and pushed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3461271807
I have just added another commit where we use the `ByteBufferCleaner` to clean up the direct byte buffers rather than resorting to GC. There is an open question about whether we should do
```scala
if (ByteBufferCleaner.isSupported)
ByteBufferCleaner.clean(inputBB)
```
on the `compress` method in `ZstdCompressor` as this will trigger a cleanup on every `onPush` of the stream. I don't know how expensive this cleanup is, there can be an argument that for this specific case we should defer it for GC as GC cleaning up many ByteBuffers as a batch would be faster, @jrudolph can you comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2473889475
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.io.ByteBufferCleaner
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArrayUnsafe())
+ inputBB.flip()
+ compressingStream.compress(inputBB)
+ val result = ByteString.fromByteBuffer(targetBuffer)
Review Comment:
I missed Netty's bytebuf here, when we release the bytebuf, it will returns to the pool, which avloid the fragment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3508037372
Currently doing work on making our own separate `ZstdDictionary` class so we don't depend on the zstd-jni one but its blocked by https://github.com/apache/pekko/pull/2464
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3578994173
@mdedetrich I think you can mark it as experimental and then polish it in a later release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2560538712
##########
.scala-steward.conf:
##########
@@ -5,6 +5,8 @@ updates.pin = [
{ groupId = "org.scala-lang", artifactId = "scala3-library", version = "3.3." }
# sbt-assembly 2.3 causes build issues (https://github.com/apache/pekko/pull/1744)
{ groupId = "com.eed3si9n", artifactId = "sbt-assembly", version = "2.2." }
+ # zstd-jni upgrades are fine as long as they are backwards compatible
+ { groupId = "com.github.luben", artifactId = "zstd-jni", version = "1."}
Review Comment:
> Shouldn't we even allow major version updates? Then we can decide on a case-by-case basis whether the incompatibility that caused them to bump the major version is a reason for us to avoid the update.
Well on the premise of it following semver doing major version updates could be problematic because it can imply behaviour and/or API changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "raboof (via GitHub)" <gi...@apache.org>.
raboof commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2560585971
##########
.scala-steward.conf:
##########
@@ -5,6 +5,8 @@ updates.pin = [
{ groupId = "org.scala-lang", artifactId = "scala3-library", version = "3.3." }
# sbt-assembly 2.3 causes build issues (https://github.com/apache/pekko/pull/1744)
{ groupId = "com.eed3si9n", artifactId = "sbt-assembly", version = "2.2." }
+ # zstd-jni upgrades are fine as long as they are backwards compatible
+ { groupId = "com.github.luben", artifactId = "zstd-jni", version = "1."}
Review Comment:
It _could_, but it's not _necessarily_ problematic. We specifically reserve the right to do major version updates of dependencies when we deem them unproblematic (https://pekko.apache.org/docs/pekko/current/common/binary-compatibility-rules.html#binary-compatibility-of-dependencies). We let scala-steward open PRs for such upgrades, and then when that happens we can decide whether to follow upstream or restrict ourselves to the previous major version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2577498059
##########
actor/src/main/java/org/apache/pekko/io/ByteBufferCleaner.java:
##########
@@ -36,7 +37,8 @@
*
* <p>See <a href=https://bugs.openjdk.java.net/browse/JDK-4724038>JDK-4724038</a>
*/
-final class ByteBufferCleaner {
+@InternalStableApi
Review Comment:
* this code doesn't fail if Unsafe is not accessible - it accepts that GC will eventually clean but it is better for users if this Unsafe code cleans the instance earlier
* it's one of those areas where the Java team keep telling us to stop using Unsafe but in this case, they have not yet given an alternative
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2472705067
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
Review Comment:
There is a ByteBufferPool, can we use that here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2472752957
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
Review Comment:
mark it as not threadsafe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2472235237
##########
stream/src/main/scala/org/apache/pekko/stream/scaladsl/Compression.scala:
##########
@@ -85,4 +87,28 @@ object Compression {
*/
def inflate(maxBytesPerChunk: Int, nowrap: Boolean): Flow[ByteString, ByteString, NotUsed] =
Flow[ByteString].via(new DeflateDecompressor(maxBytesPerChunk, nowrap)).named("inflate")
+
+ /**
+ * @since 2.0.0
+ */
+ def zstd: Flow[ByteString, ByteString, NotUsed] = zstd(Zstd.defaultCompressionLevel())
+
+ /**
+ * Same as [[zstd]] with a custom level and an optional dictionary.
+ * @param level The compression level, must be greater or equal to [[Zstd.minCompressionLevel]] and less than or equal
+ * to [[Zstd.maxCompressionLevel]]
+ * @param dictionary An optional dictionary that can be used for compression
+ * @since 2.0.0
+ */
+ def zstd(level: Int, dictionary: Option[ZstdDictCompress] = None): Flow[ByteString, ByteString, NotUsed] = {
+ require(level <= Zstd.maxCompressionLevel() && level >= Zstd.minCompressionLevel())
+ CompressionUtils.compressorFlow(() => new ZstdCompressor(level, dictionary))
+ }
+
+ /**
+ * @since 2.0.0
+ */
+ def zstdDecompress(maxBytesPerChunk: Int = MaxBytesPerChunkDefault): Flow[ByteString, ByteString, NotUsed] =
+ Flow[ByteString].via(new ZstdDecompressor(maxBytesPerChunk)).named("zstdDecompress")
Review Comment:
Why, its sitting in the same spot that all other compression flows like deflate or gzip do? If we decide to put this in another artifact then sure, but that is another debate.
Either we treat zstd as a standard widely adopted industry compression algorithm (which it is) in which this PR has it in the appropriate place or we put it in another artifact and if so we should have the discussion then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3462846239
So I have been thinking about this, and I am coming to the conclusion that I should just reimplement the zstd compressor from scratch using the low level zstd-jni API (i.e. `public int compress(byte[] dst, byte[] src)` for the respective `ZstdCompressCtx` and `ZstdDecompressCtx` classes. The reasoning is as follows
- `ZstdDirectBufferCompressingStreamNoFinalizer` and related classes were designed to work with sources that provide direct `ByteBuffer`'s i.e. Netty/Java NIO however in our case we are dealing with `ByteString` where we don't have direct `ByteBuffer`'s (ByteString.asByteBuffer` is not a direct byte buffer) so we always need up having to copy into an array that then gets pushed into a direct `ByteBuffer`. Using `ZstdCompressCtx` we can just directly feed/get the byte arrays. While for the compressor we would still likely want to use a direct `ByteBuffer` to buffer incoming `onPush` elements in order to avoid defragmentation
- Now lets get onto buffering
- When it comes to compression, whether we want to buffer or not actually depends on the size of the `ByteString` element thats incoming onto the stream.
- If the element happens to be really "big", its actually ideal to compress that element as is without feeding it into any buffer as the larger the element, the better the compression ratio is. In this case we can just use `input.toByteArrayUnsafe` and feed it directly into `ZstdCompressCtx.compress`, even better we don't need to worry about any copying in happy path case for small `ByteString`.
- If the input `ByteStrings` is "small", then we can keep on asking for elements to buffer and once it hits a constant size limit we can then feed it into `ZstdCompressCtx.compress`
- When it comes to decompression, the logic can reman as is. We don't need to use a `DirectByteBufferPool` because we only create 2 buffers per stream instantiation (**not** per element of stream) and those `ByteBuffer`'s are reused for each incoming element, albeit it should be updated so that those `ByteBuffer`'s get cleaned up with `byteBuffer.isDirect() && ByteBufferCleaner.isSupported`. Using `ZstdDecompressCtx` isn't going to gain us much net benefit as we need to use the native streaming capabilities of zstd-jni to accept incoming data until we can get to a point where we can decompress.
What are peoples thoughts here, is my thinking of the right track here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2474979326
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.io.ByteBufferCleaner
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArrayUnsafe())
+ inputBB.flip()
+ compressingStream.compress(inputBB)
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.flip()
+ if (ByteBufferCleaner.isSupported)
+ ByteBufferCleaner.clean(inputBB)
+ result
+ }
+
+ override def flush(): ByteString = {
+ targetBuffer.flip()
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.clear()
+ compressingStream.flush()
+ result
+ }
+
+ override def finish(): ByteString = {
+ compressingStream.close()
+ targetBuffer.flip()
+ val arr = Array.ofDim[Byte](targetBuffer.limit())
+ targetBuffer.get(arr)
+ val result = ByteString.fromArrayUnsafe(arr)
+ if (ByteBufferCleaner.isSupported)
+ ByteBufferCleaner.clean(targetBuffer)
+ result
+ }
+
+ override def compressAndFlush(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArray)
Review Comment:
Ah right that was an accident, in any case as per https://github.com/apache/pekko/pull/2409#issuecomment-3462846239 this is going to be recoded
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "raboof (via GitHub)" <gi...@apache.org>.
raboof commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3595823505
@mdedetrich are you still planning further changes? I see you pushed some changes yesterday but AFAICT there's still some open review comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "raboof (via GitHub)" <gi...@apache.org>.
raboof commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2576603211
##########
docs/src/main/paradox/stream/operators/Compression/zstd.md:
##########
@@ -0,0 +1,30 @@
+# Compression.zstd
+
+Creates a flow that zstd-compresses a stream of ByteStrings.
+
+@ref[Compression operators](../index.md#compression-operators)
+
+## Signature
+
+@apidoc[Compression.zstd](stream.*.Compression$) { scala="#zstd:org.apache.pekko.stream.scaladsl.Flow[org.apache.pekko.util.ByteString,org.apache.pekko.util.ByteString,org.apache.pekko.NotUsed]" java="#zstd()" }
+
+## Description
+
+Creates a flow that zstd-compresses a stream of ByteStrings. Note that the compressor
+will SYNC_FLUSH after every @apidoc[util.ByteString] so that it is guaranteed that every @apidoc[util.ByteString]
Review Comment:
But that is not the case "if the way the incoming data is cut up into `ByteString`s is not deterministic", right? E.g when the data is coming from the network?
(this is not a fault of the stream compression, but a truth that would be good to highlight)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-4194490774
@mdedetrich Thanks for the update,i think you can prepare a pr with ai help to zstd repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2474051229
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.io.ByteBufferCleaner
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
Review Comment:
This won't give as anything, as we would have to redundantly convert from an `Optional`/`Optional` to an `OptionVal` since the public scaladsl/javadsl uses `Option`/`Optional`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2474613821
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.io.ByteBufferCleaner
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArrayUnsafe())
+ inputBB.flip()
+ compressingStream.compress(inputBB)
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.flip()
+ if (ByteBufferCleaner.isSupported)
+ ByteBufferCleaner.clean(inputBB)
+ result
+ }
+
+ override def flush(): ByteString = {
+ targetBuffer.flip()
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.clear()
+ compressingStream.flush()
+ result
+ }
+
+ override def finish(): ByteString = {
+ compressingStream.close()
+ targetBuffer.flip()
+ val arr = Array.ofDim[Byte](targetBuffer.limit())
+ targetBuffer.get(arr)
+ val result = ByteString.fromArrayUnsafe(arr)
+ if (ByteBufferCleaner.isSupported)
+ ByteBufferCleaner.clean(targetBuffer)
+ result
+ }
+
+ override def compressAndFlush(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArray)
Review Comment:
input.toArrayUnsafe seems ok and better here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3460313894
@mdedetrich Not sure if it's better to be a dedicated module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2472111521
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdDecompressor.scala:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.stream.Attributes
+import pekko.stream.impl.fusing.GraphStages.SimpleLinearGraphStage
+import pekko.stream.stage.{ GraphStageLogic, InHandler, OutHandler }
+import pekko.util.ByteString
+
+import com.github.luben.zstd.ZstdDirectBufferDecompressingStreamNoFinalizer
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdDecompressor(maxBytesPerChunk: Int) extends SimpleLinearGraphStage[ByteString] {
+
+ override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = {
+ new GraphStageLogic(shape) with InHandler with OutHandler {
+
+ private val sourceBuffer = ByteBuffer.allocateDirect(maxBytesPerChunk)
+
+ // This is initialized here to avoid an allocation per onPush, remember to clear before using
+ private val outputBuffer = ByteBuffer.allocateDirect(maxBytesPerChunk)
+ private val decompressingStream = new ZstdDirectBufferDecompressingStreamNoFinalizer(sourceBuffer)
+
+ override def onPush(): Unit = {
+ sourceBuffer.clear()
+ val inputArray = grab(in).toArrayUnsafe()
+ sourceBuffer.put(inputArray)
+ sourceBuffer.flip()
+ if (sourceBuffer.hasRemaining) {
+ var result = ByteString.empty
+ while (sourceBuffer.hasRemaining) {
+ outputBuffer.clear()
+ decompressingStream.read(outputBuffer)
+ outputBuffer.flip()
+ val outputArray = new Array[Byte](outputBuffer.limit())
+ outputBuffer.get(outputArray)
+ result = result.concat(ByteString.fromArrayUnsafe(outputArray))
+ }
+
+ // Fencepost case, it's possible to still have sourceBuffer.hasRemaining and yet compression result
+ // not output anything
+ if (result.nonEmpty)
+ push(out, result)
+ else
+ pull(in)
+ } else pull(in)
+ sourceBuffer.flip()
Review Comment:
This just plain wrong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2508036176
##########
stream/src/main/scala/org/apache/pekko/stream/javadsl/compression/ZstdDictionary.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.javadsl.compression;
+
+import org.apache.pekko.annotation.InternalApi;
+import org.apache.pekko.stream.impl.io.compression.ZstdDictionaryImpl;
+import org.apache.pekko.util.ByteString;
+import org.apache.pekko.util.OptionalUtil;
+import scala.Option;
+
+import java.util.OptionalInt;
+
+/**
+ * @since 2.0.0
Review Comment:
Add java/scala doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2472229497
##########
stream/src/main/scala/org/apache/pekko/stream/scaladsl/Compression.scala:
##########
@@ -85,4 +87,28 @@ object Compression {
*/
def inflate(maxBytesPerChunk: Int, nowrap: Boolean): Flow[ByteString, ByteString, NotUsed] =
Flow[ByteString].via(new DeflateDecompressor(maxBytesPerChunk, nowrap)).named("inflate")
+
+ /**
+ * @since 2.0.0
+ */
+ def zstd: Flow[ByteString, ByteString, NotUsed] = zstd(Zstd.defaultCompressionLevel())
+
+ /**
+ * Same as [[zstd]] with a custom level and an optional dictionary.
+ * @param level The compression level, must be greater or equal to [[Zstd.minCompressionLevel]] and less than or equal
+ * to [[Zstd.maxCompressionLevel]]
+ * @param dictionary An optional dictionary that can be used for compression
+ * @since 2.0.0
+ */
+ def zstd(level: Int, dictionary: Option[ZstdDictCompress] = None): Flow[ByteString, ByteString, NotUsed] = {
+ require(level <= Zstd.maxCompressionLevel() && level >= Zstd.minCompressionLevel())
+ CompressionUtils.compressorFlow(() => new ZstdCompressor(level, dictionary))
+ }
+
+ /**
+ * @since 2.0.0
+ */
+ def zstdDecompress(maxBytesPerChunk: Int = MaxBytesPerChunkDefault): Flow[ByteString, ByteString, NotUsed] =
+ Flow[ByteString].via(new ZstdDecompressor(maxBytesPerChunk)).named("zstdDecompress")
Review Comment:
I mean, what if the user wants to use it in Android?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2472227148
##########
stream/src/main/scala/org/apache/pekko/stream/scaladsl/Compression.scala:
##########
@@ -85,4 +87,28 @@ object Compression {
*/
def inflate(maxBytesPerChunk: Int, nowrap: Boolean): Flow[ByteString, ByteString, NotUsed] =
Flow[ByteString].via(new DeflateDecompressor(maxBytesPerChunk, nowrap)).named("inflate")
+
+ /**
+ * @since 2.0.0
+ */
+ def zstd: Flow[ByteString, ByteString, NotUsed] = zstd(Zstd.defaultCompressionLevel())
+
+ /**
+ * Same as [[zstd]] with a custom level and an optional dictionary.
+ * @param level The compression level, must be greater or equal to [[Zstd.minCompressionLevel]] and less than or equal
+ * to [[Zstd.maxCompressionLevel]]
+ * @param dictionary An optional dictionary that can be used for compression
+ * @since 2.0.0
+ */
+ def zstd(level: Int, dictionary: Option[ZstdDictCompress] = None): Flow[ByteString, ByteString, NotUsed] = {
+ require(level <= Zstd.maxCompressionLevel() && level >= Zstd.minCompressionLevel())
+ CompressionUtils.compressorFlow(() => new ZstdCompressor(level, dictionary))
+ }
+
+ /**
+ * @since 2.0.0
+ */
+ def zstdDecompress(maxBytesPerChunk: Int = MaxBytesPerChunkDefault): Flow[ByteString, ByteString, NotUsed] =
+ Flow[ByteString].via(new ZstdDecompressor(maxBytesPerChunk)).named("zstdDecompress")
Review Comment:
Can we make this live inside `ZstdCompression`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2469518733
##########
project/Dependencies.scala:
##########
@@ -351,7 +351,8 @@ object Dependencies {
// pekko stream
- lazy val stream = l ++= Seq[sbt.ModuleID](reactiveStreams, TestDependencies.scalatest)
+ lazy val stream =
+ l ++= Seq[sbt.ModuleID](reactiveStreams, "com.github.luben" % "zstd-jni" % "1.5.7-6", TestDependencies.scalatest)
Review Comment:
> How about having a separate pekko-streams-ztsd or pekko-streams-compression so that pekko-streams core lib doesn't need this dependency? ZIO have done something similar where a small number of compression operators are in zio-streams but others have their own libs, eg zio-streams-compress-zstd.
You could make the same argument as to why lz4Java isn't its own artifact, or any of the other compression algorithms. Also, due to the way the current API is designed, all of the compression algorithms are contained within the `Compression` object (which we could monkey patch in its own artifact). Ontop of that, when adding zstd compression to pekko http request/response headers, we would likewise have to create yet another artifact in pekko-http.
I understand the sentiment, but honestly this is creating a huge amount of ceremony for no real benefit and the only reason we are having this discussion is because of how behind Java is in adding what are de facto standard compression algorithms to the stdlib (of which zstd has been for at least half a decade now).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3461472543
> > on the `compress` method in `ZstdCompressor` as this will trigger a cleanup on every `onPush` of the stream. I don't know how expensive this cleanup is, there can be an argument that for this specific case we should defer it for GC as GC cleaning up many ByteBuffers as a batch would be faster, @jrudolph can you comment?
>
> There used to be a reason to use buffer pools especially for direct buffers, probably not least because you might fragment the native heap. Would probably make sense to not churn through them in quick succession. Why not use the buffer pool from the zstd-jni jar?
So I just tried doing this and it doesn't work with `ZstdDirectBufferCompressingStreamNoFinalizer`/`ZstdDirectBufferDecompressingStreamNoFinalizer` as they expect direct byte buffers. zstd-jni's `RecyclingBufferPool` is used internally with `ZstdInputStreamNoFinalizer`/`ZstOutputStreamNoFinalizer` which leaves me with 2 options, either
1. Use Pekko's `ByteBufferPool` and rewrite the `ZstdCompressor` logic to deal with fixed size direct `ByteBuffer`'s (shouldn't be too bad)
or
2. Use zstd-jni's `ZstdInputStreamNoFinalizer`/`ZstOutputStreamNoFinalizer`, which would likely be simpler but also would be less performant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3506815559
Our API gateway currently uses zstd, so I know the benefits are significant, both in terms of performance and compression ratio. Although Netty's codec directly depends on zstd-jni, I don't know if Pekko-stream users will find this troublesome; personally, I think it's fine.
By standard standards, I think this PR is already good enough, but I'm unsure how much complex effort you're willing to invest. For example, in our work, some of our APIs have separate configurations, dictionary names, versions, etc., and they negotiate with the gateway during application startup. But here, I think we're probably good enough if you're not considering further complexity and extreme optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3469014680
> You can ignore the copy cost, compression is not cheap anyway. Also, the copying is likely happening anyways as you otherwise have to globally lock the heap array and interfere with GC...
Yes definitely true, my main point here is that even though the copying is already happening behind the scenes and in context of compression the copying is cheap, doing this
```scala
ByteString.fromArrayUnsafe(zstdCompressCtx.compress(input.toArrayUnsafe()))
```
is far simpler and easier to maintain then
```scala
val inputBB = ByteBuffer.allocateDirect(input.size)
inputBB.put(input.toArray)
inputBB.flip()
compressingStream.compress(inputBB)
compressingStream.flush()
targetBuffer.flip()
val arr = new Array[Byte](targetBuffer.limit())
targetBuffer.get(arr)
targetBuffer.clear()
ByteString.fromArrayUnsafe(arr)
```
And that version isn't even the ideal one since we are allocating a direct `ByteBuffer` on every element (which we shouldn't be doing since direct `ByteBuffer` allocation is extremely expensive). Rather we should get a direct `ByteBuffer` from a `DirectByteBufferPool` which will complicate the logic even further since `DirectByteBufferPool` only supports direct `ByteBuffer`'s of a static size (and in our case we have a dynamic `input.size` length)
> It's a complex topic how and when a (streaming) compressor should outputs compressed data. Roughly speaking, for best compression, you leave the decision completely to the compressor. For more interactive usage you want to send any data as early as possible (kind of a nagle situation but for compression). The compressor cannot know what you want and we cannot know what the user wants. Also note, that streaming does not equal interactive usage, you might want to stream data to run with less memory, so for the (native) compressor API you don't want to make the size of the chunks part of the decision of when to flush.
So my mentality behind is https://github.com/apache/pekko/pull/2409#issuecomment-3462846239 is that with all else being equal, its always better to compress larger chunks of data than small. Now the critical part here is all else being equal, what @raboof said earlier is entirely correct in that even if we have stream elements as small as 1 byte (or even less), for some kind of applications you want to compress and send as fast as possible due to latency reasons.
> For our streaming API, the problem is that we cannot easily provide the signal when to flush using a raw ByteString based stream. For that reason, in the deflate compressor we allow to set a flush mode and flush after each ByteString.
Unless I am missing something I don't think the flush mode is configurable as of now, in fact you even left a comment and an issue saying it should be configurable (see https://github.com/apache/pekko/blob/085bb4275a70d8db4c2c53946712ee6b54e74688/stream/src/main/scala/org/apache/pekko/stream/scaladsl/Compression.scala#L29-L34)
Also on the same point, are you saying that @raboof 's suggestion i.e.
> It's possible to avoid this problem: we could 'keep asking for elements to buffer' only when there's currently no demand from downstream, but compress and send 'everything we have' as soon as demand is signaled. That might be significantly more complex, though.
Is not possible or is it just overcomplicated/not necessary? To me, with the assumption that we can reliably figure out if downstream is demanding data this should be able to get the best of both worlds, no? i.e. if downstream is asking for data, we just compress the input element as is and if downstream is not asking for data then we can buffer elements (up to a sane limit) and we send the compressed data as soon as that limit is hit and/or downstream asks for demand again.
Also if this suggested logic ends up being implemented, it would be configurable so users can disable it which would just compress each element as it is.
> That way the user still has full control over when the flushing does happen (by buffering data in the stream if required). We should not add any inner logic on top of that. Let's not overthink the interface between our compression component and the native component. In general, if we can provide all of incoming data in one go to the compressor that's ok because it's already on the heap anyway and owned by us. But we should never hold anything back.
If I am understanding what you are saying, then essentially you are suggesting that the implementation of the compressor would be a trivial `SimpleLinearGraphStage[ByteString]` that on each incoming processing element just does `ByteString.fromArrayUnsafe(zstdCompressCtx.compress(input.toArrayUnsafe()))` and we can call it a day?
Don't have to worry about flushing at all, or buffering or anything like that and if a user wants buffering then they can just use the standard streaming operators to implement it on their own.
> Btw. zstd is somewhat different from deflate in that regard, you can only ever decompress full blocks while deflate can at least in theory start streaming while still receiving a block (see [facebook/zstd#1359](https://github.com/facebook/zstd/issues/1359) and https://fastcompression.blogspot.com/2016/04/working-with-streaming.html, the tldr; is that a compressed block is max 128kb anyway and that usually dwarfs any LZ77 context buffers you might have to keep around anyways).
Thanks for reminding, Zstd actually provides a constant called `Zstd.maxBlockSize` which should be the default value for `maxBytesPerChunk`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-4147753445
@mdedetrich what's the status of this pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "ptrdom (via GitHub) via notifications" <no...@pekko.apache.org>.
ptrdom commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-6542536794
@pjfanning We can probably close this PR, right? I understand that all of the work here was merged through a new PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "jrudolph (via GitHub)" <gi...@apache.org>.
jrudolph commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2472735096
##########
stream/src/main/scala/org/apache/pekko/stream/scaladsl/Compression.scala:
##########
@@ -85,4 +87,28 @@ object Compression {
*/
def inflate(maxBytesPerChunk: Int, nowrap: Boolean): Flow[ByteString, ByteString, NotUsed] =
Flow[ByteString].via(new DeflateDecompressor(maxBytesPerChunk, nowrap)).named("inflate")
+
+ /**
+ * @since 2.0.0
+ */
+ def zstd: Flow[ByteString, ByteString, NotUsed] = zstd(Zstd.defaultCompressionLevel())
+
+ /**
+ * Same as [[zstd]] with a custom level and an optional dictionary.
+ * @param level The compression level, must be greater or equal to [[Zstd.minCompressionLevel]] and less than or equal
+ * to [[Zstd.maxCompressionLevel]]
+ * @param dictionary An optional dictionary that can be used for compression
+ * @since 2.0.0
+ */
+ def zstd(level: Int, dictionary: Option[ZstdDictCompress] = None): Flow[ByteString, ByteString, NotUsed] = {
+ require(level <= Zstd.maxCompressionLevel() && level >= Zstd.minCompressionLevel())
+ CompressionUtils.compressorFlow(() => new ZstdCompressor(level, dictionary))
+ }
+
+ /**
+ * @since 2.0.0
+ */
+ def zstdDecompress(maxBytesPerChunk: Int = MaxBytesPerChunkDefault): Flow[ByteString, ByteString, NotUsed] =
+ Flow[ByteString].via(new ZstdDecompressor(maxBytesPerChunk)).named("zstdDecompress")
Review Comment:
One difference is that zlib is distributed differently and already part of the JDK while zstd-jni might bring extra weight that not everyone is interested in. The full jar including binaries for all platforms is >7MB. Whether that is significant or not is a different question but it goes somewhat against the usual idea to keep pekko-stream reasonably minimal and add stuff like this to the connectors (which probably does not work if we want to depend on it from http). So, I'd also slightly favor having an extra module at least in pekko core.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3469084264
I have just updated the PR, it was only two small minor changes, one is setting the default of `maxBytesPerChunk` for the zstd decompressor to be `Zstd.blockSizeMax()` which happens to be 128kb (thanks @jrudolph !) and the other is cleaning up the direct `ByteBuffer` in the `postStop` hook.
I am holding on my current work of buffering data up to a limit until I get feedback from my latest comment at https://github.com/apache/pekko/pull/2409#issuecomment-3469014680 as it sounds like it might be over-engineering the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3456590405
Since the core tests have passed (the github action says failed but thats due to a missing paradox documentation) I have added other reviewers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3501159668
@mdedetrich, rechunk seems to be able to be done with a rechunk before it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "raboof (via GitHub)" <gi...@apache.org>.
raboof commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2560575765
##########
stream/src/main/scala/org/apache/pekko/stream/javadsl/compression/ZstdDictionary.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.javadsl.compression;
+
+import org.apache.pekko.annotation.InternalApi;
+import org.apache.pekko.stream.impl.io.compression.ZstdDictionaryImpl;
+import org.apache.pekko.util.ByteString;
+import org.apache.pekko.util.OptionalUtil;
+import scala.Option;
+
+import java.util.OptionalInt;
+
+/**
+ * @since 2.0.0
+ */
+public class ZstdDictionary {
+ byte[] dictionary;
Review Comment:
Cool, would be good to add that reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2576975777
##########
docs/src/main/paradox/stream/operators/Compression/zstd.md:
##########
@@ -0,0 +1,30 @@
+# Compression.zstd
+
+Creates a flow that zstd-compresses a stream of ByteStrings.
+
+@ref[Compression operators](../index.md#compression-operators)
+
+## Signature
+
+@apidoc[Compression.zstd](stream.*.Compression$) { scala="#zstd:org.apache.pekko.stream.scaladsl.Flow[org.apache.pekko.util.ByteString,org.apache.pekko.util.ByteString,org.apache.pekko.NotUsed]" java="#zstd()" }
+
+## Description
+
+Creates a flow that zstd-compresses a stream of ByteStrings. Note that the compressor
+will SYNC_FLUSH after every @apidoc[util.ByteString] so that it is guaranteed that every @apidoc[util.ByteString]
Review Comment:
> But that is not the case "if the way the incoming data is cut up into ByteStrings is not deterministic", right? E.g when the data is coming from the network?
Yes true, I believe that comment is strictly talking about the source happens right before going into compression and not the entire "stack". All the comment is trying to convey is that if `autoFlush` is true, any buffers will get processed and cleared for each incoming element, so if you have "abc" as an incoming element, you will always get the compressed "abc" as a result, and not lets say "a" and then "bc" as a further element or some variation.
The network can always split up data as it wants, but thats an orthogonal concern
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3607222158
Relevant issue https://github.com/luben/zstd-jni/issues/324
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2473906258
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.io.ByteBufferCleaner
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
Review Comment:
what about the `compressDirectByteBuffer` method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2471832875
##########
stream/src/main/scala/org/apache/pekko/stream/scaladsl/Compression.scala:
##########
@@ -85,4 +87,28 @@ object Compression {
*/
def inflate(maxBytesPerChunk: Int, nowrap: Boolean): Flow[ByteString, ByteString, NotUsed] =
Flow[ByteString].via(new DeflateDecompressor(maxBytesPerChunk, nowrap)).named("inflate")
+
+ /**
+ * @since 2.0.0
+ */
+ def zstd: Flow[ByteString, ByteString, NotUsed] = zstd(Zstd.defaultCompressionLevel())
+
+ /**
+ * Same as [[zstd]] with a custom level and an optional dictionary.
+ * @param level The compression level, must be greater or equal to [[Zstd.minCompressionLevel]] and less than or equal
+ * to [[Zstd.maxCompressionLevel]]
+ * @param dictionary An optional dictionary that can be used for compression
+ * @since 2.0.0
Review Comment:
1.3.0 ?
##########
stream/src/main/scala/org/apache/pekko/stream/scaladsl/Compression.scala:
##########
@@ -85,4 +87,28 @@ object Compression {
*/
def inflate(maxBytesPerChunk: Int, nowrap: Boolean): Flow[ByteString, ByteString, NotUsed] =
Flow[ByteString].via(new DeflateDecompressor(maxBytesPerChunk, nowrap)).named("inflate")
+
+ /**
+ * @since 2.0.0
+ */
+ def zstd: Flow[ByteString, ByteString, NotUsed] = zstd(Zstd.defaultCompressionLevel())
+
+ /**
+ * Same as [[zstd]] with a custom level and an optional dictionary.
+ * @param level The compression level, must be greater or equal to [[Zstd.minCompressionLevel]] and less than or equal
+ * to [[Zstd.maxCompressionLevel]]
+ * @param dictionary An optional dictionary that can be used for compression
+ * @since 2.0.0
Review Comment:
1.3.0 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2469468261
##########
project/Dependencies.scala:
##########
@@ -351,7 +351,8 @@ object Dependencies {
// pekko stream
- lazy val stream = l ++= Seq[sbt.ModuleID](reactiveStreams, TestDependencies.scalatest)
+ lazy val stream =
+ l ++= Seq[sbt.ModuleID](reactiveStreams, "com.github.luben" % "zstd-jni" % "1.5.7-6", TestDependencies.scalatest)
Review Comment:
can we make the zstd dependency optional or provided?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2469518733
##########
project/Dependencies.scala:
##########
@@ -351,7 +351,8 @@ object Dependencies {
// pekko stream
- lazy val stream = l ++= Seq[sbt.ModuleID](reactiveStreams, TestDependencies.scalatest)
+ lazy val stream =
+ l ++= Seq[sbt.ModuleID](reactiveStreams, "com.github.luben" % "zstd-jni" % "1.5.7-6", TestDependencies.scalatest)
Review Comment:
> How about having a separate pekko-streams-ztsd or pekko-streams-compression so that pekko-streams core lib doesn't need this dependency? ZIO have done something similar where a small number of compression operators are in zio-streams but others have their own libs, eg zio-streams-compress-zstd.
You could make the same argument as to why lz4Java isn't its own artifact, or any of the other compression algorithms. Also, due to the way the current API is designed, all of the compression algorithms are contained within the pekko-streams `Compression` object (which we could monkey patch to add more methods as a separate artifact, but thats starting to look ugly already). Ontop of that, when adding zstd compression to pekko http request/response headers (see https://github.com/apache/pekko-http/issues/860), we would likewise have to create yet another artifact for pekko-http.
I understand the sentiment, but honestly this is creating a huge amount of ceremony for no real benefit and the only reason we are having this discussion is because of how behind Java is in adding what are de facto standard compression algorithms to the stdlib (of which zstd has been for at least half a decade now). The only reason I can see to use separate artifacts would be to support Pekko 1.x series, but zstd-jni only supports JDK 11 or higher and this would then need to be advertised to end users.
Put differently, the entire point of `Compression` object for scaladsl/javadsl is to contain the standard compression/decompression algorithms, and zstd is one of those. While having to use zstd-jni is not ideal, to the end user there is no difference as it already supports almost every architecture that would be used in the wild.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2473986166
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.io.ByteBufferCleaner
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArrayUnsafe())
+ inputBB.flip()
+ compressingStream.compress(inputBB)
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.flip()
+ if (ByteBufferCleaner.isSupported)
+ ByteBufferCleaner.clean(inputBB)
+ result
+ }
+
+ override def flush(): ByteString = {
+ targetBuffer.flip()
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.clear()
+ compressingStream.flush()
+ result
+ }
+
+ override def finish(): ByteString = {
+ compressingStream.close()
+ targetBuffer.flip()
+ val arr = Array.ofDim[Byte](targetBuffer.limit())
+ targetBuffer.get(arr)
+ val result = ByteString.fromArrayUnsafe(arr)
+ if (ByteBufferCleaner.isSupported)
+ ByteBufferCleaner.clean(targetBuffer)
+ result
+ }
+
+ override def compressAndFlush(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArray)
Review Comment:
Inconsistent use of `toArray` instead of `toArrayUnsafe()`. Line 41 uses `toArrayUnsafe()` in the `compress` method, but this method uses `toArray` which creates a defensive copy. For consistency and performance, use `toArrayUnsafe()` here as well, matching the pattern in line 41.
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdDecompressor.scala:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.io.ByteBufferCleaner
+import pekko.stream.Attributes
+import pekko.stream.impl.fusing.GraphStages.SimpleLinearGraphStage
+import pekko.stream.stage.{ GraphStageLogic, InHandler, OutHandler }
+import pekko.util.ByteString
+
+import com.github.luben.zstd.ZstdDirectBufferDecompressingStreamNoFinalizer
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdDecompressor(maxBytesPerChunk: Int) extends SimpleLinearGraphStage[ByteString] {
+
+ override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = {
+ new GraphStageLogic(shape) with InHandler with OutHandler {
+
+ private val sourceBuffer = ByteBuffer.allocateDirect(maxBytesPerChunk)
+
+ // This is initialized here to avoid an allocation per onPush, remember to clear before using
+ private val outputBuffer = ByteBuffer.allocateDirect(maxBytesPerChunk)
+ private val decompressingStream = new ZstdDirectBufferDecompressingStreamNoFinalizer(sourceBuffer)
+
+ override def onPush(): Unit = {
+ sourceBuffer.clear()
+ val inputArray = grab(in).toArrayUnsafe()
+ sourceBuffer.put(inputArray)
+ sourceBuffer.flip()
+ if (sourceBuffer.hasRemaining) {
+ var result = ByteString.empty
+ while (sourceBuffer.hasRemaining) {
+ outputBuffer.clear()
+ decompressingStream.read(outputBuffer)
+ outputBuffer.flip()
+ val outputArray = new Array[Byte](outputBuffer.limit())
+ outputBuffer.get(outputArray)
+ result = result.concat(ByteString.fromArrayUnsafe(outputArray))
+ }
+
+ // Fencepost case, it's possible to still have sourceBuffer.hasRemaining and yet decompression result
+ // not output anything
+ if (result.nonEmpty)
+ push(out, result)
+ else
+ pull(in)
+ } else pull(in)
+ sourceBuffer.flip()
Review Comment:
The `sourceBuffer.flip()` call after the if-else block appears to be incorrect. The buffer was already flipped on line 48, and all processing paths (lines 49-66) consume it. This extra flip will reset the buffer to an inconsistent state. This line should be removed.
```suggestion
```
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.io.ByteBufferCleaner
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArrayUnsafe())
+ inputBB.flip()
+ compressingStream.compress(inputBB)
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.flip()
+ if (ByteBufferCleaner.isSupported)
+ ByteBufferCleaner.clean(inputBB)
+ result
+ }
+
+ override def flush(): ByteString = {
+ targetBuffer.flip()
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.clear()
+ compressingStream.flush()
Review Comment:
The `compressingStream.flush()` is called after extracting the result and clearing the buffer. This means any data written by flush() to `targetBuffer` will be lost. The flush should be called before flipping the buffer, and the buffer should be cleared after creating the result ByteString, similar to the pattern in `compressAndFlush`.
```suggestion
compressingStream.flush()
targetBuffer.flip()
val result = ByteString.fromByteBuffer(targetBuffer)
targetBuffer.clear()
```
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.io.ByteBufferCleaner
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArrayUnsafe())
+ inputBB.flip()
+ compressingStream.compress(inputBB)
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.flip()
+ if (ByteBufferCleaner.isSupported)
+ ByteBufferCleaner.clean(inputBB)
+ result
+ }
+
+ override def flush(): ByteString = {
+ targetBuffer.flip()
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.clear()
+ compressingStream.flush()
+ result
+ }
+
+ override def finish(): ByteString = {
+ compressingStream.close()
+ targetBuffer.flip()
+ val arr = Array.ofDim[Byte](targetBuffer.limit())
+ targetBuffer.get(arr)
+ val result = ByteString.fromArrayUnsafe(arr)
+ if (ByteBufferCleaner.isSupported)
+ ByteBufferCleaner.clean(targetBuffer)
+ result
+ }
+
+ override def compressAndFlush(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArray)
+ inputBB.flip()
+ compressingStream.compress(inputBB)
+ compressingStream.flush()
+ targetBuffer.flip()
+
+ val arr = new Array[Byte](targetBuffer.limit())
+ targetBuffer.get(arr)
+ targetBuffer.clear()
+ if (ByteBufferCleaner.isSupported)
+ ByteBufferCleaner.clean(inputBB)
+ ByteString.fromArrayUnsafe(arr)
+ }
+
+ override def compressAndFinish(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArray)
Review Comment:
Inconsistent use of `toArray` instead of `toArrayUnsafe()`. Line 41 uses `toArrayUnsafe()` in the `compress` method, but this method uses `toArray` which creates a defensive copy. For consistency and performance, use `toArrayUnsafe()` here as well, matching the pattern in line 41.
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.io.ByteBufferCleaner
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArrayUnsafe())
+ inputBB.flip()
+ compressingStream.compress(inputBB)
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.flip()
Review Comment:
The order of operations is incorrect. `ByteString.fromByteBuffer` reads from the current position to the limit, but `targetBuffer` hasn't been flipped yet. The flip should occur before calling `fromByteBuffer`. This should be: `targetBuffer.flip()` followed by `val result = ByteString.fromByteBuffer(targetBuffer)`.
```suggestion
targetBuffer.flip()
val result = ByteString.fromByteBuffer(targetBuffer)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2469518733
##########
project/Dependencies.scala:
##########
@@ -351,7 +351,8 @@ object Dependencies {
// pekko stream
- lazy val stream = l ++= Seq[sbt.ModuleID](reactiveStreams, TestDependencies.scalatest)
+ lazy val stream =
+ l ++= Seq[sbt.ModuleID](reactiveStreams, "com.github.luben" % "zstd-jni" % "1.5.7-6", TestDependencies.scalatest)
Review Comment:
> How about having a separate pekko-streams-ztsd or pekko-streams-compression so that pekko-streams core lib doesn't need this dependency? ZIO have done something similar where a small number of compression operators are in zio-streams but others have their own libs, eg zio-streams-compress-zstd.
You could make the same argument as to why lz4Java isn't its own artifact, or any of the other compression algorithms. Also, due to the way the current API is designed, all of the compression algorithms are contained within the pekko-streams `Compression` object (which we could monkey patch to add more methods as a separate artifact, but thats starting to look ugly already). Ontop of that, when adding zstd compression to pekko http request/response headers (see https://github.com/apache/pekko-http/issues/860), we would likewise have to create yet another artifact for pekko-http.
`zio-streams-compress` is also not the best to compare to, since if you look at https://zio.dev/zio-streams-compress/ you will see they also have separate artifacts for `gzip` and `lz4` something is contained within pekko-streams core artifact for us. zio appears to have a different design ethos here, where they err on the side of hyper modularity and put everything in its own artifact where as pekko is on the other side.
I understand the sentiment, but honestly this is creating a huge amount of ceremony for no real benefit and the only reason we are having this discussion is because of how behind Java is in adding what are de facto standard compression algorithms to the stdlib (of which zstd has been for at least half a decade now). The only reason I can see to use separate artifacts would be to support Pekko 1.x series, but zstd-jni only supports JDK 11 or higher and this would then need to be advertised to end users.
`zio-streams-compress` is also not the best comparison, since if you look at https://zio.dev/zio-streams-compress/ you will see they also have separate artifacts for `gzip` and `lz4` something is contained within pekko-streams core artifact for us. zio appears to have a different design ethos here, where they err on the side of hyper modularity and put everything in its own artifact where as pekko is on the other side.
Put differently, the entire point of `Compression` object for scaladsl/javadsl is to contain the standard compression/decompression algorithms, and zstd is one of those. While having to use zstd-jni is not ideal, to the end user there is no difference as it already supports almost every architecture that would be used in the wild.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2472816214
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
Review Comment:
You mean with a comment or is there some annotation I am not aware of?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2472877219
##########
stream/src/main/scala/org/apache/pekko/stream/scaladsl/Compression.scala:
##########
@@ -85,4 +87,28 @@ object Compression {
*/
def inflate(maxBytesPerChunk: Int, nowrap: Boolean): Flow[ByteString, ByteString, NotUsed] =
Flow[ByteString].via(new DeflateDecompressor(maxBytesPerChunk, nowrap)).named("inflate")
+
+ /**
+ * @since 2.0.0
+ */
+ def zstd: Flow[ByteString, ByteString, NotUsed] = zstd(Zstd.defaultCompressionLevel())
+
+ /**
+ * Same as [[zstd]] with a custom level and an optional dictionary.
+ * @param level The compression level, must be greater or equal to [[Zstd.minCompressionLevel]] and less than or equal
+ * to [[Zstd.maxCompressionLevel]]
+ * @param dictionary An optional dictionary that can be used for compression
+ * @since 2.0.0
+ */
+ def zstd(level: Int, dictionary: Option[ZstdDictCompress] = None): Flow[ByteString, ByteString, NotUsed] = {
+ require(level <= Zstd.maxCompressionLevel() && level >= Zstd.minCompressionLevel())
+ CompressionUtils.compressorFlow(() => new ZstdCompressor(level, dictionary))
+ }
+
+ /**
+ * @since 2.0.0
+ */
+ def zstdDecompress(maxBytesPerChunk: Int = MaxBytesPerChunkDefault): Flow[ByteString, ByteString, NotUsed] =
+ Flow[ByteString].via(new ZstdDecompressor(maxBytesPerChunk)).named("zstdDecompress")
Review Comment:
> One difference is that zlib is distributed differently and already part of the JDK while zstd-jni might bring extra weight that not everyone is interested in. The full jar including binaries for all platforms is >7MB. Whether that is significant or not is a different question but it goes somewhat against the usual idea to keep pekko-stream reasonably minimal and add stuff like this to the connectors (which probably does not work if we want to depend on it from http). So, I'd also slightly favor having an extra module at least in pekko core.
What would you recommend here, because there are many ways to go about this? For example, it would be possible to have the `zstd`/`zstdDecompress` method in the `Compression` object so that the API is clean but then the user would have to provide their own artifact (of which we can provide a default that uses `zstd-jni`).
Or should the artifact just contain everything related to zstd and should the artifact be a `pekko-streams-compress-extra` (which initially contains a zstd) or `pekko-streams-zstd`?
One thing to note is that when doing this, I also have https://github.com/apache/pekko-http/issues/860 in mind, and I wanted to add zstd to the standard spot where all of our content encoder's are i.e. https://github.com/apache/pekko-http/blob/4833a8e42f946682a72a72a0f3bee4c4d662b9a6/http-core/src/main/scala/org/apache/pekko/http/scaladsl/model/headers/HttpEncoding.scala#L63-L76
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3500549528
@mdedetrich, in Netty, we have `ZstdOptions` too. I think we can take a look at Netty's one for some inspiration before we merge this.
I think we should have this in 1.3.0 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3506717207
> zstd is indeed quite complex. In our work, it actually includes a complete platform and toolchain for dictionary training, end-to-end deployment, version synchronization, etc.
Actually the zstd-api is incredibly simple, especially if we decide to the dumb solution of doing `ByteString.fromArrayUnsafe(zstdCompressCtx.compress(input.toArrayUnsafe()))`. In this case the compressor would just be a few lines of code and its far simpler than what is done in [gzip/deflate](https://github.com/apache/pekko/blob/b0fdac259bd57fdd481483f3fe9a7aec6e1ff38a/stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/DeflateCompressor.scala) which has to deal with arbitrary behavior in the JVM implementation.
The complexity that I am describing comes from doing something very smart that is only possible with pekko-streams, i.e. taking into account downstream demand in a stream when it comes to buffering, afaik no other implementation does this.
On the other hand the complexity you describe is from your specific usecase where you work. Training dictionaries would be entirely out of scope for this, but of course you can specify a dictionary in the same way you can specify a compression level and its a single of code.
> Therefore, could we consider releasing a separate zstd toolkit, such as pekko-zstd or pekko-http-zstd, etc.? This would allow for a more complete design and implementation, and also enable independent evolution.
Assuming the initial implementation is correct I don't see how this could evolve aside from version bumps in zstd-jni (at least with assumption that dictionary training is out of scope)
> Without a dictionary, zstd is simply a slightly faster gzip implementation.
Its not just slightly faster, it has significantly better compression ratios. And slightly is downplaying it a bit, it can be many factors faster (ofc it depends on the level and where the bottleneck is).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3508219613
> Although Netty's codec directly depends on zstd-jni, I don't know if Pekko-stream users will find this troublesome; personally, I think it's fine.
Its not just netty, even Apache Commons Compress uses zstd-jni and includes it directly as an artifact (see https://commons.apache.org/proper/commons-compress/ and https://mvnrepository.com/artifact/org.apache.commons/commons-compress/1.28.0).
The only difference is that apache-commons make its an optional maven dependency. We can do the same here, we would just have to document to end users that they have to add the zstd-jni artifact themselves (this line is also just a single change in code).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3460355246
I have finished the docs for paradox (hence completing the PR) and also fixed some minor things, all resolved comments from the copilot review (including ones which were outright wrong)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2472123454
##########
stream/src/main/scala/org/apache/pekko/stream/impl/io/compression/ZstdCompressor.scala:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.pekko.stream.impl.io.compression
+
+import java.nio.ByteBuffer
+
+import com.github.luben.zstd.{ Zstd, ZstdDictCompress, ZstdDirectBufferCompressingStreamNoFinalizer }
+
+import org.apache.pekko
+import pekko.annotation.InternalApi
+import pekko.util.ByteString
+
+/** INTERNAL API */
+@InternalApi private[pekko] class ZstdCompressor(
+ compressionLevel: Int =
+ Zstd.defaultCompressionLevel(), dictionary: Option[ZstdDictCompress] = None) extends Compressor {
+
+ private val targetBuffer = ByteBuffer.allocateDirect(65536)
+ private val compressingStream = new ZstdDirectBufferCompressingStreamNoFinalizer(targetBuffer, compressionLevel)
+
+ dictionary.foreach(compressingStream.setDict)
+
+ override def compress(input: ByteString): ByteString = {
+ val inputBB = ByteBuffer.allocateDirect(input.size)
+ inputBB.put(input.toArrayUnsafe())
+ inputBB.flip()
+ compressingStream.compress(input.asByteBuffer)
+ val result = ByteString.fromByteBuffer(targetBuffer)
+ targetBuffer.flip()
Review Comment:
The ordering here doesn't really matter, as all that matters is the `targetBuffer.flip()` happens after the read at `compressingStream.compress(inputBB)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #2409:
URL: https://github.com/apache/pekko/pull/2409#discussion_r2469518733
##########
project/Dependencies.scala:
##########
@@ -351,7 +351,8 @@ object Dependencies {
// pekko stream
- lazy val stream = l ++= Seq[sbt.ModuleID](reactiveStreams, TestDependencies.scalatest)
+ lazy val stream =
+ l ++= Seq[sbt.ModuleID](reactiveStreams, "com.github.luben" % "zstd-jni" % "1.5.7-6", TestDependencies.scalatest)
Review Comment:
> How about having a separate pekko-streams-ztsd or pekko-streams-compression so that pekko-streams core lib doesn't need this dependency? ZIO have done something similar where a small number of compression operators are in zio-streams but others have their own libs, eg zio-streams-compress-zstd.
You could make the same argument as to why lz4Java isn't its own artifact, or any of the other compression algorithms. Also, due to the way the current API is designed, all of the compression algorithms are contained within the `Compression` object (which we could monkey patch to add more methods as a separate artifact, but thats starting to look ugly already). Ontop of that, when adding zstd compression to pekko http request/response headers, we would likewise have to create yet another artifact in pekko-http.
I understand the sentiment, but honestly this is creating a huge amount of ceremony for no real benefit and the only reason we are having this discussion is because of how behind Java is in adding what are de facto standard compression algorithms to the stdlib (of which zstd has been for at least half a decade now).
Put differently, the entire point of `Compression` object for scaladsl/javadsl is to contain the standard compression/decompression algorithms, and zstd is one of those. While having to use zstd-jni is not ideal, to the end user there is no difference as it already supports almost every architecture that would be used in the wild.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3460144063
@He-Pin This cannot be backported as the zstd-jni library is built against JDK 11 and Pekko 1.x series requires every dependency to be built against JDK 1.8 or lower
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3576364036
> I have not fully digested the PR but on first read it looks reasonable. Noted a couple of small things that are potential leftovers/nits.
So I still want to get this into 2.0.0-M1 but I haven't had time with the buffer implementation I was planning to do. For now I am thinking of doing the simple dumb solution so that we at least have a implementation with a stable API which unlocks zstd-compression for http responses in pekko-http and I can always do a better version for a future milestone release of 2.0.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3596472004
@raboof So I tried to get my fancy buffering implementation done, but I am coming across a very weird issue/problem where even if I change the current implementation to something like this
```scala
override def compressAndFlush(input: ByteString): ByteString =
ByteString.fromArrayUnsafe(Zstd.compress(input.toArrayUnsafe))
```
The tests start failing because `onUpstreamFinish` never ends up being executed.
So I think we should opt for the simple option for now, which is going to be the current implementation but avoid creating a direct byte buffer on each incoming element (instead using a pool), doing the changes now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #2409:
URL: https://github.com/apache/pekko/pull/2409#issuecomment-3599163013
@raboof @pjfanning @pjfanning So here is a status update, I actually found a bug in the `ZstdDecompressor` which I am now trying to fix. The bug is quite simple, it happens when [here](https://github.com/apache/pekko/pull/2409/files#diff-36b15bf36eaf7b9bd2c64a650c0d1f602a475383340766c45c0907aedf4f9b6cR50) when the source buffer is larger than the bytebuffer causing a basic overflow. This bug didn't show in tests because individual `ByteString` elements in the tests happen always be smaller than the `maxBytesPerChunk`, but if you set `maxBytesPerChunk` to small value such as 8 then it immediately shows.
To solve this, I wanted to create an ultra simple implementation which would essentially just be `ByteString.fromArrayUnsafe(Zstd.decompress(input.toArrayUnsafe))` but this is not possible because `Zstd.decompress` actually needs to know the original size of the decompressed element (and we have no context of this in a stream).
Due to this I have to shuffle data from the input `ByteString` to that direct bytebuffer while taking into account the fact that the input `ByteString` can be of any size. There is a naive implementation that transfers one byte at a time (which is really slow), so instead the implementation copies as much as it can depending on the free amount of space in the `outputBuffer` but understandably this is a more complicated solution. Note that this solution is also what the zstd-jni test also [does](https://github.com/luben/zstd-jni/blob/9c3386d306086078155f58116a4d905e07239db4/src/test/scala/Zstd.scala#L516-L551).
The advantage here is that this is the exact same logic that is needed to avoid the `ByteBuffer.allocateDirect(input.size)` in the current PR implementation which we need to remove anyways because of how slow `ByteBuffer.allocateDirect(input.size)` is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Add zstd compression and decompression for pekko streams [pekko]
Posted by "pjfanning (via GitHub) via notifications" <no...@pekko.apache.org>.
pjfanning closed pull request #2409: Add zstd compression and decompression for pekko streams
URL: https://github.com/apache/pekko/pull/2409
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org