RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

["Denloye, Olasoji" <ol...@intel.com>]

Hi, I would like to bring this back to everyone's attention. I have made changes to the proposal to incorporate the ServiceLoader API as suggested.

https://cwiki.apache.org/confluence/x/Xgr0Ew 

Thanks

-----Original Message-----
From: Denloye, Olasoji 
Sent: Thursday, December 19, 2024 11:01 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Frédérik, Thanks for the suggestion to use the existing ServiceLoader feature in Java. It could serve as suitable mechanism for selecting the best available compression codec. 

Thanks

-----Original Message-----
From: Frédérik Rouleau <fr...@confluent.io.INVALID>
Sent: Wednesday, December 18, 2024 12:24 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Olasoji,

I think that having a way to provide some optimised/accelerated implementation is a good idea.
What not using the ServiceLoader feature of Java? That would allow anyone to provide their own optimised implementation depending on their infrastructure. We can keep the existing implementation as the default and allow Kafka to find a better implementation based on Service found. As you mentioned, we might need to enrich the interface to allow Kafka to find the best version available at runtime.

Regards,

On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hi Everyone,
>
> I would like to start a discussion on KIP-1121: Compression 
> acceleration acceleration in Kafka. KIP-1121: Compression acceleration 
> in Kafka - Apache Kafka - Apache Software Foundation< 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Compress
> ion+acceleration+in+Kafka
> >
>

RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

["Denloye, Olasoji" <ol...@intel.com>]

Hello all,

I am resurrecting the discussion thread as I got no votes previously.

KIP 1121 is a proposal to introduce pluggable compression acceleration into Kafka. This feature would allow third party plugins to be used in place of a built-in compressor. A compression plugin would accelerate any of the existing compression algorithms and could be applied asymmetrically on brokers, producers, and consumers. 

I have updated the KIP document with a section on performance which includes benchmarking data using a custom plugin that leverages QAT hardware accelerator and achieves up to 6x improvement in throughput over the built-in gzip compressor.

https://cwiki.apache.org/confluence/x/Xgr0Ew

Thanks
-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com> 
Sent: Monday, September 8, 2025 11:46 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka


Hello all, please vote for KIP-1121. 

This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Monday, August 11, 2025 8:59 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello all, please vote for KIP-1121. 

This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Thursday, July 24, 2025 8:53 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello all, 

Voting is in progress for KIP-1121. 

This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks


-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Thursday, June 26, 2025 4:00 PM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello all, please vote for KIP-1121. 

This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Friday, June 13, 2025 8:40 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello, I would like to call a vote for KIP-1121. This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, June 11, 2025 11:31 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello all, I would like to call a vote for KIP-1121. This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks


-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, June 4, 2025 9:00 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hello all,

Regarding Compression acceleration, I would like to call a vote soon so if anyone wants to contribute to the discussion or bring up other concerns, you are welcome to comment.

Thanks,
Olasoji

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Thursday, May 29, 2025 10:39 PM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hello all, any more comments?

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, May 7, 2025 10:55 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Greg,

2, 10 I figured a way to not expose any of these internal classes to the public API. I added a new interface in the place of BufferSupplier.
7. No, CompressionService#isAvailable returning false does not currently cause a ConfigException, it would cause a fall-back to the base implementation. I have changed this to checkAvailable which throws an exception if not available. If a more appropriate exception does not already exist, we could introduce "UnavailableServiceException"
9. I have updated the testing section to include additional scenarios relating to Kafka side testing. In short, we need to ensure the default behavior is unchanged and validate all new functions and configurations. 
11. the "accelerated GzipCompressor" mentioned is a reference design and need not be a part of Kafka. Is there a recommendation for hosting/distributing implementations of CompressionService?
12. Similar to the base implementation the goal is to have a separate instance of CompressionService for each I/O threads handling compression/decompression.
13. The existing implementations will remain unchanged. If the CompressionService configuration is empty then the base implementation would be used.
14. The broker configuration is currently in ServerConfigs.java and is configured statically in broker.properties (or server.properties). I have not considered dynamic configuration via kraft.

Thanks,
Olasoji

-----Original Message-----
From: Greg Harris <gr...@aiven.io.INVALID>
Sent: Wednesday, April 30, 2025 1:04 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Olasoji,

Thanks for updating the KIP.

2. I'm still not satisfied with the description of the public API in the KIP. It mentions these classes:
* org.apache.kafka.common.utils.ByteBufferOutputStream
* org.apache.kafka.common.utils.ByteBufferInputStream
* org.apache.kafka.common.utils.BufferSupplier
* org.apache.kafka.common.record.CompressionType
But not any description of these classes. Please include signatures of all new methods/variables/etc that are being exposed.

7. Does CompressionService#isAvailable returning false cause a ConfigException? Could the CompressionService provide a reason (via an
exception) of why it isn't available, instead of returning a boolean and requiring users to look through logs to debug the unavailability?
8. Thanks for clarifying the security model. This is just like any other Java application, where you need to trust the JARs that you add to your application. I don't think that the isolated classloader needs to be mentioned in the KIP, so thank you for removing that.
9. This point does not appear to have been addressed.
10. The classes mentioned above are in existing packages that are not exposed via Javadoc, and we have generally included whole packages in the API at once. I think these classes need to be moved to a different package, or removed from the API surface area.
11. The KIP states that it "also introduces an accelerated GzipCompressor".
Does this require Apache Kafka to depend on a specific accelerator implementation? The Apache Kafka project has traditionally been very averse to adding dependencies like this.
12. What sort of concurrency guarantees are expected from CompressionService implementations? Should each connection, thread, or JVM get its own instance of each CompressionService?
13. Will the existing implementations be converted to CompressionService classes, or will they be special fallbacks? Will the default for the configuration be "null" or some real class?
14. It looks like the new configurations are being added to the CommonClientConfigs to be included in both producer and consumer. Where is this configuration on the broker side? Is it a dynamic config (able to be reconfigured via KRaft) or a static config (only configurable by server.properties)?

Thanks,
Greg

On Wed, Apr 30, 2025 at 10:2 AM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hello all,
>
> I would like to call a vote soon so if anyone wants to contribute to 
> the discussion or bring up other concerns, you are welcome to comment.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Wednesday, April 23, 2025 10:22 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi All,
>
> This is a gentle reminder about open discussion on this compression 
> acceleration KIP. I would love to hear your thoughts.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Monday, April 7, 2025 8:49 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Greg,
>
> I have refactored the proposal to reduce the API surface area. I 
> introduced a new CompressionService interface which contains basic 
> compression/decompression functions. A service provider, when 
> supplied, would be wrapped in a concrete 'Compression' and run similar 
> to the existing compression codecs. The 'Compression' interface and 
> its subclasses have been decoupled from these changes.
>
> 6. based on your feedback, offloading would occur if and only if the 
> specific configuration is set, even if multiple service providers are 
> available 8. The threat model is that dynamically loading any 
> untrusted or untested jar on the classpath can be dangerous. The 
> purpose of bringing this up is to discuss how service provider
> (plugins) is distributed and installed. On one hand, a centralized 
> Kafka repository and on the other a decentralized approach where 
> plugins are developed and installed on the fly. Which is preferable?
> The choice would determine what security mitigations are needed if any.
>
> https://cwiki.apache.org/confluence/x/Xgr0Ew
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Monday, February 24, 2025 12:42 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Olasoji,
>
> Thanks for the update.
>
> 1. Could you add these as code examples to the KIP, and not as prose? 
> See [1-3] for some examples of well-written public interfaces.
>
> 2. I find the proposed API surface area excessive, and think we need 
> to narrow it down. Looking at what's merged on trunk right now and not 
> including your proposed modifications, this KIP includes:
> * 2 interfaces
> * 12 classes
> * 5 constructors
> * 8 static methods
> * 2 static variables
> * 18 unique instance methods
> Committing to long-term stability of this many classes and interfaces 
> as public API will make it hard to initially review it, and innovate 
> on it in the future.
>
> 4. Is there a way to modify the existing signatures/implementations 
> such that we don't have to expose any subclasses of Compression as 
> public API, only the CompressionTypes?
> 6. What does "first provider loaded" mean to users? AFAIU, this looks 
> arbitrary to users, as it depends on classpath ordering, filesystem 
> ordering, etc.
> 8. I'm trying to understand under what conditions this can be attacked 
> (threat model), and what mitigations Kafka needs to provide 
> (implementation). Is Kafka providing special filtering? Is Kafka 
> providing the custom class loader?
> 6, 7, 8. Please add these details to the KIP so that other reviewers 
> don't need to follow the discussion thread so closely.
> 9. The testing section looks like it is written for individuals 
> implementing accelerator plugins. Could you also write about how the 
> Kafka side of the implementation will be tested?
>
> Thanks,
> Greg
>
> [1]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Onc
> e+Delivery+and+Transactional+Messaging
> [2]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tie
> red+Storage
> [3]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism
> +for+plugins+and+connectors+to+register+metrics
>
> On Fri, Feb 21, 2025 at 12:15 AM Denloye, Olasoji < 
> olasoji.denloye@intel.com>
> wrote:
>
> > Hi Greg, Thanks for your response.
> >
> > 1. Some changes will be made to the Compression and Builder 
> > interfaces. At least one default method, isAvailable(), would be 
> > added to the Builder interface. The static methods in the 
> > Compression interface that create and return specific builders would 
> > be modified to return a new builder or a loaded builder from a 
> > service provider if it exists. The return types may need to be more 
> > generic e.g Builder<GzipCompression> vs GzipCompression.Builder.
> > 2. These classes are appropriate to add to the API surface as they 
> > may be required to develop efficient compression implementations. I 
> > will modify the KIP.
> > 4. This is because the Compression interface (currently) expects a 
> > Builder of type 'GzipCompression.Builder'. I agree that alternative 
> > implementations need not extend the base implementation with some 
> > modifications to the Compression interface. However if an 
> > alternative implementation is registered as a service provider of 
> > GzipCompression then it must be of type 'CompressionType.GZIP' and 
> > it must be fully compatible with the base implementation of GzipCompression.
> > 5. So far no, but 'Configurable' looks promising 6. If no specific 
> > configuration is set, then the first provider loaded and available 
> > will be used. If a specific configuration is set then that specific 
> > provider will be used if available else the base implementation.
> > 7. Yes, a ConfigException will be thrown if the class specified in 
> > the config is not found on the classpath.
> > 8. This can be implemented by using special filtering or a custom 
> > class loader. The concern here is that any class on the class path 
> > (trusted or
> > not) could be loaded which could intercept messages or corrupt 
> > messages by compressing using an unknown algorithm.
> >
> > -----Original Message-----
> > From: Greg Harris <gr...@aiven.io.INVALID>
> > Sent: Tuesday, February 18, 2025 1:58 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Olasoji,
> >
> > Thanks for the KIP! I find it addresses many of the concerns I had 
> > with KIP-984. Thank you for following up on that!
> >
> > 1. Since the Compression interface is internal currently, it may 
> > contain some components that were not designed with pluggability in 
> > mind, such as the static methods. Are these interfaces being 
> > published as-is, or are some changes going to be made first?
> > 2. Similarly, the ByteBufferOutputStream, CompressionType, 
> > BufferSupplier, ByteBufferInputStream, ChunkedByteStream, and 
> > GzipOutputStream classes in the signature or implementations of 
> > Compression are not public API currently, and would need to be made 
> > public along with Compression. Are these classes appropriate to add 
> > to
> our API surface?
> > 3. Could you adjust the KIP structure/formatting to make the public 
> > API surface area more clear? Some prose in "Proposed Changes" could 
> > become code examples in "Public Interfaces".
> > 4. What is the significance of this statement: "Any new service 
> > provider should extend the base implementation and thus return the 
> > same CompressionType". Are we expecting that `o instanceof 
> > GzipCompression == o.type().equals(CompressionType.GZIP)` at all 
> > times? If an alternative implementation doesn't need any part of the 
> > upstream implementation, does it make sense to force them to extend 
> > one
> another?
> > 5. Is this interface expected to be composed with any other plugin 
> > mix-ins, such as Configurable, Reconfigurable, Closeable, 
> > Monitorable, Versioned, etc?
> > 6. If there are multiple accelerators present, which one will be chosen?
> > 7. If the configuration is set but the accelerator implementation 
> > isn't available, does that cause a ConfigException?
> > 8. Could you expand on this statement: "Due to security and/or 
> > compatibility concerns it is possible to limit this feature to only 
> > load approved service providers (e.g those distributed with Kafka)".
> > How will this be implemented? What threat model does this assume?
> >
> > Thanks,
> > Greg
> >
> > On Tue, Feb 18, 2025 at 11:41 AM Denloye, Olasoji < 
> > olasoji.denloye@intel.com>
> > wrote:
> >
> > > Hi, I would like to bring this back to everyone's attention. This 
> > > KIP introduces a simple framework that enables the use of 
> > > alternative, faster implementations of the existing compression 
> > > codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji <ol...@intel.com>
> > > Sent: Thursday, February 13, 2025 1:22 PM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi, I would like to bring this back to everyone's attention. I 
> > > have made changes to the proposal to incorporate the ServiceLoader 
> > > API as
> > suggested.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji
> > > Sent: Thursday, December 19, 2024 11:01 AM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Frédérik, Thanks for the suggestion to use the existing 
> > > ServiceLoader feature in Java. It could serve as suitable 
> > > mechanism for selecting the best available compression codec.
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Frédérik Rouleau <fr...@confluent.io.INVALID>
> > > Sent: Wednesday, December 18, 2024 12:24 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Olasoji,
> > >
> > > I think that having a way to provide some optimised/accelerated 
> > > implementation is a good idea.
> > > What not using the ServiceLoader feature of Java? That would allow 
> > > anyone to provide their own optimised implementation depending on 
> > > their infrastructure. We can keep the existing implementation as 
> > > the default and allow Kafka to find a better implementation based 
> > > on Service found. As you mentioned, we might need to enrich the 
> > > interface to allow Kafka to find the best version available at runtime.
> > >
> > > Regards,
> > >
> > > On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji < 
> > > olasoji.denloye@intel.com>
> > > wrote:
> > >
> > > > Hi Everyone,
> > > >
> > > > I would like to start a discussion on KIP-1121: Compression 
> > > > acceleration acceleration in Kafka. KIP-1121: Compression 
> > > > acceleration in Kafka - Apache Kafka - Apache Software 
> > > > Foundation< 
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Co
> > > > mp
> > > > re
> > > > ss
> > > > ion+acceleration+in+Kafka
> > > > >
> > > >
> > >
> >
>

RE: [VOTE] KIP-1121: Compression acceleration in Kafka

["Denloye, Olasoji" <ol...@intel.com>]

Hello all, please vote for KIP-1121. 

This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com> 
Sent: Monday, August 11, 2025 8:59 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello all, please vote for KIP-1121. 

This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Thursday, July 24, 2025 8:53 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello all, 

Voting is in progress for KIP-1121. 

This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks


-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Thursday, June 26, 2025 4:00 PM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello all, please vote for KIP-1121. 

This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Friday, June 13, 2025 8:40 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello, I would like to call a vote for KIP-1121. This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, June 11, 2025 11:31 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello all, I would like to call a vote for KIP-1121. This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks


-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, June 4, 2025 9:00 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hello all,

Regarding Compression acceleration, I would like to call a vote soon so if anyone wants to contribute to the discussion or bring up other concerns, you are welcome to comment.

Thanks,
Olasoji

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Thursday, May 29, 2025 10:39 PM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hello all, any more comments?

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, May 7, 2025 10:55 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Greg,

2, 10 I figured a way to not expose any of these internal classes to the public API. I added a new interface in the place of BufferSupplier.
7. No, CompressionService#isAvailable returning false does not currently cause a ConfigException, it would cause a fall-back to the base implementation. I have changed this to checkAvailable which throws an exception if not available. If a more appropriate exception does not already exist, we could introduce "UnavailableServiceException"
9. I have updated the testing section to include additional scenarios relating to Kafka side testing. In short, we need to ensure the default behavior is unchanged and validate all new functions and configurations. 
11. the "accelerated GzipCompressor" mentioned is a reference design and need not be a part of Kafka. Is there a recommendation for hosting/distributing implementations of CompressionService?
12. Similar to the base implementation the goal is to have a separate instance of CompressionService for each I/O threads handling compression/decompression.
13. The existing implementations will remain unchanged. If the CompressionService configuration is empty then the base implementation would be used.
14. The broker configuration is currently in ServerConfigs.java and is configured statically in broker.properties (or server.properties). I have not considered dynamic configuration via kraft.

Thanks,
Olasoji

-----Original Message-----
From: Greg Harris <gr...@aiven.io.INVALID>
Sent: Wednesday, April 30, 2025 1:04 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Olasoji,

Thanks for updating the KIP.

2. I'm still not satisfied with the description of the public API in the KIP. It mentions these classes:
* org.apache.kafka.common.utils.ByteBufferOutputStream
* org.apache.kafka.common.utils.ByteBufferInputStream
* org.apache.kafka.common.utils.BufferSupplier
* org.apache.kafka.common.record.CompressionType
But not any description of these classes. Please include signatures of all new methods/variables/etc that are being exposed.

7. Does CompressionService#isAvailable returning false cause a ConfigException? Could the CompressionService provide a reason (via an
exception) of why it isn't available, instead of returning a boolean and requiring users to look through logs to debug the unavailability?
8. Thanks for clarifying the security model. This is just like any other Java application, where you need to trust the JARs that you add to your application. I don't think that the isolated classloader needs to be mentioned in the KIP, so thank you for removing that.
9. This point does not appear to have been addressed.
10. The classes mentioned above are in existing packages that are not exposed via Javadoc, and we have generally included whole packages in the API at once. I think these classes need to be moved to a different package, or removed from the API surface area.
11. The KIP states that it "also introduces an accelerated GzipCompressor".
Does this require Apache Kafka to depend on a specific accelerator implementation? The Apache Kafka project has traditionally been very averse to adding dependencies like this.
12. What sort of concurrency guarantees are expected from CompressionService implementations? Should each connection, thread, or JVM get its own instance of each CompressionService?
13. Will the existing implementations be converted to CompressionService classes, or will they be special fallbacks? Will the default for the configuration be "null" or some real class?
14. It looks like the new configurations are being added to the CommonClientConfigs to be included in both producer and consumer. Where is this configuration on the broker side? Is it a dynamic config (able to be reconfigured via KRaft) or a static config (only configurable by server.properties)?

Thanks,
Greg

On Wed, Apr 30, 2025 at 10:2 AM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hello all,
>
> I would like to call a vote soon so if anyone wants to contribute to 
> the discussion or bring up other concerns, you are welcome to comment.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Wednesday, April 23, 2025 10:22 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi All,
>
> This is a gentle reminder about open discussion on this compression 
> acceleration KIP. I would love to hear your thoughts.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Monday, April 7, 2025 8:49 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Greg,
>
> I have refactored the proposal to reduce the API surface area. I 
> introduced a new CompressionService interface which contains basic 
> compression/decompression functions. A service provider, when 
> supplied, would be wrapped in a concrete 'Compression' and run similar 
> to the existing compression codecs. The 'Compression' interface and 
> its subclasses have been decoupled from these changes.
>
> 6. based on your feedback, offloading would occur if and only if the 
> specific configuration is set, even if multiple service providers are 
> available 8. The threat model is that dynamically loading any 
> untrusted or untested jar on the classpath can be dangerous. The 
> purpose of bringing this up is to discuss how service provider
> (plugins) is distributed and installed. On one hand, a centralized 
> Kafka repository and on the other a decentralized approach where 
> plugins are developed and installed on the fly. Which is preferable?
> The choice would determine what security mitigations are needed if any.
>
> https://cwiki.apache.org/confluence/x/Xgr0Ew
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Monday, February 24, 2025 12:42 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Olasoji,
>
> Thanks for the update.
>
> 1. Could you add these as code examples to the KIP, and not as prose? 
> See [1-3] for some examples of well-written public interfaces.
>
> 2. I find the proposed API surface area excessive, and think we need 
> to narrow it down. Looking at what's merged on trunk right now and not 
> including your proposed modifications, this KIP includes:
> * 2 interfaces
> * 12 classes
> * 5 constructors
> * 8 static methods
> * 2 static variables
> * 18 unique instance methods
> Committing to long-term stability of this many classes and interfaces 
> as public API will make it hard to initially review it, and innovate 
> on it in the future.
>
> 4. Is there a way to modify the existing signatures/implementations 
> such that we don't have to expose any subclasses of Compression as 
> public API, only the CompressionTypes?
> 6. What does "first provider loaded" mean to users? AFAIU, this looks 
> arbitrary to users, as it depends on classpath ordering, filesystem 
> ordering, etc.
> 8. I'm trying to understand under what conditions this can be attacked 
> (threat model), and what mitigations Kafka needs to provide 
> (implementation). Is Kafka providing special filtering? Is Kafka 
> providing the custom class loader?
> 6, 7, 8. Please add these details to the KIP so that other reviewers 
> don't need to follow the discussion thread so closely.
> 9. The testing section looks like it is written for individuals 
> implementing accelerator plugins. Could you also write about how the 
> Kafka side of the implementation will be tested?
>
> Thanks,
> Greg
>
> [1]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Onc
> e+Delivery+and+Transactional+Messaging
> [2]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tie
> red+Storage
> [3]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism
> +for+plugins+and+connectors+to+register+metrics
>
> On Fri, Feb 21, 2025 at 12:15 AM Denloye, Olasoji < 
> olasoji.denloye@intel.com>
> wrote:
>
> > Hi Greg, Thanks for your response.
> >
> > 1. Some changes will be made to the Compression and Builder 
> > interfaces. At least one default method, isAvailable(), would be 
> > added to the Builder interface. The static methods in the 
> > Compression interface that create and return specific builders would 
> > be modified to return a new builder or a loaded builder from a 
> > service provider if it exists. The return types may need to be more 
> > generic e.g Builder<GzipCompression> vs GzipCompression.Builder.
> > 2. These classes are appropriate to add to the API surface as they 
> > may be required to develop efficient compression implementations. I 
> > will modify the KIP.
> > 4. This is because the Compression interface (currently) expects a 
> > Builder of type 'GzipCompression.Builder'. I agree that alternative 
> > implementations need not extend the base implementation with some 
> > modifications to the Compression interface. However if an 
> > alternative implementation is registered as a service provider of 
> > GzipCompression then it must be of type 'CompressionType.GZIP' and 
> > it must be fully compatible with the base implementation of GzipCompression.
> > 5. So far no, but 'Configurable' looks promising 6. If no specific 
> > configuration is set, then the first provider loaded and available 
> > will be used. If a specific configuration is set then that specific 
> > provider will be used if available else the base implementation.
> > 7. Yes, a ConfigException will be thrown if the class specified in 
> > the config is not found on the classpath.
> > 8. This can be implemented by using special filtering or a custom 
> > class loader. The concern here is that any class on the class path 
> > (trusted or
> > not) could be loaded which could intercept messages or corrupt 
> > messages by compressing using an unknown algorithm.
> >
> > -----Original Message-----
> > From: Greg Harris <gr...@aiven.io.INVALID>
> > Sent: Tuesday, February 18, 2025 1:58 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Olasoji,
> >
> > Thanks for the KIP! I find it addresses many of the concerns I had 
> > with KIP-984. Thank you for following up on that!
> >
> > 1. Since the Compression interface is internal currently, it may 
> > contain some components that were not designed with pluggability in 
> > mind, such as the static methods. Are these interfaces being 
> > published as-is, or are some changes going to be made first?
> > 2. Similarly, the ByteBufferOutputStream, CompressionType, 
> > BufferSupplier, ByteBufferInputStream, ChunkedByteStream, and 
> > GzipOutputStream classes in the signature or implementations of 
> > Compression are not public API currently, and would need to be made 
> > public along with Compression. Are these classes appropriate to add 
> > to
> our API surface?
> > 3. Could you adjust the KIP structure/formatting to make the public 
> > API surface area more clear? Some prose in "Proposed Changes" could 
> > become code examples in "Public Interfaces".
> > 4. What is the significance of this statement: "Any new service 
> > provider should extend the base implementation and thus return the 
> > same CompressionType". Are we expecting that `o instanceof 
> > GzipCompression == o.type().equals(CompressionType.GZIP)` at all 
> > times? If an alternative implementation doesn't need any part of the 
> > upstream implementation, does it make sense to force them to extend 
> > one
> another?
> > 5. Is this interface expected to be composed with any other plugin 
> > mix-ins, such as Configurable, Reconfigurable, Closeable, 
> > Monitorable, Versioned, etc?
> > 6. If there are multiple accelerators present, which one will be chosen?
> > 7. If the configuration is set but the accelerator implementation 
> > isn't available, does that cause a ConfigException?
> > 8. Could you expand on this statement: "Due to security and/or 
> > compatibility concerns it is possible to limit this feature to only 
> > load approved service providers (e.g those distributed with Kafka)".
> > How will this be implemented? What threat model does this assume?
> >
> > Thanks,
> > Greg
> >
> > On Tue, Feb 18, 2025 at 11:41 AM Denloye, Olasoji < 
> > olasoji.denloye@intel.com>
> > wrote:
> >
> > > Hi, I would like to bring this back to everyone's attention. This 
> > > KIP introduces a simple framework that enables the use of 
> > > alternative, faster implementations of the existing compression 
> > > codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji <ol...@intel.com>
> > > Sent: Thursday, February 13, 2025 1:22 PM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi, I would like to bring this back to everyone's attention. I 
> > > have made changes to the proposal to incorporate the ServiceLoader 
> > > API as
> > suggested.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji
> > > Sent: Thursday, December 19, 2024 11:01 AM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Frédérik, Thanks for the suggestion to use the existing 
> > > ServiceLoader feature in Java. It could serve as suitable 
> > > mechanism for selecting the best available compression codec.
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Frédérik Rouleau <fr...@confluent.io.INVALID>
> > > Sent: Wednesday, December 18, 2024 12:24 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Olasoji,
> > >
> > > I think that having a way to provide some optimised/accelerated 
> > > implementation is a good idea.
> > > What not using the ServiceLoader feature of Java? That would allow 
> > > anyone to provide their own optimised implementation depending on 
> > > their infrastructure. We can keep the existing implementation as 
> > > the default and allow Kafka to find a better implementation based 
> > > on Service found. As you mentioned, we might need to enrich the 
> > > interface to allow Kafka to find the best version available at runtime.
> > >
> > > Regards,
> > >
> > > On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji < 
> > > olasoji.denloye@intel.com>
> > > wrote:
> > >
> > > > Hi Everyone,
> > > >
> > > > I would like to start a discussion on KIP-1121: Compression 
> > > > acceleration acceleration in Kafka. KIP-1121: Compression 
> > > > acceleration in Kafka - Apache Kafka - Apache Software 
> > > > Foundation< 
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Co
> > > > mp
> > > > re
> > > > ss
> > > > ion+acceleration+in+Kafka
> > > > >
> > > >
> > >
> >
>

RE: [VOTE] KIP-1121: Compression acceleration in Kafka

["Denloye, Olasoji" <ol...@intel.com>]

Hello all, please vote for KIP-1121. 

This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com> 
Sent: Thursday, July 24, 2025 8:53 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello all, 

Voting is in progress for KIP-1121. 

This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks


-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Thursday, June 26, 2025 4:00 PM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello all, please vote for KIP-1121. 

This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Friday, June 13, 2025 8:40 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello, I would like to call a vote for KIP-1121. This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, June 11, 2025 11:31 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello all, I would like to call a vote for KIP-1121. This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks


-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, June 4, 2025 9:00 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hello all,

Regarding Compression acceleration, I would like to call a vote soon so if anyone wants to contribute to the discussion or bring up other concerns, you are welcome to comment.

Thanks,
Olasoji

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Thursday, May 29, 2025 10:39 PM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hello all, any more comments?

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, May 7, 2025 10:55 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Greg,

2, 10 I figured a way to not expose any of these internal classes to the public API. I added a new interface in the place of BufferSupplier.
7. No, CompressionService#isAvailable returning false does not currently cause a ConfigException, it would cause a fall-back to the base implementation. I have changed this to checkAvailable which throws an exception if not available. If a more appropriate exception does not already exist, we could introduce "UnavailableServiceException"
9. I have updated the testing section to include additional scenarios relating to Kafka side testing. In short, we need to ensure the default behavior is unchanged and validate all new functions and configurations. 
11. the "accelerated GzipCompressor" mentioned is a reference design and need not be a part of Kafka. Is there a recommendation for hosting/distributing implementations of CompressionService?
12. Similar to the base implementation the goal is to have a separate instance of CompressionService for each I/O threads handling compression/decompression.
13. The existing implementations will remain unchanged. If the CompressionService configuration is empty then the base implementation would be used.
14. The broker configuration is currently in ServerConfigs.java and is configured statically in broker.properties (or server.properties). I have not considered dynamic configuration via kraft.

Thanks,
Olasoji

-----Original Message-----
From: Greg Harris <gr...@aiven.io.INVALID>
Sent: Wednesday, April 30, 2025 1:04 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Olasoji,

Thanks for updating the KIP.

2. I'm still not satisfied with the description of the public API in the KIP. It mentions these classes:
* org.apache.kafka.common.utils.ByteBufferOutputStream
* org.apache.kafka.common.utils.ByteBufferInputStream
* org.apache.kafka.common.utils.BufferSupplier
* org.apache.kafka.common.record.CompressionType
But not any description of these classes. Please include signatures of all new methods/variables/etc that are being exposed.

7. Does CompressionService#isAvailable returning false cause a ConfigException? Could the CompressionService provide a reason (via an
exception) of why it isn't available, instead of returning a boolean and requiring users to look through logs to debug the unavailability?
8. Thanks for clarifying the security model. This is just like any other Java application, where you need to trust the JARs that you add to your application. I don't think that the isolated classloader needs to be mentioned in the KIP, so thank you for removing that.
9. This point does not appear to have been addressed.
10. The classes mentioned above are in existing packages that are not exposed via Javadoc, and we have generally included whole packages in the API at once. I think these classes need to be moved to a different package, or removed from the API surface area.
11. The KIP states that it "also introduces an accelerated GzipCompressor".
Does this require Apache Kafka to depend on a specific accelerator implementation? The Apache Kafka project has traditionally been very averse to adding dependencies like this.
12. What sort of concurrency guarantees are expected from CompressionService implementations? Should each connection, thread, or JVM get its own instance of each CompressionService?
13. Will the existing implementations be converted to CompressionService classes, or will they be special fallbacks? Will the default for the configuration be "null" or some real class?
14. It looks like the new configurations are being added to the CommonClientConfigs to be included in both producer and consumer. Where is this configuration on the broker side? Is it a dynamic config (able to be reconfigured via KRaft) or a static config (only configurable by server.properties)?

Thanks,
Greg

On Wed, Apr 30, 2025 at 10:2 AM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hello all,
>
> I would like to call a vote soon so if anyone wants to contribute to 
> the discussion or bring up other concerns, you are welcome to comment.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Wednesday, April 23, 2025 10:22 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi All,
>
> This is a gentle reminder about open discussion on this compression 
> acceleration KIP. I would love to hear your thoughts.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Monday, April 7, 2025 8:49 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Greg,
>
> I have refactored the proposal to reduce the API surface area. I 
> introduced a new CompressionService interface which contains basic 
> compression/decompression functions. A service provider, when 
> supplied, would be wrapped in a concrete 'Compression' and run similar 
> to the existing compression codecs. The 'Compression' interface and 
> its subclasses have been decoupled from these changes.
>
> 6. based on your feedback, offloading would occur if and only if the 
> specific configuration is set, even if multiple service providers are 
> available 8. The threat model is that dynamically loading any 
> untrusted or untested jar on the classpath can be dangerous. The 
> purpose of bringing this up is to discuss how service provider
> (plugins) is distributed and installed. On one hand, a centralized 
> Kafka repository and on the other a decentralized approach where 
> plugins are developed and installed on the fly. Which is preferable?
> The choice would determine what security mitigations are needed if any.
>
> https://cwiki.apache.org/confluence/x/Xgr0Ew
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Monday, February 24, 2025 12:42 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Olasoji,
>
> Thanks for the update.
>
> 1. Could you add these as code examples to the KIP, and not as prose? 
> See [1-3] for some examples of well-written public interfaces.
>
> 2. I find the proposed API surface area excessive, and think we need 
> to narrow it down. Looking at what's merged on trunk right now and not 
> including your proposed modifications, this KIP includes:
> * 2 interfaces
> * 12 classes
> * 5 constructors
> * 8 static methods
> * 2 static variables
> * 18 unique instance methods
> Committing to long-term stability of this many classes and interfaces 
> as public API will make it hard to initially review it, and innovate 
> on it in the future.
>
> 4. Is there a way to modify the existing signatures/implementations 
> such that we don't have to expose any subclasses of Compression as 
> public API, only the CompressionTypes?
> 6. What does "first provider loaded" mean to users? AFAIU, this looks 
> arbitrary to users, as it depends on classpath ordering, filesystem 
> ordering, etc.
> 8. I'm trying to understand under what conditions this can be attacked 
> (threat model), and what mitigations Kafka needs to provide 
> (implementation). Is Kafka providing special filtering? Is Kafka 
> providing the custom class loader?
> 6, 7, 8. Please add these details to the KIP so that other reviewers 
> don't need to follow the discussion thread so closely.
> 9. The testing section looks like it is written for individuals 
> implementing accelerator plugins. Could you also write about how the 
> Kafka side of the implementation will be tested?
>
> Thanks,
> Greg
>
> [1]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Onc
> e+Delivery+and+Transactional+Messaging
> [2]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tie
> red+Storage
> [3]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism
> +for+plugins+and+connectors+to+register+metrics
>
> On Fri, Feb 21, 2025 at 12:15 AM Denloye, Olasoji < 
> olasoji.denloye@intel.com>
> wrote:
>
> > Hi Greg, Thanks for your response.
> >
> > 1. Some changes will be made to the Compression and Builder 
> > interfaces. At least one default method, isAvailable(), would be 
> > added to the Builder interface. The static methods in the 
> > Compression interface that create and return specific builders would 
> > be modified to return a new builder or a loaded builder from a 
> > service provider if it exists. The return types may need to be more 
> > generic e.g Builder<GzipCompression> vs GzipCompression.Builder.
> > 2. These classes are appropriate to add to the API surface as they 
> > may be required to develop efficient compression implementations. I 
> > will modify the KIP.
> > 4. This is because the Compression interface (currently) expects a 
> > Builder of type 'GzipCompression.Builder'. I agree that alternative 
> > implementations need not extend the base implementation with some 
> > modifications to the Compression interface. However if an 
> > alternative implementation is registered as a service provider of 
> > GzipCompression then it must be of type 'CompressionType.GZIP' and 
> > it must be fully compatible with the base implementation of GzipCompression.
> > 5. So far no, but 'Configurable' looks promising 6. If no specific 
> > configuration is set, then the first provider loaded and available 
> > will be used. If a specific configuration is set then that specific 
> > provider will be used if available else the base implementation.
> > 7. Yes, a ConfigException will be thrown if the class specified in 
> > the config is not found on the classpath.
> > 8. This can be implemented by using special filtering or a custom 
> > class loader. The concern here is that any class on the class path 
> > (trusted or
> > not) could be loaded which could intercept messages or corrupt 
> > messages by compressing using an unknown algorithm.
> >
> > -----Original Message-----
> > From: Greg Harris <gr...@aiven.io.INVALID>
> > Sent: Tuesday, February 18, 2025 1:58 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Olasoji,
> >
> > Thanks for the KIP! I find it addresses many of the concerns I had 
> > with KIP-984. Thank you for following up on that!
> >
> > 1. Since the Compression interface is internal currently, it may 
> > contain some components that were not designed with pluggability in 
> > mind, such as the static methods. Are these interfaces being 
> > published as-is, or are some changes going to be made first?
> > 2. Similarly, the ByteBufferOutputStream, CompressionType, 
> > BufferSupplier, ByteBufferInputStream, ChunkedByteStream, and 
> > GzipOutputStream classes in the signature or implementations of 
> > Compression are not public API currently, and would need to be made 
> > public along with Compression. Are these classes appropriate to add 
> > to
> our API surface?
> > 3. Could you adjust the KIP structure/formatting to make the public 
> > API surface area more clear? Some prose in "Proposed Changes" could 
> > become code examples in "Public Interfaces".
> > 4. What is the significance of this statement: "Any new service 
> > provider should extend the base implementation and thus return the 
> > same CompressionType". Are we expecting that `o instanceof 
> > GzipCompression == o.type().equals(CompressionType.GZIP)` at all 
> > times? If an alternative implementation doesn't need any part of the 
> > upstream implementation, does it make sense to force them to extend 
> > one
> another?
> > 5. Is this interface expected to be composed with any other plugin 
> > mix-ins, such as Configurable, Reconfigurable, Closeable, 
> > Monitorable, Versioned, etc?
> > 6. If there are multiple accelerators present, which one will be chosen?
> > 7. If the configuration is set but the accelerator implementation 
> > isn't available, does that cause a ConfigException?
> > 8. Could you expand on this statement: "Due to security and/or 
> > compatibility concerns it is possible to limit this feature to only 
> > load approved service providers (e.g those distributed with Kafka)".
> > How will this be implemented? What threat model does this assume?
> >
> > Thanks,
> > Greg
> >
> > On Tue, Feb 18, 2025 at 11:41 AM Denloye, Olasoji < 
> > olasoji.denloye@intel.com>
> > wrote:
> >
> > > Hi, I would like to bring this back to everyone's attention. This 
> > > KIP introduces a simple framework that enables the use of 
> > > alternative, faster implementations of the existing compression 
> > > codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji <ol...@intel.com>
> > > Sent: Thursday, February 13, 2025 1:22 PM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi, I would like to bring this back to everyone's attention. I 
> > > have made changes to the proposal to incorporate the ServiceLoader 
> > > API as
> > suggested.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji
> > > Sent: Thursday, December 19, 2024 11:01 AM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Frédérik, Thanks for the suggestion to use the existing 
> > > ServiceLoader feature in Java. It could serve as suitable 
> > > mechanism for selecting the best available compression codec.
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Frédérik Rouleau <fr...@confluent.io.INVALID>
> > > Sent: Wednesday, December 18, 2024 12:24 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Olasoji,
> > >
> > > I think that having a way to provide some optimised/accelerated 
> > > implementation is a good idea.
> > > What not using the ServiceLoader feature of Java? That would allow 
> > > anyone to provide their own optimised implementation depending on 
> > > their infrastructure. We can keep the existing implementation as 
> > > the default and allow Kafka to find a better implementation based 
> > > on Service found. As you mentioned, we might need to enrich the 
> > > interface to allow Kafka to find the best version available at runtime.
> > >
> > > Regards,
> > >
> > > On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji < 
> > > olasoji.denloye@intel.com>
> > > wrote:
> > >
> > > > Hi Everyone,
> > > >
> > > > I would like to start a discussion on KIP-1121: Compression 
> > > > acceleration acceleration in Kafka. KIP-1121: Compression 
> > > > acceleration in Kafka - Apache Kafka - Apache Software 
> > > > Foundation< 
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Co
> > > > mp
> > > > re
> > > > ss
> > > > ion+acceleration+in+Kafka
> > > > >
> > > >
> > >
> >
>

RE: [VOTE] KIP-1121: Compression acceleration in Kafka

["Denloye, Olasoji" <ol...@intel.com>]

Hello all, 

Voting is in progress for KIP-1121. 

This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks


-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com> 
Sent: Thursday, June 26, 2025 4:00 PM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello all, please vote for KIP-1121. 

This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Friday, June 13, 2025 8:40 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello, I would like to call a vote for KIP-1121. This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, June 11, 2025 11:31 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello all, I would like to call a vote for KIP-1121. This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks


-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, June 4, 2025 9:00 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hello all,

Regarding Compression acceleration, I would like to call a vote soon so if anyone wants to contribute to the discussion or bring up other concerns, you are welcome to comment.

Thanks,
Olasoji

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Thursday, May 29, 2025 10:39 PM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hello all, any more comments?

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, May 7, 2025 10:55 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Greg,

2, 10 I figured a way to not expose any of these internal classes to the public API. I added a new interface in the place of BufferSupplier.
7. No, CompressionService#isAvailable returning false does not currently cause a ConfigException, it would cause a fall-back to the base implementation. I have changed this to checkAvailable which throws an exception if not available. If a more appropriate exception does not already exist, we could introduce "UnavailableServiceException"
9. I have updated the testing section to include additional scenarios relating to Kafka side testing. In short, we need to ensure the default behavior is unchanged and validate all new functions and configurations. 
11. the "accelerated GzipCompressor" mentioned is a reference design and need not be a part of Kafka. Is there a recommendation for hosting/distributing implementations of CompressionService?
12. Similar to the base implementation the goal is to have a separate instance of CompressionService for each I/O threads handling compression/decompression.
13. The existing implementations will remain unchanged. If the CompressionService configuration is empty then the base implementation would be used.
14. The broker configuration is currently in ServerConfigs.java and is configured statically in broker.properties (or server.properties). I have not considered dynamic configuration via kraft.

Thanks,
Olasoji

-----Original Message-----
From: Greg Harris <gr...@aiven.io.INVALID>
Sent: Wednesday, April 30, 2025 1:04 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Olasoji,

Thanks for updating the KIP.

2. I'm still not satisfied with the description of the public API in the KIP. It mentions these classes:
* org.apache.kafka.common.utils.ByteBufferOutputStream
* org.apache.kafka.common.utils.ByteBufferInputStream
* org.apache.kafka.common.utils.BufferSupplier
* org.apache.kafka.common.record.CompressionType
But not any description of these classes. Please include signatures of all new methods/variables/etc that are being exposed.

7. Does CompressionService#isAvailable returning false cause a ConfigException? Could the CompressionService provide a reason (via an
exception) of why it isn't available, instead of returning a boolean and requiring users to look through logs to debug the unavailability?
8. Thanks for clarifying the security model. This is just like any other Java application, where you need to trust the JARs that you add to your application. I don't think that the isolated classloader needs to be mentioned in the KIP, so thank you for removing that.
9. This point does not appear to have been addressed.
10. The classes mentioned above are in existing packages that are not exposed via Javadoc, and we have generally included whole packages in the API at once. I think these classes need to be moved to a different package, or removed from the API surface area.
11. The KIP states that it "also introduces an accelerated GzipCompressor".
Does this require Apache Kafka to depend on a specific accelerator implementation? The Apache Kafka project has traditionally been very averse to adding dependencies like this.
12. What sort of concurrency guarantees are expected from CompressionService implementations? Should each connection, thread, or JVM get its own instance of each CompressionService?
13. Will the existing implementations be converted to CompressionService classes, or will they be special fallbacks? Will the default for the configuration be "null" or some real class?
14. It looks like the new configurations are being added to the CommonClientConfigs to be included in both producer and consumer. Where is this configuration on the broker side? Is it a dynamic config (able to be reconfigured via KRaft) or a static config (only configurable by server.properties)?

Thanks,
Greg

On Wed, Apr 30, 2025 at 10:2 AM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hello all,
>
> I would like to call a vote soon so if anyone wants to contribute to 
> the discussion or bring up other concerns, you are welcome to comment.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Wednesday, April 23, 2025 10:22 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi All,
>
> This is a gentle reminder about open discussion on this compression 
> acceleration KIP. I would love to hear your thoughts.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Monday, April 7, 2025 8:49 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Greg,
>
> I have refactored the proposal to reduce the API surface area. I 
> introduced a new CompressionService interface which contains basic 
> compression/decompression functions. A service provider, when 
> supplied, would be wrapped in a concrete 'Compression' and run similar 
> to the existing compression codecs. The 'Compression' interface and 
> its subclasses have been decoupled from these changes.
>
> 6. based on your feedback, offloading would occur if and only if the 
> specific configuration is set, even if multiple service providers are 
> available 8. The threat model is that dynamically loading any 
> untrusted or untested jar on the classpath can be dangerous. The 
> purpose of bringing this up is to discuss how service provider
> (plugins) is distributed and installed. On one hand, a centralized 
> Kafka repository and on the other a decentralized approach where 
> plugins are developed and installed on the fly. Which is preferable?
> The choice would determine what security mitigations are needed if any.
>
> https://cwiki.apache.org/confluence/x/Xgr0Ew
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Monday, February 24, 2025 12:42 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Olasoji,
>
> Thanks for the update.
>
> 1. Could you add these as code examples to the KIP, and not as prose? 
> See [1-3] for some examples of well-written public interfaces.
>
> 2. I find the proposed API surface area excessive, and think we need 
> to narrow it down. Looking at what's merged on trunk right now and not 
> including your proposed modifications, this KIP includes:
> * 2 interfaces
> * 12 classes
> * 5 constructors
> * 8 static methods
> * 2 static variables
> * 18 unique instance methods
> Committing to long-term stability of this many classes and interfaces 
> as public API will make it hard to initially review it, and innovate 
> on it in the future.
>
> 4. Is there a way to modify the existing signatures/implementations 
> such that we don't have to expose any subclasses of Compression as 
> public API, only the CompressionTypes?
> 6. What does "first provider loaded" mean to users? AFAIU, this looks 
> arbitrary to users, as it depends on classpath ordering, filesystem 
> ordering, etc.
> 8. I'm trying to understand under what conditions this can be attacked 
> (threat model), and what mitigations Kafka needs to provide 
> (implementation). Is Kafka providing special filtering? Is Kafka 
> providing the custom class loader?
> 6, 7, 8. Please add these details to the KIP so that other reviewers 
> don't need to follow the discussion thread so closely.
> 9. The testing section looks like it is written for individuals 
> implementing accelerator plugins. Could you also write about how the 
> Kafka side of the implementation will be tested?
>
> Thanks,
> Greg
>
> [1]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Onc
> e+Delivery+and+Transactional+Messaging
> [2]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tie
> red+Storage
> [3]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism
> +for+plugins+and+connectors+to+register+metrics
>
> On Fri, Feb 21, 2025 at 12:15 AM Denloye, Olasoji < 
> olasoji.denloye@intel.com>
> wrote:
>
> > Hi Greg, Thanks for your response.
> >
> > 1. Some changes will be made to the Compression and Builder 
> > interfaces. At least one default method, isAvailable(), would be 
> > added to the Builder interface. The static methods in the 
> > Compression interface that create and return specific builders would 
> > be modified to return a new builder or a loaded builder from a 
> > service provider if it exists. The return types may need to be more 
> > generic e.g Builder<GzipCompression> vs GzipCompression.Builder.
> > 2. These classes are appropriate to add to the API surface as they 
> > may be required to develop efficient compression implementations. I 
> > will modify the KIP.
> > 4. This is because the Compression interface (currently) expects a 
> > Builder of type 'GzipCompression.Builder'. I agree that alternative 
> > implementations need not extend the base implementation with some 
> > modifications to the Compression interface. However if an 
> > alternative implementation is registered as a service provider of 
> > GzipCompression then it must be of type 'CompressionType.GZIP' and 
> > it must be fully compatible with the base implementation of GzipCompression.
> > 5. So far no, but 'Configurable' looks promising 6. If no specific 
> > configuration is set, then the first provider loaded and available 
> > will be used. If a specific configuration is set then that specific 
> > provider will be used if available else the base implementation.
> > 7. Yes, a ConfigException will be thrown if the class specified in 
> > the config is not found on the classpath.
> > 8. This can be implemented by using special filtering or a custom 
> > class loader. The concern here is that any class on the class path 
> > (trusted or
> > not) could be loaded which could intercept messages or corrupt 
> > messages by compressing using an unknown algorithm.
> >
> > -----Original Message-----
> > From: Greg Harris <gr...@aiven.io.INVALID>
> > Sent: Tuesday, February 18, 2025 1:58 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Olasoji,
> >
> > Thanks for the KIP! I find it addresses many of the concerns I had 
> > with KIP-984. Thank you for following up on that!
> >
> > 1. Since the Compression interface is internal currently, it may 
> > contain some components that were not designed with pluggability in 
> > mind, such as the static methods. Are these interfaces being 
> > published as-is, or are some changes going to be made first?
> > 2. Similarly, the ByteBufferOutputStream, CompressionType, 
> > BufferSupplier, ByteBufferInputStream, ChunkedByteStream, and 
> > GzipOutputStream classes in the signature or implementations of 
> > Compression are not public API currently, and would need to be made 
> > public along with Compression. Are these classes appropriate to add 
> > to
> our API surface?
> > 3. Could you adjust the KIP structure/formatting to make the public 
> > API surface area more clear? Some prose in "Proposed Changes" could 
> > become code examples in "Public Interfaces".
> > 4. What is the significance of this statement: "Any new service 
> > provider should extend the base implementation and thus return the 
> > same CompressionType". Are we expecting that `o instanceof 
> > GzipCompression == o.type().equals(CompressionType.GZIP)` at all 
> > times? If an alternative implementation doesn't need any part of the 
> > upstream implementation, does it make sense to force them to extend 
> > one
> another?
> > 5. Is this interface expected to be composed with any other plugin 
> > mix-ins, such as Configurable, Reconfigurable, Closeable, 
> > Monitorable, Versioned, etc?
> > 6. If there are multiple accelerators present, which one will be chosen?
> > 7. If the configuration is set but the accelerator implementation 
> > isn't available, does that cause a ConfigException?
> > 8. Could you expand on this statement: "Due to security and/or 
> > compatibility concerns it is possible to limit this feature to only 
> > load approved service providers (e.g those distributed with Kafka)".
> > How will this be implemented? What threat model does this assume?
> >
> > Thanks,
> > Greg
> >
> > On Tue, Feb 18, 2025 at 11:41 AM Denloye, Olasoji < 
> > olasoji.denloye@intel.com>
> > wrote:
> >
> > > Hi, I would like to bring this back to everyone's attention. This 
> > > KIP introduces a simple framework that enables the use of 
> > > alternative, faster implementations of the existing compression 
> > > codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji <ol...@intel.com>
> > > Sent: Thursday, February 13, 2025 1:22 PM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi, I would like to bring this back to everyone's attention. I 
> > > have made changes to the proposal to incorporate the ServiceLoader 
> > > API as
> > suggested.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji
> > > Sent: Thursday, December 19, 2024 11:01 AM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Frédérik, Thanks for the suggestion to use the existing 
> > > ServiceLoader feature in Java. It could serve as suitable 
> > > mechanism for selecting the best available compression codec.
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Frédérik Rouleau <fr...@confluent.io.INVALID>
> > > Sent: Wednesday, December 18, 2024 12:24 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Olasoji,
> > >
> > > I think that having a way to provide some optimised/accelerated 
> > > implementation is a good idea.
> > > What not using the ServiceLoader feature of Java? That would allow 
> > > anyone to provide their own optimised implementation depending on 
> > > their infrastructure. We can keep the existing implementation as 
> > > the default and allow Kafka to find a better implementation based 
> > > on Service found. As you mentioned, we might need to enrich the 
> > > interface to allow Kafka to find the best version available at runtime.
> > >
> > > Regards,
> > >
> > > On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji < 
> > > olasoji.denloye@intel.com>
> > > wrote:
> > >
> > > > Hi Everyone,
> > > >
> > > > I would like to start a discussion on KIP-1121: Compression 
> > > > acceleration acceleration in Kafka. KIP-1121: Compression 
> > > > acceleration in Kafka - Apache Kafka - Apache Software 
> > > > Foundation< 
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Co
> > > > mp
> > > > re
> > > > ss
> > > > ion+acceleration+in+Kafka
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-1121: Compression acceleration in Kafka

[Kirk True <ki...@kirktrue.pro>]

Hi,

The prevailing interpretation of the KIP voting process is to start a dedicated VOTE email thread for voting.

See:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=50859233#KafkaImprovementProposals-Process

Thanks,
Kirk

On Thu, Jun 26, 2025, at 3:59 PM, Denloye, Olasoji wrote:
> Hello all, please vote for KIP-1121. 
> 
> This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.
> 
> https://cwiki.apache.org/confluence/x/Xgr0Ew    
> 
> Thanks
> 
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com> 
> Sent: Friday, June 13, 2025 8:40 AM
> To: dev@kafka.apache.org
> Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka
> 
> Hello, I would like to call a vote for KIP-1121. This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.
> 
> https://cwiki.apache.org/confluence/x/Xgr0Ew    
> 
> Thanks
> 
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Wednesday, June 11, 2025 11:31 AM
> To: dev@kafka.apache.org
> Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka
> 
> Hello all, I would like to call a vote for KIP-1121. This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.
> 
> https://cwiki.apache.org/confluence/x/Xgr0Ew    
> 
> Thanks
> 
> 
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Wednesday, June 4, 2025 9:00 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> 
> Hello all,
> 
> Regarding Compression acceleration, I would like to call a vote soon so if anyone wants to contribute to the discussion or bring up other concerns, you are welcome to comment.
> 
> Thanks,
> Olasoji
> 
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Thursday, May 29, 2025 10:39 PM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> 
> Hello all, any more comments?
> 
> Thanks
> 
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Wednesday, May 7, 2025 10:55 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> 
> Hi Greg,
> 
> 2, 10 I figured a way to not expose any of these internal classes to the public API. I added a new interface in the place of BufferSupplier.
> 7. No, CompressionService#isAvailable returning false does not currently cause a ConfigException, it would cause a fall-back to the base implementation. I have changed this to checkAvailable which throws an exception if not available. If a more appropriate exception does not already exist, we could introduce "UnavailableServiceException"
> 9. I have updated the testing section to include additional scenarios relating to Kafka side testing. In short, we need to ensure the default behavior is unchanged and validate all new functions and configurations. 
> 11. the "accelerated GzipCompressor" mentioned is a reference design and need not be a part of Kafka. Is there a recommendation for hosting/distributing implementations of CompressionService?
> 12. Similar to the base implementation the goal is to have a separate instance of CompressionService for each I/O threads handling compression/decompression.
> 13. The existing implementations will remain unchanged. If the CompressionService configuration is empty then the base implementation would be used.
> 14. The broker configuration is currently in ServerConfigs.java and is configured statically in broker.properties (or server.properties). I have not considered dynamic configuration via kraft.
> 
> Thanks,
> Olasoji
> 
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Wednesday, April 30, 2025 1:04 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> 
> Hi Olasoji,
> 
> Thanks for updating the KIP.
> 
> 2. I'm still not satisfied with the description of the public API in the KIP. It mentions these classes:
> * org.apache.kafka.common.utils.ByteBufferOutputStream
> * org.apache.kafka.common.utils.ByteBufferInputStream
> * org.apache.kafka.common.utils.BufferSupplier
> * org.apache.kafka.common.record.CompressionType
> But not any description of these classes. Please include signatures of all new methods/variables/etc that are being exposed.
> 
> 7. Does CompressionService#isAvailable returning false cause a ConfigException? Could the CompressionService provide a reason (via an
> exception) of why it isn't available, instead of returning a boolean and requiring users to look through logs to debug the unavailability?
> 8. Thanks for clarifying the security model. This is just like any other Java application, where you need to trust the JARs that you add to your application. I don't think that the isolated classloader needs to be mentioned in the KIP, so thank you for removing that.
> 9. This point does not appear to have been addressed.
> 10. The classes mentioned above are in existing packages that are not exposed via Javadoc, and we have generally included whole packages in the API at once. I think these classes need to be moved to a different package, or removed from the API surface area.
> 11. The KIP states that it "also introduces an accelerated GzipCompressor".
> Does this require Apache Kafka to depend on a specific accelerator implementation? The Apache Kafka project has traditionally been very averse to adding dependencies like this.
> 12. What sort of concurrency guarantees are expected from CompressionService implementations? Should each connection, thread, or JVM get its own instance of each CompressionService?
> 13. Will the existing implementations be converted to CompressionService classes, or will they be special fallbacks? Will the default for the configuration be "null" or some real class?
> 14. It looks like the new configurations are being added to the CommonClientConfigs to be included in both producer and consumer. Where is this configuration on the broker side? Is it a dynamic config (able to be reconfigured via KRaft) or a static config (only configurable by server.properties)?
> 
> Thanks,
> Greg
> 
> On Wed, Apr 30, 2025 at 10:2 AM Denloye, Olasoji <ol...@intel.com>
> wrote:
> 
> > Hello all,
> >
> > I would like to call a vote soon so if anyone wants to contribute to 
> > the discussion or bring up other concerns, you are welcome to comment.
> >
> > Thanks,
> > Olasoji
> >
> > -----Original Message-----
> > From: Denloye, Olasoji <ol...@intel.com>
> > Sent: Wednesday, April 23, 2025 10:22 AM
> > To: dev@kafka.apache.org
> > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi All,
> >
> > This is a gentle reminder about open discussion on this compression 
> > acceleration KIP. I would love to hear your thoughts.
> >
> > Thanks,
> > Olasoji
> >
> > -----Original Message-----
> > From: Denloye, Olasoji <ol...@intel.com>
> > Sent: Monday, April 7, 2025 8:49 AM
> > To: dev@kafka.apache.org
> > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Greg,
> >
> > I have refactored the proposal to reduce the API surface area. I 
> > introduced a new CompressionService interface which contains basic 
> > compression/decompression functions. A service provider, when 
> > supplied, would be wrapped in a concrete 'Compression' and run similar 
> > to the existing compression codecs. The 'Compression' interface and 
> > its subclasses have been decoupled from these changes.
> >
> > 6. based on your feedback, offloading would occur if and only if the 
> > specific configuration is set, even if multiple service providers are 
> > available 8. The threat model is that dynamically loading any 
> > untrusted or untested jar on the classpath can be dangerous. The 
> > purpose of bringing this up is to discuss how service provider
> > (plugins) is distributed and installed. On one hand, a centralized 
> > Kafka repository and on the other a decentralized approach where 
> > plugins are developed and installed on the fly. Which is preferable?
> > The choice would determine what security mitigations are needed if any.
> >
> > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > Thanks,
> > Olasoji
> >
> > -----Original Message-----
> > From: Greg Harris <gr...@aiven.io.INVALID>
> > Sent: Monday, February 24, 2025 12:42 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Olasoji,
> >
> > Thanks for the update.
> >
> > 1. Could you add these as code examples to the KIP, and not as prose? 
> > See [1-3] for some examples of well-written public interfaces.
> >
> > 2. I find the proposed API surface area excessive, and think we need 
> > to narrow it down. Looking at what's merged on trunk right now and not 
> > including your proposed modifications, this KIP includes:
> > * 2 interfaces
> > * 12 classes
> > * 5 constructors
> > * 8 static methods
> > * 2 static variables
> > * 18 unique instance methods
> > Committing to long-term stability of this many classes and interfaces 
> > as public API will make it hard to initially review it, and innovate 
> > on it in the future.
> >
> > 4. Is there a way to modify the existing signatures/implementations 
> > such that we don't have to expose any subclasses of Compression as 
> > public API, only the CompressionTypes?
> > 6. What does "first provider loaded" mean to users? AFAIU, this looks 
> > arbitrary to users, as it depends on classpath ordering, filesystem 
> > ordering, etc.
> > 8. I'm trying to understand under what conditions this can be attacked 
> > (threat model), and what mitigations Kafka needs to provide 
> > (implementation). Is Kafka providing special filtering? Is Kafka 
> > providing the custom class loader?
> > 6, 7, 8. Please add these details to the KIP so that other reviewers 
> > don't need to follow the discussion thread so closely.
> > 9. The testing section looks like it is written for individuals 
> > implementing accelerator plugins. Could you also write about how the 
> > Kafka side of the implementation will be tested?
> >
> > Thanks,
> > Greg
> >
> > [1]
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Onc
> > e+Delivery+and+Transactional+Messaging
> > [2]
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tie
> > red+Storage
> > [3]
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism
> > +for+plugins+and+connectors+to+register+metrics
> >
> > On Fri, Feb 21, 2025 at 12:15 AM Denloye, Olasoji < 
> > olasoji.denloye@intel.com>
> > wrote:
> >
> > > Hi Greg, Thanks for your response.
> > >
> > > 1. Some changes will be made to the Compression and Builder 
> > > interfaces. At least one default method, isAvailable(), would be 
> > > added to the Builder interface. The static methods in the 
> > > Compression interface that create and return specific builders would 
> > > be modified to return a new builder or a loaded builder from a 
> > > service provider if it exists. The return types may need to be more 
> > > generic e.g Builder<GzipCompression> vs GzipCompression.Builder.
> > > 2. These classes are appropriate to add to the API surface as they 
> > > may be required to develop efficient compression implementations. I 
> > > will modify the KIP.
> > > 4. This is because the Compression interface (currently) expects a 
> > > Builder of type 'GzipCompression.Builder'. I agree that alternative 
> > > implementations need not extend the base implementation with some 
> > > modifications to the Compression interface. However if an 
> > > alternative implementation is registered as a service provider of 
> > > GzipCompression then it must be of type 'CompressionType.GZIP' and 
> > > it must be fully compatible with the base implementation of GzipCompression.
> > > 5. So far no, but 'Configurable' looks promising 6. If no specific 
> > > configuration is set, then the first provider loaded and available 
> > > will be used. If a specific configuration is set then that specific 
> > > provider will be used if available else the base implementation.
> > > 7. Yes, a ConfigException will be thrown if the class specified in 
> > > the config is not found on the classpath.
> > > 8. This can be implemented by using special filtering or a custom 
> > > class loader. The concern here is that any class on the class path 
> > > (trusted or
> > > not) could be loaded which could intercept messages or corrupt 
> > > messages by compressing using an unknown algorithm.
> > >
> > > -----Original Message-----
> > > From: Greg Harris <gr...@aiven.io.INVALID>
> > > Sent: Tuesday, February 18, 2025 1:58 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Olasoji,
> > >
> > > Thanks for the KIP! I find it addresses many of the concerns I had 
> > > with KIP-984. Thank you for following up on that!
> > >
> > > 1. Since the Compression interface is internal currently, it may 
> > > contain some components that were not designed with pluggability in 
> > > mind, such as the static methods. Are these interfaces being 
> > > published as-is, or are some changes going to be made first?
> > > 2. Similarly, the ByteBufferOutputStream, CompressionType, 
> > > BufferSupplier, ByteBufferInputStream, ChunkedByteStream, and 
> > > GzipOutputStream classes in the signature or implementations of 
> > > Compression are not public API currently, and would need to be made 
> > > public along with Compression. Are these classes appropriate to add 
> > > to
> > our API surface?
> > > 3. Could you adjust the KIP structure/formatting to make the public 
> > > API surface area more clear? Some prose in "Proposed Changes" could 
> > > become code examples in "Public Interfaces".
> > > 4. What is the significance of this statement: "Any new service 
> > > provider should extend the base implementation and thus return the 
> > > same CompressionType". Are we expecting that `o instanceof 
> > > GzipCompression == o.type().equals(CompressionType.GZIP)` at all 
> > > times? If an alternative implementation doesn't need any part of the 
> > > upstream implementation, does it make sense to force them to extend 
> > > one
> > another?
> > > 5. Is this interface expected to be composed with any other plugin 
> > > mix-ins, such as Configurable, Reconfigurable, Closeable, 
> > > Monitorable, Versioned, etc?
> > > 6. If there are multiple accelerators present, which one will be chosen?
> > > 7. If the configuration is set but the accelerator implementation 
> > > isn't available, does that cause a ConfigException?
> > > 8. Could you expand on this statement: "Due to security and/or 
> > > compatibility concerns it is possible to limit this feature to only 
> > > load approved service providers (e.g those distributed with Kafka)".
> > > How will this be implemented? What threat model does this assume?
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Tue, Feb 18, 2025 at 11:41 AM Denloye, Olasoji < 
> > > olasoji.denloye@intel.com>
> > > wrote:
> > >
> > > > Hi, I would like to bring this back to everyone's attention. This 
> > > > KIP introduces a simple framework that enables the use of 
> > > > alternative, faster implementations of the existing compression 
> > > > codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.
> > > >
> > > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > > >
> > > > Thanks
> > > >
> > > > -----Original Message-----
> > > > From: Denloye, Olasoji <ol...@intel.com>
> > > > Sent: Thursday, February 13, 2025 1:22 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > > >
> > > > Hi, I would like to bring this back to everyone's attention. I 
> > > > have made changes to the proposal to incorporate the ServiceLoader 
> > > > API as
> > > suggested.
> > > >
> > > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > > >
> > > > Thanks
> > > >
> > > > -----Original Message-----
> > > > From: Denloye, Olasoji
> > > > Sent: Thursday, December 19, 2024 11:01 AM
> > > > To: dev@kafka.apache.org
> > > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > > >
> > > > Hi Frédérik, Thanks for the suggestion to use the existing 
> > > > ServiceLoader feature in Java. It could serve as suitable 
> > > > mechanism for selecting the best available compression codec.
> > > >
> > > > Thanks
> > > >
> > > > -----Original Message-----
> > > > From: Frédérik Rouleau <fr...@confluent.io.INVALID>
> > > > Sent: Wednesday, December 18, 2024 12:24 AM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > > >
> > > > Hi Olasoji,
> > > >
> > > > I think that having a way to provide some optimised/accelerated 
> > > > implementation is a good idea.
> > > > What not using the ServiceLoader feature of Java? That would allow 
> > > > anyone to provide their own optimised implementation depending on 
> > > > their infrastructure. We can keep the existing implementation as 
> > > > the default and allow Kafka to find a better implementation based 
> > > > on Service found. As you mentioned, we might need to enrich the 
> > > > interface to allow Kafka to find the best version available at runtime.
> > > >
> > > > Regards,
> > > >
> > > > On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji < 
> > > > olasoji.denloye@intel.com>
> > > > wrote:
> > > >
> > > > > Hi Everyone,
> > > > >
> > > > > I would like to start a discussion on KIP-1121: Compression 
> > > > > acceleration acceleration in Kafka. KIP-1121: Compression 
> > > > > acceleration in Kafka - Apache Kafka - Apache Software 
> > > > > Foundation< 
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Co
> > > > > mp
> > > > > re
> > > > > ss
> > > > > ion+acceleration+in+Kafka
> > > > > >
> > > > >
> > > >
> > >
> >
> 

RE: [VOTE] KIP-1121: Compression acceleration in Kafka

["Denloye, Olasoji" <ol...@intel.com>]

Hello all, please vote for KIP-1121. 

This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com> 
Sent: Friday, June 13, 2025 8:40 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello, I would like to call a vote for KIP-1121. This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, June 11, 2025 11:31 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello all, I would like to call a vote for KIP-1121. This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks


-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, June 4, 2025 9:00 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hello all,

Regarding Compression acceleration, I would like to call a vote soon so if anyone wants to contribute to the discussion or bring up other concerns, you are welcome to comment.

Thanks,
Olasoji

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Thursday, May 29, 2025 10:39 PM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hello all, any more comments?

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, May 7, 2025 10:55 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Greg,

2, 10 I figured a way to not expose any of these internal classes to the public API. I added a new interface in the place of BufferSupplier.
7. No, CompressionService#isAvailable returning false does not currently cause a ConfigException, it would cause a fall-back to the base implementation. I have changed this to checkAvailable which throws an exception if not available. If a more appropriate exception does not already exist, we could introduce "UnavailableServiceException"
9. I have updated the testing section to include additional scenarios relating to Kafka side testing. In short, we need to ensure the default behavior is unchanged and validate all new functions and configurations. 
11. the "accelerated GzipCompressor" mentioned is a reference design and need not be a part of Kafka. Is there a recommendation for hosting/distributing implementations of CompressionService?
12. Similar to the base implementation the goal is to have a separate instance of CompressionService for each I/O threads handling compression/decompression.
13. The existing implementations will remain unchanged. If the CompressionService configuration is empty then the base implementation would be used.
14. The broker configuration is currently in ServerConfigs.java and is configured statically in broker.properties (or server.properties). I have not considered dynamic configuration via kraft.

Thanks,
Olasoji

-----Original Message-----
From: Greg Harris <gr...@aiven.io.INVALID>
Sent: Wednesday, April 30, 2025 1:04 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Olasoji,

Thanks for updating the KIP.

2. I'm still not satisfied with the description of the public API in the KIP. It mentions these classes:
* org.apache.kafka.common.utils.ByteBufferOutputStream
* org.apache.kafka.common.utils.ByteBufferInputStream
* org.apache.kafka.common.utils.BufferSupplier
* org.apache.kafka.common.record.CompressionType
But not any description of these classes. Please include signatures of all new methods/variables/etc that are being exposed.

7. Does CompressionService#isAvailable returning false cause a ConfigException? Could the CompressionService provide a reason (via an
exception) of why it isn't available, instead of returning a boolean and requiring users to look through logs to debug the unavailability?
8. Thanks for clarifying the security model. This is just like any other Java application, where you need to trust the JARs that you add to your application. I don't think that the isolated classloader needs to be mentioned in the KIP, so thank you for removing that.
9. This point does not appear to have been addressed.
10. The classes mentioned above are in existing packages that are not exposed via Javadoc, and we have generally included whole packages in the API at once. I think these classes need to be moved to a different package, or removed from the API surface area.
11. The KIP states that it "also introduces an accelerated GzipCompressor".
Does this require Apache Kafka to depend on a specific accelerator implementation? The Apache Kafka project has traditionally been very averse to adding dependencies like this.
12. What sort of concurrency guarantees are expected from CompressionService implementations? Should each connection, thread, or JVM get its own instance of each CompressionService?
13. Will the existing implementations be converted to CompressionService classes, or will they be special fallbacks? Will the default for the configuration be "null" or some real class?
14. It looks like the new configurations are being added to the CommonClientConfigs to be included in both producer and consumer. Where is this configuration on the broker side? Is it a dynamic config (able to be reconfigured via KRaft) or a static config (only configurable by server.properties)?

Thanks,
Greg

On Wed, Apr 30, 2025 at 10:2 AM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hello all,
>
> I would like to call a vote soon so if anyone wants to contribute to 
> the discussion or bring up other concerns, you are welcome to comment.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Wednesday, April 23, 2025 10:22 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi All,
>
> This is a gentle reminder about open discussion on this compression 
> acceleration KIP. I would love to hear your thoughts.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Monday, April 7, 2025 8:49 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Greg,
>
> I have refactored the proposal to reduce the API surface area. I 
> introduced a new CompressionService interface which contains basic 
> compression/decompression functions. A service provider, when 
> supplied, would be wrapped in a concrete 'Compression' and run similar 
> to the existing compression codecs. The 'Compression' interface and 
> its subclasses have been decoupled from these changes.
>
> 6. based on your feedback, offloading would occur if and only if the 
> specific configuration is set, even if multiple service providers are 
> available 8. The threat model is that dynamically loading any 
> untrusted or untested jar on the classpath can be dangerous. The 
> purpose of bringing this up is to discuss how service provider
> (plugins) is distributed and installed. On one hand, a centralized 
> Kafka repository and on the other a decentralized approach where 
> plugins are developed and installed on the fly. Which is preferable?
> The choice would determine what security mitigations are needed if any.
>
> https://cwiki.apache.org/confluence/x/Xgr0Ew
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Monday, February 24, 2025 12:42 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Olasoji,
>
> Thanks for the update.
>
> 1. Could you add these as code examples to the KIP, and not as prose? 
> See [1-3] for some examples of well-written public interfaces.
>
> 2. I find the proposed API surface area excessive, and think we need 
> to narrow it down. Looking at what's merged on trunk right now and not 
> including your proposed modifications, this KIP includes:
> * 2 interfaces
> * 12 classes
> * 5 constructors
> * 8 static methods
> * 2 static variables
> * 18 unique instance methods
> Committing to long-term stability of this many classes and interfaces 
> as public API will make it hard to initially review it, and innovate 
> on it in the future.
>
> 4. Is there a way to modify the existing signatures/implementations 
> such that we don't have to expose any subclasses of Compression as 
> public API, only the CompressionTypes?
> 6. What does "first provider loaded" mean to users? AFAIU, this looks 
> arbitrary to users, as it depends on classpath ordering, filesystem 
> ordering, etc.
> 8. I'm trying to understand under what conditions this can be attacked 
> (threat model), and what mitigations Kafka needs to provide 
> (implementation). Is Kafka providing special filtering? Is Kafka 
> providing the custom class loader?
> 6, 7, 8. Please add these details to the KIP so that other reviewers 
> don't need to follow the discussion thread so closely.
> 9. The testing section looks like it is written for individuals 
> implementing accelerator plugins. Could you also write about how the 
> Kafka side of the implementation will be tested?
>
> Thanks,
> Greg
>
> [1]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Onc
> e+Delivery+and+Transactional+Messaging
> [2]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tie
> red+Storage
> [3]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism
> +for+plugins+and+connectors+to+register+metrics
>
> On Fri, Feb 21, 2025 at 12:15 AM Denloye, Olasoji < 
> olasoji.denloye@intel.com>
> wrote:
>
> > Hi Greg, Thanks for your response.
> >
> > 1. Some changes will be made to the Compression and Builder 
> > interfaces. At least one default method, isAvailable(), would be 
> > added to the Builder interface. The static methods in the 
> > Compression interface that create and return specific builders would 
> > be modified to return a new builder or a loaded builder from a 
> > service provider if it exists. The return types may need to be more 
> > generic e.g Builder<GzipCompression> vs GzipCompression.Builder.
> > 2. These classes are appropriate to add to the API surface as they 
> > may be required to develop efficient compression implementations. I 
> > will modify the KIP.
> > 4. This is because the Compression interface (currently) expects a 
> > Builder of type 'GzipCompression.Builder'. I agree that alternative 
> > implementations need not extend the base implementation with some 
> > modifications to the Compression interface. However if an 
> > alternative implementation is registered as a service provider of 
> > GzipCompression then it must be of type 'CompressionType.GZIP' and 
> > it must be fully compatible with the base implementation of GzipCompression.
> > 5. So far no, but 'Configurable' looks promising 6. If no specific 
> > configuration is set, then the first provider loaded and available 
> > will be used. If a specific configuration is set then that specific 
> > provider will be used if available else the base implementation.
> > 7. Yes, a ConfigException will be thrown if the class specified in 
> > the config is not found on the classpath.
> > 8. This can be implemented by using special filtering or a custom 
> > class loader. The concern here is that any class on the class path 
> > (trusted or
> > not) could be loaded which could intercept messages or corrupt 
> > messages by compressing using an unknown algorithm.
> >
> > -----Original Message-----
> > From: Greg Harris <gr...@aiven.io.INVALID>
> > Sent: Tuesday, February 18, 2025 1:58 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Olasoji,
> >
> > Thanks for the KIP! I find it addresses many of the concerns I had 
> > with KIP-984. Thank you for following up on that!
> >
> > 1. Since the Compression interface is internal currently, it may 
> > contain some components that were not designed with pluggability in 
> > mind, such as the static methods. Are these interfaces being 
> > published as-is, or are some changes going to be made first?
> > 2. Similarly, the ByteBufferOutputStream, CompressionType, 
> > BufferSupplier, ByteBufferInputStream, ChunkedByteStream, and 
> > GzipOutputStream classes in the signature or implementations of 
> > Compression are not public API currently, and would need to be made 
> > public along with Compression. Are these classes appropriate to add 
> > to
> our API surface?
> > 3. Could you adjust the KIP structure/formatting to make the public 
> > API surface area more clear? Some prose in "Proposed Changes" could 
> > become code examples in "Public Interfaces".
> > 4. What is the significance of this statement: "Any new service 
> > provider should extend the base implementation and thus return the 
> > same CompressionType". Are we expecting that `o instanceof 
> > GzipCompression == o.type().equals(CompressionType.GZIP)` at all 
> > times? If an alternative implementation doesn't need any part of the 
> > upstream implementation, does it make sense to force them to extend 
> > one
> another?
> > 5. Is this interface expected to be composed with any other plugin 
> > mix-ins, such as Configurable, Reconfigurable, Closeable, 
> > Monitorable, Versioned, etc?
> > 6. If there are multiple accelerators present, which one will be chosen?
> > 7. If the configuration is set but the accelerator implementation 
> > isn't available, does that cause a ConfigException?
> > 8. Could you expand on this statement: "Due to security and/or 
> > compatibility concerns it is possible to limit this feature to only 
> > load approved service providers (e.g those distributed with Kafka)".
> > How will this be implemented? What threat model does this assume?
> >
> > Thanks,
> > Greg
> >
> > On Tue, Feb 18, 2025 at 11:41 AM Denloye, Olasoji < 
> > olasoji.denloye@intel.com>
> > wrote:
> >
> > > Hi, I would like to bring this back to everyone's attention. This 
> > > KIP introduces a simple framework that enables the use of 
> > > alternative, faster implementations of the existing compression 
> > > codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji <ol...@intel.com>
> > > Sent: Thursday, February 13, 2025 1:22 PM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi, I would like to bring this back to everyone's attention. I 
> > > have made changes to the proposal to incorporate the ServiceLoader 
> > > API as
> > suggested.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji
> > > Sent: Thursday, December 19, 2024 11:01 AM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Frédérik, Thanks for the suggestion to use the existing 
> > > ServiceLoader feature in Java. It could serve as suitable 
> > > mechanism for selecting the best available compression codec.
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Frédérik Rouleau <fr...@confluent.io.INVALID>
> > > Sent: Wednesday, December 18, 2024 12:24 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Olasoji,
> > >
> > > I think that having a way to provide some optimised/accelerated 
> > > implementation is a good idea.
> > > What not using the ServiceLoader feature of Java? That would allow 
> > > anyone to provide their own optimised implementation depending on 
> > > their infrastructure. We can keep the existing implementation as 
> > > the default and allow Kafka to find a better implementation based 
> > > on Service found. As you mentioned, we might need to enrich the 
> > > interface to allow Kafka to find the best version available at runtime.
> > >
> > > Regards,
> > >
> > > On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji < 
> > > olasoji.denloye@intel.com>
> > > wrote:
> > >
> > > > Hi Everyone,
> > > >
> > > > I would like to start a discussion on KIP-1121: Compression 
> > > > acceleration acceleration in Kafka. KIP-1121: Compression 
> > > > acceleration in Kafka - Apache Kafka - Apache Software 
> > > > Foundation< 
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Co
> > > > mp
> > > > re
> > > > ss
> > > > ion+acceleration+in+Kafka
> > > > >
> > > >
> > >
> >
>

RE: [VOTE] KIP-1121: Compression acceleration in Kafka

["Denloye, Olasoji" <ol...@intel.com>]

Hello, I would like to call a vote for KIP-1121. This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com> 
Sent: Wednesday, June 11, 2025 11:31 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-1121: Compression acceleration in Kafka

Hello all, I would like to call a vote for KIP-1121. This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks


-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, June 4, 2025 9:00 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hello all,

Regarding Compression acceleration, I would like to call a vote soon so if anyone wants to contribute to the discussion or bring up other concerns, you are welcome to comment.

Thanks,
Olasoji

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Thursday, May 29, 2025 10:39 PM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hello all, any more comments?

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, May 7, 2025 10:55 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Greg,

2, 10 I figured a way to not expose any of these internal classes to the public API. I added a new interface in the place of BufferSupplier.
7. No, CompressionService#isAvailable returning false does not currently cause a ConfigException, it would cause a fall-back to the base implementation. I have changed this to checkAvailable which throws an exception if not available. If a more appropriate exception does not already exist, we could introduce "UnavailableServiceException"
9. I have updated the testing section to include additional scenarios relating to Kafka side testing. In short, we need to ensure the default behavior is unchanged and validate all new functions and configurations. 
11. the "accelerated GzipCompressor" mentioned is a reference design and need not be a part of Kafka. Is there a recommendation for hosting/distributing implementations of CompressionService?
12. Similar to the base implementation the goal is to have a separate instance of CompressionService for each I/O threads handling compression/decompression.
13. The existing implementations will remain unchanged. If the CompressionService configuration is empty then the base implementation would be used.
14. The broker configuration is currently in ServerConfigs.java and is configured statically in broker.properties (or server.properties). I have not considered dynamic configuration via kraft.

Thanks,
Olasoji

-----Original Message-----
From: Greg Harris <gr...@aiven.io.INVALID>
Sent: Wednesday, April 30, 2025 1:04 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Olasoji,

Thanks for updating the KIP.

2. I'm still not satisfied with the description of the public API in the KIP. It mentions these classes:
* org.apache.kafka.common.utils.ByteBufferOutputStream
* org.apache.kafka.common.utils.ByteBufferInputStream
* org.apache.kafka.common.utils.BufferSupplier
* org.apache.kafka.common.record.CompressionType
But not any description of these classes. Please include signatures of all new methods/variables/etc that are being exposed.

7. Does CompressionService#isAvailable returning false cause a ConfigException? Could the CompressionService provide a reason (via an
exception) of why it isn't available, instead of returning a boolean and requiring users to look through logs to debug the unavailability?
8. Thanks for clarifying the security model. This is just like any other Java application, where you need to trust the JARs that you add to your application. I don't think that the isolated classloader needs to be mentioned in the KIP, so thank you for removing that.
9. This point does not appear to have been addressed.
10. The classes mentioned above are in existing packages that are not exposed via Javadoc, and we have generally included whole packages in the API at once. I think these classes need to be moved to a different package, or removed from the API surface area.
11. The KIP states that it "also introduces an accelerated GzipCompressor".
Does this require Apache Kafka to depend on a specific accelerator implementation? The Apache Kafka project has traditionally been very averse to adding dependencies like this.
12. What sort of concurrency guarantees are expected from CompressionService implementations? Should each connection, thread, or JVM get its own instance of each CompressionService?
13. Will the existing implementations be converted to CompressionService classes, or will they be special fallbacks? Will the default for the configuration be "null" or some real class?
14. It looks like the new configurations are being added to the CommonClientConfigs to be included in both producer and consumer. Where is this configuration on the broker side? Is it a dynamic config (able to be reconfigured via KRaft) or a static config (only configurable by server.properties)?

Thanks,
Greg

On Wed, Apr 30, 2025 at 10:2 AM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hello all,
>
> I would like to call a vote soon so if anyone wants to contribute to 
> the discussion or bring up other concerns, you are welcome to comment.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Wednesday, April 23, 2025 10:22 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi All,
>
> This is a gentle reminder about open discussion on this compression 
> acceleration KIP. I would love to hear your thoughts.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Monday, April 7, 2025 8:49 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Greg,
>
> I have refactored the proposal to reduce the API surface area. I 
> introduced a new CompressionService interface which contains basic 
> compression/decompression functions. A service provider, when 
> supplied, would be wrapped in a concrete 'Compression' and run similar 
> to the existing compression codecs. The 'Compression' interface and 
> its subclasses have been decoupled from these changes.
>
> 6. based on your feedback, offloading would occur if and only if the 
> specific configuration is set, even if multiple service providers are 
> available 8. The threat model is that dynamically loading any 
> untrusted or untested jar on the classpath can be dangerous. The 
> purpose of bringing this up is to discuss how service provider
> (plugins) is distributed and installed. On one hand, a centralized 
> Kafka repository and on the other a decentralized approach where 
> plugins are developed and installed on the fly. Which is preferable?
> The choice would determine what security mitigations are needed if any.
>
> https://cwiki.apache.org/confluence/x/Xgr0Ew
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Monday, February 24, 2025 12:42 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Olasoji,
>
> Thanks for the update.
>
> 1. Could you add these as code examples to the KIP, and not as prose? 
> See [1-3] for some examples of well-written public interfaces.
>
> 2. I find the proposed API surface area excessive, and think we need 
> to narrow it down. Looking at what's merged on trunk right now and not 
> including your proposed modifications, this KIP includes:
> * 2 interfaces
> * 12 classes
> * 5 constructors
> * 8 static methods
> * 2 static variables
> * 18 unique instance methods
> Committing to long-term stability of this many classes and interfaces 
> as public API will make it hard to initially review it, and innovate 
> on it in the future.
>
> 4. Is there a way to modify the existing signatures/implementations 
> such that we don't have to expose any subclasses of Compression as 
> public API, only the CompressionTypes?
> 6. What does "first provider loaded" mean to users? AFAIU, this looks 
> arbitrary to users, as it depends on classpath ordering, filesystem 
> ordering, etc.
> 8. I'm trying to understand under what conditions this can be attacked 
> (threat model), and what mitigations Kafka needs to provide 
> (implementation). Is Kafka providing special filtering? Is Kafka 
> providing the custom class loader?
> 6, 7, 8. Please add these details to the KIP so that other reviewers 
> don't need to follow the discussion thread so closely.
> 9. The testing section looks like it is written for individuals 
> implementing accelerator plugins. Could you also write about how the 
> Kafka side of the implementation will be tested?
>
> Thanks,
> Greg
>
> [1]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Onc
> e+Delivery+and+Transactional+Messaging
> [2]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tie
> red+Storage
> [3]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism
> +for+plugins+and+connectors+to+register+metrics
>
> On Fri, Feb 21, 2025 at 12:15 AM Denloye, Olasoji < 
> olasoji.denloye@intel.com>
> wrote:
>
> > Hi Greg, Thanks for your response.
> >
> > 1. Some changes will be made to the Compression and Builder 
> > interfaces. At least one default method, isAvailable(), would be 
> > added to the Builder interface. The static methods in the 
> > Compression interface that create and return specific builders would 
> > be modified to return a new builder or a loaded builder from a 
> > service provider if it exists. The return types may need to be more 
> > generic e.g Builder<GzipCompression> vs GzipCompression.Builder.
> > 2. These classes are appropriate to add to the API surface as they 
> > may be required to develop efficient compression implementations. I 
> > will modify the KIP.
> > 4. This is because the Compression interface (currently) expects a 
> > Builder of type 'GzipCompression.Builder'. I agree that alternative 
> > implementations need not extend the base implementation with some 
> > modifications to the Compression interface. However if an 
> > alternative implementation is registered as a service provider of 
> > GzipCompression then it must be of type 'CompressionType.GZIP' and 
> > it must be fully compatible with the base implementation of GzipCompression.
> > 5. So far no, but 'Configurable' looks promising 6. If no specific 
> > configuration is set, then the first provider loaded and available 
> > will be used. If a specific configuration is set then that specific 
> > provider will be used if available else the base implementation.
> > 7. Yes, a ConfigException will be thrown if the class specified in 
> > the config is not found on the classpath.
> > 8. This can be implemented by using special filtering or a custom 
> > class loader. The concern here is that any class on the class path 
> > (trusted or
> > not) could be loaded which could intercept messages or corrupt 
> > messages by compressing using an unknown algorithm.
> >
> > -----Original Message-----
> > From: Greg Harris <gr...@aiven.io.INVALID>
> > Sent: Tuesday, February 18, 2025 1:58 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Olasoji,
> >
> > Thanks for the KIP! I find it addresses many of the concerns I had 
> > with KIP-984. Thank you for following up on that!
> >
> > 1. Since the Compression interface is internal currently, it may 
> > contain some components that were not designed with pluggability in 
> > mind, such as the static methods. Are these interfaces being 
> > published as-is, or are some changes going to be made first?
> > 2. Similarly, the ByteBufferOutputStream, CompressionType, 
> > BufferSupplier, ByteBufferInputStream, ChunkedByteStream, and 
> > GzipOutputStream classes in the signature or implementations of 
> > Compression are not public API currently, and would need to be made 
> > public along with Compression. Are these classes appropriate to add 
> > to
> our API surface?
> > 3. Could you adjust the KIP structure/formatting to make the public 
> > API surface area more clear? Some prose in "Proposed Changes" could 
> > become code examples in "Public Interfaces".
> > 4. What is the significance of this statement: "Any new service 
> > provider should extend the base implementation and thus return the 
> > same CompressionType". Are we expecting that `o instanceof 
> > GzipCompression == o.type().equals(CompressionType.GZIP)` at all 
> > times? If an alternative implementation doesn't need any part of the 
> > upstream implementation, does it make sense to force them to extend 
> > one
> another?
> > 5. Is this interface expected to be composed with any other plugin 
> > mix-ins, such as Configurable, Reconfigurable, Closeable, 
> > Monitorable, Versioned, etc?
> > 6. If there are multiple accelerators present, which one will be chosen?
> > 7. If the configuration is set but the accelerator implementation 
> > isn't available, does that cause a ConfigException?
> > 8. Could you expand on this statement: "Due to security and/or 
> > compatibility concerns it is possible to limit this feature to only 
> > load approved service providers (e.g those distributed with Kafka)".
> > How will this be implemented? What threat model does this assume?
> >
> > Thanks,
> > Greg
> >
> > On Tue, Feb 18, 2025 at 11:41 AM Denloye, Olasoji < 
> > olasoji.denloye@intel.com>
> > wrote:
> >
> > > Hi, I would like to bring this back to everyone's attention. This 
> > > KIP introduces a simple framework that enables the use of 
> > > alternative, faster implementations of the existing compression 
> > > codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji <ol...@intel.com>
> > > Sent: Thursday, February 13, 2025 1:22 PM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi, I would like to bring this back to everyone's attention. I 
> > > have made changes to the proposal to incorporate the ServiceLoader 
> > > API as
> > suggested.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji
> > > Sent: Thursday, December 19, 2024 11:01 AM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Frédérik, Thanks for the suggestion to use the existing 
> > > ServiceLoader feature in Java. It could serve as suitable 
> > > mechanism for selecting the best available compression codec.
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Frédérik Rouleau <fr...@confluent.io.INVALID>
> > > Sent: Wednesday, December 18, 2024 12:24 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Olasoji,
> > >
> > > I think that having a way to provide some optimised/accelerated 
> > > implementation is a good idea.
> > > What not using the ServiceLoader feature of Java? That would allow 
> > > anyone to provide their own optimised implementation depending on 
> > > their infrastructure. We can keep the existing implementation as 
> > > the default and allow Kafka to find a better implementation based 
> > > on Service found. As you mentioned, we might need to enrich the 
> > > interface to allow Kafka to find the best version available at runtime.
> > >
> > > Regards,
> > >
> > > On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji < 
> > > olasoji.denloye@intel.com>
> > > wrote:
> > >
> > > > Hi Everyone,
> > > >
> > > > I would like to start a discussion on KIP-1121: Compression 
> > > > acceleration acceleration in Kafka. KIP-1121: Compression 
> > > > acceleration in Kafka - Apache Kafka - Apache Software 
> > > > Foundation< 
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Co
> > > > mp
> > > > re
> > > > ss
> > > > ion+acceleration+in+Kafka
> > > > >
> > > >
> > >
> >
>

RE: [VOTE] KIP-1121: Compression acceleration in Kafka

["Denloye, Olasoji" <ol...@intel.com>]

Hello all, I would like to call a vote for KIP-1121. This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks


-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com> 
Sent: Wednesday, June 4, 2025 9:00 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hello all,

Regarding Compression acceleration, I would like to call a vote soon so if anyone wants to contribute to the discussion or bring up other concerns, you are welcome to comment.

Thanks,
Olasoji

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Thursday, May 29, 2025 10:39 PM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hello all, any more comments?

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, May 7, 2025 10:55 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Greg,

2, 10 I figured a way to not expose any of these internal classes to the public API. I added a new interface in the place of BufferSupplier.
7. No, CompressionService#isAvailable returning false does not currently cause a ConfigException, it would cause a fall-back to the base implementation. I have changed this to checkAvailable which throws an exception if not available. If a more appropriate exception does not already exist, we could introduce "UnavailableServiceException"
9. I have updated the testing section to include additional scenarios relating to Kafka side testing. In short, we need to ensure the default behavior is unchanged and validate all new functions and configurations. 
11. the "accelerated GzipCompressor" mentioned is a reference design and need not be a part of Kafka. Is there a recommendation for hosting/distributing implementations of CompressionService?
12. Similar to the base implementation the goal is to have a separate instance of CompressionService for each I/O threads handling compression/decompression.
13. The existing implementations will remain unchanged. If the CompressionService configuration is empty then the base implementation would be used.
14. The broker configuration is currently in ServerConfigs.java and is configured statically in broker.properties (or server.properties). I have not considered dynamic configuration via kraft.

Thanks,
Olasoji

-----Original Message-----
From: Greg Harris <gr...@aiven.io.INVALID>
Sent: Wednesday, April 30, 2025 1:04 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Olasoji,

Thanks for updating the KIP.

2. I'm still not satisfied with the description of the public API in the KIP. It mentions these classes:
* org.apache.kafka.common.utils.ByteBufferOutputStream
* org.apache.kafka.common.utils.ByteBufferInputStream
* org.apache.kafka.common.utils.BufferSupplier
* org.apache.kafka.common.record.CompressionType
But not any description of these classes. Please include signatures of all new methods/variables/etc that are being exposed.

7. Does CompressionService#isAvailable returning false cause a ConfigException? Could the CompressionService provide a reason (via an
exception) of why it isn't available, instead of returning a boolean and requiring users to look through logs to debug the unavailability?
8. Thanks for clarifying the security model. This is just like any other Java application, where you need to trust the JARs that you add to your application. I don't think that the isolated classloader needs to be mentioned in the KIP, so thank you for removing that.
9. This point does not appear to have been addressed.
10. The classes mentioned above are in existing packages that are not exposed via Javadoc, and we have generally included whole packages in the API at once. I think these classes need to be moved to a different package, or removed from the API surface area.
11. The KIP states that it "also introduces an accelerated GzipCompressor".
Does this require Apache Kafka to depend on a specific accelerator implementation? The Apache Kafka project has traditionally been very averse to adding dependencies like this.
12. What sort of concurrency guarantees are expected from CompressionService implementations? Should each connection, thread, or JVM get its own instance of each CompressionService?
13. Will the existing implementations be converted to CompressionService classes, or will they be special fallbacks? Will the default for the configuration be "null" or some real class?
14. It looks like the new configurations are being added to the CommonClientConfigs to be included in both producer and consumer. Where is this configuration on the broker side? Is it a dynamic config (able to be reconfigured via KRaft) or a static config (only configurable by server.properties)?

Thanks,
Greg

On Wed, Apr 30, 2025 at 10:2 AM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hello all,
>
> I would like to call a vote soon so if anyone wants to contribute to 
> the discussion or bring up other concerns, you are welcome to comment.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Wednesday, April 23, 2025 10:22 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi All,
>
> This is a gentle reminder about open discussion on this compression 
> acceleration KIP. I would love to hear your thoughts.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Monday, April 7, 2025 8:49 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Greg,
>
> I have refactored the proposal to reduce the API surface area. I 
> introduced a new CompressionService interface which contains basic 
> compression/decompression functions. A service provider, when 
> supplied, would be wrapped in a concrete 'Compression' and run similar 
> to the existing compression codecs. The 'Compression' interface and 
> its subclasses have been decoupled from these changes.
>
> 6. based on your feedback, offloading would occur if and only if the 
> specific configuration is set, even if multiple service providers are 
> available 8. The threat model is that dynamically loading any 
> untrusted or untested jar on the classpath can be dangerous. The 
> purpose of bringing this up is to discuss how service provider
> (plugins) is distributed and installed. On one hand, a centralized 
> Kafka repository and on the other a decentralized approach where 
> plugins are developed and installed on the fly. Which is preferable?
> The choice would determine what security mitigations are needed if any.
>
> https://cwiki.apache.org/confluence/x/Xgr0Ew
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Monday, February 24, 2025 12:42 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Olasoji,
>
> Thanks for the update.
>
> 1. Could you add these as code examples to the KIP, and not as prose? 
> See [1-3] for some examples of well-written public interfaces.
>
> 2. I find the proposed API surface area excessive, and think we need 
> to narrow it down. Looking at what's merged on trunk right now and not 
> including your proposed modifications, this KIP includes:
> * 2 interfaces
> * 12 classes
> * 5 constructors
> * 8 static methods
> * 2 static variables
> * 18 unique instance methods
> Committing to long-term stability of this many classes and interfaces 
> as public API will make it hard to initially review it, and innovate 
> on it in the future.
>
> 4. Is there a way to modify the existing signatures/implementations 
> such that we don't have to expose any subclasses of Compression as 
> public API, only the CompressionTypes?
> 6. What does "first provider loaded" mean to users? AFAIU, this looks 
> arbitrary to users, as it depends on classpath ordering, filesystem 
> ordering, etc.
> 8. I'm trying to understand under what conditions this can be attacked 
> (threat model), and what mitigations Kafka needs to provide 
> (implementation). Is Kafka providing special filtering? Is Kafka 
> providing the custom class loader?
> 6, 7, 8. Please add these details to the KIP so that other reviewers 
> don't need to follow the discussion thread so closely.
> 9. The testing section looks like it is written for individuals 
> implementing accelerator plugins. Could you also write about how the 
> Kafka side of the implementation will be tested?
>
> Thanks,
> Greg
>
> [1]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Onc
> e+Delivery+and+Transactional+Messaging
> [2]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tie
> red+Storage
> [3]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism
> +for+plugins+and+connectors+to+register+metrics
>
> On Fri, Feb 21, 2025 at 12:15 AM Denloye, Olasoji < 
> olasoji.denloye@intel.com>
> wrote:
>
> > Hi Greg, Thanks for your response.
> >
> > 1. Some changes will be made to the Compression and Builder 
> > interfaces. At least one default method, isAvailable(), would be 
> > added to the Builder interface. The static methods in the 
> > Compression interface that create and return specific builders would 
> > be modified to return a new builder or a loaded builder from a 
> > service provider if it exists. The return types may need to be more 
> > generic e.g Builder<GzipCompression> vs GzipCompression.Builder.
> > 2. These classes are appropriate to add to the API surface as they 
> > may be required to develop efficient compression implementations. I 
> > will modify the KIP.
> > 4. This is because the Compression interface (currently) expects a 
> > Builder of type 'GzipCompression.Builder'. I agree that alternative 
> > implementations need not extend the base implementation with some 
> > modifications to the Compression interface. However if an 
> > alternative implementation is registered as a service provider of 
> > GzipCompression then it must be of type 'CompressionType.GZIP' and 
> > it must be fully compatible with the base implementation of GzipCompression.
> > 5. So far no, but 'Configurable' looks promising 6. If no specific 
> > configuration is set, then the first provider loaded and available 
> > will be used. If a specific configuration is set then that specific 
> > provider will be used if available else the base implementation.
> > 7. Yes, a ConfigException will be thrown if the class specified in 
> > the config is not found on the classpath.
> > 8. This can be implemented by using special filtering or a custom 
> > class loader. The concern here is that any class on the class path 
> > (trusted or
> > not) could be loaded which could intercept messages or corrupt 
> > messages by compressing using an unknown algorithm.
> >
> > -----Original Message-----
> > From: Greg Harris <gr...@aiven.io.INVALID>
> > Sent: Tuesday, February 18, 2025 1:58 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Olasoji,
> >
> > Thanks for the KIP! I find it addresses many of the concerns I had 
> > with KIP-984. Thank you for following up on that!
> >
> > 1. Since the Compression interface is internal currently, it may 
> > contain some components that were not designed with pluggability in 
> > mind, such as the static methods. Are these interfaces being 
> > published as-is, or are some changes going to be made first?
> > 2. Similarly, the ByteBufferOutputStream, CompressionType, 
> > BufferSupplier, ByteBufferInputStream, ChunkedByteStream, and 
> > GzipOutputStream classes in the signature or implementations of 
> > Compression are not public API currently, and would need to be made 
> > public along with Compression. Are these classes appropriate to add 
> > to
> our API surface?
> > 3. Could you adjust the KIP structure/formatting to make the public 
> > API surface area more clear? Some prose in "Proposed Changes" could 
> > become code examples in "Public Interfaces".
> > 4. What is the significance of this statement: "Any new service 
> > provider should extend the base implementation and thus return the 
> > same CompressionType". Are we expecting that `o instanceof 
> > GzipCompression == o.type().equals(CompressionType.GZIP)` at all 
> > times? If an alternative implementation doesn't need any part of the 
> > upstream implementation, does it make sense to force them to extend 
> > one
> another?
> > 5. Is this interface expected to be composed with any other plugin 
> > mix-ins, such as Configurable, Reconfigurable, Closeable, 
> > Monitorable, Versioned, etc?
> > 6. If there are multiple accelerators present, which one will be chosen?
> > 7. If the configuration is set but the accelerator implementation 
> > isn't available, does that cause a ConfigException?
> > 8. Could you expand on this statement: "Due to security and/or 
> > compatibility concerns it is possible to limit this feature to only 
> > load approved service providers (e.g those distributed with Kafka)".
> > How will this be implemented? What threat model does this assume?
> >
> > Thanks,
> > Greg
> >
> > On Tue, Feb 18, 2025 at 11:41 AM Denloye, Olasoji < 
> > olasoji.denloye@intel.com>
> > wrote:
> >
> > > Hi, I would like to bring this back to everyone's attention. This 
> > > KIP introduces a simple framework that enables the use of 
> > > alternative, faster implementations of the existing compression 
> > > codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji <ol...@intel.com>
> > > Sent: Thursday, February 13, 2025 1:22 PM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi, I would like to bring this back to everyone's attention. I 
> > > have made changes to the proposal to incorporate the ServiceLoader 
> > > API as
> > suggested.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji
> > > Sent: Thursday, December 19, 2024 11:01 AM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Frédérik, Thanks for the suggestion to use the existing 
> > > ServiceLoader feature in Java. It could serve as suitable 
> > > mechanism for selecting the best available compression codec.
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Frédérik Rouleau <fr...@confluent.io.INVALID>
> > > Sent: Wednesday, December 18, 2024 12:24 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Olasoji,
> > >
> > > I think that having a way to provide some optimised/accelerated 
> > > implementation is a good idea.
> > > What not using the ServiceLoader feature of Java? That would allow 
> > > anyone to provide their own optimised implementation depending on 
> > > their infrastructure. We can keep the existing implementation as 
> > > the default and allow Kafka to find a better implementation based 
> > > on Service found. As you mentioned, we might need to enrich the 
> > > interface to allow Kafka to find the best version available at runtime.
> > >
> > > Regards,
> > >
> > > On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji < 
> > > olasoji.denloye@intel.com>
> > > wrote:
> > >
> > > > Hi Everyone,
> > > >
> > > > I would like to start a discussion on KIP-1121: Compression 
> > > > acceleration acceleration in Kafka. KIP-1121: Compression 
> > > > acceleration in Kafka - Apache Kafka - Apache Software 
> > > > Foundation< 
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Co
> > > > mp
> > > > re
> > > > ss
> > > > ion+acceleration+in+Kafka
> > > > >
> > > >
> > >
> >
>

RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

["Denloye, Olasoji" <ol...@intel.com>]

Hello all,

Regarding Compression acceleration, I would like to call a vote soon so if anyone wants to contribute to the discussion or bring up other concerns, you are welcome to comment.

Thanks,
Olasoji

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com> 
Sent: Thursday, May 29, 2025 10:39 PM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hello all, any more comments?

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Wednesday, May 7, 2025 10:55 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Greg,

2, 10 I figured a way to not expose any of these internal classes to the public API. I added a new interface in the place of BufferSupplier.
7. No, CompressionService#isAvailable returning false does not currently cause a ConfigException, it would cause a fall-back to the base implementation. I have changed this to checkAvailable which throws an exception if not available. If a more appropriate exception does not already exist, we could introduce "UnavailableServiceException"
9. I have updated the testing section to include additional scenarios relating to Kafka side testing. In short, we need to ensure the default behavior is unchanged and validate all new functions and configurations. 
11. the "accelerated GzipCompressor" mentioned is a reference design and need not be a part of Kafka. Is there a recommendation for hosting/distributing implementations of CompressionService?
12. Similar to the base implementation the goal is to have a separate instance of CompressionService for each I/O threads handling compression/decompression.
13. The existing implementations will remain unchanged. If the CompressionService configuration is empty then the base implementation would be used.
14. The broker configuration is currently in ServerConfigs.java and is configured statically in broker.properties (or server.properties). I have not considered dynamic configuration via kraft.

Thanks,
Olasoji

-----Original Message-----
From: Greg Harris <gr...@aiven.io.INVALID>
Sent: Wednesday, April 30, 2025 1:04 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Olasoji,

Thanks for updating the KIP.

2. I'm still not satisfied with the description of the public API in the KIP. It mentions these classes:
* org.apache.kafka.common.utils.ByteBufferOutputStream
* org.apache.kafka.common.utils.ByteBufferInputStream
* org.apache.kafka.common.utils.BufferSupplier
* org.apache.kafka.common.record.CompressionType
But not any description of these classes. Please include signatures of all new methods/variables/etc that are being exposed.

7. Does CompressionService#isAvailable returning false cause a ConfigException? Could the CompressionService provide a reason (via an
exception) of why it isn't available, instead of returning a boolean and requiring users to look through logs to debug the unavailability?
8. Thanks for clarifying the security model. This is just like any other Java application, where you need to trust the JARs that you add to your application. I don't think that the isolated classloader needs to be mentioned in the KIP, so thank you for removing that.
9. This point does not appear to have been addressed.
10. The classes mentioned above are in existing packages that are not exposed via Javadoc, and we have generally included whole packages in the API at once. I think these classes need to be moved to a different package, or removed from the API surface area.
11. The KIP states that it "also introduces an accelerated GzipCompressor".
Does this require Apache Kafka to depend on a specific accelerator implementation? The Apache Kafka project has traditionally been very averse to adding dependencies like this.
12. What sort of concurrency guarantees are expected from CompressionService implementations? Should each connection, thread, or JVM get its own instance of each CompressionService?
13. Will the existing implementations be converted to CompressionService classes, or will they be special fallbacks? Will the default for the configuration be "null" or some real class?
14. It looks like the new configurations are being added to the CommonClientConfigs to be included in both producer and consumer. Where is this configuration on the broker side? Is it a dynamic config (able to be reconfigured via KRaft) or a static config (only configurable by server.properties)?

Thanks,
Greg

On Wed, Apr 30, 2025 at 10:2 AM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hello all,
>
> I would like to call a vote soon so if anyone wants to contribute to 
> the discussion or bring up other concerns, you are welcome to comment.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Wednesday, April 23, 2025 10:22 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi All,
>
> This is a gentle reminder about open discussion on this compression 
> acceleration KIP. I would love to hear your thoughts.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Monday, April 7, 2025 8:49 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Greg,
>
> I have refactored the proposal to reduce the API surface area. I 
> introduced a new CompressionService interface which contains basic 
> compression/decompression functions. A service provider, when 
> supplied, would be wrapped in a concrete 'Compression' and run similar 
> to the existing compression codecs. The 'Compression' interface and 
> its subclasses have been decoupled from these changes.
>
> 6. based on your feedback, offloading would occur if and only if the 
> specific configuration is set, even if multiple service providers are 
> available 8. The threat model is that dynamically loading any 
> untrusted or untested jar on the classpath can be dangerous. The 
> purpose of bringing this up is to discuss how service provider
> (plugins) is distributed and installed. On one hand, a centralized 
> Kafka repository and on the other a decentralized approach where 
> plugins are developed and installed on the fly. Which is preferable?
> The choice would determine what security mitigations are needed if any.
>
> https://cwiki.apache.org/confluence/x/Xgr0Ew
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Monday, February 24, 2025 12:42 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Olasoji,
>
> Thanks for the update.
>
> 1. Could you add these as code examples to the KIP, and not as prose? 
> See [1-3] for some examples of well-written public interfaces.
>
> 2. I find the proposed API surface area excessive, and think we need 
> to narrow it down. Looking at what's merged on trunk right now and not 
> including your proposed modifications, this KIP includes:
> * 2 interfaces
> * 12 classes
> * 5 constructors
> * 8 static methods
> * 2 static variables
> * 18 unique instance methods
> Committing to long-term stability of this many classes and interfaces 
> as public API will make it hard to initially review it, and innovate 
> on it in the future.
>
> 4. Is there a way to modify the existing signatures/implementations 
> such that we don't have to expose any subclasses of Compression as 
> public API, only the CompressionTypes?
> 6. What does "first provider loaded" mean to users? AFAIU, this looks 
> arbitrary to users, as it depends on classpath ordering, filesystem 
> ordering, etc.
> 8. I'm trying to understand under what conditions this can be attacked 
> (threat model), and what mitigations Kafka needs to provide 
> (implementation). Is Kafka providing special filtering? Is Kafka 
> providing the custom class loader?
> 6, 7, 8. Please add these details to the KIP so that other reviewers 
> don't need to follow the discussion thread so closely.
> 9. The testing section looks like it is written for individuals 
> implementing accelerator plugins. Could you also write about how the 
> Kafka side of the implementation will be tested?
>
> Thanks,
> Greg
>
> [1]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Onc
> e+Delivery+and+Transactional+Messaging
> [2]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tie
> red+Storage
> [3]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism
> +for+plugins+and+connectors+to+register+metrics
>
> On Fri, Feb 21, 2025 at 12:15 AM Denloye, Olasoji < 
> olasoji.denloye@intel.com>
> wrote:
>
> > Hi Greg, Thanks for your response.
> >
> > 1. Some changes will be made to the Compression and Builder 
> > interfaces. At least one default method, isAvailable(), would be 
> > added to the Builder interface. The static methods in the 
> > Compression interface that create and return specific builders would 
> > be modified to return a new builder or a loaded builder from a 
> > service provider if it exists. The return types may need to be more 
> > generic e.g Builder<GzipCompression> vs GzipCompression.Builder.
> > 2. These classes are appropriate to add to the API surface as they 
> > may be required to develop efficient compression implementations. I 
> > will modify the KIP.
> > 4. This is because the Compression interface (currently) expects a 
> > Builder of type 'GzipCompression.Builder'. I agree that alternative 
> > implementations need not extend the base implementation with some 
> > modifications to the Compression interface. However if an 
> > alternative implementation is registered as a service provider of 
> > GzipCompression then it must be of type 'CompressionType.GZIP' and 
> > it must be fully compatible with the base implementation of GzipCompression.
> > 5. So far no, but 'Configurable' looks promising 6. If no specific 
> > configuration is set, then the first provider loaded and available 
> > will be used. If a specific configuration is set then that specific 
> > provider will be used if available else the base implementation.
> > 7. Yes, a ConfigException will be thrown if the class specified in 
> > the config is not found on the classpath.
> > 8. This can be implemented by using special filtering or a custom 
> > class loader. The concern here is that any class on the class path 
> > (trusted or
> > not) could be loaded which could intercept messages or corrupt 
> > messages by compressing using an unknown algorithm.
> >
> > -----Original Message-----
> > From: Greg Harris <gr...@aiven.io.INVALID>
> > Sent: Tuesday, February 18, 2025 1:58 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Olasoji,
> >
> > Thanks for the KIP! I find it addresses many of the concerns I had 
> > with KIP-984. Thank you for following up on that!
> >
> > 1. Since the Compression interface is internal currently, it may 
> > contain some components that were not designed with pluggability in 
> > mind, such as the static methods. Are these interfaces being 
> > published as-is, or are some changes going to be made first?
> > 2. Similarly, the ByteBufferOutputStream, CompressionType, 
> > BufferSupplier, ByteBufferInputStream, ChunkedByteStream, and 
> > GzipOutputStream classes in the signature or implementations of 
> > Compression are not public API currently, and would need to be made 
> > public along with Compression. Are these classes appropriate to add 
> > to
> our API surface?
> > 3. Could you adjust the KIP structure/formatting to make the public 
> > API surface area more clear? Some prose in "Proposed Changes" could 
> > become code examples in "Public Interfaces".
> > 4. What is the significance of this statement: "Any new service 
> > provider should extend the base implementation and thus return the 
> > same CompressionType". Are we expecting that `o instanceof 
> > GzipCompression == o.type().equals(CompressionType.GZIP)` at all 
> > times? If an alternative implementation doesn't need any part of the 
> > upstream implementation, does it make sense to force them to extend 
> > one
> another?
> > 5. Is this interface expected to be composed with any other plugin 
> > mix-ins, such as Configurable, Reconfigurable, Closeable, 
> > Monitorable, Versioned, etc?
> > 6. If there are multiple accelerators present, which one will be chosen?
> > 7. If the configuration is set but the accelerator implementation 
> > isn't available, does that cause a ConfigException?
> > 8. Could you expand on this statement: "Due to security and/or 
> > compatibility concerns it is possible to limit this feature to only 
> > load approved service providers (e.g those distributed with Kafka)".
> > How will this be implemented? What threat model does this assume?
> >
> > Thanks,
> > Greg
> >
> > On Tue, Feb 18, 2025 at 11:41 AM Denloye, Olasoji < 
> > olasoji.denloye@intel.com>
> > wrote:
> >
> > > Hi, I would like to bring this back to everyone's attention. This 
> > > KIP introduces a simple framework that enables the use of 
> > > alternative, faster implementations of the existing compression 
> > > codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji <ol...@intel.com>
> > > Sent: Thursday, February 13, 2025 1:22 PM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi, I would like to bring this back to everyone's attention. I 
> > > have made changes to the proposal to incorporate the ServiceLoader 
> > > API as
> > suggested.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji
> > > Sent: Thursday, December 19, 2024 11:01 AM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Frédérik, Thanks for the suggestion to use the existing 
> > > ServiceLoader feature in Java. It could serve as suitable 
> > > mechanism for selecting the best available compression codec.
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Frédérik Rouleau <fr...@confluent.io.INVALID>
> > > Sent: Wednesday, December 18, 2024 12:24 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Olasoji,
> > >
> > > I think that having a way to provide some optimised/accelerated 
> > > implementation is a good idea.
> > > What not using the ServiceLoader feature of Java? That would allow 
> > > anyone to provide their own optimised implementation depending on 
> > > their infrastructure. We can keep the existing implementation as 
> > > the default and allow Kafka to find a better implementation based 
> > > on Service found. As you mentioned, we might need to enrich the 
> > > interface to allow Kafka to find the best version available at runtime.
> > >
> > > Regards,
> > >
> > > On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji < 
> > > olasoji.denloye@intel.com>
> > > wrote:
> > >
> > > > Hi Everyone,
> > > >
> > > > I would like to start a discussion on KIP-1121: Compression 
> > > > acceleration acceleration in Kafka. KIP-1121: Compression 
> > > > acceleration in Kafka - Apache Kafka - Apache Software 
> > > > Foundation< 
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Co
> > > > mp
> > > > re
> > > > ss
> > > > ion+acceleration+in+Kafka
> > > > >
> > > >
> > >
> >
>

RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

["Denloye, Olasoji" <ol...@intel.com>]

Hello all, any more comments?

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com> 
Sent: Wednesday, May 7, 2025 10:55 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Greg,

2, 10 I figured a way to not expose any of these internal classes to the public API. I added a new interface in the place of BufferSupplier.
7. No, CompressionService#isAvailable returning false does not currently cause a ConfigException, it would cause a fall-back to the base implementation. I have changed this to checkAvailable which throws an exception if not available. If a more appropriate exception does not already exist, we could introduce "UnavailableServiceException"
9. I have updated the testing section to include additional scenarios relating to Kafka side testing. In short, we need to ensure the default behavior is unchanged and validate all new functions and configurations. 
11. the "accelerated GzipCompressor" mentioned is a reference design and need not be a part of Kafka. Is there a recommendation for hosting/distributing implementations of CompressionService?
12. Similar to the base implementation the goal is to have a separate instance of CompressionService for each I/O threads handling compression/decompression.
13. The existing implementations will remain unchanged. If the CompressionService configuration is empty then the base implementation would be used.
14. The broker configuration is currently in ServerConfigs.java and is configured statically in broker.properties (or server.properties). I have not considered dynamic configuration via kraft.

Thanks,
Olasoji

-----Original Message-----
From: Greg Harris <gr...@aiven.io.INVALID>
Sent: Wednesday, April 30, 2025 1:04 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Olasoji,

Thanks for updating the KIP.

2. I'm still not satisfied with the description of the public API in the KIP. It mentions these classes:
* org.apache.kafka.common.utils.ByteBufferOutputStream
* org.apache.kafka.common.utils.ByteBufferInputStream
* org.apache.kafka.common.utils.BufferSupplier
* org.apache.kafka.common.record.CompressionType
But not any description of these classes. Please include signatures of all new methods/variables/etc that are being exposed.

7. Does CompressionService#isAvailable returning false cause a ConfigException? Could the CompressionService provide a reason (via an
exception) of why it isn't available, instead of returning a boolean and requiring users to look through logs to debug the unavailability?
8. Thanks for clarifying the security model. This is just like any other Java application, where you need to trust the JARs that you add to your application. I don't think that the isolated classloader needs to be mentioned in the KIP, so thank you for removing that.
9. This point does not appear to have been addressed.
10. The classes mentioned above are in existing packages that are not exposed via Javadoc, and we have generally included whole packages in the API at once. I think these classes need to be moved to a different package, or removed from the API surface area.
11. The KIP states that it "also introduces an accelerated GzipCompressor".
Does this require Apache Kafka to depend on a specific accelerator implementation? The Apache Kafka project has traditionally been very averse to adding dependencies like this.
12. What sort of concurrency guarantees are expected from CompressionService implementations? Should each connection, thread, or JVM get its own instance of each CompressionService?
13. Will the existing implementations be converted to CompressionService classes, or will they be special fallbacks? Will the default for the configuration be "null" or some real class?
14. It looks like the new configurations are being added to the CommonClientConfigs to be included in both producer and consumer. Where is this configuration on the broker side? Is it a dynamic config (able to be reconfigured via KRaft) or a static config (only configurable by server.properties)?

Thanks,
Greg

On Wed, Apr 30, 2025 at 10:24 AM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hello all,
>
> I would like to call a vote soon so if anyone wants to contribute to 
> the discussion or bring up other concerns, you are welcome to comment.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Wednesday, April 23, 2025 10:22 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi All,
>
> This is a gentle reminder about open discussion on this compression 
> acceleration KIP. I would love to hear your thoughts.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Monday, April 7, 2025 8:49 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Greg,
>
> I have refactored the proposal to reduce the API surface area. I 
> introduced a new CompressionService interface which contains basic 
> compression/decompression functions. A service provider, when 
> supplied, would be wrapped in a concrete 'Compression' and run similar 
> to the existing compression codecs. The 'Compression' interface and 
> its subclasses have been decoupled from these changes.
>
> 6. based on your feedback, offloading would occur if and only if the 
> specific configuration is set, even if multiple service providers are 
> available 8. The threat model is that dynamically loading any 
> untrusted or untested jar on the classpath can be dangerous. The 
> purpose of bringing this up is to discuss how service provider
> (plugins) is distributed and installed. On one hand, a centralized 
> Kafka repository and on the other a decentralized approach where 
> plugins are developed and installed on the fly. Which is preferable?
> The choice would determine what security mitigations are needed if any.
>
> https://cwiki.apache.org/confluence/x/Xgr0Ew
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Monday, February 24, 2025 12:42 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Olasoji,
>
> Thanks for the update.
>
> 1. Could you add these as code examples to the KIP, and not as prose? 
> See [1-3] for some examples of well-written public interfaces.
>
> 2. I find the proposed API surface area excessive, and think we need 
> to narrow it down. Looking at what's merged on trunk right now and not 
> including your proposed modifications, this KIP includes:
> * 2 interfaces
> * 12 classes
> * 5 constructors
> * 8 static methods
> * 2 static variables
> * 18 unique instance methods
> Committing to long-term stability of this many classes and interfaces 
> as public API will make it hard to initially review it, and innovate 
> on it in the future.
>
> 4. Is there a way to modify the existing signatures/implementations 
> such that we don't have to expose any subclasses of Compression as 
> public API, only the CompressionTypes?
> 6. What does "first provider loaded" mean to users? AFAIU, this looks 
> arbitrary to users, as it depends on classpath ordering, filesystem 
> ordering, etc.
> 8. I'm trying to understand under what conditions this can be attacked 
> (threat model), and what mitigations Kafka needs to provide 
> (implementation). Is Kafka providing special filtering? Is Kafka 
> providing the custom class loader?
> 6, 7, 8. Please add these details to the KIP so that other reviewers 
> don't need to follow the discussion thread so closely.
> 9. The testing section looks like it is written for individuals 
> implementing accelerator plugins. Could you also write about how the 
> Kafka side of the implementation will be tested?
>
> Thanks,
> Greg
>
> [1]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Onc
> e+Delivery+and+Transactional+Messaging
> [2]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tie
> red+Storage
> [3]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism
> +for+plugins+and+connectors+to+register+metrics
>
> On Fri, Feb 21, 2025 at 12:15 AM Denloye, Olasoji < 
> olasoji.denloye@intel.com>
> wrote:
>
> > Hi Greg, Thanks for your response.
> >
> > 1. Some changes will be made to the Compression and Builder 
> > interfaces. At least one default method, isAvailable(), would be 
> > added to the Builder interface. The static methods in the 
> > Compression interface that create and return specific builders would 
> > be modified to return a new builder or a loaded builder from a 
> > service provider if it exists. The return types may need to be more 
> > generic e.g Builder<GzipCompression> vs GzipCompression.Builder.
> > 2. These classes are appropriate to add to the API surface as they 
> > may be required to develop efficient compression implementations. I 
> > will modify the KIP.
> > 4. This is because the Compression interface (currently) expects a 
> > Builder of type 'GzipCompression.Builder'. I agree that alternative 
> > implementations need not extend the base implementation with some 
> > modifications to the Compression interface. However if an 
> > alternative implementation is registered as a service provider of 
> > GzipCompression then it must be of type 'CompressionType.GZIP' and 
> > it must be fully compatible with the base implementation of GzipCompression.
> > 5. So far no, but 'Configurable' looks promising 6. If no specific 
> > configuration is set, then the first provider loaded and available 
> > will be used. If a specific configuration is set then that specific 
> > provider will be used if available else the base implementation.
> > 7. Yes, a ConfigException will be thrown if the class specified in 
> > the config is not found on the classpath.
> > 8. This can be implemented by using special filtering or a custom 
> > class loader. The concern here is that any class on the class path 
> > (trusted or
> > not) could be loaded which could intercept messages or corrupt 
> > messages by compressing using an unknown algorithm.
> >
> > -----Original Message-----
> > From: Greg Harris <gr...@aiven.io.INVALID>
> > Sent: Tuesday, February 18, 2025 1:58 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Olasoji,
> >
> > Thanks for the KIP! I find it addresses many of the concerns I had 
> > with KIP-984. Thank you for following up on that!
> >
> > 1. Since the Compression interface is internal currently, it may 
> > contain some components that were not designed with pluggability in 
> > mind, such as the static methods. Are these interfaces being 
> > published as-is, or are some changes going to be made first?
> > 2. Similarly, the ByteBufferOutputStream, CompressionType, 
> > BufferSupplier, ByteBufferInputStream, ChunkedByteStream, and 
> > GzipOutputStream classes in the signature or implementations of 
> > Compression are not public API currently, and would need to be made 
> > public along with Compression. Are these classes appropriate to add 
> > to
> our API surface?
> > 3. Could you adjust the KIP structure/formatting to make the public 
> > API surface area more clear? Some prose in "Proposed Changes" could 
> > become code examples in "Public Interfaces".
> > 4. What is the significance of this statement: "Any new service 
> > provider should extend the base implementation and thus return the 
> > same CompressionType". Are we expecting that `o instanceof 
> > GzipCompression == o.type().equals(CompressionType.GZIP)` at all 
> > times? If an alternative implementation doesn't need any part of the 
> > upstream implementation, does it make sense to force them to extend 
> > one
> another?
> > 5. Is this interface expected to be composed with any other plugin 
> > mix-ins, such as Configurable, Reconfigurable, Closeable, 
> > Monitorable, Versioned, etc?
> > 6. If there are multiple accelerators present, which one will be chosen?
> > 7. If the configuration is set but the accelerator implementation 
> > isn't available, does that cause a ConfigException?
> > 8. Could you expand on this statement: "Due to security and/or 
> > compatibility concerns it is possible to limit this feature to only 
> > load approved service providers (e.g those distributed with Kafka)".
> > How will this be implemented? What threat model does this assume?
> >
> > Thanks,
> > Greg
> >
> > On Tue, Feb 18, 2025 at 11:41 AM Denloye, Olasoji < 
> > olasoji.denloye@intel.com>
> > wrote:
> >
> > > Hi, I would like to bring this back to everyone's attention. This 
> > > KIP introduces a simple framework that enables the use of 
> > > alternative, faster implementations of the existing compression 
> > > codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji <ol...@intel.com>
> > > Sent: Thursday, February 13, 2025 1:22 PM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi, I would like to bring this back to everyone's attention. I 
> > > have made changes to the proposal to incorporate the ServiceLoader 
> > > API as
> > suggested.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji
> > > Sent: Thursday, December 19, 2024 11:01 AM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Frédérik, Thanks for the suggestion to use the existing 
> > > ServiceLoader feature in Java. It could serve as suitable 
> > > mechanism for selecting the best available compression codec.
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Frédérik Rouleau <fr...@confluent.io.INVALID>
> > > Sent: Wednesday, December 18, 2024 12:24 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Olasoji,
> > >
> > > I think that having a way to provide some optimised/accelerated 
> > > implementation is a good idea.
> > > What not using the ServiceLoader feature of Java? That would allow 
> > > anyone to provide their own optimised implementation depending on 
> > > their infrastructure. We can keep the existing implementation as 
> > > the default and allow Kafka to find a better implementation based 
> > > on Service found. As you mentioned, we might need to enrich the 
> > > interface to allow Kafka to find the best version available at runtime.
> > >
> > > Regards,
> > >
> > > On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji < 
> > > olasoji.denloye@intel.com>
> > > wrote:
> > >
> > > > Hi Everyone,
> > > >
> > > > I would like to start a discussion on KIP-1121: Compression 
> > > > acceleration acceleration in Kafka. KIP-1121: Compression 
> > > > acceleration in Kafka - Apache Kafka - Apache Software 
> > > > Foundation< 
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Co
> > > > mp
> > > > re
> > > > ss
> > > > ion+acceleration+in+Kafka
> > > > >
> > > >
> > >
> >
>

RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

["Denloye, Olasoji" <ol...@intel.com>]

Hi Greg,

2, 10 I figured a way to not expose any of these internal classes to the public API. I added a new interface in the place of BufferSupplier.
7. No, CompressionService#isAvailable returning false does not currently cause a ConfigException, it would cause a fall-back to the base implementation. I have changed this to checkAvailable which throws an exception if not available. If a more appropriate exception does not already exist, we could introduce "UnavailableServiceException"
9. I have updated the testing section to include additional scenarios relating to Kafka side testing. In short, we need to ensure the default behavior is unchanged and validate all new functions and configurations. 
11. the "accelerated GzipCompressor" mentioned is a reference design and need not be a part of Kafka. Is there a recommendation for hosting/distributing implementations of CompressionService?
12. Similar to the base implementation the goal is to have a separate instance of CompressionService for each I/O threads handling compression/decompression.
13. The existing implementations will remain unchanged. If the CompressionService configuration is empty then the base implementation would be used.
14. The broker configuration is currently in ServerConfigs.java and is configured statically in broker.properties (or server.properties). I have not considered dynamic configuration via kraft.

Thanks,
Olasoji

-----Original Message-----
From: Greg Harris <gr...@aiven.io.INVALID> 
Sent: Wednesday, April 30, 2025 1:04 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Olasoji,

Thanks for updating the KIP.

2. I'm still not satisfied with the description of the public API in the KIP. It mentions these classes:
* org.apache.kafka.common.utils.ByteBufferOutputStream
* org.apache.kafka.common.utils.ByteBufferInputStream
* org.apache.kafka.common.utils.BufferSupplier
* org.apache.kafka.common.record.CompressionType
But not any description of these classes. Please include signatures of all new methods/variables/etc that are being exposed.

7. Does CompressionService#isAvailable returning false cause a ConfigException? Could the CompressionService provide a reason (via an
exception) of why it isn't available, instead of returning a boolean and requiring users to look through logs to debug the unavailability?
8. Thanks for clarifying the security model. This is just like any other Java application, where you need to trust the JARs that you add to your application. I don't think that the isolated classloader needs to be mentioned in the KIP, so thank you for removing that.
9. This point does not appear to have been addressed.
10. The classes mentioned above are in existing packages that are not exposed via Javadoc, and we have generally included whole packages in the API at once. I think these classes need to be moved to a different package, or removed from the API surface area.
11. The KIP states that it "also introduces an accelerated GzipCompressor".
Does this require Apache Kafka to depend on a specific accelerator implementation? The Apache Kafka project has traditionally been very averse to adding dependencies like this.
12. What sort of concurrency guarantees are expected from CompressionService implementations? Should each connection, thread, or JVM get its own instance of each CompressionService?
13. Will the existing implementations be converted to CompressionService classes, or will they be special fallbacks? Will the default for the configuration be "null" or some real class?
14. It looks like the new configurations are being added to the CommonClientConfigs to be included in both producer and consumer. Where is this configuration on the broker side? Is it a dynamic config (able to be reconfigured via KRaft) or a static config (only configurable by server.properties)?

Thanks,
Greg

On Wed, Apr 30, 2025 at 10:24 AM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hello all,
>
> I would like to call a vote soon so if anyone wants to contribute to 
> the discussion or bring up other concerns, you are welcome to comment.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Wednesday, April 23, 2025 10:22 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi All,
>
> This is a gentle reminder about open discussion on this compression 
> acceleration KIP. I would love to hear your thoughts.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Monday, April 7, 2025 8:49 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Greg,
>
> I have refactored the proposal to reduce the API surface area. I 
> introduced a new CompressionService interface which contains basic 
> compression/decompression functions. A service provider, when 
> supplied, would be wrapped in a concrete 'Compression' and run similar 
> to the existing compression codecs. The 'Compression' interface and 
> its subclasses have been decoupled from these changes.
>
> 6. based on your feedback, offloading would occur if and only if the 
> specific configuration is set, even if multiple service providers are 
> available 8. The threat model is that dynamically loading any 
> untrusted or untested jar on the classpath can be dangerous. The 
> purpose of bringing this up is to discuss how service provider 
> (plugins) is distributed and installed. On one hand, a centralized 
> Kafka repository and on the other a decentralized approach where 
> plugins are developed and installed on the fly. Which is preferable? 
> The choice would determine what security mitigations are needed if any.
>
> https://cwiki.apache.org/confluence/x/Xgr0Ew
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Monday, February 24, 2025 12:42 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Olasoji,
>
> Thanks for the update.
>
> 1. Could you add these as code examples to the KIP, and not as prose? 
> See [1-3] for some examples of well-written public interfaces.
>
> 2. I find the proposed API surface area excessive, and think we need 
> to narrow it down. Looking at what's merged on trunk right now and not 
> including your proposed modifications, this KIP includes:
> * 2 interfaces
> * 12 classes
> * 5 constructors
> * 8 static methods
> * 2 static variables
> * 18 unique instance methods
> Committing to long-term stability of this many classes and interfaces 
> as public API will make it hard to initially review it, and innovate 
> on it in the future.
>
> 4. Is there a way to modify the existing signatures/implementations 
> such that we don't have to expose any subclasses of Compression as 
> public API, only the CompressionTypes?
> 6. What does "first provider loaded" mean to users? AFAIU, this looks 
> arbitrary to users, as it depends on classpath ordering, filesystem 
> ordering, etc.
> 8. I'm trying to understand under what conditions this can be attacked 
> (threat model), and what mitigations Kafka needs to provide 
> (implementation). Is Kafka providing special filtering? Is Kafka 
> providing the custom class loader?
> 6, 7, 8. Please add these details to the KIP so that other reviewers 
> don't need to follow the discussion thread so closely.
> 9. The testing section looks like it is written for individuals 
> implementing accelerator plugins. Could you also write about how the 
> Kafka side of the implementation will be tested?
>
> Thanks,
> Greg
>
> [1]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Onc
> e+Delivery+and+Transactional+Messaging
> [2]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tie
> red+Storage
> [3]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism
> +for+plugins+and+connectors+to+register+metrics
>
> On Fri, Feb 21, 2025 at 12:15 AM Denloye, Olasoji < 
> olasoji.denloye@intel.com>
> wrote:
>
> > Hi Greg, Thanks for your response.
> >
> > 1. Some changes will be made to the Compression and Builder 
> > interfaces. At least one default method, isAvailable(), would be 
> > added to the Builder interface. The static methods in the 
> > Compression interface that create and return specific builders would 
> > be modified to return a new builder or a loaded builder from a 
> > service provider if it exists. The return types may need to be more 
> > generic e.g Builder<GzipCompression> vs GzipCompression.Builder.
> > 2. These classes are appropriate to add to the API surface as they 
> > may be required to develop efficient compression implementations. I 
> > will modify the KIP.
> > 4. This is because the Compression interface (currently) expects a 
> > Builder of type 'GzipCompression.Builder'. I agree that alternative 
> > implementations need not extend the base implementation with some 
> > modifications to the Compression interface. However if an 
> > alternative implementation is registered as a service provider of 
> > GzipCompression then it must be of type 'CompressionType.GZIP' and 
> > it must be fully compatible with the base implementation of GzipCompression.
> > 5. So far no, but 'Configurable' looks promising 6. If no specific 
> > configuration is set, then the first provider loaded and available 
> > will be used. If a specific configuration is set then that specific 
> > provider will be used if available else the base implementation.
> > 7. Yes, a ConfigException will be thrown if the class specified in 
> > the config is not found on the classpath.
> > 8. This can be implemented by using special filtering or a custom 
> > class loader. The concern here is that any class on the class path 
> > (trusted or
> > not) could be loaded which could intercept messages or corrupt 
> > messages by compressing using an unknown algorithm.
> >
> > -----Original Message-----
> > From: Greg Harris <gr...@aiven.io.INVALID>
> > Sent: Tuesday, February 18, 2025 1:58 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Olasoji,
> >
> > Thanks for the KIP! I find it addresses many of the concerns I had 
> > with KIP-984. Thank you for following up on that!
> >
> > 1. Since the Compression interface is internal currently, it may 
> > contain some components that were not designed with pluggability in 
> > mind, such as the static methods. Are these interfaces being 
> > published as-is, or are some changes going to be made first?
> > 2. Similarly, the ByteBufferOutputStream, CompressionType, 
> > BufferSupplier, ByteBufferInputStream, ChunkedByteStream, and 
> > GzipOutputStream classes in the signature or implementations of 
> > Compression are not public API currently, and would need to be made 
> > public along with Compression. Are these classes appropriate to add 
> > to
> our API surface?
> > 3. Could you adjust the KIP structure/formatting to make the public 
> > API surface area more clear? Some prose in "Proposed Changes" could 
> > become code examples in "Public Interfaces".
> > 4. What is the significance of this statement: "Any new service 
> > provider should extend the base implementation and thus return the 
> > same CompressionType". Are we expecting that `o instanceof 
> > GzipCompression == o.type().equals(CompressionType.GZIP)` at all 
> > times? If an alternative implementation doesn't need any part of the 
> > upstream implementation, does it make sense to force them to extend 
> > one
> another?
> > 5. Is this interface expected to be composed with any other plugin 
> > mix-ins, such as Configurable, Reconfigurable, Closeable, 
> > Monitorable, Versioned, etc?
> > 6. If there are multiple accelerators present, which one will be chosen?
> > 7. If the configuration is set but the accelerator implementation 
> > isn't available, does that cause a ConfigException?
> > 8. Could you expand on this statement: "Due to security and/or 
> > compatibility concerns it is possible to limit this feature to only 
> > load approved service providers (e.g those distributed with Kafka)".
> > How will this be implemented? What threat model does this assume?
> >
> > Thanks,
> > Greg
> >
> > On Tue, Feb 18, 2025 at 11:41 AM Denloye, Olasoji < 
> > olasoji.denloye@intel.com>
> > wrote:
> >
> > > Hi, I would like to bring this back to everyone's attention. This 
> > > KIP introduces a simple framework that enables the use of 
> > > alternative, faster implementations of the existing compression 
> > > codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji <ol...@intel.com>
> > > Sent: Thursday, February 13, 2025 1:22 PM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi, I would like to bring this back to everyone's attention. I 
> > > have made changes to the proposal to incorporate the ServiceLoader 
> > > API as
> > suggested.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji
> > > Sent: Thursday, December 19, 2024 11:01 AM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Frédérik, Thanks for the suggestion to use the existing 
> > > ServiceLoader feature in Java. It could serve as suitable 
> > > mechanism for selecting the best available compression codec.
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Frédérik Rouleau <fr...@confluent.io.INVALID>
> > > Sent: Wednesday, December 18, 2024 12:24 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Olasoji,
> > >
> > > I think that having a way to provide some optimised/accelerated 
> > > implementation is a good idea.
> > > What not using the ServiceLoader feature of Java? That would allow 
> > > anyone to provide their own optimised implementation depending on 
> > > their infrastructure. We can keep the existing implementation as 
> > > the default and allow Kafka to find a better implementation based 
> > > on Service found. As you mentioned, we might need to enrich the 
> > > interface to allow Kafka to find the best version available at runtime.
> > >
> > > Regards,
> > >
> > > On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji < 
> > > olasoji.denloye@intel.com>
> > > wrote:
> > >
> > > > Hi Everyone,
> > > >
> > > > I would like to start a discussion on KIP-1121: Compression 
> > > > acceleration acceleration in Kafka. KIP-1121: Compression 
> > > > acceleration in Kafka - Apache Kafka - Apache Software 
> > > > Foundation< 
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Co
> > > > mp
> > > > re
> > > > ss
> > > > ion+acceleration+in+Kafka
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

[Greg Harris <gr...@aiven.io.INVALID>]

Hi Olasoji,

Thanks for updating the KIP.

2. I'm still not satisfied with the description of the public API in the
KIP. It mentions these classes:
* org.apache.kafka.common.utils.ByteBufferOutputStream
* org.apache.kafka.common.utils.ByteBufferInputStream
* org.apache.kafka.common.utils.BufferSupplier
* org.apache.kafka.common.record.CompressionType
But not any description of these classes. Please include signatures of all
new methods/variables/etc that are being exposed.

7. Does CompressionService#isAvailable returning false cause a
ConfigException? Could the CompressionService provide a reason (via an
exception) of why it isn't available, instead of returning a boolean and
requiring users to look through logs to debug the unavailability?
8. Thanks for clarifying the security model. This is just like any other
Java application, where you need to trust the JARs that you add to your
application. I don't think that the isolated classloader needs to be
mentioned in the KIP, so thank you for removing that.
9. This point does not appear to have been addressed.
10. The classes mentioned above are in existing packages that are not
exposed via Javadoc, and we have generally included whole packages in the
API at once. I think these classes need to be moved to a different package,
or removed from the API surface area.
11. The KIP states that it "also introduces an accelerated GzipCompressor".
Does this require Apache Kafka to depend on a specific accelerator
implementation? The Apache Kafka project has traditionally been very averse
to adding dependencies like this.
12. What sort of concurrency guarantees are expected from
CompressionService implementations? Should each connection, thread, or JVM
get its own instance of each CompressionService?
13. Will the existing implementations be converted to CompressionService
classes, or will they be special fallbacks? Will the default for the
configuration be "null" or some real class?
14. It looks like the new configurations are being added to the
CommonClientConfigs to be included in both producer and consumer. Where is
this configuration on the broker side? Is it a dynamic config (able to be
reconfigured via KRaft) or a static config (only configurable by
server.properties)?

Thanks,
Greg

On Wed, Apr 30, 2025 at 10:24 AM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hello all,
>
> I would like to call a vote soon so if anyone wants to contribute to the
> discussion or bring up other concerns, you are welcome to comment.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Wednesday, April 23, 2025 10:22 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi All,
>
> This is a gentle reminder about open discussion on this compression
> acceleration KIP. I would love to hear your thoughts.
>
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Monday, April 7, 2025 8:49 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Greg,
>
> I have refactored the proposal to reduce the API surface area. I
> introduced a new CompressionService interface which contains basic
> compression/decompression functions. A service provider, when supplied,
> would be wrapped in a concrete 'Compression' and run similar to the
> existing compression codecs. The 'Compression' interface and its subclasses
> have been decoupled from these changes.
>
> 6. based on your feedback, offloading would occur if and only if the
> specific configuration is set, even if multiple service providers are
> available 8. The threat model is that dynamically loading any untrusted or
> untested jar on the classpath can be dangerous. The purpose of bringing
> this up is to discuss how service provider (plugins) is distributed and
> installed. On one hand, a centralized Kafka repository and on the other a
> decentralized approach where plugins are developed and installed on the
> fly. Which is preferable? The choice would determine what security
> mitigations are needed if any.
>
> https://cwiki.apache.org/confluence/x/Xgr0Ew
> Thanks,
> Olasoji
>
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Monday, February 24, 2025 12:42 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Olasoji,
>
> Thanks for the update.
>
> 1. Could you add these as code examples to the KIP, and not as prose? See
> [1-3] for some examples of well-written public interfaces.
>
> 2. I find the proposed API surface area excessive, and think we need to
> narrow it down. Looking at what's merged on trunk right now and not
> including your proposed modifications, this KIP includes:
> * 2 interfaces
> * 12 classes
> * 5 constructors
> * 8 static methods
> * 2 static variables
> * 18 unique instance methods
> Committing to long-term stability of this many classes and interfaces as
> public API will make it hard to initially review it, and innovate on it in
> the future.
>
> 4. Is there a way to modify the existing signatures/implementations such
> that we don't have to expose any subclasses of Compression as public API,
> only the CompressionTypes?
> 6. What does "first provider loaded" mean to users? AFAIU, this looks
> arbitrary to users, as it depends on classpath ordering, filesystem
> ordering, etc.
> 8. I'm trying to understand under what conditions this can be attacked
> (threat model), and what mitigations Kafka needs to provide
> (implementation). Is Kafka providing special filtering? Is Kafka providing
> the custom class loader?
> 6, 7, 8. Please add these details to the KIP so that other reviewers don't
> need to follow the discussion thread so closely.
> 9. The testing section looks like it is written for individuals
> implementing accelerator plugins. Could you also write about how the Kafka
> side of the implementation will be tested?
>
> Thanks,
> Greg
>
> [1]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging
> [2]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage
> [3]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism+for+plugins+and+connectors+to+register+metrics
>
> On Fri, Feb 21, 2025 at 12:15 AM Denloye, Olasoji <
> olasoji.denloye@intel.com>
> wrote:
>
> > Hi Greg, Thanks for your response.
> >
> > 1. Some changes will be made to the Compression and Builder
> > interfaces. At least one default method, isAvailable(), would be added
> > to the Builder interface. The static methods in the Compression
> > interface that create and return specific builders would be modified
> > to return a new builder or a loaded builder from a service provider if
> > it exists. The return types may need to be more generic e.g
> > Builder<GzipCompression> vs GzipCompression.Builder.
> > 2. These classes are appropriate to add to the API surface as they may
> > be required to develop efficient compression implementations. I will
> > modify the KIP.
> > 4. This is because the Compression interface (currently) expects a
> > Builder of type 'GzipCompression.Builder'. I agree that alternative
> > implementations need not extend the base implementation with some
> > modifications to the Compression interface. However if an alternative
> > implementation is registered as a service provider of GzipCompression
> > then it must be of type 'CompressionType.GZIP' and it must be fully
> > compatible with the base implementation of GzipCompression.
> > 5. So far no, but 'Configurable' looks promising 6. If no specific
> > configuration is set, then the first provider loaded and available
> > will be used. If a specific configuration is set then that specific
> > provider will be used if available else the base implementation.
> > 7. Yes, a ConfigException will be thrown if the class specified in the
> > config is not found on the classpath.
> > 8. This can be implemented by using special filtering or a custom
> > class loader. The concern here is that any class on the class path
> > (trusted or
> > not) could be loaded which could intercept messages or corrupt
> > messages by compressing using an unknown algorithm.
> >
> > -----Original Message-----
> > From: Greg Harris <gr...@aiven.io.INVALID>
> > Sent: Tuesday, February 18, 2025 1:58 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Olasoji,
> >
> > Thanks for the KIP! I find it addresses many of the concerns I had
> > with KIP-984. Thank you for following up on that!
> >
> > 1. Since the Compression interface is internal currently, it may
> > contain some components that were not designed with pluggability in
> > mind, such as the static methods. Are these interfaces being published
> > as-is, or are some changes going to be made first?
> > 2. Similarly, the ByteBufferOutputStream, CompressionType,
> > BufferSupplier, ByteBufferInputStream, ChunkedByteStream, and
> > GzipOutputStream classes in the signature or implementations of
> > Compression are not public API currently, and would need to be made
> > public along with Compression. Are these classes appropriate to add to
> our API surface?
> > 3. Could you adjust the KIP structure/formatting to make the public
> > API surface area more clear? Some prose in "Proposed Changes" could
> > become code examples in "Public Interfaces".
> > 4. What is the significance of this statement: "Any new service
> > provider should extend the base implementation and thus return the
> > same CompressionType". Are we expecting that `o instanceof
> > GzipCompression == o.type().equals(CompressionType.GZIP)` at all
> > times? If an alternative implementation doesn't need any part of the
> > upstream implementation, does it make sense to force them to extend one
> another?
> > 5. Is this interface expected to be composed with any other plugin
> > mix-ins, such as Configurable, Reconfigurable, Closeable, Monitorable,
> > Versioned, etc?
> > 6. If there are multiple accelerators present, which one will be chosen?
> > 7. If the configuration is set but the accelerator implementation
> > isn't available, does that cause a ConfigException?
> > 8. Could you expand on this statement: "Due to security and/or
> > compatibility concerns it is possible to limit this feature to only
> > load approved service providers (e.g those distributed with Kafka)".
> > How will this be implemented? What threat model does this assume?
> >
> > Thanks,
> > Greg
> >
> > On Tue, Feb 18, 2025 at 11:41 AM Denloye, Olasoji <
> > olasoji.denloye@intel.com>
> > wrote:
> >
> > > Hi, I would like to bring this back to everyone's attention. This
> > > KIP introduces a simple framework that enables the use of
> > > alternative, faster implementations of the existing compression
> > > codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji <ol...@intel.com>
> > > Sent: Thursday, February 13, 2025 1:22 PM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi, I would like to bring this back to everyone's attention. I have
> > > made changes to the proposal to incorporate the ServiceLoader API as
> > suggested.
> > >
> > > https://cwiki.apache.org/confluence/x/Xgr0Ew
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Denloye, Olasoji
> > > Sent: Thursday, December 19, 2024 11:01 AM
> > > To: dev@kafka.apache.org
> > > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Frédérik, Thanks for the suggestion to use the existing
> > > ServiceLoader feature in Java. It could serve as suitable mechanism
> > > for selecting the best available compression codec.
> > >
> > > Thanks
> > >
> > > -----Original Message-----
> > > From: Frédérik Rouleau <fr...@confluent.io.INVALID>
> > > Sent: Wednesday, December 18, 2024 12:24 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> > >
> > > Hi Olasoji,
> > >
> > > I think that having a way to provide some optimised/accelerated
> > > implementation is a good idea.
> > > What not using the ServiceLoader feature of Java? That would allow
> > > anyone to provide their own optimised implementation depending on
> > > their infrastructure. We can keep the existing implementation as the
> > > default and allow Kafka to find a better implementation based on
> > > Service found. As you mentioned, we might need to enrich the
> > > interface to allow Kafka to find the best version available at runtime.
> > >
> > > Regards,
> > >
> > > On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji <
> > > olasoji.denloye@intel.com>
> > > wrote:
> > >
> > > > Hi Everyone,
> > > >
> > > > I would like to start a discussion on KIP-1121: Compression
> > > > acceleration acceleration in Kafka. KIP-1121: Compression
> > > > acceleration in Kafka - Apache Kafka - Apache Software Foundation<
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Comp
> > > > re
> > > > ss
> > > > ion+acceleration+in+Kafka
> > > > >
> > > >
> > >
> >
>

RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

["Denloye, Olasoji" <ol...@intel.com>]

Hello all,

I would like to call a vote soon so if anyone wants to contribute to the discussion or bring up other concerns, you are welcome to comment.

Thanks,
Olasoji

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com> 
Sent: Wednesday, April 23, 2025 10:22 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi All,

This is a gentle reminder about open discussion on this compression acceleration KIP. I would love to hear your thoughts.

Thanks,
Olasoji

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com>
Sent: Monday, April 7, 2025 8:49 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Greg,

I have refactored the proposal to reduce the API surface area. I introduced a new CompressionService interface which contains basic compression/decompression functions. A service provider, when supplied, would be wrapped in a concrete 'Compression' and run similar to the existing compression codecs. The 'Compression' interface and its subclasses have been decoupled from these changes.

6. based on your feedback, offloading would occur if and only if the specific configuration is set, even if multiple service providers are available 8. The threat model is that dynamically loading any untrusted or untested jar on the classpath can be dangerous. The purpose of bringing this up is to discuss how service provider (plugins) is distributed and installed. On one hand, a centralized Kafka repository and on the other a decentralized approach where plugins are developed and installed on the fly. Which is preferable? The choice would determine what security mitigations are needed if any.

https://cwiki.apache.org/confluence/x/Xgr0Ew
Thanks,
Olasoji

-----Original Message-----
From: Greg Harris <gr...@aiven.io.INVALID>
Sent: Monday, February 24, 2025 12:42 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Olasoji,

Thanks for the update.

1. Could you add these as code examples to the KIP, and not as prose? See [1-3] for some examples of well-written public interfaces.

2. I find the proposed API surface area excessive, and think we need to narrow it down. Looking at what's merged on trunk right now and not including your proposed modifications, this KIP includes:
* 2 interfaces
* 12 classes
* 5 constructors
* 8 static methods
* 2 static variables
* 18 unique instance methods
Committing to long-term stability of this many classes and interfaces as public API will make it hard to initially review it, and innovate on it in the future.

4. Is there a way to modify the existing signatures/implementations such that we don't have to expose any subclasses of Compression as public API, only the CompressionTypes?
6. What does "first provider loaded" mean to users? AFAIU, this looks arbitrary to users, as it depends on classpath ordering, filesystem ordering, etc.
8. I'm trying to understand under what conditions this can be attacked (threat model), and what mitigations Kafka needs to provide (implementation). Is Kafka providing special filtering? Is Kafka providing the custom class loader?
6, 7, 8. Please add these details to the KIP so that other reviewers don't need to follow the discussion thread so closely.
9. The testing section looks like it is written for individuals implementing accelerator plugins. Could you also write about how the Kafka side of the implementation will be tested?

Thanks,
Greg

[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging
[2]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage
[3]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism+for+plugins+and+connectors+to+register+metrics

On Fri, Feb 21, 2025 at 12:15 AM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hi Greg, Thanks for your response.
>
> 1. Some changes will be made to the Compression and Builder 
> interfaces. At least one default method, isAvailable(), would be added 
> to the Builder interface. The static methods in the Compression 
> interface that create and return specific builders would be modified 
> to return a new builder or a loaded builder from a service provider if 
> it exists. The return types may need to be more generic e.g 
> Builder<GzipCompression> vs GzipCompression.Builder.
> 2. These classes are appropriate to add to the API surface as they may 
> be required to develop efficient compression implementations. I will 
> modify the KIP.
> 4. This is because the Compression interface (currently) expects a 
> Builder of type 'GzipCompression.Builder'. I agree that alternative 
> implementations need not extend the base implementation with some 
> modifications to the Compression interface. However if an alternative 
> implementation is registered as a service provider of GzipCompression 
> then it must be of type 'CompressionType.GZIP' and it must be fully 
> compatible with the base implementation of GzipCompression.
> 5. So far no, but 'Configurable' looks promising 6. If no specific 
> configuration is set, then the first provider loaded and available 
> will be used. If a specific configuration is set then that specific 
> provider will be used if available else the base implementation.
> 7. Yes, a ConfigException will be thrown if the class specified in the 
> config is not found on the classpath.
> 8. This can be implemented by using special filtering or a custom 
> class loader. The concern here is that any class on the class path 
> (trusted or
> not) could be loaded which could intercept messages or corrupt 
> messages by compressing using an unknown algorithm.
>
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Tuesday, February 18, 2025 1:58 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Olasoji,
>
> Thanks for the KIP! I find it addresses many of the concerns I had 
> with KIP-984. Thank you for following up on that!
>
> 1. Since the Compression interface is internal currently, it may 
> contain some components that were not designed with pluggability in 
> mind, such as the static methods. Are these interfaces being published 
> as-is, or are some changes going to be made first?
> 2. Similarly, the ByteBufferOutputStream, CompressionType, 
> BufferSupplier, ByteBufferInputStream, ChunkedByteStream, and 
> GzipOutputStream classes in the signature or implementations of 
> Compression are not public API currently, and would need to be made 
> public along with Compression. Are these classes appropriate to add to our API surface?
> 3. Could you adjust the KIP structure/formatting to make the public 
> API surface area more clear? Some prose in "Proposed Changes" could 
> become code examples in "Public Interfaces".
> 4. What is the significance of this statement: "Any new service 
> provider should extend the base implementation and thus return the 
> same CompressionType". Are we expecting that `o instanceof 
> GzipCompression == o.type().equals(CompressionType.GZIP)` at all 
> times? If an alternative implementation doesn't need any part of the 
> upstream implementation, does it make sense to force them to extend one another?
> 5. Is this interface expected to be composed with any other plugin 
> mix-ins, such as Configurable, Reconfigurable, Closeable, Monitorable, 
> Versioned, etc?
> 6. If there are multiple accelerators present, which one will be chosen?
> 7. If the configuration is set but the accelerator implementation 
> isn't available, does that cause a ConfigException?
> 8. Could you expand on this statement: "Due to security and/or 
> compatibility concerns it is possible to limit this feature to only 
> load approved service providers (e.g those distributed with Kafka)".
> How will this be implemented? What threat model does this assume?
>
> Thanks,
> Greg
>
> On Tue, Feb 18, 2025 at 11:41 AM Denloye, Olasoji < 
> olasoji.denloye@intel.com>
> wrote:
>
> > Hi, I would like to bring this back to everyone's attention. This 
> > KIP introduces a simple framework that enables the use of 
> > alternative, faster implementations of the existing compression 
> > codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.
> >
> > https://cwiki.apache.org/confluence/x/Xgr0Ew
> >
> > Thanks
> >
> > -----Original Message-----
> > From: Denloye, Olasoji <ol...@intel.com>
> > Sent: Thursday, February 13, 2025 1:22 PM
> > To: dev@kafka.apache.org
> > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi, I would like to bring this back to everyone's attention. I have 
> > made changes to the proposal to incorporate the ServiceLoader API as
> suggested.
> >
> > https://cwiki.apache.org/confluence/x/Xgr0Ew
> >
> > Thanks
> >
> > -----Original Message-----
> > From: Denloye, Olasoji
> > Sent: Thursday, December 19, 2024 11:01 AM
> > To: dev@kafka.apache.org
> > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Frédérik, Thanks for the suggestion to use the existing 
> > ServiceLoader feature in Java. It could serve as suitable mechanism 
> > for selecting the best available compression codec.
> >
> > Thanks
> >
> > -----Original Message-----
> > From: Frédérik Rouleau <fr...@confluent.io.INVALID>
> > Sent: Wednesday, December 18, 2024 12:24 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Olasoji,
> >
> > I think that having a way to provide some optimised/accelerated 
> > implementation is a good idea.
> > What not using the ServiceLoader feature of Java? That would allow 
> > anyone to provide their own optimised implementation depending on 
> > their infrastructure. We can keep the existing implementation as the 
> > default and allow Kafka to find a better implementation based on 
> > Service found. As you mentioned, we might need to enrich the 
> > interface to allow Kafka to find the best version available at runtime.
> >
> > Regards,
> >
> > On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji < 
> > olasoji.denloye@intel.com>
> > wrote:
> >
> > > Hi Everyone,
> > >
> > > I would like to start a discussion on KIP-1121: Compression 
> > > acceleration acceleration in Kafka. KIP-1121: Compression 
> > > acceleration in Kafka - Apache Kafka - Apache Software Foundation< 
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Comp
> > > re
> > > ss
> > > ion+acceleration+in+Kafka
> > > >
> > >
> >
>

RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

["Denloye, Olasoji" <ol...@intel.com>]

Hi All,

This is a gentle reminder about open discussion on this compression acceleration KIP. I would love to hear your thoughts.

Thanks,
Olasoji

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com> 
Sent: Monday, April 7, 2025 8:49 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Greg,

I have refactored the proposal to reduce the API surface area. I introduced a new CompressionService interface which contains basic compression/decompression functions. A service provider, when supplied, would be wrapped in a concrete 'Compression' and run similar to the existing compression codecs. The 'Compression' interface and its subclasses have been decoupled from these changes.

6. based on your feedback, offloading would occur if and only if the specific configuration is set, even if multiple service providers are available 8. The threat model is that dynamically loading any untrusted or untested jar on the classpath can be dangerous. The purpose of bringing this up is to discuss how service provider (plugins) is distributed and installed. On one hand, a centralized Kafka repository and on the other a decentralized approach where plugins are developed and installed on the fly. Which is preferable? The choice would determine what security mitigations are needed if any.

https://cwiki.apache.org/confluence/x/Xgr0Ew
Thanks,
Olasoji

-----Original Message-----
From: Greg Harris <gr...@aiven.io.INVALID>
Sent: Monday, February 24, 2025 12:42 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Olasoji,

Thanks for the update.

1. Could you add these as code examples to the KIP, and not as prose? See [1-3] for some examples of well-written public interfaces.

2. I find the proposed API surface area excessive, and think we need to narrow it down. Looking at what's merged on trunk right now and not including your proposed modifications, this KIP includes:
* 2 interfaces
* 12 classes
* 5 constructors
* 8 static methods
* 2 static variables
* 18 unique instance methods
Committing to long-term stability of this many classes and interfaces as public API will make it hard to initially review it, and innovate on it in the future.

4. Is there a way to modify the existing signatures/implementations such that we don't have to expose any subclasses of Compression as public API, only the CompressionTypes?
6. What does "first provider loaded" mean to users? AFAIU, this looks arbitrary to users, as it depends on classpath ordering, filesystem ordering, etc.
8. I'm trying to understand under what conditions this can be attacked (threat model), and what mitigations Kafka needs to provide (implementation). Is Kafka providing special filtering? Is Kafka providing the custom class loader?
6, 7, 8. Please add these details to the KIP so that other reviewers don't need to follow the discussion thread so closely.
9. The testing section looks like it is written for individuals implementing accelerator plugins. Could you also write about how the Kafka side of the implementation will be tested?

Thanks,
Greg

[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging
[2]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage
[3]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism+for+plugins+and+connectors+to+register+metrics

On Fri, Feb 21, 2025 at 12:15 AM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hi Greg, Thanks for your response.
>
> 1. Some changes will be made to the Compression and Builder 
> interfaces. At least one default method, isAvailable(), would be added 
> to the Builder interface. The static methods in the Compression 
> interface that create and return specific builders would be modified 
> to return a new builder or a loaded builder from a service provider if 
> it exists. The return types may need to be more generic e.g 
> Builder<GzipCompression> vs GzipCompression.Builder.
> 2. These classes are appropriate to add to the API surface as they may 
> be required to develop efficient compression implementations. I will 
> modify the KIP.
> 4. This is because the Compression interface (currently) expects a 
> Builder of type 'GzipCompression.Builder'. I agree that alternative 
> implementations need not extend the base implementation with some 
> modifications to the Compression interface. However if an alternative 
> implementation is registered as a service provider of GzipCompression 
> then it must be of type 'CompressionType.GZIP' and it must be fully 
> compatible with the base implementation of GzipCompression.
> 5. So far no, but 'Configurable' looks promising 6. If no specific 
> configuration is set, then the first provider loaded and available 
> will be used. If a specific configuration is set then that specific 
> provider will be used if available else the base implementation.
> 7. Yes, a ConfigException will be thrown if the class specified in the 
> config is not found on the classpath.
> 8. This can be implemented by using special filtering or a custom 
> class loader. The concern here is that any class on the class path 
> (trusted or
> not) could be loaded which could intercept messages or corrupt 
> messages by compressing using an unknown algorithm.
>
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Tuesday, February 18, 2025 1:58 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Olasoji,
>
> Thanks for the KIP! I find it addresses many of the concerns I had 
> with KIP-984. Thank you for following up on that!
>
> 1. Since the Compression interface is internal currently, it may 
> contain some components that were not designed with pluggability in 
> mind, such as the static methods. Are these interfaces being published 
> as-is, or are some changes going to be made first?
> 2. Similarly, the ByteBufferOutputStream, CompressionType, 
> BufferSupplier, ByteBufferInputStream, ChunkedByteStream, and 
> GzipOutputStream classes in the signature or implementations of 
> Compression are not public API currently, and would need to be made 
> public along with Compression. Are these classes appropriate to add to our API surface?
> 3. Could you adjust the KIP structure/formatting to make the public 
> API surface area more clear? Some prose in "Proposed Changes" could 
> become code examples in "Public Interfaces".
> 4. What is the significance of this statement: "Any new service 
> provider should extend the base implementation and thus return the 
> same CompressionType". Are we expecting that `o instanceof 
> GzipCompression == o.type().equals(CompressionType.GZIP)` at all 
> times? If an alternative implementation doesn't need any part of the 
> upstream implementation, does it make sense to force them to extend one another?
> 5. Is this interface expected to be composed with any other plugin 
> mix-ins, such as Configurable, Reconfigurable, Closeable, Monitorable, 
> Versioned, etc?
> 6. If there are multiple accelerators present, which one will be chosen?
> 7. If the configuration is set but the accelerator implementation 
> isn't available, does that cause a ConfigException?
> 8. Could you expand on this statement: "Due to security and/or 
> compatibility concerns it is possible to limit this feature to only 
> load approved service providers (e.g those distributed with Kafka)".
> How will this be implemented? What threat model does this assume?
>
> Thanks,
> Greg
>
> On Tue, Feb 18, 2025 at 11:41 AM Denloye, Olasoji < 
> olasoji.denloye@intel.com>
> wrote:
>
> > Hi, I would like to bring this back to everyone's attention. This 
> > KIP introduces a simple framework that enables the use of 
> > alternative, faster implementations of the existing compression 
> > codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.
> >
> > https://cwiki.apache.org/confluence/x/Xgr0Ew
> >
> > Thanks
> >
> > -----Original Message-----
> > From: Denloye, Olasoji <ol...@intel.com>
> > Sent: Thursday, February 13, 2025 1:22 PM
> > To: dev@kafka.apache.org
> > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi, I would like to bring this back to everyone's attention. I have 
> > made changes to the proposal to incorporate the ServiceLoader API as
> suggested.
> >
> > https://cwiki.apache.org/confluence/x/Xgr0Ew
> >
> > Thanks
> >
> > -----Original Message-----
> > From: Denloye, Olasoji
> > Sent: Thursday, December 19, 2024 11:01 AM
> > To: dev@kafka.apache.org
> > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Frédérik, Thanks for the suggestion to use the existing 
> > ServiceLoader feature in Java. It could serve as suitable mechanism 
> > for selecting the best available compression codec.
> >
> > Thanks
> >
> > -----Original Message-----
> > From: Frédérik Rouleau <fr...@confluent.io.INVALID>
> > Sent: Wednesday, December 18, 2024 12:24 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Olasoji,
> >
> > I think that having a way to provide some optimised/accelerated 
> > implementation is a good idea.
> > What not using the ServiceLoader feature of Java? That would allow 
> > anyone to provide their own optimised implementation depending on 
> > their infrastructure. We can keep the existing implementation as the 
> > default and allow Kafka to find a better implementation based on 
> > Service found. As you mentioned, we might need to enrich the 
> > interface to allow Kafka to find the best version available at runtime.
> >
> > Regards,
> >
> > On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji < 
> > olasoji.denloye@intel.com>
> > wrote:
> >
> > > Hi Everyone,
> > >
> > > I would like to start a discussion on KIP-1121: Compression 
> > > acceleration acceleration in Kafka. KIP-1121: Compression 
> > > acceleration in Kafka - Apache Kafka - Apache Software Foundation< 
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Comp
> > > re
> > > ss
> > > ion+acceleration+in+Kafka
> > > >
> > >
> >
>

RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

["Denloye, Olasoji" <ol...@intel.com>]

Hi Greg,

I have refactored the proposal to reduce the API surface area. I introduced a new CompressionService interface which contains basic compression/decompression functions. A service provider, when supplied, would be wrapped in a concrete 'Compression' and run similar to the existing compression codecs. The 'Compression' interface and its subclasses have been decoupled from these changes.

6. based on your feedback, offloading would occur if and only if the specific configuration is set, even if multiple service providers are available
8. The threat model is that dynamically loading any untrusted or untested jar on the classpath can be dangerous. The purpose of bringing this up is to discuss how service provider (plugins) is distributed and installed. On one hand, a centralized Kafka repository and on the other a decentralized approach where plugins are developed and installed on the fly. Which is preferable? The choice would determine what security mitigations are needed if any.

https://cwiki.apache.org/confluence/x/Xgr0Ew 
Thanks,
Olasoji

-----Original Message-----
From: Greg Harris <gr...@aiven.io.INVALID> 
Sent: Monday, February 24, 2025 12:42 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Olasoji,

Thanks for the update.

1. Could you add these as code examples to the KIP, and not as prose? See [1-3] for some examples of well-written public interfaces.

2. I find the proposed API surface area excessive, and think we need to narrow it down. Looking at what's merged on trunk right now and not including your proposed modifications, this KIP includes:
* 2 interfaces
* 12 classes
* 5 constructors
* 8 static methods
* 2 static variables
* 18 unique instance methods
Committing to long-term stability of this many classes and interfaces as public API will make it hard to initially review it, and innovate on it in the future.

4. Is there a way to modify the existing signatures/implementations such that we don't have to expose any subclasses of Compression as public API, only the CompressionTypes?
6. What does "first provider loaded" mean to users? AFAIU, this looks arbitrary to users, as it depends on classpath ordering, filesystem ordering, etc.
8. I'm trying to understand under what conditions this can be attacked (threat model), and what mitigations Kafka needs to provide (implementation). Is Kafka providing special filtering? Is Kafka providing the custom class loader?
6, 7, 8. Please add these details to the KIP so that other reviewers don't need to follow the discussion thread so closely.
9. The testing section looks like it is written for individuals implementing accelerator plugins. Could you also write about how the Kafka side of the implementation will be tested?

Thanks,
Greg

[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging
[2]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage
[3]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism+for+plugins+and+connectors+to+register+metrics

On Fri, Feb 21, 2025 at 12:15 AM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hi Greg, Thanks for your response.
>
> 1. Some changes will be made to the Compression and Builder 
> interfaces. At least one default method, isAvailable(), would be added 
> to the Builder interface. The static methods in the Compression 
> interface that create and return specific builders would be modified 
> to return a new builder or a loaded builder from a service provider if 
> it exists. The return types may need to be more generic e.g 
> Builder<GzipCompression> vs GzipCompression.Builder.
> 2. These classes are appropriate to add to the API surface as they may 
> be required to develop efficient compression implementations. I will 
> modify the KIP.
> 4. This is because the Compression interface (currently) expects a 
> Builder of type 'GzipCompression.Builder'. I agree that alternative 
> implementations need not extend the base implementation with some 
> modifications to the Compression interface. However if an alternative 
> implementation is registered as a service provider of GzipCompression 
> then it must be of type 'CompressionType.GZIP' and it must be fully 
> compatible with the base implementation of GzipCompression.
> 5. So far no, but 'Configurable' looks promising 6. If no specific 
> configuration is set, then the first provider loaded and available 
> will be used. If a specific configuration is set then that specific 
> provider will be used if available else the base implementation.
> 7. Yes, a ConfigException will be thrown if the class specified in the 
> config is not found on the classpath.
> 8. This can be implemented by using special filtering or a custom 
> class loader. The concern here is that any class on the class path 
> (trusted or
> not) could be loaded which could intercept messages or corrupt 
> messages by compressing using an unknown algorithm.
>
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Tuesday, February 18, 2025 1:58 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Olasoji,
>
> Thanks for the KIP! I find it addresses many of the concerns I had 
> with KIP-984. Thank you for following up on that!
>
> 1. Since the Compression interface is internal currently, it may 
> contain some components that were not designed with pluggability in 
> mind, such as the static methods. Are these interfaces being published 
> as-is, or are some changes going to be made first?
> 2. Similarly, the ByteBufferOutputStream, CompressionType, 
> BufferSupplier, ByteBufferInputStream, ChunkedByteStream, and 
> GzipOutputStream classes in the signature or implementations of 
> Compression are not public API currently, and would need to be made 
> public along with Compression. Are these classes appropriate to add to our API surface?
> 3. Could you adjust the KIP structure/formatting to make the public 
> API surface area more clear? Some prose in "Proposed Changes" could 
> become code examples in "Public Interfaces".
> 4. What is the significance of this statement: "Any new service 
> provider should extend the base implementation and thus return the 
> same CompressionType". Are we expecting that `o instanceof 
> GzipCompression == o.type().equals(CompressionType.GZIP)` at all 
> times? If an alternative implementation doesn't need any part of the 
> upstream implementation, does it make sense to force them to extend one another?
> 5. Is this interface expected to be composed with any other plugin 
> mix-ins, such as Configurable, Reconfigurable, Closeable, Monitorable, 
> Versioned, etc?
> 6. If there are multiple accelerators present, which one will be chosen?
> 7. If the configuration is set but the accelerator implementation 
> isn't available, does that cause a ConfigException?
> 8. Could you expand on this statement: "Due to security and/or 
> compatibility concerns it is possible to limit this feature to only 
> load approved service providers (e.g those distributed with Kafka)". 
> How will this be implemented? What threat model does this assume?
>
> Thanks,
> Greg
>
> On Tue, Feb 18, 2025 at 11:41 AM Denloye, Olasoji < 
> olasoji.denloye@intel.com>
> wrote:
>
> > Hi, I would like to bring this back to everyone's attention. This 
> > KIP introduces a simple framework that enables the use of 
> > alternative, faster implementations of the existing compression 
> > codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.
> >
> > https://cwiki.apache.org/confluence/x/Xgr0Ew
> >
> > Thanks
> >
> > -----Original Message-----
> > From: Denloye, Olasoji <ol...@intel.com>
> > Sent: Thursday, February 13, 2025 1:22 PM
> > To: dev@kafka.apache.org
> > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi, I would like to bring this back to everyone's attention. I have 
> > made changes to the proposal to incorporate the ServiceLoader API as
> suggested.
> >
> > https://cwiki.apache.org/confluence/x/Xgr0Ew
> >
> > Thanks
> >
> > -----Original Message-----
> > From: Denloye, Olasoji
> > Sent: Thursday, December 19, 2024 11:01 AM
> > To: dev@kafka.apache.org
> > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Frédérik, Thanks for the suggestion to use the existing 
> > ServiceLoader feature in Java. It could serve as suitable mechanism 
> > for selecting the best available compression codec.
> >
> > Thanks
> >
> > -----Original Message-----
> > From: Frédérik Rouleau <fr...@confluent.io.INVALID>
> > Sent: Wednesday, December 18, 2024 12:24 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Olasoji,
> >
> > I think that having a way to provide some optimised/accelerated 
> > implementation is a good idea.
> > What not using the ServiceLoader feature of Java? That would allow 
> > anyone to provide their own optimised implementation depending on 
> > their infrastructure. We can keep the existing implementation as the 
> > default and allow Kafka to find a better implementation based on 
> > Service found. As you mentioned, we might need to enrich the 
> > interface to allow Kafka to find the best version available at runtime.
> >
> > Regards,
> >
> > On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji < 
> > olasoji.denloye@intel.com>
> > wrote:
> >
> > > Hi Everyone,
> > >
> > > I would like to start a discussion on KIP-1121: Compression 
> > > acceleration acceleration in Kafka. KIP-1121: Compression 
> > > acceleration in Kafka - Apache Kafka - Apache Software Foundation< 
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Comp
> > > re
> > > ss
> > > ion+acceleration+in+Kafka
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

[Greg Harris <gr...@aiven.io.INVALID>]

Hi Olasoji,

Thanks for the update.

1. Could you add these as code examples to the KIP, and not as prose? See
[1-3] for some examples of well-written public interfaces.

2. I find the proposed API surface area excessive, and think we need to
narrow it down. Looking at what's merged on trunk right now and not
including your proposed modifications, this KIP includes:
* 2 interfaces
* 12 classes
* 5 constructors
* 8 static methods
* 2 static variables
* 18 unique instance methods
Committing to long-term stability of this many classes and interfaces as
public API will make it hard to initially review it, and innovate on it in
the future.

4. Is there a way to modify the existing signatures/implementations such
that we don't have to expose any subclasses of Compression as public API,
only the CompressionTypes?
6. What does "first provider loaded" mean to users? AFAIU, this looks
arbitrary to users, as it depends on classpath ordering, filesystem
ordering, etc.
8. I'm trying to understand under what conditions this can be attacked
(threat model), and what mitigations Kafka needs to provide
(implementation). Is Kafka providing special filtering? Is Kafka providing
the custom class loader?
6, 7, 8. Please add these details to the KIP so that other reviewers don't
need to follow the discussion thread so closely.
9. The testing section looks like it is written for individuals
implementing accelerator plugins. Could you also write about how the Kafka
side of the implementation will be tested?

Thanks,
Greg

[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging
[2]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage
[3]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism+for+plugins+and+connectors+to+register+metrics

On Fri, Feb 21, 2025 at 12:15 AM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hi Greg, Thanks for your response.
>
> 1. Some changes will be made to the Compression and Builder interfaces. At
> least one default method, isAvailable(), would be added to the Builder
> interface. The static methods in the Compression interface that create and
> return specific builders would be modified to return a new builder or a
> loaded builder from a service provider if it exists. The return types may
> need to be more generic e.g Builder<GzipCompression> vs
> GzipCompression.Builder.
> 2. These classes are appropriate to add to the API surface as they may be
> required to develop efficient compression implementations. I will modify
> the KIP.
> 4. This is because the Compression interface (currently) expects a Builder
> of type 'GzipCompression.Builder'. I agree that alternative implementations
> need not extend the base implementation with some modifications to the
> Compression interface. However if an alternative implementation is
> registered as a service provider of GzipCompression then it must be of type
> 'CompressionType.GZIP' and it must be fully compatible with the base
> implementation of GzipCompression.
> 5. So far no, but 'Configurable' looks promising
> 6. If no specific configuration is set, then the first provider loaded and
> available will be used. If a specific configuration is set then that
> specific provider will be used if available else the base implementation.
> 7. Yes, a ConfigException will be thrown if the class specified in the
> config is not found on the classpath.
> 8. This can be implemented by using special filtering or a custom class
> loader. The concern here is that any class on the class path (trusted or
> not) could be loaded which could intercept messages or corrupt messages by
> compressing using an unknown algorithm.
>
> -----Original Message-----
> From: Greg Harris <gr...@aiven.io.INVALID>
> Sent: Tuesday, February 18, 2025 1:58 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Olasoji,
>
> Thanks for the KIP! I find it addresses many of the concerns I had with
> KIP-984. Thank you for following up on that!
>
> 1. Since the Compression interface is internal currently, it may contain
> some components that were not designed with pluggability in mind, such as
> the static methods. Are these interfaces being published as-is, or are some
> changes going to be made first?
> 2. Similarly, the ByteBufferOutputStream, CompressionType, BufferSupplier,
> ByteBufferInputStream, ChunkedByteStream, and GzipOutputStream classes in
> the signature or implementations of Compression are not public API
> currently, and would need to be made public along with Compression. Are
> these classes appropriate to add to our API surface?
> 3. Could you adjust the KIP structure/formatting to make the public API
> surface area more clear? Some prose in "Proposed Changes" could become code
> examples in "Public Interfaces".
> 4. What is the significance of this statement: "Any new service provider
> should extend the base implementation and thus return the same
> CompressionType". Are we expecting that `o instanceof GzipCompression ==
> o.type().equals(CompressionType.GZIP)` at all times? If an alternative
> implementation doesn't need any part of the upstream implementation, does
> it make sense to force them to extend one another?
> 5. Is this interface expected to be composed with any other plugin
> mix-ins, such as Configurable, Reconfigurable, Closeable, Monitorable,
> Versioned, etc?
> 6. If there are multiple accelerators present, which one will be chosen?
> 7. If the configuration is set but the accelerator implementation isn't
> available, does that cause a ConfigException?
> 8. Could you expand on this statement: "Due to security and/or
> compatibility concerns it is possible to limit this feature to only load
> approved service providers (e.g those distributed with Kafka)". How will
> this be implemented? What threat model does this assume?
>
> Thanks,
> Greg
>
> On Tue, Feb 18, 2025 at 11:41 AM Denloye, Olasoji <
> olasoji.denloye@intel.com>
> wrote:
>
> > Hi, I would like to bring this back to everyone's attention. This KIP
> > introduces a simple framework that enables the use of alternative,
> > faster implementations of the existing compression codecs (Gzip,
> > Snappy, Zstandard and LZ4) in Kafka.
> >
> > https://cwiki.apache.org/confluence/x/Xgr0Ew
> >
> > Thanks
> >
> > -----Original Message-----
> > From: Denloye, Olasoji <ol...@intel.com>
> > Sent: Thursday, February 13, 2025 1:22 PM
> > To: dev@kafka.apache.org
> > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi, I would like to bring this back to everyone's attention. I have
> > made changes to the proposal to incorporate the ServiceLoader API as
> suggested.
> >
> > https://cwiki.apache.org/confluence/x/Xgr0Ew
> >
> > Thanks
> >
> > -----Original Message-----
> > From: Denloye, Olasoji
> > Sent: Thursday, December 19, 2024 11:01 AM
> > To: dev@kafka.apache.org
> > Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Frédérik, Thanks for the suggestion to use the existing
> > ServiceLoader feature in Java. It could serve as suitable mechanism
> > for selecting the best available compression codec.
> >
> > Thanks
> >
> > -----Original Message-----
> > From: Frédérik Rouleau <fr...@confluent.io.INVALID>
> > Sent: Wednesday, December 18, 2024 12:24 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
> >
> > Hi Olasoji,
> >
> > I think that having a way to provide some optimised/accelerated
> > implementation is a good idea.
> > What not using the ServiceLoader feature of Java? That would allow
> > anyone to provide their own optimised implementation depending on
> > their infrastructure. We can keep the existing implementation as the
> > default and allow Kafka to find a better implementation based on
> > Service found. As you mentioned, we might need to enrich the interface
> > to allow Kafka to find the best version available at runtime.
> >
> > Regards,
> >
> > On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji <
> > olasoji.denloye@intel.com>
> > wrote:
> >
> > > Hi Everyone,
> > >
> > > I would like to start a discussion on KIP-1121: Compression
> > > acceleration acceleration in Kafka. KIP-1121: Compression
> > > acceleration in Kafka - Apache Kafka - Apache Software Foundation<
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Compre
> > > ss
> > > ion+acceleration+in+Kafka
> > > >
> > >
> >
>

RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

["Denloye, Olasoji" <ol...@intel.com>]

Hi Greg, Thanks for your response.

1. Some changes will be made to the Compression and Builder interfaces. At least one default method, isAvailable(), would be added to the Builder interface. The static methods in the Compression interface that create and return specific builders would be modified to return a new builder or a loaded builder from a service provider if it exists. The return types may need to be more generic e.g Builder<GzipCompression> vs GzipCompression.Builder.
2. These classes are appropriate to add to the API surface as they may be required to develop efficient compression implementations. I will modify the KIP.
4. This is because the Compression interface (currently) expects a Builder of type 'GzipCompression.Builder'. I agree that alternative implementations need not extend the base implementation with some modifications to the Compression interface. However if an alternative implementation is registered as a service provider of GzipCompression then it must be of type 'CompressionType.GZIP' and it must be fully compatible with the base implementation of GzipCompression.
5. So far no, but 'Configurable' looks promising
6. If no specific configuration is set, then the first provider loaded and available will be used. If a specific configuration is set then that specific provider will be used if available else the base implementation.
7. Yes, a ConfigException will be thrown if the class specified in the config is not found on the classpath.
8. This can be implemented by using special filtering or a custom class loader. The concern here is that any class on the class path (trusted or not) could be loaded which could intercept messages or corrupt messages by compressing using an unknown algorithm.

-----Original Message-----
From: Greg Harris <gr...@aiven.io.INVALID> 
Sent: Tuesday, February 18, 2025 1:58 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Olasoji,

Thanks for the KIP! I find it addresses many of the concerns I had with KIP-984. Thank you for following up on that!

1. Since the Compression interface is internal currently, it may contain some components that were not designed with pluggability in mind, such as the static methods. Are these interfaces being published as-is, or are some changes going to be made first?
2. Similarly, the ByteBufferOutputStream, CompressionType, BufferSupplier, ByteBufferInputStream, ChunkedByteStream, and GzipOutputStream classes in the signature or implementations of Compression are not public API currently, and would need to be made public along with Compression. Are these classes appropriate to add to our API surface?
3. Could you adjust the KIP structure/formatting to make the public API surface area more clear? Some prose in "Proposed Changes" could become code examples in "Public Interfaces".
4. What is the significance of this statement: "Any new service provider should extend the base implementation and thus return the same CompressionType". Are we expecting that `o instanceof GzipCompression == o.type().equals(CompressionType.GZIP)` at all times? If an alternative implementation doesn't need any part of the upstream implementation, does it make sense to force them to extend one another?
5. Is this interface expected to be composed with any other plugin mix-ins, such as Configurable, Reconfigurable, Closeable, Monitorable, Versioned, etc?
6. If there are multiple accelerators present, which one will be chosen?
7. If the configuration is set but the accelerator implementation isn't available, does that cause a ConfigException?
8. Could you expand on this statement: "Due to security and/or compatibility concerns it is possible to limit this feature to only load approved service providers (e.g those distributed with Kafka)". How will this be implemented? What threat model does this assume?

Thanks,
Greg

On Tue, Feb 18, 2025 at 11:41 AM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hi, I would like to bring this back to everyone's attention. This KIP 
> introduces a simple framework that enables the use of alternative, 
> faster implementations of the existing compression codecs (Gzip, 
> Snappy, Zstandard and LZ4) in Kafka.
>
> https://cwiki.apache.org/confluence/x/Xgr0Ew
>
> Thanks
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Thursday, February 13, 2025 1:22 PM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi, I would like to bring this back to everyone's attention. I have 
> made changes to the proposal to incorporate the ServiceLoader API as suggested.
>
> https://cwiki.apache.org/confluence/x/Xgr0Ew
>
> Thanks
>
> -----Original Message-----
> From: Denloye, Olasoji
> Sent: Thursday, December 19, 2024 11:01 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Frédérik, Thanks for the suggestion to use the existing 
> ServiceLoader feature in Java. It could serve as suitable mechanism 
> for selecting the best available compression codec.
>
> Thanks
>
> -----Original Message-----
> From: Frédérik Rouleau <fr...@confluent.io.INVALID>
> Sent: Wednesday, December 18, 2024 12:24 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Olasoji,
>
> I think that having a way to provide some optimised/accelerated 
> implementation is a good idea.
> What not using the ServiceLoader feature of Java? That would allow 
> anyone to provide their own optimised implementation depending on 
> their infrastructure. We can keep the existing implementation as the 
> default and allow Kafka to find a better implementation based on 
> Service found. As you mentioned, we might need to enrich the interface 
> to allow Kafka to find the best version available at runtime.
>
> Regards,
>
> On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji < 
> olasoji.denloye@intel.com>
> wrote:
>
> > Hi Everyone,
> >
> > I would like to start a discussion on KIP-1121: Compression 
> > acceleration acceleration in Kafka. KIP-1121: Compression 
> > acceleration in Kafka - Apache Kafka - Apache Software Foundation< 
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Compre
> > ss
> > ion+acceleration+in+Kafka
> > >
> >
>

Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

[Greg Harris <gr...@aiven.io.INVALID>]

Hi Olasoji,

Thanks for the KIP! I find it addresses many of the concerns I had with
KIP-984. Thank you for following up on that!

1. Since the Compression interface is internal currently, it may contain
some components that were not designed with pluggability in mind, such as
the static methods. Are these interfaces being published as-is, or are some
changes going to be made first?
2. Similarly, the ByteBufferOutputStream, CompressionType, BufferSupplier,
ByteBufferInputStream, ChunkedByteStream, and GzipOutputStream classes in
the signature or implementations of Compression are not public API
currently, and would need to be made public along with Compression. Are
these classes appropriate to add to our API surface?
3. Could you adjust the KIP structure/formatting to make the public API
surface area more clear? Some prose in "Proposed Changes" could become code
examples in "Public Interfaces".
4. What is the significance of this statement: "Any new service provider
should extend the base implementation and thus return the same
CompressionType". Are we expecting that `o instanceof GzipCompression ==
o.type().equals(CompressionType.GZIP)` at all times? If an alternative
implementation doesn't need any part of the upstream implementation, does
it make sense to force them to extend one another?
5. Is this interface expected to be composed with any other plugin mix-ins,
such as Configurable, Reconfigurable, Closeable, Monitorable, Versioned,
etc?
6. If there are multiple accelerators present, which one will be chosen?
7. If the configuration is set but the accelerator implementation isn't
available, does that cause a ConfigException?
8. Could you expand on this statement: "Due to security and/or
compatibility concerns it is possible to limit this feature to only load
approved service providers (e.g those distributed with Kafka)". How will
this be implemented? What threat model does this assume?

Thanks,
Greg

On Tue, Feb 18, 2025 at 11:41 AM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hi, I would like to bring this back to everyone's attention. This KIP
> introduces a simple framework that enables the use of alternative, faster
> implementations of the existing compression codecs (Gzip, Snappy, Zstandard
> and LZ4) in Kafka.
>
> https://cwiki.apache.org/confluence/x/Xgr0Ew
>
> Thanks
>
> -----Original Message-----
> From: Denloye, Olasoji <ol...@intel.com>
> Sent: Thursday, February 13, 2025 1:22 PM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi, I would like to bring this back to everyone's attention. I have made
> changes to the proposal to incorporate the ServiceLoader API as suggested.
>
> https://cwiki.apache.org/confluence/x/Xgr0Ew
>
> Thanks
>
> -----Original Message-----
> From: Denloye, Olasoji
> Sent: Thursday, December 19, 2024 11:01 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Frédérik, Thanks for the suggestion to use the existing ServiceLoader
> feature in Java. It could serve as suitable mechanism for selecting the
> best available compression codec.
>
> Thanks
>
> -----Original Message-----
> From: Frédérik Rouleau <fr...@confluent.io.INVALID>
> Sent: Wednesday, December 18, 2024 12:24 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka
>
> Hi Olasoji,
>
> I think that having a way to provide some optimised/accelerated
> implementation is a good idea.
> What not using the ServiceLoader feature of Java? That would allow anyone
> to provide their own optimised implementation depending on their
> infrastructure. We can keep the existing implementation as the default and
> allow Kafka to find a better implementation based on Service found. As you
> mentioned, we might need to enrich the interface to allow Kafka to find the
> best version available at runtime.
>
> Regards,
>
> On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji <
> olasoji.denloye@intel.com>
> wrote:
>
> > Hi Everyone,
> >
> > I would like to start a discussion on KIP-1121: Compression
> > acceleration acceleration in Kafka. KIP-1121: Compression acceleration
> > in Kafka - Apache Kafka - Apache Software Foundation<
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Compress
> > ion+acceleration+in+Kafka
> > >
> >
>

RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

["Denloye, Olasoji" <ol...@intel.com>]

Hi, I would like to bring this back to everyone's attention. This KIP introduces a simple framework that enables the use of alternative, faster implementations of the existing compression codecs (Gzip, Snappy, Zstandard and LZ4) in Kafka.

https://cwiki.apache.org/confluence/x/Xgr0Ew    

Thanks

-----Original Message-----
From: Denloye, Olasoji <ol...@intel.com> 
Sent: Thursday, February 13, 2025 1:22 PM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi, I would like to bring this back to everyone's attention. I have made changes to the proposal to incorporate the ServiceLoader API as suggested.

https://cwiki.apache.org/confluence/x/Xgr0Ew 

Thanks

-----Original Message-----
From: Denloye, Olasoji
Sent: Thursday, December 19, 2024 11:01 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Frédérik, Thanks for the suggestion to use the existing ServiceLoader feature in Java. It could serve as suitable mechanism for selecting the best available compression codec. 

Thanks

-----Original Message-----
From: Frédérik Rouleau <fr...@confluent.io.INVALID>
Sent: Wednesday, December 18, 2024 12:24 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi Olasoji,

I think that having a way to provide some optimised/accelerated implementation is a good idea.
What not using the ServiceLoader feature of Java? That would allow anyone to provide their own optimised implementation depending on their infrastructure. We can keep the existing implementation as the default and allow Kafka to find a better implementation based on Service found. As you mentioned, we might need to enrich the interface to allow Kafka to find the best version available at runtime.

Regards,

On Tue, Dec 17, 2024 at 6:36 PM Denloye, Olasoji <ol...@intel.com>
wrote:

> Hi Everyone,
>
> I would like to start a discussion on KIP-1121: Compression 
> acceleration acceleration in Kafka. KIP-1121: Compression acceleration 
> in Kafka - Apache Kafka - Apache Software Foundation< 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1121%3A+Compress
> ion+acceleration+in+Kafka
> >
>

Re: [DISCUSS] Change log.segment.bytes config type from int to long

[Mikhail Fesenko <pr...@gmail.com>]

This is interesting, could be a huge barrier for this change, thanks for
your clarification

On Fri, 21 Feb 2025 at 18:52, Jun Rao <ju...@confluent.io.invalid> wrote:

> Hi, Mikhail,
>
> Currently, the index file uses 4 bytes to represent the file position of a
> record batch within a segment file. This is based on the assumption that
> each segment is capped at 2GB. RemoteLogSegmentMetadata is a public
> interface and currently uses int to represent segmentSizeInBytes. If we
> want to increase the segment size, we need to consider the impact to those
> places too.
>
> Thanks,
>
> Jun
>
> On Thu, Feb 13, 2025 at 1:32 PM Mikhail Fesenko <pr...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > I’d like to propose a change the data type of log.segment.bytes from int
> to
> > long, allowing segment sizes beyond the current 2GB limit.
> >
> > Rationale:
> >
> > Currently, the maximum segment size is constrained by the int type,
> capping
> > it at 2GB (max int). However, with modern hardware—such as large-scale
> > deployments on machines with multiple high-capacity disks (e.g., EPYC
> > servers with 4×15TB drives)—this limitation makes segment sizes
> > inefficiently
> > small. And too many handles (log, index, metadata). So Increasing the
> limit
> > to 3–4GB (or more) would better align with today’s storage capabilities.
> >
> > Would love to hear your thoughts !
> >
> > Best,
> >
> > Mikhail Fesenko
> >
>

Re: [DISCUSS] Change log.segment.bytes config type from int to long

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Mikhail,

Currently, the index file uses 4 bytes to represent the file position of a
record batch within a segment file. This is based on the assumption that
each segment is capped at 2GB. RemoteLogSegmentMetadata is a public
interface and currently uses int to represent segmentSizeInBytes. If we
want to increase the segment size, we need to consider the impact to those
places too.

Thanks,

Jun

On Thu, Feb 13, 2025 at 1:32 PM Mikhail Fesenko <pr...@gmail.com> wrote:

> Hi everyone,
>
> I’d like to propose a change the data type of log.segment.bytes from int to
> long, allowing segment sizes beyond the current 2GB limit.
>
> Rationale:
>
> Currently, the maximum segment size is constrained by the int type, capping
> it at 2GB (max int). However, with modern hardware—such as large-scale
> deployments on machines with multiple high-capacity disks (e.g., EPYC
> servers with 4×15TB drives)—this limitation makes segment sizes
> inefficiently
> small. And too many handles (log, index, metadata). So Increasing the limit
> to 3–4GB (or more) would better align with today’s storage capabilities.
>
> Would love to hear your thoughts !
>
> Best,
>
> Mikhail Fesenko
>

Re: [DISCUSS] Change log.segment.bytes config type from int to long

[Mikhail Fesenko <pr...@gmail.com>]

Also here you can see guys have same idea

https://forum.confluent.io/t/what-happens-if-i-increase-log-segment-bytes/5845

On Tue, 22 Apr 2025 at 15:41, Mikhail Fesenko <pr...@gmail.com> wrote:

> Currently we have big amount of topics and big data, and using high disk
> storage, so rotate files too fast in case of high traffic means less sense,
> kafka has 50-100k open files index/offset/log so decrease amount of
> files/improve cache for them, it can give many improvements for big
> kafka topics
>
> On Fri, 21 Feb 2025 at 16:14, Chia-Ping Tsai <ch...@gmail.com> wrote:
>
>> hi Mikhail
>>
>> > So Increasing the limit to 3–4GB (or more) would better align with
>> today’s storage capabilities.
>> Out of curiosity, what is the benefit of using the larger segment?
>>
>> While changing the type from int to long is a minor change, allowing or
>> encouraging users to create large segments requires stronger
>> justification.
>>
>> thanks,
>> Chia-Ping
>>
>>
>> Mikhail Fesenko <pr...@gmail.com> 於 2025年2月21日 週五 下午10:31寫道：
>>
>> > Guys, any thoughts on that? Please let me know!
>> >
>> > On Thu, 13 Feb 2025 at 22:31, Mikhail Fesenko <pr...@gmail.com>
>> wrote:
>> >
>> > > Hi everyone,
>> > >
>> > > I’d like to propose a change the data type of log.segment.bytes from
>> int
>> > > to long, allowing segment sizes beyond the current 2GB limit.
>> > >
>> > > Rationale:
>> > >
>> > > Currently, the maximum segment size is constrained by the int type,
>> > > capping it at 2GB (max int). However, with modern hardware—such as
>> > > large-scale deployments on machines with multiple high-capacity disks
>> > > (e.g., EPYC servers with 4×15TB drives)—this limitation makes segment
>> > > sizes inefficiently small. And too many handles (log, index,
>> metadata).
>> > > So Increasing the limit to 3–4GB (or more) would better align with
>> > today’s
>> > > storage capabilities.
>> > >
>> > > Would love to hear your thoughts !
>> > >
>> > > Best,
>> > >
>> > > Mikhail Fesenko
>> > >
>> >
>>
>

Re: [DISCUSS] Change log.segment.bytes config type from int to long

[Mikhail Fesenko <pr...@gmail.com>]

Currently we have big amount of topics and big data, and using high disk
storage, so rotate files too fast in case of high traffic means less sense,
kafka has 50-100k open files index/offset/log so decrease amount of
files/improve cache for them, it can give many improvements for big
kafka topics

On Fri, 21 Feb 2025 at 16:14, Chia-Ping Tsai <ch...@gmail.com> wrote:

> hi Mikhail
>
> > So Increasing the limit to 3–4GB (or more) would better align with
> today’s storage capabilities.
> Out of curiosity, what is the benefit of using the larger segment?
>
> While changing the type from int to long is a minor change, allowing or
> encouraging users to create large segments requires stronger justification.
>
> thanks,
> Chia-Ping
>
>
> Mikhail Fesenko <pr...@gmail.com> 於 2025年2月21日 週五 下午10:31寫道：
>
> > Guys, any thoughts on that? Please let me know!
> >
> > On Thu, 13 Feb 2025 at 22:31, Mikhail Fesenko <pr...@gmail.com> wrote:
> >
> > > Hi everyone,
> > >
> > > I’d like to propose a change the data type of log.segment.bytes from
> int
> > > to long, allowing segment sizes beyond the current 2GB limit.
> > >
> > > Rationale:
> > >
> > > Currently, the maximum segment size is constrained by the int type,
> > > capping it at 2GB (max int). However, with modern hardware—such as
> > > large-scale deployments on machines with multiple high-capacity disks
> > > (e.g., EPYC servers with 4×15TB drives)—this limitation makes segment
> > > sizes inefficiently small. And too many handles (log, index, metadata).
> > > So Increasing the limit to 3–4GB (or more) would better align with
> > today’s
> > > storage capabilities.
> > >
> > > Would love to hear your thoughts !
> > >
> > > Best,
> > >
> > > Mikhail Fesenko
> > >
> >
>

Re: [DISCUSS] Change log.segment.bytes config type from int to long

[Chia-Ping Tsai <ch...@gmail.com>]

hi Mikhail

> So Increasing the limit to 3–4GB (or more) would better align with
today’s storage capabilities.
Out of curiosity, what is the benefit of using the larger segment?

While changing the type from int to long is a minor change, allowing or
encouraging users to create large segments requires stronger justification.

thanks,
Chia-Ping


Mikhail Fesenko <pr...@gmail.com> 於 2025年2月21日 週五 下午10:31寫道：

> Guys, any thoughts on that? Please let me know!
>
> On Thu, 13 Feb 2025 at 22:31, Mikhail Fesenko <pr...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > I’d like to propose a change the data type of log.segment.bytes from int
> > to long, allowing segment sizes beyond the current 2GB limit.
> >
> > Rationale:
> >
> > Currently, the maximum segment size is constrained by the int type,
> > capping it at 2GB (max int). However, with modern hardware—such as
> > large-scale deployments on machines with multiple high-capacity disks
> > (e.g., EPYC servers with 4×15TB drives)—this limitation makes segment
> > sizes inefficiently small. And too many handles (log, index, metadata).
> > So Increasing the limit to 3–4GB (or more) would better align with
> today’s
> > storage capabilities.
> >
> > Would love to hear your thoughts !
> >
> > Best,
> >
> > Mikhail Fesenko
> >
>

Re: [DISCUSS] Change log.segment.bytes config type from int to long

[Mikhail Fesenko <pr...@gmail.com>]

Guys, any thoughts on that? Please let me know!

On Thu, 13 Feb 2025 at 22:31, Mikhail Fesenko <pr...@gmail.com> wrote:

> Hi everyone,
>
> I’d like to propose a change the data type of log.segment.bytes from int
> to long, allowing segment sizes beyond the current 2GB limit.
>
> Rationale:
>
> Currently, the maximum segment size is constrained by the int type,
> capping it at 2GB (max int). However, with modern hardware—such as
> large-scale deployments on machines with multiple high-capacity disks
> (e.g., EPYC servers with 4×15TB drives)—this limitation makes segment
> sizes inefficiently small. And too many handles (log, index, metadata).
> So Increasing the limit to 3–4GB (or more) would better align with today’s
> storage capabilities.
>
> Would love to hear your thoughts !
>
> Best,
>
> Mikhail Fesenko
>

[DISCUSS] Change log.segment.bytes config type from int to long

[Mikhail Fesenko <pr...@gmail.com>]

Hi everyone,

I’d like to propose a change the data type of log.segment.bytes from int to
long, allowing segment sizes beyond the current 2GB limit.

Rationale:

Currently, the maximum segment size is constrained by the int type, capping
it at 2GB (max int). However, with modern hardware—such as large-scale
deployments on machines with multiple high-capacity disks (e.g., EPYC
servers with 4×15TB drives)—this limitation makes segment sizes inefficiently
small. And too many handles (log, index, metadata). So Increasing the limit
to 3–4GB (or more) would better align with today’s storage capabilities.

Would love to hear your thoughts !

Best,

Mikhail Fesenko