You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Guozhang Wang <gu...@linkedin.com> on 2014/02/20 03:58:59 UTC
Review Request 18299: Proposed In-place Compression on MemoryRecords
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description
-------
KAFKA-1253
Diffs
-----
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java PRE-CREATION
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
Thanks,
Guozhang Wang
Re: Review Request 18299: Proposed In-place Compression on MemoryRecords
Posted by Guozhang Wang <gu...@linkedin.com>.
> On Feb. 23, 2014, 10:38 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java, lines 101-112
> > <https://reviews.apache.org/r/18299/diff/3/?file=500841#file500841line101>
> >
> > This is actually not the right place to add the offset and the crc for the shallow iterator. This method is probably intended for writing the produce request to the socket without first serializing the request into a separate buffer. However, we don't do that now and this method is not currently being used. On the producer side, when a Records is ready to be sent, we get its underlying ByteBuffer and use it in the ProducerRequest, which then serializes the bytes into a send ByteBuffer.
> >
> > So, we need to add the offset/crc for the shallow message when a Records is ready to be sent. Potentially, we can add a close() method in Record and call it in RecordAccumlator.ready() when a Records is determined to be ready. The offset/crc for the shallow message can be added in close().
Yep, thanks Jun.
> On Feb. 23, 2014, 10:38 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java, lines 149-160
> > <https://reviews.apache.org/r/18299/diff/3/?file=500841#file500841line149>
> >
> > Instead of copying the data to a new ByteBuffer, we can let Record reuse the portion of the original ByteBuffer passed into CompressedRecordsIterator.
Not sure how it can be done: the uncompressed message size is most likely larger than the compressed version, so the recordBuffer's size would most likely be larger than the size we just read out the compressed data.
> On Feb. 23, 2014, 10:38 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Record.java, lines 87-101
> > <https://reviews.apache.org/r/18299/diff/3/?file=500843#file500843line87>
> >
> > This doesn't handle compressed codec.
Yes and that is because this function will only called by the MessageRecords.
- Guozhang
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review35245
-----------------------------------------------------------
On Feb. 22, 2014, 1:57 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated Feb. 22, 2014, 1:57 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> In-place compression with Crc32 pre-computation.
>
> In-place decompression embedded in nested iterator.
>
>
> Diffs
> -----
>
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 9c34e7dc82f33df7406cad0e64eb6a896d068dc6
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Proposed In-place Compression on MemoryRecords
Posted by Jun Rao <ju...@gmail.com>.
> On Feb. 23, 2014, 10:38 p.m., Jun Rao wrote:
> > A couple of high level comments:
> >
> > 1. RecordBatch: If compression is enabled in the producer, each Records will be of CompressedMemoryRecord. The tricky thing will be in tryAppend(). In the corner case, it could happen that the compressed message is bigger than the uncompressed version. So, records.append() would fail even though records.hasRoom() returns true. When this happens, ByteBuffer.put() throws BufferOverflowException, which is RuntimeException. So, in this corner case, we need to catch this exception and treat it as if there is no more room.
> >
> > 2. RecordAccumulator.append(): There is another tricky case in calculating the size when creating a new RecordBatch. Again, in the corner case, it could happen that the compressed message is bigger than the uncompressed version. So, in the rare case, we may not be able to put in a single compressed record into a newly allocated RecordBatch. Not sure what's the best way to handle this. We could (1) just send this record as uncompressed (since compressing doesn't really help); or (2) try to keep allocating a bigger buffer until the record can be put in.
> >
> >
Another thing is that if we do hit BufferOverflowException while calling records.append(), we probably need to rewind the underlying ByteBuffer to remove the partial writes.
- Jun
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review35245
-----------------------------------------------------------
On Feb. 22, 2014, 1:57 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated Feb. 22, 2014, 1:57 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> In-place compression with Crc32 pre-computation.
>
> In-place decompression embedded in nested iterator.
>
>
> Diffs
> -----
>
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 9c34e7dc82f33df7406cad0e64eb6a896d068dc6
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Proposed In-place Compression on MemoryRecords
Posted by Jun Rao <ju...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review35245
-----------------------------------------------------------
A couple of high level comments:
1. RecordBatch: If compression is enabled in the producer, each Records will be of CompressedMemoryRecord. The tricky thing will be in tryAppend(). In the corner case, it could happen that the compressed message is bigger than the uncompressed version. So, records.append() would fail even though records.hasRoom() returns true. When this happens, ByteBuffer.put() throws BufferOverflowException, which is RuntimeException. So, in this corner case, we need to catch this exception and treat it as if there is no more room.
2. RecordAccumulator.append(): There is another tricky case in calculating the size when creating a new RecordBatch. Again, in the corner case, it could happen that the compressed message is bigger than the uncompressed version. So, in the rare case, we may not be able to put in a single compressed record into a newly allocated RecordBatch. Not sure what's the best way to handle this. We could (1) just send this record as uncompressed (since compressing doesn't really help); or (2) try to keep allocating a bigger buffer until the record can be put in.
clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java
<https://reviews.apache.org/r/18299/#comment65699>
unused import.
clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java
<https://reviews.apache.org/r/18299/#comment65698>
This method should be private.
clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java
<https://reviews.apache.org/r/18299/#comment65704>
This is actually not the right place to add the offset and the crc for the shallow iterator. This method is probably intended for writing the produce request to the socket without first serializing the request into a separate buffer. However, we don't do that now and this method is not currently being used. On the producer side, when a Records is ready to be sent, we get its underlying ByteBuffer and use it in the ProducerRequest, which then serializes the bytes into a send ByteBuffer.
So, we need to add the offset/crc for the shallow message when a Records is ready to be sent. Potentially, we can add a close() method in Record and call it in RecordAccumlator.ready() when a Records is determined to be ready. The offset/crc for the shallow message can be added in close().
clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java
<https://reviews.apache.org/r/18299/#comment65705>
Instead of copying the data to a new ByteBuffer, we can let Record reuse the portion of the original ByteBuffer passed into CompressedRecordsIterator.
clients/src/main/java/org/apache/kafka/common/record/Record.java
<https://reviews.apache.org/r/18299/#comment65702>
This doesn't handle compressed codec.
- Jun Rao
On Feb. 22, 2014, 1:57 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated Feb. 22, 2014, 1:57 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> In-place compression with Crc32 pre-computation.
>
> In-place decompression embedded in nested iterator.
>
>
> Diffs
> -----
>
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 9c34e7dc82f33df7406cad0e64eb6a896d068dc6
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Proposed In-place Compression on MemoryRecords
Posted by Jun Rao <ju...@gmail.com>.
> On March 2, 2014, 2:15 a.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java, lines 151-165
> > <https://reviews.apache.org/r/18299/diff/5/?file=505393#file505393line151>
> >
> > Shouldn't we set the limit for buffer to pos at the end so that RecordAccumulator.ready() can pick up this MemoryRecords after closed?
Actually, just realized that this is ok.
- Jun
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review35908
-----------------------------------------------------------
On Feb. 27, 2014, 1:33 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated Feb. 27, 2014, 1:33 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Incorporated Jun's comments.
>
> TODOs:
>
> Class loader for Snappy.
>
>
> Diffs
> -----
>
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java e4bc97279585818860487a39a93b6481742b91db
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java d8e35e7d0e4cd27aad9a8d4bf14bc97458da9417
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java ce5cf27efa08b79e501439cf79bc8666054a5429
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java eb16f6d236e07b16654623606294a051531b5f58
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 9c34e7dc82f33df7406cad0e64eb6a896d068dc6
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 34baa8c6c7a15bb4aa93c286604f0eb7b19cd58e
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Proposed In-place Compression on MemoryRecords
Posted by Guozhang Wang <gu...@linkedin.com>.
> On March 2, 2014, 2:15 a.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java, lines 155-157
> > <https://reviews.apache.org/r/18299/diff/5/?file=505388#file505388line155>
> >
> > My earlier comment about whether we should close the RecordBatch immediately after this record is appended, is not due to synchronization. My concern is that if we don't close this RecordBatch, the next message could be added as an uncompressed one when it can be added as a compressed one. Not sure if it's a big concern though.
Got it. Agreed.
> On March 2, 2014, 2:15 a.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java, lines 62-63
> > <https://reviews.apache.org/r/18299/diff/5/?file=505393#file505393line62>
> >
> > What is the TODO item?
What I am trying to declare is that this hand-written logic is dependent on the fact that compressed shallow message has key==null. If it is changed in the future this logic will break. Probably I should just leave it as normal comments?
> On March 2, 2014, 2:15 a.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Record.java, lines 156-158
> > <https://reviews.apache.org/r/18299/diff/5/?file=505394#file505394line156>
> >
> > ???
Sorry, placeholder function that should be removed..
> On March 2, 2014, 2:15 a.m., Jun Rao wrote:
> > clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java, lines 84-85
> > <https://reviews.apache.org/r/18299/diff/5/?file=505396#file505396line84>
> >
> > Line 81 not needed?
Should be removed..
- Guozhang
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review35908
-----------------------------------------------------------
On Feb. 27, 2014, 1:33 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated Feb. 27, 2014, 1:33 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Incorporated Jun's comments.
>
> TODOs:
>
> Class loader for Snappy.
>
>
> Diffs
> -----
>
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java e4bc97279585818860487a39a93b6481742b91db
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java d8e35e7d0e4cd27aad9a8d4bf14bc97458da9417
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java ce5cf27efa08b79e501439cf79bc8666054a5429
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java eb16f6d236e07b16654623606294a051531b5f58
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 9c34e7dc82f33df7406cad0e64eb6a896d068dc6
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 34baa8c6c7a15bb4aa93c286604f0eb7b19cd58e
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Proposed In-place Compression on MemoryRecords
Posted by Jun Rao <ju...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review35908
-----------------------------------------------------------
Could you rebase?
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
<https://reviews.apache.org/r/18299/#comment66722>
Could you add the logging?
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
<https://reviews.apache.org/r/18299/#comment66723>
My earlier comment about whether we should close the RecordBatch immediately after this record is appended, is not due to synchronization. My concern is that if we don't close this RecordBatch, the next message could be added as an uncompressed one when it can be added as a compressed one. Not sure if it's a big concern though.
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java
<https://reviews.apache.org/r/18299/#comment66724>
Could we just use Class.forName()? Could we also add a comment on why we want to load the class dynamically?
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment66726>
What is the TODO item?
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment66728>
BufferOverflowException is a RuntimeException, not an IOException.
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment66727>
Could you add the logging?
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment66730>
Should we call maybeInitForAppend() or assert that appendStream is not null?
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment66732>
Shouldn't we set the limit for buffer to pos at the end so that RecordAccumulator.ready() can pick up this MemoryRecords after closed?
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment66731>
What's the TODO item? The key for the shallow message is always null.
clients/src/main/java/org/apache/kafka/common/record/Record.java
<https://reviews.apache.org/r/18299/#comment66725>
???
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java
<https://reviews.apache.org/r/18299/#comment66733>
Line 81 not needed?
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala
<https://reviews.apache.org/r/18299/#comment66734>
Could we verify the content of the fetched messages too?
- Jun Rao
On Feb. 27, 2014, 1:33 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated Feb. 27, 2014, 1:33 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Incorporated Jun's comments.
>
> TODOs:
>
> Class loader for Snappy.
>
>
> Diffs
> -----
>
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java e4bc97279585818860487a39a93b6481742b91db
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java d8e35e7d0e4cd27aad9a8d4bf14bc97458da9417
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java ce5cf27efa08b79e501439cf79bc8666054a5429
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java eb16f6d236e07b16654623606294a051531b5f58
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 9c34e7dc82f33df7406cad0e64eb6a896d068dc6
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 34baa8c6c7a15bb4aa93c286604f0eb7b19cd58e
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Proposed In-place Compression on MemoryRecords
Posted by Jun Rao <ju...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review35910
-----------------------------------------------------------
Could we add a unit test that covers the case that the compressed message size is larger than the batch size?
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java
<https://reviews.apache.org/r/18299/#comment66735>
Could we add a test for snappy too?
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala
<https://reviews.apache.org/r/18299/#comment66736>
Could we add a test for snappy too?
- Jun Rao
On Feb. 27, 2014, 1:33 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated Feb. 27, 2014, 1:33 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Incorporated Jun's comments.
>
> TODOs:
>
> Class loader for Snappy.
>
>
> Diffs
> -----
>
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java e4bc97279585818860487a39a93b6481742b91db
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java d8e35e7d0e4cd27aad9a8d4bf14bc97458da9417
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java ce5cf27efa08b79e501439cf79bc8666054a5429
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java eb16f6d236e07b16654623606294a051531b5f58
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 9c34e7dc82f33df7406cad0e64eb6a896d068dc6
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 34baa8c6c7a15bb4aa93c286604f0eb7b19cd58e
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
> On March 7, 2014, 5:58 p.m., Jun Rao wrote:
> > build.gradle, lines 40-71
> > <https://reviews.apache.org/r/18299/diff/6/?file=513281#file513281line40>
> >
> > This should be uncommented.
The un-commentating missed my commit last time, will fix.
> On March 7, 2014, 5:58 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Compressor.java, lines 98-99
> > <https://reviews.apache.org/r/18299/diff/6/?file=513289#file513289line98>
> >
> > BufferOverflowException is not an IOException. It's a RuntimeException. Is that handled properly?
>
> Guozhang Wang wrote:
> In my tests I observed that when writes out of the boundary IOException can be thrown, but I did not observe BufferOverflowException thrown.
I re-test it with batch size = 1 and it does throw BufferOverflowException now. I think you are right.
> On March 7, 2014, 5:58 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Compressor.java, lines 125-126
> > <https://reviews.apache.org/r/18299/diff/6/?file=513289#file513289line125>
> >
> > Do we need to log the exception? Why is this not consistent with the logging in putInt()?
I tried to centralize the exception handling using callable parameters, but gave up eventually finding out they do not help much in reducing LOC but increased extra cost of creating a new callable object each time.
> On March 7, 2014, 5:58 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Record.java, lines 87-91
> > <https://reviews.apache.org/r/18299/diff/6/?file=513291#file513291line87>
> >
> > Since we always interpret the bytes as uncompressed, do we need to pass in the codec in the constructor?
You are right, like I said in comments Record construction should always write a record as un-compressed, i.e. a record can either be 1) un-compressed or 2) indicated compressed in the attribute and payload wrapping a batch of messages. But in some unit tests in core some times a Message with compression codec in attributes but the payload content are actually un-compressed is generated, and which would use this function. The new Record function migrated from the Message class in core and inherited this "property". I think we can just leave it here for compatibility.
- Guozhang
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review36525
-----------------------------------------------------------
On March 7, 2014, 1:49 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 7, 2014, 1:49 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Refactored Compression logic into Compressor
>
> GZIP/Snappy Integration Tests
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java f8748960fe9a02d52f7caae675abec503df91a39
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 616e1006f21c54af8260e84a96928cb8893ceb7c
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
> On March 7, 2014, 5:58 p.m., Jun Rao wrote:
> > Does the unit test pass? It seems to hang on ProducerFailureHandlingTest.
It does pass on my machine. Is the handing issue persistent or it is transient?
> On March 7, 2014, 5:58 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java, lines 158-159
> > <https://reviews.apache.org/r/18299/diff/6/?file=513284#file513284line158>
> >
> > Could this be part of the constructor of MemoryRecords?
I use init() here for lazy construction of the stream since it may be only used for read (on the consumer side).
> On March 7, 2014, 5:58 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Compressor.java, lines 98-99
> > <https://reviews.apache.org/r/18299/diff/6/?file=513289#file513289line98>
> >
> > BufferOverflowException is not an IOException. It's a RuntimeException. Is that handled properly?
In my tests I observed that when writes out of the boundary IOException can be thrown, but I did not observe BufferOverflowException thrown.
> On March 7, 2014, 5:58 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java, lines 73-75
> > <https://reviews.apache.org/r/18299/diff/6/?file=513290#file513290line73>
> >
> > Throwing exception is a bit expensive. We probably should avoid writing to the compressor after the first failure.
If the first part of && is false, the second part will not be executed I thought?
- Guozhang
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review36525
-----------------------------------------------------------
On March 7, 2014, 1:49 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 7, 2014, 1:49 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Refactored Compression logic into Compressor
>
> GZIP/Snappy Integration Tests
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java f8748960fe9a02d52f7caae675abec503df91a39
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 616e1006f21c54af8260e84a96928cb8893ceb7c
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Jun Rao <ju...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review36525
-----------------------------------------------------------
Does the unit test pass? It seems to hang on ProducerFailureHandlingTest.
build.gradle
<https://reviews.apache.org/r/18299/#comment67519>
This should be uncommented.
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
<https://reviews.apache.org/r/18299/#comment67521>
Why is this in error?
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
<https://reviews.apache.org/r/18299/#comment67524>
Could this be part of the constructor of MemoryRecords?
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
<https://reviews.apache.org/r/18299/#comment67523>
This probably should be warning.
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
<https://reviews.apache.org/r/18299/#comment67534>
Could this be part of the constructor of MemoryRecords?
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment67529>
BufferOverflowException is not an IOException. It's a RuntimeException. Is that handled properly?
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment67525>
Since this is just a warning, we probably shouldn't include error in the message.
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment67526>
Ditto as the above.
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment67527>
Do we need to log the exception? Why is this not consistent with the logging in putInt()?
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment67530>
Ditto as the above.
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment67533>
Could this be part of the constructor?
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment67535>
Throwing exception is a bit expensive. We probably should avoid writing to the compressor after the first failure.
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment67537>
This check seems unnecessary. If there are fewer bytes in the stream than size, a BufferUnderFlowException will be thrown.
clients/src/main/java/org/apache/kafka/common/record/Record.java
<https://reviews.apache.org/r/18299/#comment67541>
Since we always interpret the bytes as uncompressed, do we need to pass in the codec in the constructor?
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala
<https://reviews.apache.org/r/18299/#comment67544>
I am wondering if it's better to break the test into two. In the first test, we just test the normal case with compression. We can probably just write one parameterized test (supported in junit) and bind it with gzip and snappy. In the second test, we test the overflow case.
- Jun Rao
On March 7, 2014, 1:49 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 7, 2014, 1:49 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Refactored Compression logic into Compressor
>
> GZIP/Snappy Integration Tests
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java f8748960fe9a02d52f7caae675abec503df91a39
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 616e1006f21c54af8260e84a96928cb8893ceb7c
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
> On March 10, 2014, 5:23 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Compressor.java, lines 100-101
> > <https://reviews.apache.org/r/18299/diff/7/?file=514016#file514016line100>
> >
> > Would it be better to write the code as:
> >
> > catch (Exception e) {
> > if (e instanceof IOException || e instanceof IOException) {
> > // do sth
> > }
> >
> > This avoids duplicating the code and we can just say I/O exception in the log.
I would personally prefer the current code layout since 1) IOException and BufferOverflowException are actually for different cases, the first one is not expected while the other is; we handle IOException here only to not expose them in MemoryRecords; 2) If we catch Exception, then for "else" we'd better re-throw them, which will make the function having to be declared as throws Exceptions.
> On March 10, 2014, 5:23 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Compressor.java, lines 111-112
> > <https://reviews.apache.org/r/18299/diff/7/?file=514016#file514016line111>
> >
> > Do we need the input to be final?
Not necessarily, I put them here just to make sure they are not altered at any cases. Do you think there is a reason we should better remove them?
> On March 10, 2014, 5:23 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java, lines 50-53
> > <https://reviews.apache.org/r/18299/diff/7/?file=514017#file514017line50>
> >
> > If we have a separate init(), every caller has to remember calling this after the construction of MemoryRecord. Is this worth doing lazy init for compressor? If it's really worth doing, perhaps we can do the lazy init in each of the func (e.g. append) that needs the compressor.
I think it is necessary to do lazy init, but just efficient. Since while doing the init for compression the header will be re-written, which we do not want while reading a memory records. I can move the init() to Compressor and check that every time though.
> On March 10, 2014, 5:23 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java, lines 178-179
> > <https://reviews.apache.org/r/18299/diff/7/?file=514017#file514017line178>
> >
> > Do we need to wrap it or just let it pass through?
If this is the only place that could throw BufferUnderflowException in consumers/mirrormakers, then probably we do not need to wrap it; otherwise I would prefer to indicate where this exception is thrown. How about keeping it here for now and remove it later when we realize that this is really not necessary?
> On March 10, 2014, 5:23 p.m., Jun Rao wrote:
> > core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala, lines 145-146
> > <https://reviews.apache.org/r/18299/diff/7/?file=514024#file514024line145>
> >
> > Do we actually hit the Overflow exception with this partition size?
I have confirmed this in my tests with trace logging.
> On March 10, 2014, 5:23 p.m., Jun Rao wrote:
> > core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala, lines 119-121
> > <https://reviews.apache.org/r/18299/diff/7/?file=514024#file514024line119>
> >
> > Could we also verify the content of the fetched messages?
Ack.
> On March 10, 2014, 5:23 p.m., Jun Rao wrote:
> > core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala, lines 170-171
> > <https://reviews.apache.org/r/18299/diff/7/?file=514024#file514024line170>
> >
> > Could we also verify the content of the fetched messages?
Ack.
> On March 10, 2014, 5:23 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Record.java, lines 91-94
> > <https://reviews.apache.org/r/18299/diff/7/?file=514018#file514018line91>
> >
> > It's still weird to always write the data as uncompressed. In RecordTest, the compressed type is not really being exercised since the record is created as uncompressed. Perhaps we should just change the unit test. If the compression type is not none, we will have to decompress the value to get the uncompressed content.
Added the comments for clearer explanation.
- Guozhang
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review36658
-----------------------------------------------------------
On March 8, 2014, 12:36 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 8, 2014, 12:36 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Incorporate Jun's comments round two
>
> Refactored compression logic into Compressor
>
> GZIP/Snappy integration tests
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 616e1006f21c54af8260e84a96928cb8893ceb7c
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Jun Rao <ju...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review36658
-----------------------------------------------------------
Could we add the new compression property to existing tools like consoleProducer and ProducerPerformance that support compressed input?
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment67707>
Could the init be part of the constructor?
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment67691>
Would it be better to write the code as:
catch (Exception e) {
if (e instanceof IOException || e instanceof IOException) {
// do sth
}
This avoids duplicating the code and we can just say I/O exception in the log.
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment67692>
Do we need the input to be final?
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment67703>
If we have a separate init(), every caller has to remember calling this after the construction of MemoryRecord. Is this worth doing lazy init for compressor? If it's really worth doing, perhaps we can do the lazy init in each of the func (e.g. append) that needs the compressor.
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment67696>
Do we need to wrap it or just let it pass through?
clients/src/main/java/org/apache/kafka/common/record/Record.java
<https://reviews.apache.org/r/18299/#comment67708>
It's still weird to always write the data as uncompressed. In RecordTest, the compressed type is not really being exercised since the record is created as uncompressed. Perhaps we should just change the unit test. If the compression type is not none, we will have to decompress the value to get the uncompressed content.
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala
<https://reviews.apache.org/r/18299/#comment67699>
Could we also verify the content of the fetched messages?
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala
<https://reviews.apache.org/r/18299/#comment67701>
Do we actually hit the Overflow exception with this partition size?
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala
<https://reviews.apache.org/r/18299/#comment67702>
Could we also verify the content of the fetched messages?
- Jun Rao
On March 8, 2014, 12:36 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 8, 2014, 12:36 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Incorporate Jun's comments round two
>
> Refactored compression logic into Compressor
>
> GZIP/Snappy integration tests
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 616e1006f21c54af8260e84a96928cb8893ceb7c
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Jay Kreps <bo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review36732
-----------------------------------------------------------
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
<https://reviews.apache.org/r/18299/#comment67843>
I don't see why we need this. We always take the max of the batch size and the base data size, no?
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
<https://reviews.apache.org/r/18299/#comment67842>
This logic should happen inside tryAppend, no?
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
<https://reviews.apache.org/r/18299/#comment67840>
This is a check method, we can't change the state.
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java
<https://reviews.apache.org/r/18299/#comment67844>
This seems like it doesn't belong in the enum.
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java
<https://reviews.apache.org/r/18299/#comment67847>
I'm concerned about wrapping with DataOutputStream as it adds a layer of synchronization.
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java
<https://reviews.apache.org/r/18299/#comment67845>
Do we really need to do this? I think just using new should be okay, no?
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java
<https://reviews.apache.org/r/18299/#comment67846>
This also seems like it doesn't belong in the enum.
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment67852>
Why is this lazy?
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment67849>
These interfaces are pretty gross, is there no other way to do this?
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment67851>
If this is expected don't log it as warn log it as debug. WARN==user should do something different in their code.
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment67854>
Code style: please make two lines:
if (written)
compressor.written()
clients/src/main/java/org/apache/kafka/common/utils/Utils.java
<https://reviews.apache.org/r/18299/#comment67856>
Why isn't this part of the CRC class?
- Jay Kreps
On March 10, 2014, 10:28 p.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 10, 2014, 10:28 p.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Incorporate Jun's comments round three
>
> Refactored compression logic into Compressor
>
> GZIP/Snappy integration tests
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 616e1006f21c54af8260e84a96928cb8893ceb7c
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala f061dbabb49044f8fe94b8fd7dc0153c33bedeee
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review37302
-----------------------------------------------------------
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
<https://reviews.apache.org/r/18299/#comment68747>
I changed the deallocation function to be one-at-a-time since currently no callers give it a list of buffers at a time. If we are not going to make BufferPool out of producer I think this change is OK.
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java
<https://reviews.apache.org/r/18299/#comment68748>
I doubled the space in re-allocation, but we can be also conservative.
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java
<https://reviews.apache.org/r/18299/#comment68749>
Personally I do not like this to be in CompressionType, but moving the init-rate to CompressionFactory also requires some awkwardness. Any suggestions are welcomed.
clients/src/main/java/org/apache/kafka/common/utils/Utils.java
<https://reviews.apache.org/r/18299/#comment68750>
This function may not be best here, but I do not really have a better place to locate it.
- Guozhang Wang
On March 14, 2014, 8:54 p.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 14, 2014, 8:54 p.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java 5bed60730ea108684bea2440af5a008feb0ada61
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 7a03f389cc6a4d56152b882555d7498af9a04d20
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> config/tools-log4j.properties 52f07c96019b4083fc78f62cfb0a81080327e847
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala f12a45becb11a8bed586024866235b268630fec6
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
> On March 18, 2014, 7:07 p.m., Jun Rao wrote:
> > 1. There are two unit test failures.
> >
> > RecordAccumulatorTest. testFull
> > RecordAccumulatorTest. testLinger
> >
> > 2. The issue with compressed message size exceeding the request size is still not resolved.
Hi Jun,
This patch is not final yet, I am still doing some debugging for issues found in the system test and perf tests. Could you take another look once I uploaded the final version? This version is just for bookkeeping, sorry for the confusion.
- Guozhang
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review37585
-----------------------------------------------------------
On March 17, 2014, 10:56 p.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 17, 2014, 10:56 p.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java 5bed60730ea108684bea2440af5a008feb0ada61
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 7a03f389cc6a4d56152b882555d7498af9a04d20
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala f12a45becb11a8bed586024866235b268630fec6
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
> On March 18, 2014, 7:07 p.m., Jun Rao wrote:
> > 1. There are two unit test failures.
> >
> > RecordAccumulatorTest. testFull
> > RecordAccumulatorTest. testLinger
> >
> > 2. The issue with compressed message size exceeding the request size is still not resolved.
>
> Guozhang Wang wrote:
> Hi Jun,
>
> This patch is not final yet, I am still doing some debugging for issues found in the system test and perf tests. Could you take another look once I uploaded the final version? This version is just for bookkeeping, sorry for the confusion.
These two tests do not fail any more in my latest patch, maybe related to some bugs I found during system tests.
When a single batch is larger than the max request size, will drop it on the floor and log the error.
> On March 18, 2014, 7:07 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Compressor.java, lines 169-170
> > <https://reviews.apache.org/r/18299/diff/13/?file=525728#file525728line169>
> >
> > If we want to be a bit conservative, shouldn't we take the max of the two estimates?
According to the perf test, the min is already quite effective in reducing reallocation possibilities; on the other hand, more conservative means larger memory waste.
> On March 18, 2014, 7:07 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java, lines 78-79
> > <https://reviews.apache.org/r/18299/diff/13/?file=525729#file525729line78>
> >
> > Why do we need to mark this position?
It is not ByteBuffer.mark, rather Compressor.mark() that tells the compressor that one record with the size has been written. I will change the name to avoid confusion.
> On March 18, 2014, 7:07 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java, lines 172-173
> > <https://reviews.apache.org/r/18299/diff/13/?file=525729#file525729line172>
> >
> > This comment is still not clear to me.
Reworded again.
> On March 18, 2014, 7:07 p.m., Jun Rao wrote:
> > core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala, lines 40-43
> > <https://reviews.apache.org/r/18299/diff/13/?file=525737#file525737line40>
> >
> > Not sure how this test works. We send messages randomly to one of the four partitions, yet we only read data from partition 0. Do we need to have 4 partitions in the topic? Do we need more than 1 broker?
When the topic is created, only one partition is used. That is why this test works, but I agree that we do not actually needs 4 partitions and 2 servers. Will change that.
- Guozhang
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review37585
-----------------------------------------------------------
On March 17, 2014, 10:56 p.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 17, 2014, 10:56 p.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java 5bed60730ea108684bea2440af5a008feb0ada61
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 7a03f389cc6a4d56152b882555d7498af9a04d20
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala f12a45becb11a8bed586024866235b268630fec6
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Jun Rao <ju...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review37585
-----------------------------------------------------------
1. There are two unit test failures.
RecordAccumulatorTest. testFull
RecordAccumulatorTest. testLinger
2. The issue with compressed message size exceeding the request size is still not resolved.
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java
<https://reviews.apache.org/r/18299/#comment69180>
Could we write that as a private util function expandBuffer()?
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment69219>
Should we add a debug level logging for those unexpected exceptions?
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment69189>
If we want to be a bit conservative, shouldn't we take the max of the two estimates?
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment69210>
Why do we need to mark this position?
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment69211>
Ditto as the above?
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment69198>
Shouldn't >= be <= ?
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment69212>
This comment is still not clear to me.
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala
<https://reviews.apache.org/r/18299/#comment69216>
Not sure how this test works. We send messages randomly to one of the four partitions, yet we only read data from partition 0. Do we need to have 4 partitions in the topic? Do we need more than 1 broker?
- Jun Rao
On March 17, 2014, 10:56 p.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 17, 2014, 10:56 p.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java 5bed60730ea108684bea2440af5a008feb0ada61
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 7a03f389cc6a4d56152b882555d7498af9a04d20
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala f12a45becb11a8bed586024866235b268630fec6
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
> On March 24, 2014, 11:50 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java, lines 246-251
> > <https://reviews.apache.org/r/18299/diff/16/?file=533373#file533373line246>
> >
> > We can't just drop those records on the floor since we need to inform the client about the failure. Perhaps we could just form a request with a single RecordBatch and send it. The request will fail and the client will be notified.
I agree, and in the long run we probably should back-off the batch size.
I think that before we eventually fix this issue, currently we would probably just log it when this issue happens and move on. Throwing an runtime exception is generally doing the same as it does not necessarily stop the producer, but throwing it has the side-effect of undefined behavior of this sender iteration. So I decided to just log it and let the producer proceed. Let me know if you have any preferences.
> On March 24, 2014, 11:50 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Compressor.java, lines 187-188
> > <https://reviews.apache.org/r/18299/diff/16/?file=533379#file533379line187>
> >
> > So, we are switching to a static estimated compression rate? Will that be effective in terms of reducing re-allocation rate?
We still update the compression rate in close().
- Guozhang
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review38386
-----------------------------------------------------------
On March 23, 2014, 12:55 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 23, 2014, 12:55 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Incorporated Jay's comments
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Crc32.java 153c5a6d345293aa0ba2cf513373323a6e9f2467
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> shallow cluster tests with rolling bounces
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Jun Rao <ju...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review38386
-----------------------------------------------------------
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
<https://reviews.apache.org/r/18299/#comment70483>
We can't just drop those records on the floor since we need to inform the client about the failure. Perhaps we could just form a request with a single RecordBatch and send it. The request will fail and the client will be notified.
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment70511>
So, we are switching to a static estimated compression rate? Will that be effective in terms of reducing re-allocation rate?
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala
<https://reviews.apache.org/r/18299/#comment70505>
Is this still valid?
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala
<https://reviews.apache.org/r/18299/#comment70508>
Indentation.
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala
<https://reviews.apache.org/r/18299/#comment70509>
We probably can put "value" + i in a separate list and reuse.
- Jun Rao
On March 23, 2014, 12:55 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 23, 2014, 12:55 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Incorporated Jay's comments
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Crc32.java 153c5a6d345293aa0ba2cf513373323a6e9f2467
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> shallow cluster tests with rolling bounces
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Jay Kreps <bo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review38500
-----------------------------------------------------------
1. We need to fix the RecordAccumulator request size issue--that limit need not be a hard limit. I recommend we just change drain to not block if the request size exceeds the max and only one batch is selected.
2. Compressor: Can we fix the public/private/package annotations on the methods? Also is there a better name than wrapForOutput?
3. Can we do a before and after performance run on trunk with and w/o this patch using 100 byte messages against localhost?
- Jay Kreps
On March 23, 2014, 12:55 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 23, 2014, 12:55 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Incorporated Jay's comments
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Crc32.java 153c5a6d345293aa0ba2cf513373323a6e9f2467
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> shallow cluster tests with rolling bounces
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Jun Rao <ju...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review38694
-----------------------------------------------------------
Ship it!
+1.
config/log4j.properties
<https://reviews.apache.org/r/18299/#comment70991>
This probably should be uncommented?
- Jun Rao
On March 26, 2014, 5:59 p.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 26, 2014, 5:59 p.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Jay's minor comment
>
>
> Dummy
>
>
> Dummy
>
>
> Incorporate Jun and Jay's comments
>
>
> Minor change in ConsoleProducer
>
>
> Add estimation conservativeness factor
>
>
> Dummy
>
>
> Incorporate Jay and Timothy's comments
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Incorporate Jun's comments round four
>
>
> Dummy
>
>
> KAFKA-1253.v5
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Small fix on ProducerPerformance
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Incorporate Jun's comments one
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy
>
>
> Dummy 2
>
>
> Dummy
>
>
> KAFKA-1253
>
>
> Diffs
> -----
>
> build.gradle d6fd28758c92e0dce3efcf10ab2c540e8064650f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Crc32.java 153c5a6d345293aa0ba2cf513373323a6e9f2467
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> config/log4j.properties baa698b64259b18efed372cf009a04613b6c53cf
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> local system tests
>
> shallow cluster tests with rolling bounces
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 26, 2014, 5:59 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Jay's minor comment
Dummy
Dummy
Incorporate Jun and Jay's comments
Minor change in ConsoleProducer
Add estimation conservativeness factor
Dummy
Incorporate Jay and Timothy's comments
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Incorporate Jun's comments round four
Dummy
KAFKA-1253.v5
Dummy
Dummy
Dummy
Small fix on ProducerPerformance
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Incorporate Jun's comments one
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy 2
Dummy
KAFKA-1253
Diffs (updated)
-----
build.gradle d6fd28758c92e0dce3efcf10ab2c540e8064650f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Crc32.java 153c5a6d345293aa0ba2cf513373323a6e9f2467
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
config/log4j.properties baa698b64259b18efed372cf009a04613b6c53cf
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
local system tests
shallow cluster tests with rolling bounces
integration tests
unit tests
stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Jay Kreps <bo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review38558
-----------------------------------------------------------
I'm +1 on this with the one minor cleanup. Jun, let me know if you have no further suggestions and I will change that statement and apply.
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
<https://reviews.apache.org/r/18299/#comment70820>
Can you change the condition to be !ready.isEmpty() rather than using size...I think that is less cryptic.
Also the comment is wrong, we will not in general expect an error unless this happens to be larger than the servers maximum request size which is an unrelated parameter.
- Jay Kreps
On March 25, 2014, 8:51 p.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 25, 2014, 8:51 p.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Incorporated Jun and Jay's comments
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle d6fd28758c92e0dce3efcf10ab2c540e8064650f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Crc32.java 153c5a6d345293aa0ba2cf513373323a6e9f2467
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> config/log4j.properties baa698b64259b18efed372cf009a04613b6c53cf
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> local system tests
>
> shallow cluster tests with rolling bounces
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 25, 2014, 8:51 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Incorporated Jun and Jay's comments
In-place compression with
1) Dynamic reallocation in the underlying byte buffer
2) Written bytes estimate to reduce reallocation probabilities
3) Deallocation in buffer pool following the original capacity
Diffs
-----
build.gradle d6fd28758c92e0dce3efcf10ab2c540e8064650f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Crc32.java 153c5a6d345293aa0ba2cf513373323a6e9f2467
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
config/log4j.properties baa698b64259b18efed372cf009a04613b6c53cf
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
Diff: https://reviews.apache.org/r/18299/diff/
Testing (updated)
-------
local system tests
shallow cluster tests with rolling bounces
integration tests
unit tests
stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 25, 2014, 8:48 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Dummy
Dummy
Incorporate Jun and Jay's comments
Minor change in ConsoleProducer
Add estimation conservativeness factor
Dummy
Incorporate Jay and Timothy's comments
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Incorporate Jun's comments round four
Dummy
KAFKA-1253.v5
Dummy
Dummy
Dummy
Small fix on ProducerPerformance
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Incorporate Jun's comments one
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy 2
Dummy
KAFKA-1253
Diffs (updated)
-----
build.gradle d6fd28758c92e0dce3efcf10ab2c540e8064650f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Crc32.java 153c5a6d345293aa0ba2cf513373323a6e9f2467
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
config/log4j.properties baa698b64259b18efed372cf009a04613b6c53cf
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
shallow cluster tests with rolling bounces
integration tests
unit tests
stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 23, 2014, 12:55 a.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Incorporated Jay's comments
In-place compression with
1) Dynamic reallocation in the underlying byte buffer
2) Written bytes estimate to reduce reallocation probabilities
3) Deallocation in buffer pool following the original capacity
Diffs
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Crc32.java 153c5a6d345293aa0ba2cf513373323a6e9f2467
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
Diff: https://reviews.apache.org/r/18299/diff/
Testing (updated)
-------
shallow cluster tests with rolling bounces
integration tests
unit tests
stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 23, 2014, 12:53 a.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Minor change in ConsoleProducer
Add estimation conservativeness factor
Dummy
Incorporate Jay and Timothy's comments
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Incorporate Jun's comments round four
Dummy
KAFKA-1253.v5
Dummy
Dummy
Dummy
Small fix on ProducerPerformance
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Incorporate Jun's comments one
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy 2
Dummy
KAFKA-1253
Diffs (updated)
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Crc32.java 153c5a6d345293aa0ba2cf513373323a6e9f2467
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
unit tests
stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Timothy Chen <tn...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review37837
-----------------------------------------------------------
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java
<https://reviews.apache.org/r/18299/#comment69589>
Perhaps only provide a getter to avoid the stream state getting modified.
- Timothy Chen
On March 19, 2014, 11:32 p.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 19, 2014, 11:32 p.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Incorporated Jay's comments
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Crc32.java 153c5a6d345293aa0ba2cf513373323a6e9f2467
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Jay Kreps <bo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review37846
-----------------------------------------------------------
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment69605>
Maybe check the static/public stuff and see if we need it and if it can be private...
- Jay Kreps
On March 19, 2014, 11:32 p.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 19, 2014, 11:32 p.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Incorporated Jay's comments
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Crc32.java 153c5a6d345293aa0ba2cf513373323a6e9f2467
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Jay Kreps <bo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review37838
-----------------------------------------------------------
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment69596>
Can you move your member variables to the top of the file?
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment69595>
Can you move your constructors to the top of the file?
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment69594>
The compressor instance is single-threaded and per records instance, right? So can we just allocate an 8 byte array as a member variable and use that for all the writes?
- Jay Kreps
On March 19, 2014, 11:32 p.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 19, 2014, 11:32 p.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Incorporated Jay's comments
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Crc32.java 153c5a6d345293aa0ba2cf513373323a6e9f2467
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 19, 2014, 11:32 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Incorporated Jay's comments
In-place compression with
1) Dynamic reallocation in the underlying byte buffer
2) Written bytes estimate to reduce reallocation probabilities
3) Deallocation in buffer pool following the original capacity
Diffs
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Crc32.java 153c5a6d345293aa0ba2cf513373323a6e9f2467
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
unit tests
stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 19, 2014, 11:31 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Incorporate Jun's comments round four
Dummy
KAFKA-1253.v5
Dummy
Dummy
Dummy
Small fix on ProducerPerformance
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Incorporate Jun's comments one
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy 2
Dummy
KAFKA-1253
Diffs (updated)
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Crc32.java 153c5a6d345293aa0ba2cf513373323a6e9f2467
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
unit tests
stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Jay Kreps <bo...@gmail.com>.
> On March 19, 2014, 3:17 a.m., Jay Kreps wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Compressor.java, line 103
> > <https://reviews.apache.org/r/18299/diff/14/?file=526588#file526588line103>
> >
> > These should not be swallowed.
>
> Guozhang Wang wrote:
> The only reason I swallow them here is that IOExceptions should theoretically never be thrown, since the underlying stream used a byte buffer, which does not do any IO and hence not throwing IOException. But the signature of the function does declare throws IOException, that is why I swallow them here. I could also wrap it with the runtime KafkaException. Let me know which one do you prefer.
Yeah please do wrap it in KafkaException or another RuntimeException and rethrow. The problem is that if we are wrong and this exception is thrown for some reason then we will be silently swallowing it and it will be hard to figure out what is going on...
> On March 19, 2014, 3:17 a.m., Jay Kreps wrote:
> > clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java, line 46
> > <https://reviews.apache.org/r/18299/diff/14/?file=526586#file526586line46>
> >
> > I don't think this is needed. Inside an if statement you should be able to refer to a class. The class only loads if that branch is taken.
>
> Guozhang Wang wrote:
> Just tested and verified that if the library is imported in the class file, then whenever that class is ever executed the library will be loaded. So we have t use the class loader.
Here was the test I did last night to verify prior to my comment:
jkreps-mn:tmp jkreps$ cat Test.java
class Test {
public static void main(String[] args) throws Exception {
if(args.length == 0)
return;
else
new org.xerial.snappy.SnappyOutputStream(System.out);
}
}jkreps-mn:tmp jkreps$ javac -cp /Users/jkreps/.m2/repository/org/xerial/snappy/snappy-java/1.0.3.2/snappy-java-1.0.3.2.jar Test.java
jkreps-mn:tmp jkreps$ java Test
jkreps-mn:tmp jkreps$ java Test 0
Exception in thread "main" java.lang.NoClassDefFoundError: org/xerial/snappy/SnappyOutputStream
at Test.main(Test.java:8)
Caused by: java.lang.ClassNotFoundException: org.xerial.snappy.SnappyOutputStream
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 1 more
> On March 19, 2014, 3:17 a.m., Jay Kreps wrote:
> > clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java, line 58
> > <https://reviews.apache.org/r/18299/diff/14/?file=526582#file526582line58>
> >
> > What is the reason for this check?
>
> Guozhang Wang wrote:
> There is a corner case that the heuristics used to estimate the written bytes would return a value larger than the capacity already if we use the compressor's buffer size. For now since I removed this heuristics this may not hit, but I think it is better leaving it here in case we change to some other heuristics in the future.
If it isn't being used let's remove it. If you want to keep it it needs to be intelligible. I.e. a person reading the code needs to know why that check needs to be there. Even with your explanation I don't really get it. Maybe a comment or something?
- Jay
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review37679
-----------------------------------------------------------
On March 19, 2014, 11:32 p.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 19, 2014, 11:32 p.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Incorporated Jay's comments
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Crc32.java 153c5a6d345293aa0ba2cf513373323a6e9f2467
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
> On March 19, 2014, 3:17 a.m., Jay Kreps wrote:
> > clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java, line 46
> > <https://reviews.apache.org/r/18299/diff/14/?file=526586#file526586line46>
> >
> > I don't think this is needed. Inside an if statement you should be able to refer to a class. The class only loads if that branch is taken.
Just tested and verified that if the library is imported in the class file, then whenever that class is ever executed the library will be loaded. So we have t use the class loader.
- Guozhang
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review37679
-----------------------------------------------------------
On March 19, 2014, 12:11 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 19, 2014, 12:11 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
> On March 19, 2014, 3:17 a.m., Jay Kreps wrote:
> > clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java, line 58
> > <https://reviews.apache.org/r/18299/diff/14/?file=526582#file526582line58>
> >
> > What is the reason for this check?
There is a corner case that the heuristics used to estimate the written bytes would return a value larger than the capacity already if we use the compressor's buffer size. For now since I removed this heuristics this may not hit, but I think it is better leaving it here in case we change to some other heuristics in the future.
> On March 19, 2014, 3:17 a.m., Jay Kreps wrote:
> > clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java, line 69
> > <https://reviews.apache.org/r/18299/diff/14/?file=526589#file526589line69>
> >
> > Do we actually need this method? This is yet another place we define the message format.
This function is used in some unit tests. Besides I think this does not actually access the message format, but just the logentry format.
> On March 19, 2014, 3:17 a.m., Jay Kreps wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Compressor.java, line 71
> > <https://reviews.apache.org/r/18299/diff/14/?file=526588#file526588line71>
> >
> > Why isn't this done in MemoryRecords? This means we are defining the format in multiple places.
As I understands it, the MemoryRecords does not actually know the format of the message, all it knows is the format of the LogEntry, which is
[offset, message-size, message]
And inside message, the format is completely abstract from MemoryRecords but handles in the Compressor/Record:
[crc, magic, attribute, key-size, key, value-size, value]
When compression is used, we need to make the shallow message out of the compressed messages, and I think this logic fits naturally in Compressor.
> On March 19, 2014, 3:17 a.m., Jay Kreps wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Compressor.java, line 103
> > <https://reviews.apache.org/r/18299/diff/14/?file=526588#file526588line103>
> >
> > These should not be swallowed.
The only reason I swallow them here is that IOExceptions should theoretically never be thrown, since the underlying stream used a byte buffer, which does not do any IO and hence not throwing IOException. But the signature of the function does declare throws IOException, that is why I swallow them here. I could also wrap it with the runtime KafkaException. Let me know which one do you prefer.
> On March 19, 2014, 3:17 a.m., Jay Kreps wrote:
> > clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java, line 31
> > <https://reviews.apache.org/r/18299/diff/14/?file=526586#file526586line31>
> >
> > I strongly recommend you inline all of this into Compressor. This isn't really a factory for Compressions it is an OutputStreamFactory but splitting it into another class doesn't seem to buy us much.
I did this trying to isolate the Compressor from the specific compression types. I can change it back by merging Compressor with CompressionFactory.
> On March 19, 2014, 3:17 a.m., Jay Kreps wrote:
> > clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java, line 62
> > <https://reviews.apache.org/r/18299/diff/14/?file=526589#file526589line62>
> >
> > Consider replacing this with a factory method
> > MemoryRecords.emptyRecords(int size)
> >
> > The problem with boolean flags in java is that there is no named parameters so it is really hard to remember what the parameter is when you see
> > new MemoryRecords(buffer, type, true)
> >
Great point.
> On March 19, 2014, 3:17 a.m., Jay Kreps wrote:
> > clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java, line 88
> > <https://reviews.apache.org/r/18299/diff/14/?file=526586#file526586line88>
> >
> > ConcurrentModificationException. Make this an array of floats.
Using a list instead.
> On March 19, 2014, 3:17 a.m., Jay Kreps wrote:
> > clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java, line 39
> > <https://reviews.apache.org/r/18299/diff/14/?file=526586#file526586line39>
> >
> > I am concerned about the use of DataOutputStream as it adds another layer of synchronization.
OK, removing the DataOutputStream.
- Guozhang
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review37679
-----------------------------------------------------------
On March 19, 2014, 12:11 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 19, 2014, 12:11 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
> On March 19, 2014, 3:17 a.m., Jay Kreps wrote:
> > clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java, line 46
> > <https://reviews.apache.org/r/18299/diff/14/?file=526586#file526586line46>
> >
> > I don't think this is needed. Inside an if statement you should be able to refer to a class. The class only loads if that branch is taken.
>
> Guozhang Wang wrote:
> Just tested and verified that if the library is imported in the class file, then whenever that class is ever executed the library will be loaded. So we have t use the class loader.
>
> Jay Kreps wrote:
> Here was the test I did last night to verify prior to my comment:
>
> jkreps-mn:tmp jkreps$ cat Test.java
>
> class Test {
>
> public static void main(String[] args) throws Exception {
> if(args.length == 0)
> return;
> else
> new org.xerial.snappy.SnappyOutputStream(System.out);
> }
> }jkreps-mn:tmp jkreps$ javac -cp /Users/jkreps/.m2/repository/org/xerial/snappy/snappy-java/1.0.3.2/snappy-java-1.0.3.2.jar Test.java
> jkreps-mn:tmp jkreps$ java Test
> jkreps-mn:tmp jkreps$ java Test 0
> Exception in thread "main" java.lang.NoClassDefFoundError: org/xerial/snappy/SnappyOutputStream
> at Test.main(Test.java:8)
> Caused by: java.lang.ClassNotFoundException: org.xerial.snappy.SnappyOutputStream
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 1 more
Tried various ways including
1) if-else
2) explicit reference of org.xerial.snappy.SnappyOutputStream inside case(if) statements
Nothing works. I think it has something to do with the eager loader:
http://javarevisited.blogspot.com/2012/07/when-class-loading-initialization-java-example.html
- Guozhang
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review37679
-----------------------------------------------------------
On March 19, 2014, 11:32 p.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 19, 2014, 11:32 p.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Incorporated Jay's comments
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Crc32.java 153c5a6d345293aa0ba2cf513373323a6e9f2467
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Jay Kreps <bo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review37679
-----------------------------------------------------------
High-level comment:
I'm not sure if I get the code organization strategy between MemoryRecords, Compressor, and CompressionFactory. I'm pretty sure CompressionFactory and Compressor should be combined. It needs to be the case that the byte format is defined in a single place, not split between two or three places. Can you think if there is a cleaner way to divide these up--maybe just combining Compressor/CompressionFactory and moving the close() logic in Compressor into MemoryRecords?
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
<https://reviews.apache.org/r/18299/#comment69308>
What is the reason for this check?
clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java
<https://reviews.apache.org/r/18299/#comment69313>
I strongly recommend you inline all of this into Compressor. This isn't really a factory for Compressions it is an OutputStreamFactory but splitting it into another class doesn't seem to buy us much.
clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java
<https://reviews.apache.org/r/18299/#comment69319>
This isn't much damping. .9?
clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java
<https://reviews.apache.org/r/18299/#comment69315>
I am concerned about the use of DataOutputStream as it adds another layer of synchronization.
clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java
<https://reviews.apache.org/r/18299/#comment69314>
I don't think this is needed. Inside an if statement you should be able to refer to a class. The class only loads if that branch is taken.
clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java
<https://reviews.apache.org/r/18299/#comment69317>
ConcurrentModificationException. Make this an array of floats.
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment69324>
Why isn't this done in MemoryRecords? This means we are defining the format in multiple places.
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment69318>
These should not be swallowed.
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment69320>
Consider making this private.
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment69323>
You need to cap the compression block size to something reasonable. If I use 256K batches that doesn't mean I want you to allocate 512K of memory. There can't possibly be value in a block size of more than 1-4k, right?
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment69321>
Consider replacing this with a factory method
MemoryRecords.emptyRecords(int size)
The problem with boolean flags in java is that there is no named parameters so it is really hard to remember what the parameter is when you see
new MemoryRecords(buffer, type, true)
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment69325>
Do we actually need this method? This is yet another place we define the message format.
clients/src/main/java/org/apache/kafka/common/record/Record.java
<https://reviews.apache.org/r/18299/#comment69322>
Please no TODOs unless we are planning on doing them in the next few weeks, it makes the code messy. I don't think we are going to break the protocol, but if you think we should let's file a JIRA.
clients/src/main/java/org/apache/kafka/common/utils/Utils.java
<https://reviews.apache.org/r/18299/#comment69306>
Let's move this into CRC.java too as a static method.
clients/src/main/java/org/apache/kafka/common/utils/Utils.java
<https://reviews.apache.org/r/18299/#comment69305>
This should go in Crc.java as a non-static method, no?
- Jay Kreps
On March 19, 2014, 12:11 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 19, 2014, 12:11 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 19, 2014, 12:11 a.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
In-place compression with
1) Dynamic reallocation in the underlying byte buffer
2) Written bytes estimate to reduce reallocation probabilities
3) Deallocation in buffer pool following the original capacity
Diffs
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
unit tests
stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 19, 2014, 12:10 a.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Dummy
Dummy
Dummy
Incorporate Jun's comments round four
Dummy
KAFKA-1253.v5
Dummy
Dummy
Dummy
Small fix on ProducerPerformance
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Incorporate Jun's comments one
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy 2
Dummy
KAFKA-1253
Diffs (updated)
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java b69866a9fb9a8b4e1e78d304a20eda3cbf178c6f
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 673b2962771c28ceb3c7a6c0fd6f69521bd7ed16
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala 3df0d130308a35fca96184adc21eeee2eea1488f
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
unit tests
stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 17, 2014, 10:56 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
In-place compression with
1) Dynamic reallocation in the underlying byte buffer
2) Written bytes estimate to reduce reallocation probabilities
3) Deallocation in buffer pool following the original capacity
Diffs
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java 5bed60730ea108684bea2440af5a008feb0ada61
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 7a03f389cc6a4d56152b882555d7498af9a04d20
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala f12a45becb11a8bed586024866235b268630fec6
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
unit tests
stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 17, 2014, 10:56 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Dummy
Incorporate Jun's comments round four
Dummy
KAFKA-1253.v5
Dummy
Dummy
Dummy
Small fix on ProducerPerformance
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Incorporate Jun's comments one
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy 2
Dummy
KAFKA-1253
Diffs (updated)
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java 5bed60730ea108684bea2440af5a008feb0ada61
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 7a03f389cc6a4d56152b882555d7498af9a04d20
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala f12a45becb11a8bed586024866235b268630fec6
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
unit tests
stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review37304
-----------------------------------------------------------
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala
<https://reviews.apache.org/r/18299/#comment68753>
We can use the callback in Utils after it is checked in.
- Guozhang Wang
On March 15, 2014, 12:40 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 15, 2014, 12:40 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java 5bed60730ea108684bea2440af5a008feb0ada61
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 7a03f389cc6a4d56152b882555d7498af9a04d20
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala f12a45becb11a8bed586024866235b268630fec6
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
> On March 16, 2014, 6:42 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Compressor.java, lines 49-50
> > <https://reviews.apache.org/r/18299/diff/12/?file=520198#file520198line49>
> >
> > We probably can call Compressor.putRecord() directly since when key and value are null, Record.write and Compressor.putRecord are doing the same thing. Currently, it's a bit weird that Compressor calls Record.write, which calls putRecord back in Compressor.
That is great point. The problem here is that this in inside Compressor's constructor function now, and it should use a compressor with compression-type NONE.
I can either copy the write logic of Record with NONE compression to Compressor, but which will make it in two places; so I would prefer to put it just as is, unless you have some better suggestions?
> On March 16, 2014, 6:42 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/Compressor.java, lines 169-170
> > <https://reviews.apache.org/r/18299/diff/12/?file=520198#file520198line169>
> >
> > The latter estimate seems more accurate. Could you explain why we need the first estimate?
I think the latter is not always more accurate since it assumes the whole block size (default to the same as the batch size, 16KB) is full, which is rarely the case. Anyways, I will do some perf tests on the effectiveness of these heuristics.
> On March 16, 2014, 6:42 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java, lines 42-43
> > <https://reviews.apache.org/r/18299/diff/12/?file=520199#file520199line42>
> >
> > It seems that in all callers, writable is always true.
For now, yes. This is used for consumer, where writable will be false. I added this boolean also to avoid maybeInit for each append call.
> On March 16, 2014, 6:42 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java, lines 168-169
> > <https://reviews.apache.org/r/18299/diff/12/?file=520199#file520199line168>
> >
> > Not sure that I understand the comment on depending on remaining.
Updated the comments. Does this sounds more understandable?
> On March 16, 2014, 6:42 p.m., Jun Rao wrote:
> > core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala, lines 124-125
> > <https://reviews.apache.org/r/18299/diff/12/?file=520207#file520207line124>
> >
> > Instead of duplicating the input messages, perhaps we could store them in an array initially?
Unfortunately we cannot for now, since on the producer it is ProduceRecord while on the consumer it is Message, and it is hard to mutually translate them. And I think we do not really need to avoid duplication across test cases of a parameterized test?
- Guozhang
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review37311
-----------------------------------------------------------
On March 15, 2014, 12:40 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 15, 2014, 12:40 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java 5bed60730ea108684bea2440af5a008feb0ada61
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 7a03f389cc6a4d56152b882555d7498af9a04d20
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala f12a45becb11a8bed586024866235b268630fec6
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Jun Rao <ju...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review37311
-----------------------------------------------------------
1. Compilation error when running unit tests.
/Users/jrao/Intellij/kafka_gradle/core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala:135: Missing closing brace `}' assumed here
object ProducerCompressionTest {
^
one error found
FAILED
2. Thought about another corner case. Now that MemoryRecord can expand, it's possible for a compressed MemoryRecord to be larger that the max produce request size. When this happens, we will never be able to drain that MemoryRecord when calling RecordAccumulator.drain(). Not sure what's the best way to address this.
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java
<https://reviews.apache.org/r/18299/#comment68771>
Wouldn't capacity() << 1 be always >= than capacity() + 1?
clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java
<https://reviews.apache.org/r/18299/#comment68772>
Shouldn't we pass in bufferSize instead of 512?
clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java
<https://reviews.apache.org/r/18299/#comment68773>
Should we use double to get more precision?
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment68823>
We probably can call Compressor.putRecord() directly since when key and value are null, Record.write and Compressor.putRecord are doing the same thing. Currently, it's a bit weird that Compressor calls Record.write, which calls putRecord back in Compressor.
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment68774>
Should we avoid this calculation if compression type is None? Division on float could be a bit expensive.
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment68817>
Could we name this estimatedBytesWritten()?
clients/src/main/java/org/apache/kafka/common/record/Compressor.java
<https://reviews.apache.org/r/18299/#comment68818>
The latter estimate seems more accurate. Could you explain why we need the first estimate?
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment68781>
It seems that in all callers, writable is always true.
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment68783>
Is this comment still valid?
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment68782>
Could we call this isFull()?
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment68821>
Not sure that I understand the comment on depending on remaining.
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment68820>
rec.limit() is probably only needed in the uncompressed case.
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
<https://reviews.apache.org/r/18299/#comment68822>
has => have, new compressed => new uncompressed
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java
<https://reviews.apache.org/r/18299/#comment68824>
Could we just merge these into a single parameterized unit test?
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java
<https://reviews.apache.org/r/18299/#comment68825>
Instead of storing keyArray, could we just use this.key.array()?
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala
<https://reviews.apache.org/r/18299/#comment68826>
Could we just use a single consumer?
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala
<https://reviews.apache.org/r/18299/#comment68827>
Instead of duplicating the input messages, perhaps we could store them in an array initially?
- Jun Rao
On March 15, 2014, 12:40 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated March 15, 2014, 12:40 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> In-place compression with
>
> 1) Dynamic reallocation in the underlying byte buffer
> 2) Written bytes estimate to reduce reallocation probabilities
> 3) Deallocation in buffer pool following the original capacity
>
>
> Diffs
> -----
>
> build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java 5bed60730ea108684bea2440af5a008feb0ada61
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 7a03f389cc6a4d56152b882555d7498af9a04d20
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
> clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
> core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
> core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
> core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
> perf/src/main/scala/kafka/perf/ProducerPerformance.scala f12a45becb11a8bed586024866235b268630fec6
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
> integration tests
>
> unit tests
>
> stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
>
> snappy dynamic load test
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 15, 2014, 12:40 a.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
In-place compression with
1) Dynamic reallocation in the underlying byte buffer
2) Written bytes estimate to reduce reallocation probabilities
3) Deallocation in buffer pool following the original capacity
Diffs
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java 5bed60730ea108684bea2440af5a008feb0ada61
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 7a03f389cc6a4d56152b882555d7498af9a04d20
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala f12a45becb11a8bed586024866235b268630fec6
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
unit tests
stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 15, 2014, 12:40 a.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Dummy
KAFKA-1253.v5
Dummy
Dummy
Dummy
Small fix on ProducerPerformance
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Incorporate Jun's comments one
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy 2
Dummy
KAFKA-1253
Diffs (updated)
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java 5bed60730ea108684bea2440af5a008feb0ada61
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 7a03f389cc6a4d56152b882555d7498af9a04d20
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionFactory.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala f12a45becb11a8bed586024866235b268630fec6
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
unit tests
stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 14, 2014, 8:54 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description
-------
In-place compression with
1) Dynamic reallocation in the underlying byte buffer
2) Written bytes estimate to reduce reallocation probabilities
3) Deallocation in buffer pool following the original capacity
Diffs
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java 5bed60730ea108684bea2440af5a008feb0ada61
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 7a03f389cc6a4d56152b882555d7498af9a04d20
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
config/tools-log4j.properties 52f07c96019b4083fc78f62cfb0a81080327e847
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala f12a45becb11a8bed586024866235b268630fec6
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
unit tests
stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 14, 2014, 8:51 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description
-------
In-place compression with
1) Dynamic reallocation in the underlying byte buffer
2) Written bytes estimate to reduce reallocation probabilities
3) Deallocation in buffer pool following the original capacity
Diffs
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java 5bed60730ea108684bea2440af5a008feb0ada61
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 7a03f389cc6a4d56152b882555d7498af9a04d20
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
config/tools-log4j.properties 52f07c96019b4083fc78f62cfb0a81080327e847
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala f12a45becb11a8bed586024866235b268630fec6
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
unit tests
stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 14, 2014, 8:50 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
In-place compression with
1) Dynamic reallocation in the underlying byte buffer
2) Written bytes estimate to reduce reallocation probabilities
3) Deallocation in buffer pool following the original capacity
Diffs
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java 5bed60730ea108684bea2440af5a008feb0ada61
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 7a03f389cc6a4d56152b882555d7498af9a04d20
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
config/tools-log4j.properties 52f07c96019b4083fc78f62cfb0a81080327e847
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala f12a45becb11a8bed586024866235b268630fec6
Diff: https://reviews.apache.org/r/18299/diff/
Testing (updated)
-------
integration tests
unit tests
stress tests (1K message size, 1M messages in producer performance with ack = 1, linger time = 0ms/500ms, random bit/all-ones messages)
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 14, 2014, 8:46 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
KAFKA-1253.v5
Dummy
Dummy
Dummy
Small fix on ProducerPerformance
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Incorporate Jun's comments one
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy 2
Dummy
KAFKA-1253
Diffs (updated)
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java 5bed60730ea108684bea2440af5a008feb0ada61
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 7a03f389cc6a4d56152b882555d7498af9a04d20
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
config/tools-log4j.properties 52f07c96019b4083fc78f62cfb0a81080327e847
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala f12a45becb11a8bed586024866235b268630fec6
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 10, 2014, 10:28 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Incorporate Jun's comments round three
Refactored compression logic into Compressor
GZIP/Snappy integration tests
Diffs
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 616e1006f21c54af8260e84a96928cb8893ceb7c
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala f061dbabb49044f8fe94b8fd7dc0153c33bedeee
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 10, 2014, 10:28 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Dummy
Small fix on ProducerPerformance
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Incorporate Jun's comments one
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy 2
Dummy
KAFKA-1253
Diffs (updated)
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 616e1006f21c54af8260e84a96928cb8893ceb7c
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
perf/src/main/scala/kafka/perf/ProducerPerformance.scala f061dbabb49044f8fe94b8fd7dc0153c33bedeee
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 10, 2014, 9:40 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Incorporate Jun's comments round three
Refactored compression logic into Compressor
GZIP/Snappy integration tests
Diffs
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 616e1006f21c54af8260e84a96928cb8893ceb7c
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 10, 2014, 9:40 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Small fix on ProducerPerformance
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Incorporate Jun's comments one
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy 2
Dummy
KAFKA-1253
Diffs (updated)
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 616e1006f21c54af8260e84a96928cb8893ceb7c
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 10, 2014, 9:37 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description
-------
Incorporate Jun's comments round three
Refactored compression logic into Compressor
GZIP/Snappy integration tests
Diffs
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 616e1006f21c54af8260e84a96928cb8893ceb7c
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 10, 2014, 9:36 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Incorporate Jun's comments round three
Refactored compression logic into Compressor
GZIP/Snappy integration tests
Diffs
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 616e1006f21c54af8260e84a96928cb8893ceb7c
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 10, 2014, 9:36 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Incorporate Jun's comments one
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy 2
Dummy
KAFKA-1253
Diffs (updated)
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 616e1006f21c54af8260e84a96928cb8893ceb7c
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java 3ebbb804242be6a001b3bae6524afccc85a87602
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/main/scala/kafka/producer/ConsoleProducer.scala dd39ff22c918fe5b05f04582b748e32349b2055f
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 8, 2014, 12:36 a.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Incorporate Jun's comments round two
Refactored compression logic into Compressor
GZIP/Snappy integration tests
Diffs
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 616e1006f21c54af8260e84a96928cb8893ceb7c
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 8, 2014, 12:34 a.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Incorporate Jun's comments one
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy 2
Dummy
KAFKA-1253
Diffs (updated)
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 32e12ad149f6d70c96a498d0a390976f77bf9e2a
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 616e1006f21c54af8260e84a96928cb8893ceb7c
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala PRE-CREATION
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 7, 2014, 1:49 a.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Refactored Compression logic into Compressor
GZIP/Snappy Integration Tests
Diffs
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java f8748960fe9a02d52f7caae675abec503df91a39
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 616e1006f21c54af8260e84a96928cb8893ceb7c
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
Diff: https://reviews.apache.org/r/18299/diff/
Testing (updated)
-------
integration tests
snappy dynamic load test
Thanks,
Guozhang Wang
Re: Review Request 18299: Fix KAFKA-1253
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated March 7, 2014, 1:48 a.m.)
Review request for kafka.
Summary (updated)
-----------------
Fix KAFKA-1253
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Dummy
Dummy
Dummy
Dummy
Dummy
Incorporate Jun's comments one
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy 2
Dummy
KAFKA-1253
Diffs (updated)
-----
build.gradle 84fa0d6b5f7405af755c5d7ff7bdd7592bb8668f
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 1ac69436f117800815b8d50f042e9e2a29364b43
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java f8748960fe9a02d52f7caae675abec503df91a39
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 616e1006f21c54af8260e84a96928cb8893ceb7c
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 038a05a94b795ec0a95b2d40a89222394b5a74c4
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/Compressor.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 0c6b3656375721a718fb4de10118170aacce0ea9
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala c002f5ea38ece66ad559fadb18ffaf40ac2026aa
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 66ea76b9b6c0f8839f715c845fb9b9671b8f35c1
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
integration tests
Thanks,
Guozhang Wang
Re: Review Request 18299: Proposed In-place Compression on MemoryRecords
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated Feb. 27, 2014, 1:33 a.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Incorporated Jun's comments.
TODOs:
Class loader for Snappy.
Diffs
-----
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java e4bc97279585818860487a39a93b6481742b91db
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java d8e35e7d0e4cd27aad9a8d4bf14bc97458da9417
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java ce5cf27efa08b79e501439cf79bc8666054a5429
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java eb16f6d236e07b16654623606294a051531b5f58
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 9c34e7dc82f33df7406cad0e64eb6a896d068dc6
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 34baa8c6c7a15bb4aa93c286604f0eb7b19cd58e
Diff: https://reviews.apache.org/r/18299/diff/
Testing (updated)
-------
integration tests
Thanks,
Guozhang Wang
Re: Review Request 18299: Proposed In-place Compression on MemoryRecords
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated Feb. 27, 2014, 1:31 a.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy
Dummy 2
Dummy
KAFKA-1253
Diffs (updated)
-----
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java e4bc97279585818860487a39a93b6481742b91db
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java d8e35e7d0e4cd27aad9a8d4bf14bc97458da9417
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java ce5cf27efa08b79e501439cf79bc8666054a5429
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java eb16f6d236e07b16654623606294a051531b5f58
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 9c34e7dc82f33df7406cad0e64eb6a896d068dc6
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 34baa8c6c7a15bb4aa93c286604f0eb7b19cd58e
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
Thanks,
Guozhang Wang
Re: Review Request 18299: Proposed In-place Compression on MemoryRecords
Posted by Guozhang Wang <gu...@linkedin.com>.
> On Feb. 25, 2014, 6:22 p.m., Jun Rao wrote:
> > Could we extend ProducerSendTest to test sending/reading compressed messages?
Yep
> On Feb. 25, 2014, 6:22 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java, lines 72-73
> > <https://reviews.apache.org/r/18299/diff/4/?file=502748#file502748line72>
> >
> > Does it really do the rewinding?
Ack
> On Feb. 25, 2014, 6:22 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java, lines 74-76
> > <https://reviews.apache.org/r/18299/diff/4/?file=502748#file502748line74>
> >
> > Not sure we need to do this since we don't expect any other exceptions.
If we do not catch IOException and wrap with KafkaException, we need to declare throwables in the function signature, and any functions that call it.
> On Feb. 25, 2014, 6:22 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/record/CompressionType.java, lines 61-71
> > <https://reviews.apache.org/r/18299/diff/4/?file=502749#file502749line61>
> >
> > Could we just catch Exception?
Ack
> On Feb. 25, 2014, 6:22 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java, lines 145-155
> > <https://reviews.apache.org/r/18299/diff/4/?file=502744#file502744line145>
> >
> > When this happens, should we close the RecordBatch immediately after this record is appended?
Talked with Jay about closing a record batch, and it is actually not an issue. Since the appending into a MemoryRecords and the getting of a MemoryRecords from a dequeue is sycnchronized, appending and writing to socket will not happen at the same time.
- Guozhang
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review35422
-----------------------------------------------------------
On Feb. 24, 2014, 9:34 p.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2014, 9:34 p.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Incorporated Jun's comments.
>
> In-place compression with Crc32 pre-computation.
>
> In-place decompression embedded in nested iterator.
>
> Dynamic class loader for Snappy.
>
> TODOs:
>
> 1. Mix CompressedMemoryRecords into MemoryRecords.
> 2. Integration Test with GZIP.
> 3. Class loader for Snappy.
>
>
> Diffs
> -----
>
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java e4bc97279585818860487a39a93b6481742b91db
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java d8e35e7d0e4cd27aad9a8d4bf14bc97458da9417
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java ce5cf27efa08b79e501439cf79bc8666054a5429
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java eb16f6d236e07b16654623606294a051531b5f58
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 9c34e7dc82f33df7406cad0e64eb6a896d068dc6
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Proposed In-place Compression on MemoryRecords
Posted by Jun Rao <ju...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review35422
-----------------------------------------------------------
Could we extend ProducerSendTest to test sending/reading compressed messages?
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
<https://reviews.apache.org/r/18299/#comment65908>
When this happens, should we close the RecordBatch immediately after this record is appended?
clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java
<https://reviews.apache.org/r/18299/#comment65923>
Does it really do the rewinding?
clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java
<https://reviews.apache.org/r/18299/#comment65919>
Not sure we need to do this since we don't expect any other exceptions.
clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java
<https://reviews.apache.org/r/18299/#comment65924>
Ditto as the above.
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java
<https://reviews.apache.org/r/18299/#comment65912>
Could we just catch Exception?
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java
<https://reviews.apache.org/r/18299/#comment65913>
Ditto as the above.
- Jun Rao
On Feb. 24, 2014, 9:34 p.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2014, 9:34 p.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Incorporated Jun's comments.
>
> In-place compression with Crc32 pre-computation.
>
> In-place decompression embedded in nested iterator.
>
> Dynamic class loader for Snappy.
>
> TODOs:
>
> 1. Mix CompressedMemoryRecords into MemoryRecords.
> 2. Integration Test with GZIP.
> 3. Class loader for Snappy.
>
>
> Diffs
> -----
>
> clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java e4bc97279585818860487a39a93b6481742b91db
> clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java d8e35e7d0e4cd27aad9a8d4bf14bc97458da9417
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java ce5cf27efa08b79e501439cf79bc8666054a5429
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java eb16f6d236e07b16654623606294a051531b5f58
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
> clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
> clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 9c34e7dc82f33df7406cad0e64eb6a896d068dc6
> clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
> clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
> clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
> clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Guozhang Wang
>
>
Re: Review Request 18299: Proposed In-place Compression on MemoryRecords
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated Feb. 24, 2014, 9:34 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Incorporated Jun's comments.
In-place compression with Crc32 pre-computation.
In-place decompression embedded in nested iterator.
Dynamic class loader for Snappy.
TODOs:
1. Mix CompressedMemoryRecords into MemoryRecords.
2. Integration Test with GZIP.
3. Class loader for Snappy.
Diffs
-----
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java e4bc97279585818860487a39a93b6481742b91db
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java d8e35e7d0e4cd27aad9a8d4bf14bc97458da9417
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java ce5cf27efa08b79e501439cf79bc8666054a5429
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java eb16f6d236e07b16654623606294a051531b5f58
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 9c34e7dc82f33df7406cad0e64eb6a896d068dc6
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
Thanks,
Guozhang Wang
Re: Review Request 18299: Proposed In-place Compression on MemoryRecords
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated Feb. 24, 2014, 9:31 p.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Dummy
Dummy 2
Dummy
KAFKA-1253
Diffs (updated)
-----
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java e4bc97279585818860487a39a93b6481742b91db
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java d8e35e7d0e4cd27aad9a8d4bf14bc97458da9417
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java ce5cf27efa08b79e501439cf79bc8666054a5429
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java eb16f6d236e07b16654623606294a051531b5f58
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressionType.java 906da02d02c03aadd8ab73ed2fc9a1898acb8d72
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 9c34e7dc82f33df7406cad0e64eb6a896d068dc6
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
Thanks,
Guozhang Wang
Re: Review Request 18299: Proposed In-place Compression on MemoryRecords
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated Feb. 22, 2014, 1:57 a.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
In-place compression with Crc32 pre-computation.
In-place decompression embedded in nested iterator.
Diffs
-----
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 9c34e7dc82f33df7406cad0e64eb6a896d068dc6
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
Thanks,
Guozhang Wang
Re: Review Request 18299: Proposed In-place Compression on MemoryRecords
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated Feb. 22, 2014, 1:56 a.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Dummy 2
Dummy
KAFKA-1253
Diffs (updated)
-----
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 9c34e7dc82f33df7406cad0e64eb6a896d068dc6
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java PRE-CREATION
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
Thanks,
Guozhang Wang
Re: Review Request 18299: Proposed In-place Compression on MemoryRecords
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated Feb. 22, 2014, 12:16 a.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Pre-compute the Crc value to enable sequential writes on compressed streams.
Diffs
-----
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 9c34e7dc82f33df7406cad0e64eb6a896d068dc6
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
Thanks,
Guozhang Wang
Re: Review Request 18299: Proposed In-place Compression on MemoryRecords
Posted by Guozhang Wang <gu...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/
-----------------------------------------------------------
(Updated Feb. 22, 2014, 12:15 a.m.)
Review request for kafka.
Bugs: KAFKA-1253
https://issues.apache.org/jira/browse/KAFKA-1253
Repository: kafka
Description (updated)
-------
Dummy
KAFKA-1253
Diffs (updated)
-----
clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java PRE-CREATION
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 9d8935fa3beeb2a78b109a41ed76fd4374239560
clients/src/main/java/org/apache/kafka/common/record/Record.java f1dc9778502cbdfe982254fb6e25947842622239
clients/src/main/java/org/apache/kafka/common/utils/Utils.java 9c34e7dc82f33df7406cad0e64eb6a896d068dc6
clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java b0745b528cef929c4273f7e2ac4de1476cfc25ad
clients/src/test/java/org/apache/kafka/common/record/RecordTest.java ae54d67da9907b0a043180c7395a1370b3d0528d
clients/src/test/java/org/apache/kafka/test/TestUtils.java 36cfc0fda742eb501af2c2c0330e3f461cf1f40c
Diff: https://reviews.apache.org/r/18299/diff/
Testing
-------
Thanks,
Guozhang Wang
Re: Review Request 18299: Proposed In-place Compression on MemoryRecords
Posted by Jay Kreps <bo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18299/#review34977
-----------------------------------------------------------
clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java
<https://reviews.apache.org/r/18299/#comment65371>
This should maybe be lazy?
clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java
<https://reviews.apache.org/r/18299/#comment65370>
Writing the full record array is not actually correct. The ByteBuffer just represents a slice of the backing array so if you write the full backing array you may be writing other data.
clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java
<https://reviews.apache.org/r/18299/#comment65368>
I don't think this will fly, we should refactor in some way to do the right in place.
clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java
<https://reviews.apache.org/r/18299/#comment65369>
We should ideally do this with out double-copying.
- Jay Kreps
On Feb. 20, 2014, 2:58 a.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18299/
> -----------------------------------------------------------
>
> (Updated Feb. 20, 2014, 2:58 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1253
> https://issues.apache.org/jira/browse/KAFKA-1253
>
>
> Repository: kafka
>
>
> Description
> -------
>
> KAFKA-1253
>
>
> Diffs
> -----
>
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java PRE-CREATION
> clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/18299/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Guozhang Wang
>
>