You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Joel Koshy <jj...@gmail.com> on 2015/05/19 09:54:45 UTC
Review Request 34397: Patch for KAFKA-1374
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34397/
-----------------------------------------------------------
Review request for kafka.
Bugs: KAFKA-1374
https://issues.apache.org/jira/browse/KAFKA-1374
Repository: kafka
Description
-------
Rebased patch; incorporate edits and further changes from Manikumar
Diffs
-----
core/src/main/scala/kafka/log/LogCleaner.scala abea8b251895a5cc0788c6e25b112a2935a3f631
core/src/main/scala/kafka/message/ByteBufferMessageSet.scala 9dfe914991aaf82162e5e300c587c794555d5fd0
core/src/test/scala/kafka/tools/TestLogCleaning.scala 844589427cb9337acd89a5239a98b811ee58118e
core/src/test/scala/unit/kafka/log/LogCleanerIntegrationTest.scala 3b5aa9dc3b7ac5893c1d281ae1326be0e9ed8aad
core/src/test/scala/unit/kafka/log/LogTest.scala 76d3bfd378f32fd2b216b3ebdec86e2070491924
Diff: https://reviews.apache.org/r/34397/diff/
Testing
-------
Thanks,
Joel Koshy
Re: Review Request 34397: Patch for KAFKA-1374
Posted by Guozhang Wang <wa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34397/#review84545
-----------------------------------------------------------
Ship it!
Assuming some integration tests will be added in the follow-up patches.
- Guozhang Wang
On May 20, 2015, 12:20 a.m., Joel Koshy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34397/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 12:20 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1374
> https://issues.apache.org/jira/browse/KAFKA-1374
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Remove no-compression constraint for compacted topics
>
>
> Fix log cleaner integration test
>
>
> Incorporate edits from latest patch
>
>
> More minor edits
>
>
> Incorporate Guozhang's comments
>
>
> Diffs
> -----
>
> core/src/main/scala/kafka/log/LogCleaner.scala abea8b251895a5cc0788c6e25b112a2935a3f631
> core/src/main/scala/kafka/message/ByteBufferMessageSet.scala 9dfe914991aaf82162e5e300c587c794555d5fd0
> core/src/test/scala/kafka/tools/TestLogCleaning.scala 844589427cb9337acd89a5239a98b811ee58118e
> core/src/test/scala/unit/kafka/log/LogCleanerIntegrationTest.scala 3b5aa9dc3b7ac5893c1d281ae1326be0e9ed8aad
> core/src/test/scala/unit/kafka/log/LogTest.scala 76d3bfd378f32fd2b216b3ebdec86e2070491924
>
> Diff: https://reviews.apache.org/r/34397/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Joel Koshy
>
>
Re: Review Request 34397: Patch for KAFKA-1374
Posted by Joel Koshy <jj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34397/
-----------------------------------------------------------
(Updated May 20, 2015, 12:20 a.m.)
Review request for kafka.
Bugs: KAFKA-1374
https://issues.apache.org/jira/browse/KAFKA-1374
Repository: kafka
Description (updated)
-------
Remove no-compression constraint for compacted topics
Fix log cleaner integration test
Incorporate edits from latest patch
More minor edits
Incorporate Guozhang's comments
Diffs (updated)
-----
core/src/main/scala/kafka/log/LogCleaner.scala abea8b251895a5cc0788c6e25b112a2935a3f631
core/src/main/scala/kafka/message/ByteBufferMessageSet.scala 9dfe914991aaf82162e5e300c587c794555d5fd0
core/src/test/scala/kafka/tools/TestLogCleaning.scala 844589427cb9337acd89a5239a98b811ee58118e
core/src/test/scala/unit/kafka/log/LogCleanerIntegrationTest.scala 3b5aa9dc3b7ac5893c1d281ae1326be0e9ed8aad
core/src/test/scala/unit/kafka/log/LogTest.scala 76d3bfd378f32fd2b216b3ebdec86e2070491924
Diff: https://reviews.apache.org/r/34397/diff/
Testing
-------
Thanks,
Joel Koshy
Re: Review Request 34397: Patch for KAFKA-1374
Posted by Joel Koshy <jj...@gmail.com>.
> On May 19, 2015, 9:41 p.m., Guozhang Wang wrote:
> > About duplicating the compression logic, one way would be:
> >
> > 1. Add the following functions in ByteBufferMessageSet as:
> >
> > fill(buffer: ByteBuffer, compressionCodec: CompressionCodec, messages: MessageAndOffset*): Int
> >
> > 2. Change ByteBufferMessageSet.create() as:
> >
> > create(offsetCounter: AtomicLong, compressionCodec: CompressionCodec, messages: Message*): ByteBuffer = {
> >
> > // 1. pair messages to MessageAndOffset with offsetCounter
> > // 2. create the buffer as ByteBuffer.allocate(MessageSet.messageSetSize(messages))
> > // 3. call fill()
> > }
Yeah I thought of doing this, but thought it may be better to defer on it because:
- The duplicate code isolates these immediate changes for log compaction. i.e., the existing producer request handling will be unaffected.
- It forces an additional iteration for the common case. i.e., create on messages will do one iteration to assign offsets and create MessageAndOffset; and then a subsequent iteration in fill
- We should evaluate using the client compressor due to the other issue I mentioned (i.e., different compression code on broker vs client)
So just to be clear, I'm going to file a follow-up jira to investigate the above and also fix the broker compression codec issue (i.e., recompress to configured codec during log compaction).
> On May 19, 2015, 9:41 p.m., Guozhang Wang wrote:
> > core/src/main/scala/kafka/message/ByteBufferMessageSet.scala, lines 261-264
> > <https://reviews.apache.org/r/34397/diff/1/?file=963488#file963488line261>
> >
> > Do we need to modify this comment as well?
Yes these comments could use an edit.
> On May 19, 2015, 9:41 p.m., Guozhang Wang wrote:
> > core/src/main/scala/kafka/log/LogCleaner.scala, lines 432-433
> > <https://reviews.apache.org/r/34397/diff/1/?file=963487#file963487line432>
> >
> > I think the coding style convention is
> >
> > if {
> >
> > } else {
> >
> > }
Will do.
- Joel
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34397/#review84390
-----------------------------------------------------------
On May 20, 2015, 12:20 a.m., Joel Koshy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34397/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 12:20 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1374
> https://issues.apache.org/jira/browse/KAFKA-1374
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Remove no-compression constraint for compacted topics
>
>
> Fix log cleaner integration test
>
>
> Incorporate edits from latest patch
>
>
> More minor edits
>
>
> Incorporate Guozhang's comments
>
>
> Diffs
> -----
>
> core/src/main/scala/kafka/log/LogCleaner.scala abea8b251895a5cc0788c6e25b112a2935a3f631
> core/src/main/scala/kafka/message/ByteBufferMessageSet.scala 9dfe914991aaf82162e5e300c587c794555d5fd0
> core/src/test/scala/kafka/tools/TestLogCleaning.scala 844589427cb9337acd89a5239a98b811ee58118e
> core/src/test/scala/unit/kafka/log/LogCleanerIntegrationTest.scala 3b5aa9dc3b7ac5893c1d281ae1326be0e9ed8aad
> core/src/test/scala/unit/kafka/log/LogTest.scala 76d3bfd378f32fd2b216b3ebdec86e2070491924
>
> Diff: https://reviews.apache.org/r/34397/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Joel Koshy
>
>
Re: Review Request 34397: Patch for KAFKA-1374
Posted by Guozhang Wang <wa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34397/#review84390
-----------------------------------------------------------
About duplicating the compression logic, one way would be:
1. Add the following functions in ByteBufferMessageSet as:
fill(buffer: ByteBuffer, compressionCodec: CompressionCodec, messages: MessageAndOffset*): Int
2. Change ByteBufferMessageSet.create() as:
create(offsetCounter: AtomicLong, compressionCodec: CompressionCodec, messages: Message*): ByteBuffer = {
// 1. pair messages to MessageAndOffset with offsetCounter
// 2. create the buffer as ByteBuffer.allocate(MessageSet.messageSetSize(messages))
// 3. call fill()
}
core/src/main/scala/kafka/log/LogCleaner.scala
<https://reviews.apache.org/r/34397/#comment135612>
I think the coding style convention is
if {
} else {
}
core/src/main/scala/kafka/message/ByteBufferMessageSet.scala
<https://reviews.apache.org/r/34397/#comment135613>
Do we need to modify this comment as well?
- Guozhang Wang
On May 19, 2015, 7:54 a.m., Joel Koshy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34397/
> -----------------------------------------------------------
>
> (Updated May 19, 2015, 7:54 a.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1374
> https://issues.apache.org/jira/browse/KAFKA-1374
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Rebased patch; incorporate edits and further changes from Manikumar
>
>
> Diffs
> -----
>
> core/src/main/scala/kafka/log/LogCleaner.scala abea8b251895a5cc0788c6e25b112a2935a3f631
> core/src/main/scala/kafka/message/ByteBufferMessageSet.scala 9dfe914991aaf82162e5e300c587c794555d5fd0
> core/src/test/scala/kafka/tools/TestLogCleaning.scala 844589427cb9337acd89a5239a98b811ee58118e
> core/src/test/scala/unit/kafka/log/LogCleanerIntegrationTest.scala 3b5aa9dc3b7ac5893c1d281ae1326be0e9ed8aad
> core/src/test/scala/unit/kafka/log/LogTest.scala 76d3bfd378f32fd2b216b3ebdec86e2070491924
>
> Diff: https://reviews.apache.org/r/34397/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Joel Koshy
>
>