[jira] [Commented] (KAFKA-2066) Replace FetchRequest / FetchResponse with their org.apache.kafka.common.requests equivalents

["David Jacot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15104883#comment-15104883 ] 

David Jacot commented on KAFKA-2066:
------------------------------------

In the current implementation, FetchResponse contains a map of FetchResponsePartitionData where each FetchResponsePartitionData contains a MessageSet. Then, FetchResponseSend is used to directly transfers bytes of each MessageSet to the channel. This is where the "zero-copy" happens.

Whereas, in the new FetchResponse, messages are represented as a ByteBuffer which brings all bytes in the user space and therefore breaks the "zero-copy". In order to keep it, I propose to use Records instead of a ByteBuffer in FetchResponse and extend the serialization mechanism to support channels. I need to work out the details but, roughly, It would require to:

1) update FetchResponse (and ProduceRequest to stay consistent);
2) introduce a new Type for Records;
3) add a new method in Type which work a the channel level with a default implementation which uses the current way; and
4) add some methods in Records interface for the serialization (basically what is in MessageSet)
4) make MessageSet inherit from Records to make them interoperable (serialization wise).

[~ijuma], [~granthenke] What do you think?

> Replace FetchRequest / FetchResponse with their org.apache.kafka.common.requests equivalents
> --------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-2066
>                 URL: https://issues.apache.org/jira/browse/KAFKA-2066
>             Project: Kafka
>          Issue Type: Sub-task
>            Reporter: Gwen Shapira
>            Assignee: David Jacot
>
> Replace FetchRequest / FetchResponse with their org.apache.kafka.common.requests equivalents.
> Note that they can't be completely removed until we deprecate the SimpleConsumer API (and it will require very careful patchwork for the places where core modules actually use the SimpleConsumer API).
> This also requires a solution on how to stream from memory-mapped files (similar to what existing code does with FileMessageSet. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)