[jira] [Resolved] (KAFKA-3744) Message format needs to identify serializer

["Jason Gustafson (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-3744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Gustafson resolved KAFKA-3744.
------------------------------------
    Resolution: Won't Fix

Closing this as "won't fix" since Kafka now supports message headers for use cases like this. Please feel free to reopen if you think we should reconsider.

> Message format needs to identify serializer
> -------------------------------------------
>
>                 Key: KAFKA-3744
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3744
>             Project: Kafka
>          Issue Type: Improvement
>            Reporter: David Kay
>            Priority: Minor
>
> https://issues.apache.org/jira/browse/KAFKA-3698 was recently resolved with https://github.com/apache/kafka/commit/27a19b964af35390d78e1b3b50bc03d23327f4d0.
> But Kafka documentation on message formats needs to be more explicit for new users. Section 1.3 Step 4 says: "Send some messages" and takes lines of text from the command line. Beginner's guide (http://www.slideshare.net/miguno/apache-kafka-08-basic-training-verisign Slide 104 says:
> {noformat}
>    Kafka does not care about data format of msg payload
>    Up to developer to handle serialization/deserialization
>       Common choices: Avro, JSON
> {noformat}
> If one producer sends lines of console text, another producer sends Avro, a third producer sends JSON, and a fourth sends CBOR, how does the consumer identify which deserializer to use for the payload?  The commit includes an opaque K byte Key that could potentially include a codec identifier, but provides no guidance on how to use it:
> {quote}
> "Leaving the key and value opaque is the right decision: there is a great deal of progress being made on serialization libraries right now, and any particular choice is unlikely to be right for all uses. Needless to say a particular application using Kafka would likely mandate a particular serialization type as part of its usage."
> {quote}
> Mandating any particular serialization is as unrealistic as mandating a single mime-type for all web content.  There must be a way to signal the serialization used to produce this message's V byte payload, and documenting the existence of even a rudimentary codec registry with a few values (text, Avro, JSON, CBOR) would establish the pattern to be used for future serialization libraries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)