producer changes

[Jay Kreps <ja...@gmail.com>]

Hey guys,

Two quick things:
1. There is a proposed change to ProducerData to allow us to retain the
partition key with the message here:
   https://issues.apache.org/jira/browse/KAFKA-544
This would also mean changing the Encoder/Decoder interface. Since this
impacts the user-facing APIs I think it would be useful to get any feedback
up front before I write a bunch of code.
2. There is another JIRA to make the producer return the offset(s) to the
caller. Since these are both in the same chunk of code, I would like to
grab this if no one else is actively working on it to avoid merge hell.
  https://issues.apache.org/jira/browse/KAFKA-496

-Jay

Re: producer changes

[Jay Kreps <ja...@gmail.com>]

Okay, I looked into this more. I think there are actually a number of
changes that should be made here, and it might be better to really think it
through and do a more serious project to get this right. So three things to
discuss:

   - I propose we make no change in the producer api in 0.8. Returning the
   offset is a nice feature but not really a blocker. It can come in 0.8 or
   0.9. Storing the key can be done using the existing ProducerData
   definition. This is not ideal since it is unintuitive to have the same key
   for many values, but since most people will not be using the
   key de-duplication I think this is a tolerable intermediate state.
   - I wrote up a few thoughts on what we should do to the producer API (in
   0.8.1 or 0.9 or whatever) here:
   https://cwiki.apache.org/confluence/display/KAFKA/Producer+API+Proposal

The rationale is that I think a good producer API is actually a little more
involved. Properly handling async requests means making each response a
future, which while not rocket science will involve pretty big changes to
the producer internals. Making this change without refactoring the existing
code would be very hard to do, but the refactoring is probably fairly
involved. I suspect if we try to cram this into 0.8 we will probably do a
half-assed job and just end up redoing it again.

-Jay


On Thu, Oct 4, 2012 at 1:24 PM, Jay Kreps <ja...@gmail.com> wrote:

> Hey guys,
>
> Two quick things:
> 1. There is a proposed change to ProducerData to allow us to retain the
> partition key with the message here:
>    https://issues.apache.org/jira/browse/KAFKA-544
> This would also mean changing the Encoder/Decoder interface. Since this
> impacts the user-facing APIs I think it would be useful to get any feedback
> up front before I write a bunch of code.
> 2. There is another JIRA to make the producer return the offset(s) to the
> caller. Since these are both in the same chunk of code, I would like to
> grab this if no one else is actively working on it to avoid merge hell.
>   https://issues.apache.org/jira/browse/KAFKA-496
>
> -Jay
>