versionId in responses (and general API versioning questions)

[David Arthur <mu...@gmail.com>]

Just noticed that most of the API response structs do not include the 
ApiVersion. This will make it hard for clients to determine how to 
handle responses once the APIs start getting updated (which is pretty 
much inevitable).

I'd propose we update the standard response envelope to include it

Response => ApiVersion CorrelationId ResponseMessage
   ApiVersion => int16
   CorrelationId => int32
   ResponseMessage => ...

Otherwise, it seems we must freeze the response formats forever.

Another thought is that the server should return the same version of the 
API that was requested (if a client sends in v1, then presumably it does 
know about any future versions...). How will this work? This seems to 
imply backwards compat for the APIs like Avro.

Thoughts?

-David

Re: versionId in responses (and general API versioning questions)

[Jay Kreps <ja...@gmail.com>]

Awesome, thanks!

-Jay

On Tue, Feb 19, 2013 at 10:34 AM, David Arthur <mu...@gmail.com> wrote:
> I have created https://issues.apache.org/jira/browse/KAFKA-759 to remove
> versionId as discussed
>
>
> On 2/13/13 7:40 PM, David Arthur wrote:
>>
>> On 2/13/13 4:08 PM, Jay Kreps wrote:
>>>
>>> Hey David,
>>>
>>> We ended up not versioning the response, instead the version must
>>> correspond to the request version.
>>>
>>> This makes sense from the client point of view. If you send a request
>>> using version X of the protocol you know you will get a response in
>>> format X. Separately versioning the response would seem to indicate
>>> that the server is allowed to send back a different version of the
>>> response. This means the client has to check this and handle old
>>> response versions (and what would it even do with newer versions?).
>>
>> This makes complete sense. I was confused since OffsetCommit/Fetch are
>> including it in the responses
>>>
>>> Instead we thought it makes more sense to make the server deal with
>>> compatibility only. So the versioning is at the request/response pair
>>> and the server is required to always send the correct version of the
>>> response for all supported request versions.
>>>
>>> I just noticed that your responses for the offset apis actually have a
>>> version. We should probably remove that before the release.
>>
>> Yea, agreed.
>>>
>>>
>>> -Jay
>>>
>>> On Wed, Feb 13, 2013 at 12:48 PM, David Arthur <mu...@gmail.com> wrote:
>>>>
>>>> Just noticed that most of the API response structs do not include the
>>>> ApiVersion. This will make it hard for clients to determine how to
>>>> handle
>>>> responses once the APIs start getting updated (which is pretty much
>>>> inevitable).
>>>>
>>>> I'd propose we update the standard response envelope to include it
>>>>
>>>> Response => ApiVersion CorrelationId ResponseMessage
>>>>    ApiVersion => int16
>>>>    CorrelationId => int32
>>>>    ResponseMessage => ...
>>>>
>>>> Otherwise, it seems we must freeze the response formats forever.
>>>>
>>>> Another thought is that the server should return the same version of the
>>>> API
>>>> that was requested (if a client sends in v1, then presumably it does
>>>> know
>>>> about any future versions...). How will this work? This seems to imply
>>>> backwards compat for the APIs like Avro.
>>>>
>>>> Thoughts?
>>>>
>>>> -David
>>
>>
>

Re: versionId in responses (and general API versioning questions)

[David Arthur <mu...@gmail.com>]

I have created https://issues.apache.org/jira/browse/KAFKA-759 to remove 
versionId as discussed

On 2/13/13 7:40 PM, David Arthur wrote:
> On 2/13/13 4:08 PM, Jay Kreps wrote:
>> Hey David,
>>
>> We ended up not versioning the response, instead the version must
>> correspond to the request version.
>>
>> This makes sense from the client point of view. If you send a request
>> using version X of the protocol you know you will get a response in
>> format X. Separately versioning the response would seem to indicate
>> that the server is allowed to send back a different version of the
>> response. This means the client has to check this and handle old
>> response versions (and what would it even do with newer versions?).
> This makes complete sense. I was confused since OffsetCommit/Fetch are 
> including it in the responses
>> Instead we thought it makes more sense to make the server deal with
>> compatibility only. So the versioning is at the request/response pair
>> and the server is required to always send the correct version of the
>> response for all supported request versions.
>>
>> I just noticed that your responses for the offset apis actually have a
>> version. We should probably remove that before the release.
> Yea, agreed.
>>
>> -Jay
>>
>> On Wed, Feb 13, 2013 at 12:48 PM, David Arthur <mu...@gmail.com> wrote:
>>> Just noticed that most of the API response structs do not include the
>>> ApiVersion. This will make it hard for clients to determine how to 
>>> handle
>>> responses once the APIs start getting updated (which is pretty much
>>> inevitable).
>>>
>>> I'd propose we update the standard response envelope to include it
>>>
>>> Response => ApiVersion CorrelationId ResponseMessage
>>>    ApiVersion => int16
>>>    CorrelationId => int32
>>>    ResponseMessage => ...
>>>
>>> Otherwise, it seems we must freeze the response formats forever.
>>>
>>> Another thought is that the server should return the same version of 
>>> the API
>>> that was requested (if a client sends in v1, then presumably it does 
>>> know
>>> about any future versions...). How will this work? This seems to imply
>>> backwards compat for the APIs like Avro.
>>>
>>> Thoughts?
>>>
>>> -David
>

Re: versionId in responses (and general API versioning questions)

[David Arthur <mu...@gmail.com>]

On 2/13/13 4:08 PM, Jay Kreps wrote:
> Hey David,
>
> We ended up not versioning the response, instead the version must
> correspond to the request version.
>
> This makes sense from the client point of view. If you send a request
> using version X of the protocol you know you will get a response in
> format X. Separately versioning the response would seem to indicate
> that the server is allowed to send back a different version of the
> response. This means the client has to check this and handle old
> response versions (and what would it even do with newer versions?).
This makes complete sense. I was confused since OffsetCommit/Fetch are 
including it in the responses
> Instead we thought it makes more sense to make the server deal with
> compatibility only. So the versioning is at the request/response pair
> and the server is required to always send the correct version of the
> response for all supported request versions.
>
> I just noticed that your responses for the offset apis actually have a
> version. We should probably remove that before the release.
Yea, agreed.
>
> -Jay
>
> On Wed, Feb 13, 2013 at 12:48 PM, David Arthur <mu...@gmail.com> wrote:
>> Just noticed that most of the API response structs do not include the
>> ApiVersion. This will make it hard for clients to determine how to handle
>> responses once the APIs start getting updated (which is pretty much
>> inevitable).
>>
>> I'd propose we update the standard response envelope to include it
>>
>> Response => ApiVersion CorrelationId ResponseMessage
>>    ApiVersion => int16
>>    CorrelationId => int32
>>    ResponseMessage => ...
>>
>> Otherwise, it seems we must freeze the response formats forever.
>>
>> Another thought is that the server should return the same version of the API
>> that was requested (if a client sends in v1, then presumably it does know
>> about any future versions...). How will this work? This seems to imply
>> backwards compat for the APIs like Avro.
>>
>> Thoughts?
>>
>> -David

Re: versionId in responses (and general API versioning questions)

[Jay Kreps <ja...@gmail.com>]

Hey David,

We ended up not versioning the response, instead the version must
correspond to the request version.

This makes sense from the client point of view. If you send a request
using version X of the protocol you know you will get a response in
format X. Separately versioning the response would seem to indicate
that the server is allowed to send back a different version of the
response. This means the client has to check this and handle old
response versions (and what would it even do with newer versions?).
Instead we thought it makes more sense to make the server deal with
compatibility only. So the versioning is at the request/response pair
and the server is required to always send the correct version of the
response for all supported request versions.

I just noticed that your responses for the offset apis actually have a
version. We should probably remove that before the release.

-Jay

On Wed, Feb 13, 2013 at 12:48 PM, David Arthur <mu...@gmail.com> wrote:
> Just noticed that most of the API response structs do not include the
> ApiVersion. This will make it hard for clients to determine how to handle
> responses once the APIs start getting updated (which is pretty much
> inevitable).
>
> I'd propose we update the standard response envelope to include it
>
> Response => ApiVersion CorrelationId ResponseMessage
>   ApiVersion => int16
>   CorrelationId => int32
>   ResponseMessage => ...
>
> Otherwise, it seems we must freeze the response formats forever.
>
> Another thought is that the server should return the same version of the API
> that was requested (if a client sends in v1, then presumably it does know
> about any future versions...). How will this work? This seems to imply
> backwards compat for the APIs like Avro.
>
> Thoughts?
>
> -David