You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Jay Kreps <ja...@gmail.com> on 2012/03/30 23:17:27 UTC
error code in ByteBufferMessageSet
There are a number of methods on ByteBufferMessageSet:
def getInitialOffset = initialOffset
def getBuffer = buffer
def getErrorCode = errorCode
def getSerialized(): ByteBuffer = buffer
We don't generally have getters in kafka, since it automatically adds
them. Is there a reason for these?
Also, there is no documentation on getSerialized() which does the same
thing as getBuffer().
Also why does a message set have an initial offset? Isn't it the case
that a messageset is independent of a particular offset to which it
might be written?
It isn't clear to me why a MessageSet would have an error code at all.
It is a set of messages, that is what it is. Error codes are something
that have to do with APIs...did we just stick that in for convenience?
I have a feeling that some of this was stuck in due to not having a
proper consumer response object in which to put the fields maybe. Now
that we have that in 0.8 can we clean this up? I feel if we junk up
these core classes we will soon have a real mess on our hands...
-Jay
Re: error code in ByteBufferMessageSet
Posted by Jay Kreps <ja...@gmail.com>.
I guess the issue, though, is that MessageSet is meant to represent a
set of messages even prior to their being published. As a result it is
not necessarily the case that there is an offset yet. Likewise it is
possible to write a single message set twice to two separate topics. I
guess a fully correct modeling would have MessageSet which implements
Iterator[Message] and PublishedMessageSet[MessageAndOffsetInfo]
(probably need to work on the name) which represents a chunk of a
published log. This might be a bit of overkill, though.
-Jay
On Mon, Apr 2, 2012 at 7:55 AM, Prashanth Menon
<pr...@gmail.com> wrote:
> My two cents here:
>
> Agreed on the getters and the errorCode: the former is unncessary and the
> latter should be solved with the new fetch response. With respect to the
> initialOffset, from the writer perspective it seems unnecessary. It's only
> really used from the consumer perspective as a convenience to iterate over
> the messages tieing each to an offset (ie. MessageAndOffset). A simple
> solution would be have MessageSet iterate over its Message and to pull out
> the initialOffset into a new class that wraps a MessageSet and an
> initialOffset that consumers use allowing simple iteration over
> MessageAndOffset? This will probably make the API cleaner.
>
> - Prashanth
>
> On Fri, Mar 30, 2012 at 8:53 PM, Jun Rao <ju...@gmail.com> wrote:
>
>> Yes, they should all be fixable with the new consumer response object.
>> Could you open a jira to track this?
>>
>> Thanks,
>>
>> Jun
>>
>> On Fri, Mar 30, 2012 at 2:17 PM, Jay Kreps <ja...@gmail.com> wrote:
>>
>> > There are a number of methods on ByteBufferMessageSet:
>> > def getInitialOffset = initialOffset
>> > def getBuffer = buffer
>> > def getErrorCode = errorCode
>> > def getSerialized(): ByteBuffer = buffer
>> >
>> > We don't generally have getters in kafka, since it automatically adds
>> > them. Is there a reason for these?
>> >
>> > Also, there is no documentation on getSerialized() which does the same
>> > thing as getBuffer().
>> >
>> > Also why does a message set have an initial offset? Isn't it the case
>> > that a messageset is independent of a particular offset to which it
>> > might be written?
>> >
>> > It isn't clear to me why a MessageSet would have an error code at all.
>> > It is a set of messages, that is what it is. Error codes are something
>> > that have to do with APIs...did we just stick that in for convenience?
>> >
>> > I have a feeling that some of this was stuck in due to not having a
>> > proper consumer response object in which to put the fields maybe. Now
>> > that we have that in 0.8 can we clean this up? I feel if we junk up
>> > these core classes we will soon have a real mess on our hands...
>> >
>> > -Jay
>> >
>>
Re: error code in ByteBufferMessageSet
Posted by Prashanth Menon <pr...@gmail.com>.
My two cents here:
Agreed on the getters and the errorCode: the former is unncessary and the
latter should be solved with the new fetch response. With respect to the
initialOffset, from the writer perspective it seems unnecessary. It's only
really used from the consumer perspective as a convenience to iterate over
the messages tieing each to an offset (ie. MessageAndOffset). A simple
solution would be have MessageSet iterate over its Message and to pull out
the initialOffset into a new class that wraps a MessageSet and an
initialOffset that consumers use allowing simple iteration over
MessageAndOffset? This will probably make the API cleaner.
- Prashanth
On Fri, Mar 30, 2012 at 8:53 PM, Jun Rao <ju...@gmail.com> wrote:
> Yes, they should all be fixable with the new consumer response object.
> Could you open a jira to track this?
>
> Thanks,
>
> Jun
>
> On Fri, Mar 30, 2012 at 2:17 PM, Jay Kreps <ja...@gmail.com> wrote:
>
> > There are a number of methods on ByteBufferMessageSet:
> > def getInitialOffset = initialOffset
> > def getBuffer = buffer
> > def getErrorCode = errorCode
> > def getSerialized(): ByteBuffer = buffer
> >
> > We don't generally have getters in kafka, since it automatically adds
> > them. Is there a reason for these?
> >
> > Also, there is no documentation on getSerialized() which does the same
> > thing as getBuffer().
> >
> > Also why does a message set have an initial offset? Isn't it the case
> > that a messageset is independent of a particular offset to which it
> > might be written?
> >
> > It isn't clear to me why a MessageSet would have an error code at all.
> > It is a set of messages, that is what it is. Error codes are something
> > that have to do with APIs...did we just stick that in for convenience?
> >
> > I have a feeling that some of this was stuck in due to not having a
> > proper consumer response object in which to put the fields maybe. Now
> > that we have that in 0.8 can we clean this up? I feel if we junk up
> > these core classes we will soon have a real mess on our hands...
> >
> > -Jay
> >
>
Re: error code in ByteBufferMessageSet
Posted by Jun Rao <ju...@gmail.com>.
Yes, they should all be fixable with the new consumer response object.
Could you open a jira to track this?
Thanks,
Jun
On Fri, Mar 30, 2012 at 2:17 PM, Jay Kreps <ja...@gmail.com> wrote:
> There are a number of methods on ByteBufferMessageSet:
> def getInitialOffset = initialOffset
> def getBuffer = buffer
> def getErrorCode = errorCode
> def getSerialized(): ByteBuffer = buffer
>
> We don't generally have getters in kafka, since it automatically adds
> them. Is there a reason for these?
>
> Also, there is no documentation on getSerialized() which does the same
> thing as getBuffer().
>
> Also why does a message set have an initial offset? Isn't it the case
> that a messageset is independent of a particular offset to which it
> might be written?
>
> It isn't clear to me why a MessageSet would have an error code at all.
> It is a set of messages, that is what it is. Error codes are something
> that have to do with APIs...did we just stick that in for convenience?
>
> I have a feeling that some of this was stuck in due to not having a
> proper consumer response object in which to put the fields maybe. Now
> that we have that in 0.8 can we clean this up? I feel if we junk up
> these core classes we will soon have a real mess on our hands...
>
> -Jay
>