You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Stevo Slavić <ss...@gmail.com> on 2015/07/02 17:32:20 UTC
Indication in FetchResponse that fetch size is too small
Hello Apache Kafka community,
Couldn't broker return a special error code in FetchResponse for a given
partition(s) where it detects that there was something to return/read from
partition but actual FetchResponse contains no messages for that partition
since fetch size in FetchRequest for that partition is too small?
I don't see it already in existing Kafka wire protocol docs
<https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol>.
It doesn't seem to be supported, since docs for broker and consumer
configuration mention that the maximums should be in sync, otherwise too
large message could block consumer.
If it was available this feature would be useful for simple consumer API
users, in the cases when range of possible message sizes on same topic is
relatively large (e.g. most very small messages, but some small fraction of
relatively large messages). Would help with being able to soft and hard
maximums, so most of the time one could read with soft max, and only in
case of this error code is on, attempt a read using hard max limit.
Assuming this feature is not there, I guess what would need to be done at
minimum is:
- minimal change in protocol and docs, just one new error code
- accompanying detection of this condition on broker side, so extend
handling of FetchRequest
- maybe in high level consumer make use of this error code in a special way
(e.g. just log a warning/error that there is potential misconfiguration).
Kind regards,
Stevo Slavic.
Re: Indication in FetchResponse that fetch size is too small
Posted by Stevo Slavić <ss...@gmail.com>.
Thanks for heads up Joel!
Kind regards,
Stevo Slavic.
On Thu, Jul 2, 2015 at 10:17 PM, Joel Koshy <jj...@gmail.com> wrote:
> A somewhat related request came up in another thread and I think it is
> reasonable to provide this. However, there are already some indicators
> that you can use:
> - The consumer iterator throws a MessageSizeTooLargeException if it
> cannot extract any messages out of the next chunk.
> - If you are using the simple consumer, the fetch response includes
> the high watermark - so if the HW > your fetch offset but there are
> no iterable messages, then you will know that your fetch size needs
> to increase.
>
> Thanks,
>
> Joel
>
> On Thu, Jul 02, 2015 at 05:32:20PM +0200, Stevo Slavić wrote:
> > Hello Apache Kafka community,
> >
> > Couldn't broker return a special error code in FetchResponse for a given
> > partition(s) where it detects that there was something to return/read
> from
> > partition but actual FetchResponse contains no messages for that
> partition
> > since fetch size in FetchRequest for that partition is too small?
> >
> > I don't see it already in existing Kafka wire protocol docs
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
> >.
> > It doesn't seem to be supported, since docs for broker and consumer
> > configuration mention that the maximums should be in sync, otherwise too
> > large message could block consumer.
> >
> > If it was available this feature would be useful for simple consumer API
> > users, in the cases when range of possible message sizes on same topic is
> > relatively large (e.g. most very small messages, but some small fraction
> of
> > relatively large messages). Would help with being able to soft and hard
> > maximums, so most of the time one could read with soft max, and only in
> > case of this error code is on, attempt a read using hard max limit.
> >
> > Assuming this feature is not there, I guess what would need to be done at
> > minimum is:
> > - minimal change in protocol and docs, just one new error code
> > - accompanying detection of this condition on broker side, so extend
> > handling of FetchRequest
> > - maybe in high level consumer make use of this error code in a special
> way
> > (e.g. just log a warning/error that there is potential misconfiguration).
> >
> > Kind regards,
> > Stevo Slavic.
>
>
Re: Indication in FetchResponse that fetch size is too small
Posted by Joel Koshy <jj...@gmail.com>.
A somewhat related request came up in another thread and I think it is
reasonable to provide this. However, there are already some indicators
that you can use:
- The consumer iterator throws a MessageSizeTooLargeException if it
cannot extract any messages out of the next chunk.
- If you are using the simple consumer, the fetch response includes
the high watermark - so if the HW > your fetch offset but there are
no iterable messages, then you will know that your fetch size needs
to increase.
Thanks,
Joel
On Thu, Jul 02, 2015 at 05:32:20PM +0200, Stevo Slavić wrote:
> Hello Apache Kafka community,
>
> Couldn't broker return a special error code in FetchResponse for a given
> partition(s) where it detects that there was something to return/read from
> partition but actual FetchResponse contains no messages for that partition
> since fetch size in FetchRequest for that partition is too small?
>
> I don't see it already in existing Kafka wire protocol docs
> <https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol>.
> It doesn't seem to be supported, since docs for broker and consumer
> configuration mention that the maximums should be in sync, otherwise too
> large message could block consumer.
>
> If it was available this feature would be useful for simple consumer API
> users, in the cases when range of possible message sizes on same topic is
> relatively large (e.g. most very small messages, but some small fraction of
> relatively large messages). Would help with being able to soft and hard
> maximums, so most of the time one could read with soft max, and only in
> case of this error code is on, attempt a read using hard max limit.
>
> Assuming this feature is not there, I guess what would need to be done at
> minimum is:
> - minimal change in protocol and docs, just one new error code
> - accompanying detection of this condition on broker side, so extend
> handling of FetchRequest
> - maybe in high level consumer make use of this error code in a special way
> (e.g. just log a warning/error that there is potential misconfiguration).
>
> Kind regards,
> Stevo Slavic.