[jira] [Commented] (KAFKA-5465) FetchResponse v0 does not return any messages when max_bytes smaller than v2 message set

["Ismael Juma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053101#comment-16053101 ] 

Ismael Juma commented on KAFKA-5465:
------------------------------------

Jason has a PR in progress for this, but it's a little tricky and we'll probably need some time to review and discuss. It's also late in the process, so we have to be particularly careful not to introduce regressions.

It's worth pointing out that all of the following need to be true for this to happen:

1. Pre KIP-74 consumer (KIP-74 was introduced in 0.10.1.0)

2. Broker configured to use message format v2. Even though this is the default, the upgrade instructions recommend that users only switch to the new message format after most consumers have been upgraded to avoid the loss of zero-copy.

3. partition max bytes is less than the v2 record batch size. For this to happen with the Java producer, the producer's max request size (default 1 MB for the Java producer) must be larger than the consumer's max partition size (default 1 MB). Note that I didn't mention the producer's batch size. The reason is that we allow a single record to be sent if it's larger than the batch size, but smaller than max request size. Increasing the max request size to be larger than the consumer's  max partition size is unsafe pre KIP-74 as one could send a single message that's larger than what the consumer can fetch causing the latter to get stuck.

Given the above, it seems to me that users with default or recommended settings for the Java clients won't run into the problem. It would be good to understand if it's the same for non Java clients (cc [~edenhill [~dana.powers]). The workaround of setting the broker message format version to v1 also seems straightforward (and it's what the upgrade instructions recommend if consumers that don't support the new message format are still being used).

I will proceed with RC1. If we agree to include this, we can spin RC2.

> FetchResponse v0 does not return any messages when max_bytes smaller than v2 message set 
> -----------------------------------------------------------------------------------------
>
>                 Key: KAFKA-5465
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5465
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.11.0.0
>            Reporter: Dana Powers
>            Assignee: Jason Gustafson
>            Priority: Blocker
>             Fix For: 0.11.0.0
>
>
> In prior releases, when consuming uncompressed messages, FetchResponse v0 will return a message if it is smaller than the max_bytes sent in the FetchRequest. In 0.11.0.0 RC0, when messages are stored as v2 internally, the response will be empty unless the full MessageSet is smaller than max_bytes. In some configurations, this may cause old consumers to get stuck on large messages where previously they were able to make progress one message at a time.
> For example, when I produce 10 5KB messages using ProduceRequest v0 and then attempt FetchRequest v0 with partition max bytes = 6KB (larger than a single message but smaller than all 10 messages together), I get an empty message set from 0.11.0.0. Previous brokers would have returned a single message.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)