[jira] [Updated] (KAFKA-21077) Java client does not honor GetTelemetrySubscriptionsResponse.TelemetryMaxBytes

["Steven Schlansker (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-21077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steven Schlansker updated KAFKA-21077:
--------------------------------------
    Description: 
KIP-714 says a client that receives TELEMETRY_TOO_LARGE must "Reduce the size of the metrics payload so its size does not exceed GetTelemetrySubscriptionsResponse.TelemetryMaxBytes." The broker sends the value in every subscription response ({{{}ClientMetricsManager{}}} line 383, {{{}setTelemetryMaxBytes{}}}).

{{ClientTelemetryReporter}} never reads it. {{createPushRequest}} collects, serializes, compresses, and sends the payload with no size check, and {{ClientTelemetrySubscription}} does not store the value. On TELEMETRY_TOO_LARGE the client only reschedules a similarly-sized payload ({{{}maybeFetchErrorIntervalMs{}}}), so it repeats a push the broker will reject every interval, and the broker rejects with {{TELEMETRY_TOO_LARGE}} every interval.

Suggested fix: keep {{telemetryMaxBytes}} in {{{}ClientTelemetrySubscription{}}}, compare the compressed payload against it before sending, and when the payload is too large either log a warning with the size and the metric count and skip the push, or drop lower-priority metric groups until it fits.

  was:
KIP-714 says a client that receives TELEMETRY_TOO_LARGE must "Reduce the size of the metrics payload so its size does not exceed GetTelemetrySubscriptionsResponse.TelemetryMaxBytes." The broker sends the value in every subscription response (\{{ClientMetricsManager}} line 383, \{{setTelemetryMaxBytes}}).

{\{ClientTelemetryReporter}} never reads it. \{{createPushRequest}} collects, serializes, compresses, and sends the payload with no size check, and \{{ClientTelemetrySubscription}} does not store the value. On TELEMETRY_TOO_LARGE the client only reschedules the same payload (\{{maybeFetchErrorIntervalMs}}), so it repeats a push the broker will reject every interval, and the broker counts a plugin error each time.

Suggested fix: keep \{{telemetryMaxBytes}} in \{{ClientTelemetrySubscription}}, compare the compressed payload against it before sending, and when the payload is too large either log a warning with the size and the metric count and skip the push, or drop lower-priority metric groups until it fits.


> Java client does not honor GetTelemetrySubscriptionsResponse.TelemetryMaxBytes
> ------------------------------------------------------------------------------
>
>                 Key: KAFKA-21077
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21077
>             Project: Kafka
>          Issue Type: Bug
>          Components: clients
>    Affects Versions: 4.3.1
>            Reporter: Steven Schlansker
>            Priority: Minor
>
> KIP-714 says a client that receives TELEMETRY_TOO_LARGE must "Reduce the size of the metrics payload so its size does not exceed GetTelemetrySubscriptionsResponse.TelemetryMaxBytes." The broker sends the value in every subscription response ({{{}ClientMetricsManager{}}} line 383, {{{}setTelemetryMaxBytes{}}}).
> {{ClientTelemetryReporter}} never reads it. {{createPushRequest}} collects, serializes, compresses, and sends the payload with no size check, and {{ClientTelemetrySubscription}} does not store the value. On TELEMETRY_TOO_LARGE the client only reschedules a similarly-sized payload ({{{}maybeFetchErrorIntervalMs{}}}), so it repeats a push the broker will reject every interval, and the broker rejects with {{TELEMETRY_TOO_LARGE}} every interval.
> Suggested fix: keep {{telemetryMaxBytes}} in {{{}ClientTelemetrySubscription{}}}, compare the compressed payload against it before sending, and when the payload is too large either log a warning with the size and the metric count and skip the push, or drop lower-priority metric groups until it fits.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)