[jira] [Created] (KAFKA-20898) PushTelemetry sets terminating flag on validation failure, permanently locking out client instance

["Ken Huang (Jira)" <ji...@apache.org>]

Ken Huang created KAFKA-20898:
---------------------------------

             Summary: PushTelemetry sets terminating flag on validation failure, permanently locking out client instance
                 Key: KAFKA-20898
                 URL: https://issues.apache.org/jira/browse/KAFKA-20898
             Project: Kafka
          Issue Type: Bug
            Reporter: Ken Huang
            Assignee: Ken Huang


[https://github.com/apache/kafka/blob/89ccd6a1260130e1a5ea83a0aa98a7d397482ace/server/src/main/java/org/apache/kafka/server/ClientMetricsManager.java#L209]
[https://cwiki.apache.org/confluence/spaces/KAFKA/pages/173085915/KIP-714+Client+metrics+and+observability]
{quote}*Client termination*
When a client with an active metrics subscription is being shut down, it should send its final metrics without waiting for the PushIntervalMs time.
To avoid the receiving broker’s metrics rate-limiter discarding this out-of-profile push, the {{PushTelemetryRequest.Terminating}} field must be set to true. A broker must only allow one such unthrottled metrics push for each combination of client instance ID and SubscriptionId.
In the event that the client’s metric subscription has changed and the final metrics push fails with error code {{UNKNOWN_SUBSCRIPTION_ID}} , the terminating client can choose to obtain a new subscription ID by sending a GetTelemetrySubscriptionsRequest and then immediately sending a PushTelemetryRequest with the Terminating flag set to true, or it can choose to abandon sending a final metrics push.{quote}

The implementation here differs from the behavior described in the KIP. The second terminating request will fail.

The flow is as follows:
 # The consumer is running normally, and the previous {{GetTelemetrySubscriptions}} request returns {{{}subscriptionId = X{}}}.
 # The admin runs {{{}kafka-client-metrics.sh --alter --interval 60000{}}}, causing the broker to recalculate the subscription ID, which changes from *X* to {*}Y{*}.
 # The consumer begins shutting down and sends its final {{PushTelemetry}} request with {{terminating=true}} and the old {{{}subscriptionId = X{}}}.
 # The broker fails the subscription ID validation and returns {{{}UNKNOWN_SUBSCRIPTION_ID{}}}, but due to a bug, the {{terminating=true}} state is still recorded.
 # The consumer attempts to recover by calling {{GetTelemetrySubscriptions}} again, obtains the new {{{}subscriptionId = Y{}}}, and sends another {{PushTelemetry}} request with {{{}terminating=true{}}}.
 # The broker sees that the instance has already been marked as {{terminating=true}} and returns {{{}INVALID_REQUEST{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)