[jira] [Created] (FLINK-29939) Add metrics for Kubernetes Client Response 5xx count and rate

["Zhou Jiang (Jira)" <ji...@apache.org>]

Zhou Jiang created FLINK-29939:
----------------------------------

             Summary: Add metrics for Kubernetes Client Response 5xx count and rate
                 Key: FLINK-29939
                 URL: https://issues.apache.org/jira/browse/FLINK-29939
             Project: Flink
          Issue Type: Improvement
          Components: Kubernetes Operator
    Affects Versions: kubernetes-operator-1.3.0
            Reporter: Zhou Jiang


Operator now publishes k8s client response count by response code. In addition to the accumulative count, adding rate for k8s client error responses could help to setup alerts detect underlying cluster API server status proactively. This is for enhancement of metrics when Flink Operator is deployed to shared / multi-tenant k8s clusters. 

 

Why is rate needed for certain response codes?

To detect issues proactively by setting up alerts in certain cases. It could not the total number but the rate indicates the start / end of unavailability issue.

 

Why do some 4xx matter in prod?

For example - noisy neighbor issue may happen at random time in shared clusters, and operator may start to see increased number of 429 if cluster does not have fairness in rate limiting. Another example is about churn: when the cluster has namespaces quota defined and namespace is under pod churn, there could be increasing number of 409. In these cases, metrics and alerting on count / rate of certain 4xx is critical to understand start / end of prod outage.

 

Why is 5xx needed ?

For faster identify infrastructure issue. With 5xx response count + rate, It's more straightforward than enumerating possible 5xx codes.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)