Re: [PR] Read Kubernetes metadata in the background, off the threads analyzing ALS, eBPF access logs and MAL [skywalking]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng commented on PR #14135:
URL: https://github.com/apache/skywalking/pull/14135#issuecomment-6184502575

   ## Updated lookup rules
   
   This revision changes how the Kubernetes metadata lookups retry, cache and wait. The PR description is updated to match.
   
   **What changed**
   - **A failed read is no longer cached like "nothing found".** A failure (403, 5xx, the API server not answering or refusing the connection) is retried on the first use 3 minutes later. A read that succeeds and matches nothing is read again on the first use 1 minute later, as the key may be new to the API server or to the lists OAP matched it against.
   - **New pods are found within about a minute.** A pod found by IP but not in the endpoints read (Envoy ALS), or selected by no service in the services read (MAL), makes the reader read that list again once it is older than 1 minute. Before, the 3-minute lists could hide a new pod for 3–6 minutes.
   - **Cached data lives longer.** It is still read again on its first use 3 minutes after it was written, but is now dropped 21 minutes after its last write (7 refresh periods), per cache: `als` for Envoy ALS, `mal` for MAL and the eBPF access log. A read again that finds nothing or fails keeps the data and is retried a refresh period later.
   - **MAL waits at most 2 seconds per `retagByK8sMeta`, not per pod.** The step queues the reads of all its pods first, then waits for them within one `malMetadataReadTimeout`, now 2000 ms by default. A pod's wait counts from when a step first waited for it, so the several `retagByK8sMeta` of one Export over the same pods wait 2 s at most in all, whatever happens to their reads meanwhile (a full lane admitting them later, an entry expiring). A cached pod is not waited for, even while it is read again. Its lookups after that never wait. That keeps an Export under the OpenTelemetry Collector's 5-second export timeout, after which it sends the data again. Pods not read by then get an empty service for this Export, and are read in the background for the next one.
   - **Failed lists are not read once per IP.** A list the API server does not serve (404, e.g. no Istio) is an empty list. A failed list read keeps the list read before; with none, the failure is kept. After either, the list is not read again for 3 minutes, even for a pod not in it. So a 403 on ServiceEntries is read once, not once per new IP.
   - **An eBPF connection resolves its unknown addresses again.** The later messages of a connection do not carry it again; a connection whose addresses were unknown now resolves them again on those messages once their metadata is read, instead of staying unknown (or dropped as same-service traffic) for the rest of the stream. A resolved address is kept.
   - **Every setting has a code default and is written out in `application.yml` with the same value.** An invalid value, including a bare number for an `als=…,mal=…` key, stops OAP at boot with a message showing the form.
   
   **Lookup rules** (defaults; keys in `core.default`)
   
   | | Envoy ALS `k8s-mesh` (and Istio ServiceEntries), eBPF access log | MAL `retagByK8sMeta`, process naming |
   |---|---|---|
   | Key not cached | unknown now; the read is queued (`K8sLookup` lane); an eBPF connection resolves it again on its later messages | all the step's reads queued first (`K8sLookup-waiting` lane), then waited for up to `malMetadataReadTimeout` (2000 ms) in all, the steps of one Export together |
   | Data cached | the data; read again in the background on the first use `k8sMetadataRefreshPeriod` (3 min) after it was written | the same |
   | Nothing found | unknown; read again on the first use `k8sMetadataNotFoundRetryPeriod` (1 min) after it was written; endpoints read again for a pod not in them once that old | empty service; the same, with the services list |
   | Read failed (403, 5xx, not answering, refused) | unknown; retried on the first use `k8sMetadataRefreshPeriod` (3 min) after it failed | empty service; the same |
   | Read again of data finds nothing or fails | data kept; retried on a use a refresh period later | the same |
   | Dropped | `k8sMetadataCacheTime` `als` (21 min) after the last write | `k8sMetadataCacheTime` `mal` (21 min) after the last write |
   | Lane full (`k8sMetadataReadQueueSize`) | unknown, not cached; queued again by the next lookup | empty service at once; the same |
   
   **Settings**
   
   | Key | Env | Default |
   |---|---|---|
   | `malMetadataReadTimeout` | `SW_CORE_MAL_METADATA_READ_TIMEOUT` | `2000` ms |
   | `k8sMetadataCacheTime` | `SW_CORE_K8S_METADATA_CACHE_TIME` | `als=21,mal=21` minutes |
   | `k8sMetadataRefreshPeriod` | `SW_CORE_K8S_METADATA_REFRESH_PERIOD` | `3` minutes |
   | `k8sMetadataNotFoundRetryPeriod` | `SW_CORE_K8S_METADATA_NOT_FOUND_RETRY_PERIOD` | `1` minute |
   | `k8sMetadataReadQueueSize` | `SW_CORE_K8S_METADATA_READ_QUEUE_SIZE` | empty (no limit) |
   
   Two reader threads serve all of it, one per lane. Each runs one read at a time, is created on demand and ends after 60 s idle (a virtual thread on JDK 25+). Refreshes are triggered by use, not by a timer.
   
   **Verified against a real API server** (kind, with the API server's audit log as the record of each request OAP sent):
   - **New pod:** a new pod's IP was first looked up 1 s after the pod started, and found nothing, as the endpoints read 24 s earlier lacked it. The use 66 s later read the endpoints again and resolved it.
   - **403:** nothing was read again at +5 s or +75 s; everything was read again at +186 s. ServiceEntries were listed once, not once per IP.
   - **MAL with the API server unresponsive:** one Export of 50 unread pods, through four `retagByK8sMeta`, took 2.2 s; the same pods again took 0.2 s; 41 cached pods took 0.2 s. With `mal=1`, 6 unread pods took 2.3 s.
   - **Short cache (1-minute refresh, 2-minute cache time):** pods in use were read again at +66 s and +129 s and stayed cached. Pods unused for 2.5 minutes had expired.
   - **Envoy ALS never waited:** 0.2–0.6 s per batch, healthy, paused or 403, on JDK 11 and 25.
   - **Load with the API server unresponsive:** 2000 unread IPs over 50 streams, JDK 11. No thread grew; only the two readers were in Kubernetes code.
   - **Boot:** every invalid setting stopped OAP with a message showing the expected form.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org