You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "cobed95 (via GitHub)" <gi...@apache.org> on 2026/08/28 02:04:02 UTC
[I] Polaris crash-loops on local Kubernetes: the OTel GCP resource detector blocks startup for ~135s [polaris]
cobed95 opened a new issue, #5401:
URL: https://github.com/apache/polaris/issues/5401
### Describe the bug
### Summary
Where `metadata.google.internal` resolves to an address that silently drops packets, the `main` thread blocks ~135s in a socket `connect()` to the GCP metadata server, from an OpenTelemetry resource detector run during Quarkus startup. Nothing binds 8181/8182 until it returns.
This happens **even with Polaris's shipped `quarkus.otel.sdk.disabled=true` default**: OTel autoconfigure runs resource providers before honoring that flag.
A bare pod recovers after the stall. Under the Helm chart it does not — the default liveness probe kills the container ~60s in, so the pod crash-loops indefinitely. There is no error and no stack trace, just an empty pod log.
### To Reproduce
### Reproduced on
* `main` 4b4218638 (Quarkus 3.38.3, JDK 21) and published apache/polaris:1.7.0
* kind, k3d/k3s, minikube, OrbStack Kubernetes, and plain `docker run`
* Not reproducible on Amazon EKS — see below
### Steps to reproduce
```shell
docker run --rm apache/polaris:1.7.0
```
### Actual Behavior
No `started in` line for ~135s.
### Measured
Same image (`apache/polaris:1.7.0`); "startup" is Quarkus's own `started in` figure, excluding image pull.
| Environment | `metadata.google.internal` | `169.254.169.254` | Startup |
| --- | --- | --- | --- |
| Amazon EKS v1.34.9 | **NXDOMAIN** | live IMDS (`401`/`200`, `connect=0.0002s`) | **2.772s** |
| kind v0.32.0 | 169.254.169.254 | no response | **134.326s** |
| k3d v5.9.0 / k3s v1.35.5 | 169.254.169.254 | no response | **135.607s** |
| minikube v1.38.1 (docker) | 169.254.169.254 | no response | **137.166s** |
| OrbStack Kubernetes v1.35.6 | 169.254.169.254 | no response | **136.217s** |
| OrbStack Docker | 169.254.169.254 | no response | **137.740s** |
The affected environments agree at ~134-138s: a TCP connect timeout, not an unbounded hang.
EKS is protected twice over. `getent hosts metadata.google.internal` returns NXDOMAIN (`.internal` is reserved and never publicly delegated), and even if it resolved, `169.254.169.254` there is the live IMDS endpoint answering in 0.2ms. The stall needs a *silent drop*, not merely an absent metadata service. Running the same image with `--dns 1.1.1.1` locally confirms the NXDOMAIN path: 1.589s.
At risk, then: local/VM Kubernetes and container runtimes that map the metadata name to an unreachable link-local address, plus split-horizon or wildcard DNS that returns a sinkhole A record instead of NXDOMAIN.
### Impact under the helm chart
The chart's `livenessProbe` (`/q/health/live`, `initialDelaySeconds: 5`, `periodSeconds: 10`, `failureThreshold: 3`) starts probing while nothing is listening. Deployed with those settings, the pod restarted 5 times in 5 minutes, never healthy — `failed liveness probe, will be restarted`, `connection refused` on 8182, `exitCode=137`. It gets ~60s and needs ~135s, so it can never win that race.
### Expected Behavior
`started in` should show up within ~1.5s.
### Additional context
### Root cause
SIGQUIT thread dump of the stalled `main` thread:
```
sun.nio.ch.Net.connect0(Native Method)
...
com.google.cloud.opentelemetry.detection.GCPMetadataConfig.fetchAttribute(GCPMetadataConfig.java:155)
com.google.cloud.opentelemetry.detection.GCPMetadataConfig.getProjectId(GCPMetadataConfig.java:52)
com.google.cloud.opentelemetry.detection.GCPPlatformDetector.isRunningOnGcp(GCPPlatformDetector.java:65)
io.opentelemetry.contrib.gcp.resource.GCPResourceProvider.getAttributes(GCPResourceProvider.java:88)
io.opentelemetry.sdk.autoconfigure.ResourceConfiguration.configureResource(ResourceConfiguration.java:81)
io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$3.apply(OpenTelemetryRecorder.java:116)
```
`GCPMetadataConfig` hardcodes `http://metadata.google.internal/computeMetadata/v1/` with no connect timeout, so two conditions must hold together:
1. `metadata.google.internal` resolves, **and**
2. that address silently drops packets, so `connect()` blocks for the full TCP
SYN-retry budget instead of failing.
That is why it reproduces in some environments and not others.
### Why it reaches the classpath
Not a direct dependency — it arrives transitively with GCS support:
```
:polaris-runtime-service -> :polaris-core
-> com.google.cloud:google-cloud-storage:2.70.0
-> io.opentelemetry.contrib:opentelemetry-gcp-resources:1.37.0-alpha
-> com.google.cloud.opentelemetry:detector-resources-support:0.33.0
```
`opentelemetry-gcp-resources` registers `GCPResourceProvider` in `META-INF/services/io.opentelemetry.sdk.autoconfigure.spi.ResourceProvider`, so autoconfigure discovers and runs it. Nothing in Polaris opts in. No AWS resource detectors are on the classpath, so only the GCP one is implicated.
### Workaround
```
OTEL_JAVA_DISABLED_RESOURCE_PROVIDERS=io.opentelemetry.contrib.gcp.resource.GCPResourceProvider
```
Same build: `started in 1.809s`.
### Suggested fix
Production cloud deployments (EKS/EC2, and GCP itself) are unaffected, so this likely does not warrant changing behaviour for everyone.
1. **Document it.** The main ask. The symptom offers nothing to search for. A short note in the Helm / getting-started docs — symptom, the `GCPMetadataConfig.fetchAttribute` thread-dump signature, and the env var above — would make it self-service.
2. **A chart toggle.** `tracing.enabled: false` (the default) is already the branch setting `quarkus.otel.sdk.disabled=true` in `helm/polaris/templates/configmap.yaml:211`. Either have that branch also disable the cloud resource providers (an operator who turned tracing off never wants a metadata probe blocking startup), or expose `tracing.disabledResourceProviders` for those who enable tracing but run in an affected environment. No strong preference.
**Not suggested:** disabling `GCPResourceProvider` globally in `runtime/defaults` — that costs resource attributes for legitimate GCP users to fix a sinkhole-DNS problem.
Happy to open a PR if maintainers agree on the approach.
### System information
OS: macOS 26.2 (arm64) — host for every local environment below
Polaris Catalog Version: apache/polaris:1.7.0 (published image), and main 4b4218638 built locally (polaris-server 1.7.0-SNAPSHOT)
Object storage & setup: none — default in-memory persistence, no catalog or object storage configured. The bug occurs during Quarkus startup, before any storage is touched.
Runtime detail, since the trigger is environmental rather than configuration:
JVM: OpenJDK 21.0.11 (in image) / 21.0.12 (local build)
Quarkus: 3.37.4 (image) / 3.38.3 (main @ 4b4218638)
Container runtime: OrbStack 29.4.0 (Docker-compatible)
Affected: kind v0.32.0, k3d v5.9.0 / k3s v1.35.5,
minikube v1.38.1 (docker driver),
OrbStack Kubernetes v1.35.6, plain `docker run`
Not affected: Amazon EKS v1.34.9
The decisive variable is DNS, not the Polaris configuration: whether `metadata.google.internal` resolves, and whether that address answers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Polaris crash-loops on local Kubernetes: the OTel GCP resource detector blocks startup for ~135s [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on issue #5401:
URL: https://github.com/apache/polaris/issues/5401#issuecomment-5450512416
Hi @cobed95 thank you for the report. I think that disabling the cloud resources in the chart when tracing is disabled would make sense. Looking forward to your PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Polaris crash-loops on local Kubernetes: the OTel GCP resource detector blocks startup for ~135s [polaris]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on issue #5401:
URL: https://github.com/apache/polaris/issues/5401#issuecomment-5454231574
I will work on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Polaris crash-loops on local Kubernetes: the OTel GCP resource detector blocks startup for ~135s [polaris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5401:
URL: https://github.com/apache/polaris/issues/5401#issuecomment-6431715877
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment if you think it's still relevant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Polaris crash-loops on local Kubernetes: the OTel GCP resource detector blocks startup for ~135s [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on issue #5401:
URL: https://github.com/apache/polaris/issues/5401#issuecomment-5826148263
@zhang-arvin are you still working on this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org