You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@pekko.apache.org by "thjaeckle (via GitHub)" <gi...@apache.org> on 2024/10/15 12:09:20 UTC
[I] Supporting to fetch pods in Kubernetes based discovery with `gzip` compression [pekko-management]
thjaeckle opened a new issue, #335:
URL: https://github.com/apache/pekko-management/issues/335
Hello Pekko maintainers.
We currently face an issue where the response of Kubernete's HTTP API is >1 MB (as we e.g. have 15 Pekko pods in our cluster and the `api/v1/namespaces/$NAMESPACE/pods` endpoint also returns all environment variables of all pods - which can become quite big.
So I looked at options and found that the [Kubernetes HTTP API supports compression for responses](https://kubernetes.io/docs/reference/using-api/api-concepts/#response-compression) since 1.16, simply by setting the HTTP header `Accept-Encoding: gzip`.
In our case that brings down the response size a lot:
```
-rw-r--r--. 1 ubuntu ubuntu 112709 Oct 15 13:59 out2.tar.gz
-rw-r--r--. 1 ubuntu ubuntu 1056800 Oct 15 13:37 out.txt
```
Would you accept a contribution to make gzip compression (and decoding) configurable at this place:
https://github.com/apache/pekko-management/blob/5703a2a0459f941c22b9f2fa768c36f33f727703/discovery-kubernetes-api/src/main/scala/org/apache/pekko/discovery/kubernetes/KubernetesApiServiceDiscovery.scala#L229-L240
I would be happy to contribute that.
Our alternative would be to increase the default payload limits of http responses to greater 1 MB (which is the default for pekko-http I think) - however for the future I see potential problems with that, if our cluster grows in number of pods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [I] Supporting to fetch pods in Kubernetes based discovery with `gzip` compression [pekko-management]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on issue #335:
URL: https://github.com/apache/pekko-management/issues/335#issuecomment-2413800211
Supporting gzip compression seems useful.
With the max-chunk-size, I suggest trying out different values but it seems plausible that this setting is in play.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [I] Supporting to fetch pods in Kubernetes based discovery with `gzip` compression [pekko-management]
Posted by "thjaeckle (via GitHub)" <gi...@apache.org>.
thjaeckle commented on issue #335:
URL: https://github.com/apache/pekko-management/issues/335#issuecomment-2413908770
To make this issue better findable via search machines, I paste the error message we saw when this happened:
```
Resolve attempt failed! Cause: org.apache.pekko.http.scaladsl.model.EntityStreamException: HTTP chunk size exceeds the configured limit of 1048576 bytes
```
Logged by logger: `org.apache.pekko.management.cluster.bootstrap.internal.BootstrapCoordinator`
Pekko source: `pekko://ditto-cluster/system/bootstrapCoordinator`
But great .. as short-term fix I will increase our `-Dpekko.http.client.parsing.max-chunk-size=2m` - as long-term fix I try to contribute the gzip compression ..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [I] Supporting to fetch pods in Kubernetes based discovery with `gzip` compression [pekko-management]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on issue #335:
URL: https://github.com/apache/pekko-management/issues/335#issuecomment-2413803470
https://pekko.apache.org/docs/pekko-http/1.1/common/encoding.html may be of interest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [I] Supporting to fetch pods in Kubernetes based discovery with `gzip` compression [pekko-management]
Posted by "thjaeckle (via GitHub)" <gi...@apache.org>.
thjaeckle commented on issue #335:
URL: https://github.com/apache/pekko-management/issues/335#issuecomment-2413774487
I assume that the 1MB default is configured here as `max-chunk-size`:
https://github.com/apache/pekko-http/blob/91315b324e7ca45c9a4892b09a0609bb69afa391/http-core/src/main/resources/reference.conf#L693C5-L693C19
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [I] Supporting to fetch pods in Kubernetes based discovery with `gzip` compression [pekko-management]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning closed issue #335: Supporting to fetch pods in Kubernetes based discovery with `gzip` compression
URL: https://github.com/apache/pekko-management/issues/335
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org