You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@pekko.apache.org by "mdedetrich (via GitHub)" <gi...@apache.org> on 2025/11/17 14:32:07 UTC
[PR] Get token on every request [pekko-management]
mdedetrich opened a new pull request, #546:
URL: https://github.com/apache/pekko-management/pull/546
Resolves: https://github.com/apache/pekko-management/issues/543
The core issue here is that getting the token is just a `lazy val` (and so are computing the headers) which means that after the first call the token is just cached indefintiely.
We can explore using a cache, albeit I do think that reading from a file should be fast/cheap enough that its not necessary and I don't think these API calls are a bottleneck. Can also write a test if requested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #546:
URL: https://github.com/apache/pekko-management/pull/546#discussion_r2539139009
##########
lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/internal/AbstractKubernetesApiImpl.scala:
##########
@@ -66,20 +67,32 @@ import scala.util.control.NonFatal
private lazy val clientSslContext: HttpsConnectionContext = ConnectionContext.httpsClient(sslContext)
- protected val namespace: String =
- settings.namespace.orElse(readConfigVarFromFilesystem(settings.namespacePath, "namespace")).getOrElse("default")
+ protected val namespace: Future[String] = {
+ settings.namespace match {
+ case Some(nSpace) => Future.successful(nSpace)
+ case _ =>
+ readConfigVarFromFilesystem(settings.namespacePath, "namespace").map(_.getOrElse("default"))(
+ ExecutionContext.parasitic)
+ }
+ }
protected val scheme: String = if (settings.secure) "https" else "http"
- private lazy val apiToken = readConfigVarFromFilesystem(settings.apiTokenPath, "api-token").getOrElse("")
- private lazy val headers = if (settings.secure) immutable.Seq(Authorization(OAuth2BearerToken(apiToken))) else Nil
+ private[pekko] def apiToken() = readConfigVarFromFilesystem(settings.apiTokenPath, "api-token").map(
+ _.getOrElse(""))(ExecutionContext.parasitic)
+ private def headers() = if (settings.secure) {
+ apiToken().map { token =>
+ immutable.Seq(Authorization(OAuth2BearerToken(token)))
+ }(ExecutionContext.parasitic)
+ } else
+ Future.successful(Nil)
log.debug("kubernetes access namespace: {}. Secure: {}", namespace, settings.secure)
Review Comment:
This makes no sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3542500418
> Hmm I made a regression, let me look into it
Disregard this, its a flaky test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #546:
URL: https://github.com/apache/pekko-management/pull/546#discussion_r2539137838
##########
lease-kubernetes/src/main/resources/reference.conf:
##########
@@ -52,4 +52,19 @@ pekko.coordination.lease.kubernetes {
# server that are required. If this timeout is hit then the lease *may* be taken due to the response being lost
# on the way back from the API server but will be reported as not taken and can be safely retried.
lease-operation-timeout = 5s
+
+ # Settings that are specific to retrying requests with 401 responses due to possible token rotation
+ token-rotation-retry {
+ # Number of total attempts to make
+ attempts = 5
Review Comment:
Fixed and pushed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3546285500
I have just updated the PR so that `readConfigVarFromFilesystem` from `AbstractKubernetesApiImpl` returns a `Future[Option[String]]` instead of `Future[Option[ByteString]]` so that the API is easier to work with and the changeset is smaller
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3560403511
> * I can't find any docs that describe the overlap behaviour of old and new tokens.
> * I don't think that the 401 retries have a downside. I would keep the code. It may be justifiable to reduce the default max-attempts to 0 or some slightly higher value like 1 or 2. Little reason to go for a default value like 5. I would favour a value like 2.
> * I don't see a real benefit to failing fast on a 401.
Fair enough, I am going to go ahead and merge this and will follow up with a cherry pick for 1.2.x branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #546:
URL: https://github.com/apache/pekko-management/pull/546#discussion_r2536922111
##########
lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/internal/AbstractKubernetesApiImpl.scala:
##########
@@ -148,9 +160,11 @@ import scala.util.control.NonFatal
protected def requestForPath(
path: Uri.Path,
method: HttpMethod = HttpMethods.GET,
- entity: RequestEntity = HttpEntity.Empty): HttpRequest = {
+ entity: RequestEntity = HttpEntity.Empty): Future[HttpRequest] = {
val uri = Uri.from(scheme = scheme, host = settings.apiServerHost, port = settings.apiServerPort).withPath(path)
- HttpRequest(uri = uri, headers = headers, method = method, entity = entity)
+ headers().map { headers =>
+ HttpRequest(uri = uri, headers = headers, method = method, entity = entity)
+ }
Review Comment:
Thanks, fixed and pushed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #546:
URL: https://github.com/apache/pekko-management/pull/546#discussion_r2537782233
##########
lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/internal/AbstractKubernetesApiImpl.scala:
##########
@@ -148,17 +160,34 @@ import scala.util.control.NonFatal
protected def requestForPath(
path: Uri.Path,
method: HttpMethod = HttpMethods.GET,
- entity: RequestEntity = HttpEntity.Empty): HttpRequest = {
+ entity: RequestEntity = HttpEntity.Empty): Future[HttpRequest] = {
val uri = Uri.from(scheme = scheme, host = settings.apiServerHost, port = settings.apiServerPort).withPath(path)
- HttpRequest(uri = uri, headers = headers, method = method, entity = entity)
+ headers().map { headers =>
+ HttpRequest(uri = uri, headers = headers, method = method, entity = entity)
+ }(ExecutionContext.parasitic)
+ }
+
+ private[pekko] def makeRawRequest(request: HttpRequest): Future[HttpResponse] = {
+ if (settings.secure)
+ http.singleRequest(request, clientSslContext)
+ else
+ http.singleRequest(request)
}
protected def makeRequest(request: HttpRequest, timeoutMsg: String): Future[HttpResponse] = {
- val response =
- if (settings.secure)
- http.singleRequest(request, clientSslContext)
- else
- http.singleRequest(request)
+ // It's possible to legitimately get a 401 response due to kubernetes doing a token rotation
+ implicit val scheduler: Scheduler = system.scheduler
+ val response = RetrySupport.retry(
Review Comment:
When cherry picking I back to 1.2.x branch I will need to use [this](https://github.com/apache/pekko/blob/870d4499cc49630eebfcecb3f10d0be9dde59b5a/actor/src/main/scala/org/apache/pekko/pattern/RetrySupport.scala#L86-L128) version of `RetrySupport.retry` since that is available in Pekko 1.0.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #546:
URL: https://github.com/apache/pekko-management/pull/546#discussion_r2539140405
##########
lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/internal/AbstractKubernetesApiImpl.scala:
##########
@@ -148,17 +160,34 @@ import scala.util.control.NonFatal
protected def requestForPath(
path: Uri.Path,
method: HttpMethod = HttpMethods.GET,
- entity: RequestEntity = HttpEntity.Empty): HttpRequest = {
+ entity: RequestEntity = HttpEntity.Empty): Future[HttpRequest] = {
val uri = Uri.from(scheme = scheme, host = settings.apiServerHost, port = settings.apiServerPort).withPath(path)
- HttpRequest(uri = uri, headers = headers, method = method, entity = entity)
+ headers().map { headers =>
+ HttpRequest(uri = uri, headers = headers, method = method, entity = entity)
+ }(ExecutionContext.parasitic)
+ }
+
+ private[pekko] def makeRawRequest(request: HttpRequest): Future[HttpResponse] = {
+ if (settings.secure)
+ http.singleRequest(request, clientSslContext)
+ else
+ http.singleRequest(request)
}
protected def makeRequest(request: HttpRequest, timeoutMsg: String): Future[HttpResponse] = {
- val response =
- if (settings.secure)
- http.singleRequest(request, clientSslContext)
- else
- http.singleRequest(request)
+ // It's possible to legitimately get a 401 response due to kubernetes doing a token rotation
+ implicit val scheduler: Scheduler = system.scheduler
+ val response = RetrySupport.retry(
+ () => makeRawRequest(request: HttpRequest),
+ (response: HttpResponse, _: Throwable) => {
+ log.warning("Received status code 401 as response, retrying due to possible token rotation")
+ response.status == StatusCodes.Unauthorized
+ },
+ settings.tokenRetrySettings.attempts,
+ settings.tokenRetrySettings.minBackoff,
+ settings.tokenRetrySettings.maxBackoff,
+ settings.tokenRetrySettings.randomFactor
+ )
Review Comment:
This is also wrong, we know that the token will refresh (there is also a test for it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on code in PR #546:
URL: https://github.com/apache/pekko-management/pull/546#discussion_r2534624912
##########
lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/internal/AbstractKubernetesApiImpl.scala:
##########
@@ -19,21 +19,22 @@ import pekko.actor.ActorSystem
import pekko.annotation.InternalApi
import pekko.coordination.lease.kubernetes.{ KubernetesApi, KubernetesSettings, LeaseResource }
import pekko.coordination.lease.{ LeaseException, LeaseTimeoutException }
+import pekko.dispatch.ExecutionContexts
Review Comment:
in main branch, you should use scala ExecutionContext methods because we don't support scala 2.12
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #546:
URL: https://github.com/apache/pekko-management/pull/546#discussion_r2534647253
##########
lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/internal/AbstractKubernetesApiImpl.scala:
##########
@@ -19,21 +19,22 @@ import pekko.actor.ActorSystem
import pekko.annotation.InternalApi
import pekko.coordination.lease.kubernetes.{ KubernetesApi, KubernetesSettings, LeaseResource }
import pekko.coordination.lease.{ LeaseException, LeaseTimeoutException }
+import pekko.dispatch.ExecutionContexts
Review Comment:
Fixed, didn't realize that we also changed pekko-management for 2.0.x. I will use `pekko.dispatch.ExecutionContexts` when I cherry pick to the 1.x branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3546348547
Ill add a test to actually confirm that this works and will then merge the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3547258668
@danischroeter @pjfanning @He-Pin Can you re-review? I have updated the PR with the retry mechanism plus added 2 tests to make sure its behaving as expected, i.e. that we do an exponential backoff retry when we hit 401 but if we exceed that retry we still return the current unauthoirzed `LeaseException`.
I tried to pick sensible defaults for the various token retry settings
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #546:
URL: https://github.com/apache/pekko-management/pull/546#discussion_r2539140985
##########
lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/internal/AbstractKubernetesApiImpl.scala:
##########
@@ -148,17 +160,34 @@ import scala.util.control.NonFatal
protected def requestForPath(
path: Uri.Path,
method: HttpMethod = HttpMethods.GET,
- entity: RequestEntity = HttpEntity.Empty): HttpRequest = {
+ entity: RequestEntity = HttpEntity.Empty): Future[HttpRequest] = {
val uri = Uri.from(scheme = scheme, host = settings.apiServerHost, port = settings.apiServerPort).withPath(path)
- HttpRequest(uri = uri, headers = headers, method = method, entity = entity)
+ headers().map { headers =>
+ HttpRequest(uri = uri, headers = headers, method = method, entity = entity)
+ }(ExecutionContext.parasitic)
+ }
+
+ private[pekko] def makeRawRequest(request: HttpRequest): Future[HttpResponse] = {
+ if (settings.secure)
+ http.singleRequest(request, clientSslContext)
+ else
+ http.singleRequest(request)
}
protected def makeRequest(request: HttpRequest, timeoutMsg: String): Future[HttpResponse] = {
- val response =
- if (settings.secure)
- http.singleRequest(request, clientSslContext)
- else
- http.singleRequest(request)
+ // It's possible to legitimately get a 401 response due to kubernetes doing a token rotation
+ implicit val scheduler: Scheduler = system.scheduler
+ val response = RetrySupport.retry(
+ () => makeRawRequest(request: HttpRequest),
+ (response: HttpResponse, _: Throwable) => {
Review Comment:
This is also wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #546:
URL: https://github.com/apache/pekko-management/pull/546#discussion_r2539133838
##########
lease-kubernetes/src/main/resources/reference.conf:
##########
@@ -52,4 +52,19 @@ pekko.coordination.lease.kubernetes {
# server that are required. If this timeout is hit then the lease *may* be taken due to the response being lost
# on the way back from the API server but will be reported as not taken and can be safely retried.
lease-operation-timeout = 5s
+
+ # Settings that are specific to retrying requests with 401 responses due to possible token rotation
+ token-rotation-retry {
+ # Number of total attempts to make
+ attempts = 5
Review Comment:
`max-attempts` is clear, doing this now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3562308582
> Should we call it max-retries, Spring 7 just changed to that
`max-attempts` is what the Pekko terminology is afaik
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #546:
URL: https://github.com/apache/pekko-management/pull/546#discussion_r2537782233
##########
lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/internal/AbstractKubernetesApiImpl.scala:
##########
@@ -148,17 +160,34 @@ import scala.util.control.NonFatal
protected def requestForPath(
path: Uri.Path,
method: HttpMethod = HttpMethods.GET,
- entity: RequestEntity = HttpEntity.Empty): HttpRequest = {
+ entity: RequestEntity = HttpEntity.Empty): Future[HttpRequest] = {
val uri = Uri.from(scheme = scheme, host = settings.apiServerHost, port = settings.apiServerPort).withPath(path)
- HttpRequest(uri = uri, headers = headers, method = method, entity = entity)
+ headers().map { headers =>
+ HttpRequest(uri = uri, headers = headers, method = method, entity = entity)
+ }(ExecutionContext.parasitic)
+ }
+
+ private[pekko] def makeRawRequest(request: HttpRequest): Future[HttpResponse] = {
+ if (settings.secure)
+ http.singleRequest(request, clientSslContext)
+ else
+ http.singleRequest(request)
}
protected def makeRequest(request: HttpRequest, timeoutMsg: String): Future[HttpResponse] = {
- val response =
- if (settings.secure)
- http.singleRequest(request, clientSslContext)
- else
- http.singleRequest(request)
+ // It's possible to legitimately get a 401 response due to kubernetes doing a token rotation
+ implicit val scheduler: Scheduler = system.scheduler
+ val response = RetrySupport.retry(
Review Comment:
When cherry picking I back to 1.2.x branch I will need to refactor and use [this](https://github.com/apache/pekko/blob/870d4499cc49630eebfcecb3f10d0be9dde59b5a/actor/src/main/scala/org/apache/pekko/pattern/RetrySupport.scala#L86-L128) version of `RetrySupport.retry` since that is available in Pekko 1.0.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3543788316
Main branch targets 2.0.0. We can backport this to 1.2.x branch from it is merged. That backport PR can have 1.2.0 milestone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on code in PR #546:
URL: https://github.com/apache/pekko-management/pull/546#discussion_r2538707268
##########
lease-kubernetes/src/main/resources/reference.conf:
##########
@@ -52,4 +52,19 @@ pekko.coordination.lease.kubernetes {
# server that are required. If this timeout is hit then the lease *may* be taken due to the response being lost
# on the way back from the API server but will be reported as not taken and can be safely retried.
lease-operation-timeout = 5s
+
+ # Settings that are specific to retrying requests with 401 responses due to possible token rotation
+ token-rotation-retry {
+ # Number of total attempts to make
+ attempts = 5
Review Comment:
max-attempts-num ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3542346624
will check when on a laptop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "laurens-gross-mw (via GitHub)" <gi...@apache.org>.
laurens-gross-mw commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3556135624
Hi @mdedetrich,
regarding:
> implies there may be some bottleneck/performance considerations in how often this API method is called.
> I mean if the API is called every now and then, even it being blocking wouldn't be a real issue in production??? If there is a performance consideration then I probably should implement a cache (but in doing so I need to know how long it is before an old token that is rotated out expires). @raboof @laurens-gross-mw maybe you have some context here?
A token is 1h valid and has ~5min overlap with a new token in our system. So from our side, no caching is needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3546978955
> One point raised is that there still might be timing issues where you read the token file but it rotates just before you use it. According to the AI I used, the old token is invalidated at rotation time. That is, you are unlucky and the API call happens at same time as a token rotation event and you start by reading the token file moments before it is rotated so you could make the API call with a recently invalidated token.
>
> So, maybe the best enhancement is to handle 401s by retrying.
>
> We could come back to caching later.
I am implementing a retry mechanism now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3562301162
Should we call it max-retries, Spring 7 just changed to that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on code in PR #546:
URL: https://github.com/apache/pekko-management/pull/546#discussion_r2536144794
##########
lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/internal/AbstractKubernetesApiImpl.scala:
##########
@@ -148,9 +160,11 @@ import scala.util.control.NonFatal
protected def requestForPath(
path: Uri.Path,
method: HttpMethod = HttpMethods.GET,
- entity: RequestEntity = HttpEntity.Empty): HttpRequest = {
+ entity: RequestEntity = HttpEntity.Empty): Future[HttpRequest] = {
val uri = Uri.from(scheme = scheme, host = settings.apiServerHost, port = settings.apiServerPort).withPath(path)
- HttpRequest(uri = uri, headers = headers, method = method, entity = entity)
+ headers().map { headers =>
+ HttpRequest(uri = uri, headers = headers, method = method, entity = entity)
+ }
Review Comment:
(ExecutionContext.parasitic)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #546:
URL: https://github.com/apache/pekko-management/pull/546#discussion_r2538733844
##########
lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/internal/AbstractKubernetesApiImpl.scala:
##########
@@ -66,20 +67,32 @@ import scala.util.control.NonFatal
private lazy val clientSslContext: HttpsConnectionContext = ConnectionContext.httpsClient(sslContext)
- protected val namespace: String =
- settings.namespace.orElse(readConfigVarFromFilesystem(settings.namespacePath, "namespace")).getOrElse("default")
+ protected val namespace: Future[String] = {
+ settings.namespace match {
+ case Some(nSpace) => Future.successful(nSpace)
+ case _ =>
+ readConfigVarFromFilesystem(settings.namespacePath, "namespace").map(_.getOrElse("default"))(
+ ExecutionContext.parasitic)
+ }
+ }
protected val scheme: String = if (settings.secure) "https" else "http"
- private lazy val apiToken = readConfigVarFromFilesystem(settings.apiTokenPath, "api-token").getOrElse("")
- private lazy val headers = if (settings.secure) immutable.Seq(Authorization(OAuth2BearerToken(apiToken))) else Nil
+ private[pekko] def apiToken() = readConfigVarFromFilesystem(settings.apiTokenPath, "api-token").map(
+ _.getOrElse(""))(ExecutionContext.parasitic)
+ private def headers() = if (settings.secure) {
+ apiToken().map { token =>
+ immutable.Seq(Authorization(OAuth2BearerToken(token)))
+ }(ExecutionContext.parasitic)
+ } else
+ Future.successful(Nil)
log.debug("kubernetes access namespace: {}. Secure: {}", namespace, settings.secure)
Review Comment:
The `namespace` field is now a `Future[String]`, but it's being logged directly. This will log the Future object representation (e.g., `Future(<not completed>)`) instead of the actual namespace value. This logging should either be removed or wrapped in a callback that logs the value once the Future completes.
```suggestion
namespace.foreach { ns =>
log.debug("kubernetes access namespace: {}. Secure: {}", ns, settings.secure)
}(system.dispatcher)
```
##########
lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/internal/AbstractKubernetesApiImpl.scala:
##########
@@ -148,17 +160,34 @@ import scala.util.control.NonFatal
protected def requestForPath(
path: Uri.Path,
method: HttpMethod = HttpMethods.GET,
- entity: RequestEntity = HttpEntity.Empty): HttpRequest = {
+ entity: RequestEntity = HttpEntity.Empty): Future[HttpRequest] = {
val uri = Uri.from(scheme = scheme, host = settings.apiServerHost, port = settings.apiServerPort).withPath(path)
- HttpRequest(uri = uri, headers = headers, method = method, entity = entity)
+ headers().map { headers =>
+ HttpRequest(uri = uri, headers = headers, method = method, entity = entity)
+ }(ExecutionContext.parasitic)
+ }
+
+ private[pekko] def makeRawRequest(request: HttpRequest): Future[HttpResponse] = {
+ if (settings.secure)
+ http.singleRequest(request, clientSslContext)
+ else
+ http.singleRequest(request)
}
protected def makeRequest(request: HttpRequest, timeoutMsg: String): Future[HttpResponse] = {
- val response =
- if (settings.secure)
- http.singleRequest(request, clientSslContext)
- else
- http.singleRequest(request)
+ // It's possible to legitimately get a 401 response due to kubernetes doing a token rotation
+ implicit val scheduler: Scheduler = system.scheduler
+ val response = RetrySupport.retry(
+ () => makeRawRequest(request: HttpRequest),
+ (response: HttpResponse, _: Throwable) => {
+ log.warning("Received status code 401 as response, retrying due to possible token rotation")
+ response.status == StatusCodes.Unauthorized
+ },
+ settings.tokenRetrySettings.attempts,
+ settings.tokenRetrySettings.minBackoff,
+ settings.tokenRetrySettings.maxBackoff,
+ settings.tokenRetrySettings.randomFactor
+ )
Review Comment:
The retry logic retries with the same `HttpRequest` object that was built with the old token. When a 401 occurs due to token rotation, retrying with the same request (containing the old token in headers) will keep failing. The request should be rebuilt on each retry by calling `requestForPath` again to fetch the fresh token. Consider restructuring to pass a function that builds the request instead of the request itself.
##########
lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/internal/AbstractKubernetesApiImpl.scala:
##########
@@ -148,17 +160,34 @@ import scala.util.control.NonFatal
protected def requestForPath(
path: Uri.Path,
method: HttpMethod = HttpMethods.GET,
- entity: RequestEntity = HttpEntity.Empty): HttpRequest = {
+ entity: RequestEntity = HttpEntity.Empty): Future[HttpRequest] = {
val uri = Uri.from(scheme = scheme, host = settings.apiServerHost, port = settings.apiServerPort).withPath(path)
- HttpRequest(uri = uri, headers = headers, method = method, entity = entity)
+ headers().map { headers =>
+ HttpRequest(uri = uri, headers = headers, method = method, entity = entity)
+ }(ExecutionContext.parasitic)
+ }
+
+ private[pekko] def makeRawRequest(request: HttpRequest): Future[HttpResponse] = {
+ if (settings.secure)
+ http.singleRequest(request, clientSslContext)
+ else
+ http.singleRequest(request)
}
protected def makeRequest(request: HttpRequest, timeoutMsg: String): Future[HttpResponse] = {
- val response =
- if (settings.secure)
- http.singleRequest(request, clientSslContext)
- else
- http.singleRequest(request)
+ // It's possible to legitimately get a 401 response due to kubernetes doing a token rotation
+ implicit val scheduler: Scheduler = system.scheduler
+ val response = RetrySupport.retry(
+ () => makeRawRequest(request: HttpRequest),
+ (response: HttpResponse, _: Throwable) => {
Review Comment:
When retrying on a 401 response, the response body is not being consumed before retrying. HTTP response entities should be drained using `response.discardEntityBytes()` to prevent resource leaks and connection pool issues. Add `response.discardEntityBytes()` before the retry condition check.
```suggestion
(response: HttpResponse, _: Throwable) => {
response.discardEntityBytes()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3542162727
Hmm just noticed https://github.com/apache/pekko-management/blob/606f90024f7e8a57385c4f170cc4c897714dc0cc/lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/internal/AbstractKubernetesApiImpl.scala#L172-L174, let me update PR to use non blocking IO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3556338140
> Hi @mdedetrich, regarding:
>
> > implies there may be some bottleneck/performance considerations in how often this API method is called.
> > I mean if the API is called every now and then, even it being blocking wouldn't be a real issue in production??? If there is a performance consideration then I probably should implement a cache (but in doing so I need to know how long it is before an old token that is rotated out expires). @raboof @laurens-gross-mw maybe you have some context here?
>
> A token is 1h valid and has ~5min overlap with a new token in our system. So from our side, no caching is needed
Is this setup where tokens have an overlap standard across all k8s deployments? Can it be configured to not have an overlap (if so it sounds like by default we should not be doing a retry on 401 but a user can configure this if they need to).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #546:
URL: https://github.com/apache/pekko-management/pull/546#discussion_r2537788857
##########
lease-kubernetes/src/test/scala/org/apache/pekko/coordination/lease/kubernetes/KubernetesApiSpec.scala:
##########
@@ -269,6 +275,87 @@ class KubernetesApiSpec
underTest.removeLease(lease).failed.futureValue.getMessage shouldEqual
s"Timed out removing lease [$lease]. It is not known if the remove happened. Is the API server up?"
}
- }
+ "not cache token so it can be rotated" in {
+ val path = File.createTempFile("kubernetes-api-spec", null)
+ path.deleteOnExit()
+
+ val newSettings = new KubernetesSettings(
+ "",
+ path.getAbsolutePath,
+ "localhost",
+ wireMockServer.port(),
+ namespace = Some("lease"),
+ "",
+ apiServerRequestTimeout = 1.second,
+ secure = false)
+
+ val tokenTest = new KubernetesApiImpl(system, newSettings)
+
+ val firstTokenValue = "first"
+ val secondTokenValue = "second"
+
+ Files.write(path.toPath, firstTokenValue.getBytes)
+ tokenTest.apiToken().futureValue shouldEqual firstTokenValue
+ Files.write(path.toPath, secondTokenValue.getBytes)
+ tokenTest.apiToken().futureValue shouldEqual secondTokenValue
+ }
+
+ "retry on 401 to handle token timeout" in {
+ val newSettings = new KubernetesSettings(
+ "",
+ "",
+ "localhost",
+ wireMockServer.port(),
+ namespace = Some("lease"),
+ "",
+ apiServerRequestTimeout = 1.second,
+ secure = false)
+
+ val toFail = new AtomicBoolean(true)
+ val retryUnauthorized = new KubernetesApiImpl(system, newSettings) {
+ // avoid touching slow CI filesystem
+ override protected def readConfigVarFromFilesystem(path: String, name: String): Future[Option[String]] =
+ Future.successful(None)
+
+ override def makeRawRequest(request: HttpRequest): Future[HttpResponse] =
+ if (toFail.getAndSet(false))
+ Future.successful(HttpResponse(
+ StatusCodes.Unauthorized
+ ))
+ else
+ Future.successful(HttpResponse(
+ StatusCodes.NotFound
+ ))
+ }
+
+ retryUnauthorized.getLeaseResource("").futureValue shouldEqual None
+ }
+
+ "eventually return unauthorized LeaseException when token rotation is not happening" in {
+ val newSettings = new KubernetesSettings(
+ "",
+ "",
+ "localhost",
+ wireMockServer.port(),
+ namespace = Some("lease"),
+ "",
+ apiServerRequestTimeout = 1.second,
+ secure = false)
+
+ val retryUnauthorized = new KubernetesApiImpl(system, newSettings) {
+ // avoid touching slow CI filesystem
+ override protected def readConfigVarFromFilesystem(path: String, name: String): Future[Option[String]] =
+ Future.successful(None)
+
+ override def makeRawRequest(request: HttpRequest): Future[HttpResponse] =
+ Future.successful(HttpResponse(
+ StatusCodes.Unauthorized
+ ))
+ }
+
+ retryUnauthorized.getLeaseResource("").failed.futureValue shouldBe an[LeaseException]
Review Comment:
This is handling that we hit [this](https://github.com/apache/pekko-management/blob/25470d6a11c754adf44d8eddb3d14f89be0418e4/lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/internal/AbstractKubernetesApiImpl.scala#L151-L158) code path eventually if the retry is exceeded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3547278682
still looks good to me - the 401 retries are a good improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "laurens-gross-mw (via GitHub)" <gi...@apache.org>.
laurens-gross-mw commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3557915009
> > Hi @mdedetrich, regarding:
> > > implies there may be some bottleneck/performance considerations in how often this API method is called.
> > > I mean if the API is called every now and then, even it being blocking wouldn't be a real issue in production??? If there is a performance consideration then I probably should implement a cache (but in doing so I need to know how long it is before an old token that is rotated out expires). @raboof @laurens-gross-mw maybe you have some context here?
> >
> >
> > A token is 1h valid and has ~5min overlap with a new token in our system. So from our side, no caching is needed
>
> Is this setup where tokens have an overlap duration standard across all k8s deployments? Can it be configured to not have an overlap? (if so it sounds like by default we should not be doing a retry on 401 but a user can configure this if they need to otherwise I should just remove the retry functionality altogether as its then pointless).
As fare as I know we didn't configured the overlap and I am also unsure if it's possible to configure it without an overlap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich merged PR #546:
URL: https://github.com/apache/pekko-management/pull/546
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3546439355
So I have just added a regressing test, but I am still slightly apprehensive in merging as the comment at https://github.com/apache/pekko-management/blob/606f90024f7e8a57385c4f170cc4c897714dc0cc/lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/internal/AbstractKubernetesApiImpl.scala#L172-L174 implies there may be some bottleneck/performance considerations in how often this API method is called.
I mean if the API is called every now and then, even it being blocking wouldn't be a real issue in production so the comment seems to imply that the API is called frequently??? If so then I should implement a cache (but in doing so I need to know how long it is before a token that is rotated out expires). @raboof @laurens-gross-mw maybe you have some context here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3558095584
* I can't find any docs that describe the overlap behaviour of old and new tokens.
* I don't think that the 401 retries have a downside. I would keep the code. It may be justifiable to reduce the default max-attempts to 0 or some slightly higher value like 1 or 2. Little reason to go for a default value like 5. I would favour a value like 2.
* I don't see a real benefit to failing fast on a 401.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3546589819
I asked an AI agent about not caching.
> Summary: Yes, read the token file on every API call
>
> It's the expected and correct behavior
> Required for auto-rotation to work seamlessly
> Zero meaningful performance cost
> Used by official clients (client-go, python-kubernetes, etc.)
> Avoids outages from expired tokens
>
> Do it. Don’t cache long-term. Your cluster thanks you.
One point raised is that there still might be timing issues where you read the token file but it rotates just before you use it. According to the AI I used, the old token is invalidated at rotation time.
So, maybe the best enhancement is to handle 401s by retrying.
We could come back to caching later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #546:
URL: https://github.com/apache/pekko-management/pull/546#discussion_r2537788857
##########
lease-kubernetes/src/test/scala/org/apache/pekko/coordination/lease/kubernetes/KubernetesApiSpec.scala:
##########
@@ -269,6 +275,87 @@ class KubernetesApiSpec
underTest.removeLease(lease).failed.futureValue.getMessage shouldEqual
s"Timed out removing lease [$lease]. It is not known if the remove happened. Is the API server up?"
}
- }
+ "not cache token so it can be rotated" in {
+ val path = File.createTempFile("kubernetes-api-spec", null)
+ path.deleteOnExit()
+
+ val newSettings = new KubernetesSettings(
+ "",
+ path.getAbsolutePath,
+ "localhost",
+ wireMockServer.port(),
+ namespace = Some("lease"),
+ "",
+ apiServerRequestTimeout = 1.second,
+ secure = false)
+
+ val tokenTest = new KubernetesApiImpl(system, newSettings)
+
+ val firstTokenValue = "first"
+ val secondTokenValue = "second"
+
+ Files.write(path.toPath, firstTokenValue.getBytes)
+ tokenTest.apiToken().futureValue shouldEqual firstTokenValue
+ Files.write(path.toPath, secondTokenValue.getBytes)
+ tokenTest.apiToken().futureValue shouldEqual secondTokenValue
+ }
+
+ "retry on 401 to handle token timeout" in {
+ val newSettings = new KubernetesSettings(
+ "",
+ "",
+ "localhost",
+ wireMockServer.port(),
+ namespace = Some("lease"),
+ "",
+ apiServerRequestTimeout = 1.second,
+ secure = false)
+
+ val toFail = new AtomicBoolean(true)
+ val retryUnauthorized = new KubernetesApiImpl(system, newSettings) {
+ // avoid touching slow CI filesystem
+ override protected def readConfigVarFromFilesystem(path: String, name: String): Future[Option[String]] =
+ Future.successful(None)
+
+ override def makeRawRequest(request: HttpRequest): Future[HttpResponse] =
+ if (toFail.getAndSet(false))
+ Future.successful(HttpResponse(
+ StatusCodes.Unauthorized
+ ))
+ else
+ Future.successful(HttpResponse(
+ StatusCodes.NotFound
+ ))
+ }
+
+ retryUnauthorized.getLeaseResource("").futureValue shouldEqual None
+ }
+
+ "eventually return unauthorized LeaseException when token rotation is not happening" in {
+ val newSettings = new KubernetesSettings(
+ "",
+ "",
+ "localhost",
+ wireMockServer.port(),
+ namespace = Some("lease"),
+ "",
+ apiServerRequestTimeout = 1.second,
+ secure = false)
+
+ val retryUnauthorized = new KubernetesApiImpl(system, newSettings) {
+ // avoid touching slow CI filesystem
+ override protected def readConfigVarFromFilesystem(path: String, name: String): Future[Option[String]] =
+ Future.successful(None)
+
+ override def makeRawRequest(request: HttpRequest): Future[HttpResponse] =
+ Future.successful(HttpResponse(
+ StatusCodes.Unauthorized
+ ))
+ }
+
+ retryUnauthorized.getLeaseResource("").failed.futureValue shouldBe an[LeaseException]
Review Comment:
This is handling that we hit [this](https://github.com/apache/pekko-management/blob/25470d6a11c754adf44d8eddb3d14f89be0418e4/lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/internal/AbstractKubernetesApiImpl.scala#L151-L158) code path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3562679612
https://github.com/spring-projects/spring-ai/commit/a2a5e1b5f25d184104527267186f1a574de17875
@mdedetrich cc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3547288218
I'm not suggesting this but if we ever did find that reading the token file was a bottleneck, instead of a time based expiring cache, we could let API calls 401 and let the retry mechanism be used to refresh the cached token.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #546:
URL: https://github.com/apache/pekko-management/pull/546#discussion_r2537782233
##########
lease-kubernetes/src/main/scala/org/apache/pekko/coordination/lease/kubernetes/internal/AbstractKubernetesApiImpl.scala:
##########
@@ -148,17 +160,34 @@ import scala.util.control.NonFatal
protected def requestForPath(
path: Uri.Path,
method: HttpMethod = HttpMethods.GET,
- entity: RequestEntity = HttpEntity.Empty): HttpRequest = {
+ entity: RequestEntity = HttpEntity.Empty): Future[HttpRequest] = {
val uri = Uri.from(scheme = scheme, host = settings.apiServerHost, port = settings.apiServerPort).withPath(path)
- HttpRequest(uri = uri, headers = headers, method = method, entity = entity)
+ headers().map { headers =>
+ HttpRequest(uri = uri, headers = headers, method = method, entity = entity)
+ }(ExecutionContext.parasitic)
+ }
+
+ private[pekko] def makeRawRequest(request: HttpRequest): Future[HttpResponse] = {
+ if (settings.secure)
+ http.singleRequest(request, clientSslContext)
+ else
+ http.singleRequest(request)
}
protected def makeRequest(request: HttpRequest, timeoutMsg: String): Future[HttpResponse] = {
- val response =
- if (settings.secure)
- http.singleRequest(request, clientSslContext)
- else
- http.singleRequest(request)
+ // It's possible to legitimately get a 401 response due to kubernetes doing a token rotation
+ implicit val scheduler: Scheduler = system.scheduler
+ val response = RetrySupport.retry(
Review Comment:
When cherry picking back to 1.2.x branch I will need to refactor and use [this](https://github.com/apache/pekko/blob/870d4499cc49630eebfcecb3f10d0be9dde59b5a/actor/src/main/scala/org/apache/pekko/pattern/RetrySupport.scala#L86-L128) version of `RetrySupport.retry` since that is available in Pekko 1.0.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3548192641
So I will leave this PR open for another couple/few days for people to comment on and if there are no objections I will merge it at the end of the week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3542297863
So I have updated the PR to make all filesystem requests to get tokens/other details from the pod asynchronous/non blocking
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Get token on every request and make fs calls async [pekko-management]
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #546:
URL: https://github.com/apache/pekko-management/pull/546#issuecomment-3542438387
Hmm I made a regression, let me look into it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org